public inbox for gcc-cvs@sourceware.org
help / color / mirror / Atom feed
* [gcc(refs/users/meissner/heads/work161-dmf)] PowerPC: Add support for accumulators in DMR registers.
@ 2024-02-29 19:28 Michael Meissner
0 siblings, 0 replies; 4+ messages in thread
From: Michael Meissner @ 2024-02-29 19:28 UTC (permalink / raw)
To: gcc-cvs
https://gcc.gnu.org/g:31ba8aaaa6620cb8a1cf9c22aaf54ac0a9e2a862
commit 31ba8aaaa6620cb8a1cf9c22aaf54ac0a9e2a862
Author: Michael Meissner <meissner@linux.ibm.com>
Date: Thu Feb 29 14:23:25 2024 -0500
PowerPC: Add support for accumulators in DMR registers.
The MMA subsystem added the notion of accumulator registers as an optional
feature of ISA 3.1 (power10). In ISA 3.1, these accumulators overlapped with
the VSX registers 0..31, but logically the accumulator registers were separate
from the FPR registers. In ISA 3.1, it was anticipated that in future systems,
the accumulator registers may no overlap with the FPR registers. This patch
adds the support for dense math registers as separate registers.
This particular patch does not change the MMA support to use the accumulators
within the dense math registers. This patch just adds the basic support for
having separate DMRs. The next patch will switch the MMA support to use the
accumulators if -mcpu=future is used.
For testing purposes, I added an undocumented option '-mdense-math' to enable
or disable the dense math support.
This patch adds a new constraint (wD). If MMA is selected but dense math is
not selected (i.e. -mcpu=power10), the wD constraint will allow access to
accumulators that overlap with VSX registers 0..31. If both MMA and dense math
are selected (i.e. -mcpu=future), the wD constraint will only allow dense math
registers.
This patch modifies the existing %A output modifier. If MMA is selected but
dense math is not selected, then %A output modifier converts the VSX register
number to the accumulator number, by dividing it by 4. If both MMA and dense
math are selected, then %A will map the separate DMR registers into 0..7.
The intention is that user code using extended asm can be modified to run on
both MMA without dense math and MMA with dense math:
1) If possible, don't use extended asm, but instead use the MMA built-in
functions;
2) If you do need to write extended asm, change the d constraints
targetting accumulators should now use wD;
3) Only use the built-in zero, assemble and disassemble functions create
move data between vector quad types and dense math accumulators.
I.e. do not use the xxmfacc, xxmtacc, and xxsetaccz directly in the
extended asm code. The reason is these instructions assume there is a
1-to-1 correspondence between 4 adjacent FPR registers and an
accumulator that overlaps with those instructions. With accumulators
now being separate registers, there no longer is a 1-to-1
correspondence.
It is possible that the mangling for DMRs and the GDB register numbers may
produce other changes in the future.
2024-02-29 Michael Meissner <meissner@linux.ibm.com>
gcc/
* config/rs6000/constraints.md (wD constraint): New constraint.
* config/rs6000/mma.md (UNSPEC_DM_ASSEMBLE): New unspec.
(movxo): Convert into define_expand.
(movxo_nodm): Version of movxo where accumulators overlap with VSX vector
registers 0..31.
(movxo_dm): Verson of movxo that supports separate dense math
accumulators.
(mma_assemble_acc): Add dense math support to define_expand.
(mma_assemble_acc_nodm): Rename from mma_assemble_acc, and restrict it
to non dense math systems.
(mma_assemble_acc_dm): Dense math version of mma_assemble_acc.
(mma_disassemble_acc): Add dense math support to define_expand.
(mma_disassemble_acc_nodm): Rename from mma_disassemble_acc, and
restrict it to non dense math systems.
(mma_disassemble_acc_dm): Dense math version of mma_disassemble_acc.
* config/rs6000/predicates.md (dmr_operand): New predicate.
(accumulator_operand): Likewise.
* config/rs6000/rs6000.cc (enum rs6000_reg_type): Add DMR_REG_TYPE.
(enum rs6000_reload_reg_type): Add RELOAD_REG_DMR.
(LAST_RELOAD_REG_CLASS): Add support for DMR registers and the wD
constraint.
(reload_reg_map): Likewise.
(rs6000_reg_names): Likewise.
(alt_reg_names): Likewise.
(rs6000_hard_regno_nregs_internal): Likewise.
(rs6000_hard_regno_mode_ok_uncached): Likewise.
(rs6000_debug_reg_global): Likewise.
(rs6000_setup_reg_addr_masks): Likewise.
(rs6000_init_hard_regno_mode_ok): Likewise.
(rs6000_secondary_reload_memory): Add support for DMR registers.
(rs6000_secondary_reload_simple_move): Likewise.
(rs6000_preferred_reload_class): Likewise.
(rs6000_secondary_reload_class): Likewise.
(print_operand): Make %A handle both FPRs and DMRs.
(rs6000_dmr_register_move_cost): New helper function.
(rs6000_register_move_cost): Add support for DMR registers.
(rs6000_memory_move_cost): Likewise.
(rs6000_compute_pressure_classes): Likewise.
(rs6000_debugger_regno): Likewise.
(rs6000_split_multireg_move): Add support for DMRs.
* config/rs6000/rs6000.h (TARGET_DENSE_MATH): New macro.
(TARGET_MMA_NO_DENSE_MATH): Likewise
(UNITS_PER_DMR_WORD): New macro.
(FIRST_PSEUDO_REGISTER): Update for DMRs.
(FIXED_REGISTERS): Add DMRs.
(CALL_REALLY_USED_REGISTERS): Likewise.
(REG_ALLOC_ORDER): Likewise.
(DMR_REGNO_P): New macro.
(enum reg_class): Add DM_REGS.
(REG_CLASS_NAMES): Likewise.
(REG_CLASS_CONTENTS): Likewise.
(enum r6000_reg_class_enum): Add RS6000_CONSTRAINT_wD.
(REGISTER_NAMES): Add DMR registers.
(ADDITIONAL_REGISTER_NAMES): Likewise.
* config/rs6000/rs6000.md (FIRST_DMR_REGNO): New constant.
(LAST_DMR_REGNO): Likewise.
(isa attribute): Add 'dm' and 'not_dm' attributes.
(enabled attribute): Support 'dm' and 'not_dm' attributes.
* doc/md.texi (PowerPC constraints): Document wD constraint.
Diff:
---
gcc/config/rs6000/constraints.md | 3 +
gcc/config/rs6000/mma.md | 115 ++++++++++++++++++-------
gcc/config/rs6000/predicates.md | 32 +++++++
gcc/config/rs6000/rs6000.cc | 180 ++++++++++++++++++++++++++++++++++-----
gcc/config/rs6000/rs6000.h | 42 +++++++--
gcc/config/rs6000/rs6000.md | 12 ++-
gcc/doc/md.texi | 7 ++
7 files changed, 332 insertions(+), 59 deletions(-)
diff --git a/gcc/config/rs6000/constraints.md b/gcc/config/rs6000/constraints.md
index 369a7b75042..277a30a8245 100644
--- a/gcc/config/rs6000/constraints.md
+++ b/gcc/config/rs6000/constraints.md
@@ -107,6 +107,9 @@
(match_test "TARGET_P8_VECTOR")
(match_operand 0 "s5bit_cint_operand")))
+(define_register_constraint "wD" "rs6000_constraints[RS6000_CONSTRAINT_wD]"
+ "Accumulator register.")
+
(define_constraint "wE"
"@internal Vector constant that can be loaded with the XXSPLTIB instruction."
(match_test "xxspltib_constant_nosplit (op, mode)"))
diff --git a/gcc/config/rs6000/mma.md b/gcc/config/rs6000/mma.md
index 04e2d0066df..9bc9895bf2d 100644
--- a/gcc/config/rs6000/mma.md
+++ b/gcc/config/rs6000/mma.md
@@ -91,6 +91,7 @@
UNSPEC_MMA_XVI8GER4SPP
UNSPEC_MMA_XXMFACC
UNSPEC_MMA_XXMTACC
+ UNSPEC_DM_ASSEMBLE
])
(define_c_enum "unspecv"
@@ -314,7 +315,9 @@
(set_attr "length" "*,*,8")])
\f
-;; Vector quad support. XOmode can only live in FPRs.
+;; Vector quad support. Under the original MMA, XOmode can only live in VSX
+;; registers 0..31. With dense math, XOmode can live in either VSX registers
+;; (0..63) or DMR registers.
(define_expand "movxo"
[(set (match_operand:XO 0 "nonimmediate_operand")
(match_operand:XO 1 "input_operand"))]
@@ -339,10 +342,10 @@
gcc_assert (false);
})
-(define_insn_and_split "*movxo"
+(define_insn_and_split "*movxo_nodm"
[(set (match_operand:XO 0 "nonimmediate_operand" "=d,ZwO,d")
(match_operand:XO 1 "input_operand" "ZwO,d,d"))]
- "TARGET_MMA
+ "TARGET_MMA_NO_DENSE_MATH
&& (gpc_reg_operand (operands[0], XOmode)
|| gpc_reg_operand (operands[1], XOmode))"
"@
@@ -359,6 +362,31 @@
(set_attr "length" "*,*,16")
(set_attr "max_prefixed_insns" "2,2,*")])
+(define_insn_and_split "*movxo_dm"
+ [(set (match_operand:XO 0 "nonimmediate_operand" "=wa,QwO,wa,wD,wD,wa")
+ (match_operand:XO 1 "input_operand" "QwO,wa, wa,wa,wD,wD"))]
+ "TARGET_DENSE_MATH
+ && (gpc_reg_operand (operands[0], XOmode)
+ || gpc_reg_operand (operands[1], XOmode))"
+ "@
+ #
+ #
+ #
+ dmxxinstdmr512 %0,%1,%Y1,0
+ dmmr %0,%1
+ dmxxextfdmr512 %0,%Y0,%1,0"
+ "&& reload_completed
+ && !dmr_operand (operands[0], XOmode)
+ && !dmr_operand (operands[1], XOmode)"
+ [(const_int 0)]
+{
+ rs6000_split_multireg_move (operands[0], operands[1]);
+ DONE;
+}
+ [(set_attr "type" "vecload,vecstore,veclogical,mma,mma,mma")
+ (set_attr "length" "*,*,16,*,*,*")
+ (set_attr "max_prefixed_insns" "2,2,*,*,*,*")])
+
(define_expand "vsx_assemble_pair"
[(match_operand:OO 0 "vsx_register_operand")
(match_operand:V16QI 1 "mma_assemble_input_operand")
@@ -426,25 +454,38 @@
})
(define_expand "mma_assemble_acc"
- [(match_operand:XO 0 "fpr_reg_operand")
+ [(match_operand:XO 0 "accumulator_operand")
(match_operand:V16QI 1 "mma_assemble_input_operand")
(match_operand:V16QI 2 "mma_assemble_input_operand")
(match_operand:V16QI 3 "mma_assemble_input_operand")
(match_operand:V16QI 4 "mma_assemble_input_operand")]
"TARGET_MMA"
{
- rtx src = gen_rtx_UNSPEC_VOLATILE (XOmode,
- gen_rtvec (4, operands[1], operands[2],
- operands[3], operands[4]),
- UNSPECV_MMA_ASSEMBLE);
- emit_move_insn (operands[0], src);
+ rtx op0 = operands[0];
+ rtx op1 = operands[1];
+ rtx op2 = operands[2];
+ rtx op3 = operands[3];
+ rtx op4 = operands[4];
+
+ if (TARGET_DENSE_MATH)
+ {
+ rtx vpair1 = gen_reg_rtx (OOmode);
+ rtx vpair2 = gen_reg_rtx (OOmode);
+ emit_insn (gen_vsx_assemble_pair (vpair1, op1, op2));
+ emit_insn (gen_vsx_assemble_pair (vpair2, op3, op4));
+ emit_insn (gen_mma_assemble_acc_dm (op0, vpair1, vpair2));
+ }
+
+ else
+ emit_insn (gen_mma_assemble_acc_nodm (op0, op1, op2, op3, op4));
+
DONE;
})
;; We cannot update the four output registers atomically, so mark the output
-;; as an early clobber so we don't accidentally clobber the input operands. */
+;; as an early clobber so we don't accidentally clobber the input operands.
-(define_insn_and_split "*mma_assemble_acc"
+(define_insn_and_split "mma_assemble_acc_nodm"
[(set (match_operand:XO 0 "fpr_reg_operand" "=&d")
(unspec_volatile:XO
[(match_operand:V16QI 1 "mma_assemble_input_operand" "mwa")
@@ -452,7 +493,7 @@
(match_operand:V16QI 3 "mma_assemble_input_operand" "mwa")
(match_operand:V16QI 4 "mma_assemble_input_operand" "mwa")]
UNSPECV_MMA_ASSEMBLE))]
- "TARGET_MMA
+ "TARGET_MMA_NO_DENSE_MATH
&& fpr_reg_operand (operands[0], XOmode)"
"#"
"&& reload_completed"
@@ -466,28 +507,31 @@
DONE;
})
+;; On a system with dense math, we build the accumulators from two vector
+;; pairs.
+
+(define_insn "mma_assemble_acc_dm"
+ [(set (match_operand:XO 0 "dmr_operand" "=wD")
+ (unspec:XO [(match_operand:OO 1 "vsx_register_operand" "wa")
+ (match_operand:OO 2 "vsx_register_operand" "wa")]
+ UNSPEC_DM_ASSEMBLE))]
+ "TARGET_DENSE_MATH"
+ "dmxxinstdmr512 %0,%1,%2,0"
+ [(set_attr "type" "mma")])
+
(define_expand "mma_disassemble_acc"
- [(match_operand:V16QI 0 "mma_disassemble_output_operand")
- (match_operand:XO 1 "fpr_reg_operand")
- (match_operand 2 "const_0_to_3_operand")]
- "TARGET_MMA"
-{
- rtx src;
- int regoff = INTVAL (operands[2]);
- src = gen_rtx_UNSPEC (V16QImode,
- gen_rtvec (2, operands[1], GEN_INT (regoff)),
- UNSPEC_MMA_EXTRACT);
- emit_move_insn (operands[0], src);
- DONE;
-})
+ [(set (match_operand:V16QI 0 "register_operand")
+ (unspec:V16QI [(match_operand:XO 1 "register_operand")
+ (match_operand 2 "const_0_to_3_operand")]
+ UNSPEC_MMA_EXTRACT))]
+ "TARGET_MMA")
-(define_insn_and_split "*mma_disassemble_acc"
+(define_insn_and_split "*mma_disassemble_acc_nodm"
[(set (match_operand:V16QI 0 "mma_disassemble_output_operand" "=mwa")
- (unspec:V16QI [(match_operand:XO 1 "fpr_reg_operand" "d")
- (match_operand 2 "const_0_to_3_operand")]
+ (unspec:V16QI [(match_operand:XO 1 "fpr_reg_operand" "d")
+ (match_operand 2 "const_0_to_3_operand")]
UNSPEC_MMA_EXTRACT))]
- "TARGET_MMA
- && fpr_reg_operand (operands[1], XOmode)"
+ "TARGET_MMA_NO_DENSE_MATH"
"#"
"&& reload_completed"
[(const_int 0)]
@@ -499,9 +543,14 @@
DONE;
})
-;; MMA instructions that do not use their accumulators as an input, still
-;; must not allow their vector operands to overlap the registers used by
-;; the accumulator. We enforce this by marking the output as early clobber.
+(define_insn "*mma_disassemble_acc_dm"
+ [(set (match_operand:V16QI 0 "vsx_register_operand" "=wa")
+ (unspec:V16QI [(match_operand:XO 1 "dmr_operand" "wD")
+ (match_operand 2 "const_0_to_3_operand")]
+ UNSPEC_MMA_EXTRACT))]
+ "TARGET_DENSE_MATH"
+ "dmxxextfdmr256 %0,%1,2"
+ [(set_attr "type" "mma")])
(define_insn "mma_<acc>"
[(set (match_operand:XO 0 "fpr_reg_operand" "=&d")
diff --git a/gcc/config/rs6000/predicates.md b/gcc/config/rs6000/predicates.md
index d23ce9a77a3..3040dcd50a3 100644
--- a/gcc/config/rs6000/predicates.md
+++ b/gcc/config/rs6000/predicates.md
@@ -186,6 +186,38 @@
return VLOGICAL_REGNO_P (REGNO (op));
})
+;; Return 1 if op is a DMR register
+(define_predicate "dmr_operand"
+ (match_operand 0 "register_operand")
+{
+ if (!REG_P (op))
+ return 0;
+
+ if (!HARD_REGISTER_P (op))
+ return 1;
+
+ return DMR_REGNO_P (REGNO (op));
+})
+
+;; Return 1 if op is an accumulator. On power10 systems, the accumulators
+;; overlap with the FPRs, while on systems with dense math, the accumulators
+;; are separate dense math registers and do not overlap with the FPR
+;; registers..
+(define_predicate "accumulator_operand"
+ (match_operand 0 "register_operand")
+{
+ if (!REG_P (op))
+ return 0;
+
+ if (!HARD_REGISTER_P (op))
+ return 1;
+
+ int r = REGNO (op);
+ return (TARGET_DENSE_MATH
+ ? DMR_REGNO_P (r)
+ : FP_REGNO_P (r) && (r & 3) == 0);
+})
+
;; Return 1 if op is the carry register.
(define_predicate "ca_operand"
(match_operand 0 "register_operand")
diff --git a/gcc/config/rs6000/rs6000.cc b/gcc/config/rs6000/rs6000.cc
index b1ad49d3734..190d3828085 100644
--- a/gcc/config/rs6000/rs6000.cc
+++ b/gcc/config/rs6000/rs6000.cc
@@ -292,7 +292,8 @@ enum rs6000_reg_type {
ALTIVEC_REG_TYPE,
FPR_REG_TYPE,
SPR_REG_TYPE,
- CR_REG_TYPE
+ CR_REG_TYPE,
+ DMR_REG_TYPE
};
/* Map register class to register type. */
@@ -306,22 +307,23 @@ static enum rs6000_reg_type reg_class_to_reg_type[N_REG_CLASSES];
/* Register classes we care about in secondary reload or go if legitimate
- address. We only need to worry about GPR, FPR, and Altivec registers here,
- along an ANY field that is the OR of the 3 register classes. */
+ address. We only need to worry about GPR, FPR, Altivec, and DMR registers
+ here, along an ANY field that is the OR of the 4 register classes. */
enum rs6000_reload_reg_type {
RELOAD_REG_GPR, /* General purpose registers. */
RELOAD_REG_FPR, /* Traditional floating point regs. */
RELOAD_REG_VMX, /* Altivec (VMX) registers. */
- RELOAD_REG_ANY, /* OR of GPR, FPR, Altivec masks. */
+ RELOAD_REG_DMR, /* DMR registers. */
+ RELOAD_REG_ANY, /* OR of GPR/FPR/VMX/DMR masks. */
N_RELOAD_REG
};
-/* For setting up register classes, loop through the 3 register classes mapping
+/* For setting up register classes, loop through the 4 register classes mapping
into real registers, and skip the ANY class, which is just an OR of the
bits. */
#define FIRST_RELOAD_REG_CLASS RELOAD_REG_GPR
-#define LAST_RELOAD_REG_CLASS RELOAD_REG_VMX
+#define LAST_RELOAD_REG_CLASS RELOAD_REG_DMR
/* Map reload register type to a register in the register class. */
struct reload_reg_map_type {
@@ -333,6 +335,7 @@ static const struct reload_reg_map_type reload_reg_map[N_RELOAD_REG] = {
{ "Gpr", FIRST_GPR_REGNO }, /* RELOAD_REG_GPR. */
{ "Fpr", FIRST_FPR_REGNO }, /* RELOAD_REG_FPR. */
{ "VMX", FIRST_ALTIVEC_REGNO }, /* RELOAD_REG_VMX. */
+ { "DMR", FIRST_DMR_REGNO }, /* RELOAD_REG_DMR. */
{ "Any", -1 }, /* RELOAD_REG_ANY. */
};
@@ -1226,6 +1229,8 @@ char rs6000_reg_names[][8] =
"0", "1", "2", "3", "4", "5", "6", "7",
/* vrsave vscr sfp */
"vrsave", "vscr", "sfp",
+ /* DMRs */
+ "0", "1", "2", "3", "4", "5", "6", "7",
};
#ifdef TARGET_REGNAMES
@@ -1252,6 +1257,8 @@ static const char alt_reg_names[][8] =
"%cr0", "%cr1", "%cr2", "%cr3", "%cr4", "%cr5", "%cr6", "%cr7",
/* vrsave vscr sfp */
"vrsave", "vscr", "sfp",
+ /* DMRs */
+ "%dmr0", "%dmr1", "%dmr2", "%dmr3", "%dmr4", "%dmr5", "%dmr6", "%dmr7",
};
#endif
@@ -1836,6 +1843,9 @@ rs6000_hard_regno_nregs_internal (int regno, machine_mode mode)
else if (ALTIVEC_REGNO_P (regno))
reg_size = UNITS_PER_ALTIVEC_WORD;
+ else if (DMR_REGNO_P (regno))
+ reg_size = UNITS_PER_DMR_WORD;
+
else
reg_size = UNITS_PER_WORD;
@@ -1857,9 +1867,36 @@ rs6000_hard_regno_mode_ok_uncached (int regno, machine_mode mode)
if (mode == OOmode)
return (TARGET_MMA && VSX_REGNO_P (regno) && (regno & 1) == 0);
- /* MMA accumulator modes need FPR registers divisible by 4. */
+ /* On ISA 3.1 (power10), MMA accumulator modes need FPR registers divisible
+ by 4.
+
+ If dense math is enabled, allow all VSX registers plus the DMR registers.
+ We need to make sure we don't cross between the boundary of FPRs and
+ traditional Altiviec registers. */
if (mode == XOmode)
- return (TARGET_MMA && FP_REGNO_P (regno) && (regno & 3) == 0);
+ {
+ if (TARGET_MMA && !TARGET_DENSE_MATH)
+ return (FP_REGNO_P (regno) && (regno & 3) == 0);
+
+ else if (TARGET_DENSE_MATH)
+ {
+ if (DMR_REGNO_P (regno))
+ return 1;
+
+ if (FP_REGNO_P (regno))
+ return ((regno & 1) == 0 && regno <= LAST_FPR_REGNO - 3);
+
+ if (ALTIVEC_REGNO_P (regno))
+ return ((regno & 1) == 0 && regno <= LAST_ALTIVEC_REGNO - 3);
+ }
+
+ else
+ return 0;
+ }
+
+ /* No other types other than XOmode can go in DMRs. */
+ if (DMR_REGNO_P (regno))
+ return 0;
/* PTImode can only go in GPRs. Quad word memory operations require even/odd
register combinations, and use PTImode where we need to deal with quad
@@ -2302,6 +2339,7 @@ rs6000_debug_reg_global (void)
rs6000_debug_reg_print (FIRST_ALTIVEC_REGNO,
LAST_ALTIVEC_REGNO,
"vs");
+ rs6000_debug_reg_print (FIRST_DMR_REGNO, LAST_DMR_REGNO, "dmr");
rs6000_debug_reg_print (LR_REGNO, LR_REGNO, "lr");
rs6000_debug_reg_print (CTR_REGNO, CTR_REGNO, "ctr");
rs6000_debug_reg_print (CR0_REGNO, CR7_REGNO, "cr");
@@ -2322,6 +2360,7 @@ rs6000_debug_reg_global (void)
"wr reg_class = %s\n"
"wx reg_class = %s\n"
"wA reg_class = %s\n"
+ "wD reg_class = %s\n"
"\n",
reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_d]],
reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_v]],
@@ -2329,7 +2368,8 @@ rs6000_debug_reg_global (void)
reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_we]],
reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_wr]],
reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_wx]],
- reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_wA]]);
+ reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_wA]],
+ reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_wD]]);
nl = "\n";
for (m = 0; m < NUM_MACHINE_MODES; ++m)
@@ -2626,6 +2666,21 @@ rs6000_setup_reg_addr_masks (void)
addr_mask = 0;
reg = reload_reg_map[rc].reg;
+ /* Special case DMR registers. */
+ if (rc == RELOAD_REG_DMR)
+ {
+ if (TARGET_DENSE_MATH && m2 == XOmode)
+ {
+ addr_mask = RELOAD_REG_VALID;
+ reg_addr[m].addr_mask[rc] = addr_mask;
+ any_addr_mask |= addr_mask;
+ }
+ else
+ reg_addr[m].addr_mask[rc] = 0;
+
+ continue;
+ }
+
/* Can mode values go in the GPR/FPR/Altivec registers? */
if (reg >= 0 && rs6000_hard_regno_mode_ok_p[m][reg])
{
@@ -2776,6 +2831,9 @@ rs6000_init_hard_regno_mode_ok (bool global_init_p)
for (r = CR1_REGNO; r <= CR7_REGNO; ++r)
rs6000_regno_regclass[r] = CR_REGS;
+ for (r = FIRST_DMR_REGNO; r <= LAST_DMR_REGNO; ++r)
+ rs6000_regno_regclass[r] = DM_REGS;
+
rs6000_regno_regclass[LR_REGNO] = LINK_REGS;
rs6000_regno_regclass[CTR_REGNO] = CTR_REGS;
rs6000_regno_regclass[CA_REGNO] = NO_REGS;
@@ -2800,6 +2858,7 @@ rs6000_init_hard_regno_mode_ok (bool global_init_p)
reg_class_to_reg_type[(int)LINK_OR_CTR_REGS] = SPR_REG_TYPE;
reg_class_to_reg_type[(int)CR_REGS] = CR_REG_TYPE;
reg_class_to_reg_type[(int)CR0_REGS] = CR_REG_TYPE;
+ reg_class_to_reg_type[(int)DM_REGS] = DMR_REG_TYPE;
if (TARGET_VSX)
{
@@ -2986,6 +3045,13 @@ rs6000_init_hard_regno_mode_ok (bool global_init_p)
if (TARGET_DIRECT_MOVE_128)
rs6000_constraints[RS6000_CONSTRAINT_we] = VSX_REGS;
+ /* Support for the accumulator registers, either FPR registers (aka original
+ mma) or DMR registers (dense math). */
+ if (TARGET_DENSE_MATH)
+ rs6000_constraints[RS6000_CONSTRAINT_wD] = DM_REGS;
+ else if (TARGET_MMA)
+ rs6000_constraints[RS6000_CONSTRAINT_wD] = FLOAT_REGS;
+
/* Set up the reload helper and direct move functions. */
if (TARGET_VSX || TARGET_ALTIVEC)
{
@@ -12311,6 +12377,11 @@ rs6000_secondary_reload_memory (rtx addr,
addr_mask = (reg_addr[mode].addr_mask[RELOAD_REG_VMX]
& ~RELOAD_REG_AND_M16);
+ /* DMR registers use VSX registers, and need to generate some extra
+ instructions. */
+ else if (rclass == DM_REGS)
+ return 2;
+
/* If the register allocator hasn't made up its mind yet on the register
class to use, settle on defaults to use. */
else if (rclass == NO_REGS)
@@ -12639,6 +12710,13 @@ rs6000_secondary_reload_simple_move (enum rs6000_reg_type to_type,
|| (to_type == SPR_REG_TYPE && from_type == GPR_REG_TYPE)))
return true;
+ /* We can transfer between VSX registers and DMR registers without needing
+ extra registers. */
+ if (TARGET_DENSE_MATH && mode == XOmode
+ && ((to_type == DMR_REG_TYPE && from_type == VSX_REG_TYPE)
+ || (to_type == VSX_REG_TYPE && from_type == DMR_REG_TYPE)))
+ return true;
+
return false;
}
@@ -13333,6 +13411,10 @@ rs6000_preferred_reload_class (rtx x, enum reg_class rclass)
machine_mode mode = GET_MODE (x);
bool is_constant = CONSTANT_P (x);
+ /* DMR registers can't be loaded or stored. */
+ if (rclass == DM_REGS)
+ return NO_REGS;
+
/* If a mode can't go in FPR/ALTIVEC/VSX registers, don't return a preferred
reload class for it. */
if ((rclass == ALTIVEC_REGS || rclass == VSX_REGS)
@@ -13429,7 +13511,7 @@ rs6000_preferred_reload_class (rtx x, enum reg_class rclass)
return VSX_REGS;
if (mode == XOmode)
- return FLOAT_REGS;
+ return TARGET_DENSE_MATH ? VSX_REGS : FLOAT_REGS;
if (GET_MODE_CLASS (mode) == MODE_INT)
return GENERAL_REGS;
@@ -13554,6 +13636,11 @@ rs6000_secondary_reload_class (enum reg_class rclass, machine_mode mode,
else
regno = -1;
+ /* DMR registers don't have loads or stores. We have to go through the VSX
+ registers to load XOmode (vector quad). */
+ if (TARGET_DENSE_MATH && rclass == DM_REGS)
+ return VSX_REGS;
+
/* If we have VSX register moves, prefer moving scalar values between
Altivec registers and GPR by going via an FPR (and then via memory)
instead of reloading the secondary memory address for Altivec moves. */
@@ -14067,8 +14154,14 @@ print_operand (FILE *file, rtx x, int code)
output_operand. */
case 'A':
- /* Write the MMA accumulator number associated with VSX register X. */
- if (!REG_P (x) || !FP_REGNO_P (REGNO (x)) || (REGNO (x) % 4) != 0)
+ /* Write the MMA accumulator number associated with VSX register X. On
+ dense math systems, only allow DMR accumulators, not accumulators
+ overlapping with the FPR registers. */
+ if (!REG_P (x))
+ output_operand_lossage ("invalid %%A value");
+ else if (TARGET_DENSE_MATH && DMR_REGNO_P (REGNO (x)))
+ fprintf (file, "%d", REGNO (x) - FIRST_DMR_REGNO);
+ else if (!FP_REGNO_P (REGNO (x)) || (REGNO (x) % 4) != 0)
output_operand_lossage ("invalid %%A value");
else
fprintf (file, "%d", (REGNO (x) - FIRST_FPR_REGNO) / 4);
@@ -22738,6 +22831,31 @@ rs6000_debug_address_cost (rtx x, machine_mode mode,
}
+/* Subroutine to determine the move cost of dense math registers. If we are
+ moving to/from VSX_REGISTER registers, the cost is either 1 move (for
+ 512-bit accumulators) or 2 moves (for 1,024 dmr registers). If we are
+ moving to anything else like GPR registers, make the cost very high. */
+
+static int
+rs6000_dmr_register_move_cost (machine_mode mode, reg_class_t rclass)
+{
+ const int reg_move_base = 2;
+ HARD_REG_SET vsx_set = (reg_class_contents[rclass]
+ & reg_class_contents[VSX_REGS]);
+
+ if (TARGET_DENSE_MATH && !hard_reg_set_empty_p (vsx_set))
+ {
+ /* __vector_quad (i.e. XOmode) is tranfered in 1 instruction. */
+ if (mode == XOmode)
+ return reg_move_base;
+
+ else
+ return reg_move_base * 2 * hard_regno_nregs (FIRST_DMR_REGNO, mode);
+ }
+
+ return 1000 * 2 * hard_regno_nregs (FIRST_DMR_REGNO, mode);
+}
+
/* A C expression returning the cost of moving data from a register of class
CLASS1 to one of CLASS2. */
@@ -22751,17 +22869,28 @@ rs6000_register_move_cost (machine_mode mode,
if (TARGET_DEBUG_COST)
dbg_cost_ctrl++;
+ HARD_REG_SET to_vsx, from_vsx;
+ to_vsx = reg_class_contents[to] & reg_class_contents[VSX_REGS];
+ from_vsx = reg_class_contents[from] & reg_class_contents[VSX_REGS];
+
+ /* Special case DMR registers, that can only move to/from VSX registers. */
+ if (from == DM_REGS && to == DM_REGS)
+ ret = 2 * hard_regno_nregs (FIRST_DMR_REGNO, mode);
+
+ else if (from == DM_REGS)
+ ret = rs6000_dmr_register_move_cost (mode, to);
+
+ else if (to == DM_REGS)
+ ret = rs6000_dmr_register_move_cost (mode, from);
+
/* If we have VSX, we can easily move between FPR or Altivec registers,
otherwise we can only easily move within classes.
Do this first so we give best-case answers for union classes
containing both gprs and vsx regs. */
- HARD_REG_SET to_vsx, from_vsx;
- to_vsx = reg_class_contents[to] & reg_class_contents[VSX_REGS];
- from_vsx = reg_class_contents[from] & reg_class_contents[VSX_REGS];
- if (!hard_reg_set_empty_p (to_vsx)
- && !hard_reg_set_empty_p (from_vsx)
- && (TARGET_VSX
- || hard_reg_set_intersect_p (to_vsx, from_vsx)))
+ else if (!hard_reg_set_empty_p (to_vsx)
+ && !hard_reg_set_empty_p (from_vsx)
+ && (TARGET_VSX
+ || hard_reg_set_intersect_p (to_vsx, from_vsx)))
{
int reg = FIRST_FPR_REGNO;
if (TARGET_VSX
@@ -22856,6 +22985,9 @@ rs6000_memory_move_cost (machine_mode mode, reg_class_t rclass,
ret = 4 * hard_regno_nregs (32, mode);
else if (reg_classes_intersect_p (rclass, ALTIVEC_REGS))
ret = 4 * hard_regno_nregs (FIRST_ALTIVEC_REGNO, mode);
+ else if (reg_classes_intersect_p (rclass, DM_REGS))
+ ret = (rs6000_dmr_register_move_cost (mode, VSX_REGS)
+ + rs6000_memory_move_cost (mode, VSX_REGS, false));
else
ret = 4 + rs6000_register_move_cost (mode, rclass, GENERAL_REGS);
@@ -24064,6 +24196,8 @@ rs6000_compute_pressure_classes (enum reg_class *pressure_classes)
if (TARGET_HARD_FLOAT)
pressure_classes[n++] = FLOAT_REGS;
}
+ if (TARGET_DENSE_MATH)
+ pressure_classes[n++] = DM_REGS;
pressure_classes[n++] = CR_REGS;
pressure_classes[n++] = SPECIAL_REGS;
@@ -24228,6 +24362,10 @@ rs6000_debugger_regno (unsigned int regno, unsigned int format)
return 67;
if (regno == 64)
return 64;
+ /* XXX: This is a guess. The GCC register number for FIRST_DMR_REGNO is 111,
+ but the frame pointer regnum uses that. */
+ if (DMR_REGNO_P (regno))
+ return regno - FIRST_DMR_REGNO + 112;
gcc_unreachable ();
}
@@ -27519,7 +27657,9 @@ rs6000_split_multireg_move (rtx dst, rtx src)
|| XINT (src, 1) == UNSPECV_MMA_ASSEMBLE);
gcc_assert (REG_P (dst));
if (GET_MODE (src) == XOmode)
- gcc_assert (FP_REGNO_P (REGNO (dst)));
+ gcc_assert ((TARGET_DENSE_MATH
+ ? VSX_REGNO_P (REGNO (dst))
+ : FP_REGNO_P (REGNO (dst))));
if (GET_MODE (src) == OOmode)
gcc_assert (VSX_REGNO_P (REGNO (dst)));
diff --git a/gcc/config/rs6000/rs6000.h b/gcc/config/rs6000/rs6000.h
index 79ce1a8cbf1..bea5db90fab 100644
--- a/gcc/config/rs6000/rs6000.h
+++ b/gcc/config/rs6000/rs6000.h
@@ -562,6 +562,10 @@ extern int rs6000_vector_align[];
&& TARGET_P8_VECTOR \
&& TARGET_POWERPC64)
+/* Whether we have dense math support. */
+#define TARGET_DENSE_MATH (TARGET_MMA && TARGET_FUTURE)
+#define TARGET_MMA_NO_DENSE_MATH (TARGET_MMA && !TARGET_FUTURE)
+
/* Inlining allows targets to define the meanings of bits in target_info
field of ipa_fn_summary by itself, the used bits for rs6000 are listed
below. */
@@ -659,6 +663,7 @@ extern unsigned char rs6000_recip_bits[];
#define UNITS_PER_FP_WORD 8
#define UNITS_PER_ALTIVEC_WORD 16
#define UNITS_PER_VSX_WORD 16
+#define UNITS_PER_DMR_WORD 128
/* Type used for ptrdiff_t, as a string used in a declaration. */
#define PTRDIFF_TYPE "int"
@@ -786,7 +791,7 @@ enum data_align { align_abi, align_opt, align_both };
Another pseudo (not included in DWARF_FRAME_REGISTERS) is soft frame
pointer, which is eventually eliminated in favor of SP or FP. */
-#define FIRST_PSEUDO_REGISTER 111
+#define FIRST_PSEUDO_REGISTER 119
/* Use standard DWARF numbering for DWARF debugging information. */
#define DEBUGGER_REGNO(REGNO) rs6000_debugger_regno ((REGNO), 0)
@@ -823,7 +828,9 @@ enum data_align { align_abi, align_opt, align_both };
/* cr0..cr7 */ \
0, 0, 0, 0, 0, 0, 0, 0, \
/* vrsave vscr sfp */ \
- 1, 1, 1 \
+ 1, 1, 1, \
+ /* DMR registers. */ \
+ 0, 0, 0, 0, 0, 0, 0, 0 \
}
/* Like `CALL_USED_REGISTERS' except this macro doesn't require that
@@ -847,7 +854,9 @@ enum data_align { align_abi, align_opt, align_both };
/* cr0..cr7 */ \
1, 1, 0, 0, 0, 1, 1, 1, \
/* vrsave vscr sfp */ \
- 0, 0, 0 \
+ 0, 0, 0, \
+ /* DMR registers. */ \
+ 0, 0, 0, 0, 0, 0, 0, 0 \
}
#define TOTAL_ALTIVEC_REGS (LAST_ALTIVEC_REGNO - FIRST_ALTIVEC_REGNO + 1)
@@ -884,6 +893,7 @@ enum data_align { align_abi, align_opt, align_both };
v2 (not saved; incoming vector arg reg; return value)
v19 - v14 (not saved or used for anything)
v31 - v20 (saved; order given to save least number)
+ dmr0 - dmr7 (not saved)
vrsave, vscr (fixed)
sfp (fixed)
*/
@@ -926,6 +936,9 @@ enum data_align { align_abi, align_opt, align_both };
66, \
83, 82, 81, 80, 79, 78, \
95, 94, 93, 92, 91, 90, 89, 88, 87, 86, 85, 84, \
+ /* DMR registers. */ \
+ 111, 112, 113, 114, 115, 116, 117, 118, \
+ /* Vrsave, vscr, sfp. */ \
108, 109, \
110 \
}
@@ -952,6 +965,9 @@ enum data_align { align_abi, align_opt, align_both };
/* True if register is a VSX register. */
#define VSX_REGNO_P(N) (FP_REGNO_P (N) || ALTIVEC_REGNO_P (N))
+/* True if register is a DMR register. */
+#define DMR_REGNO_P(N) ((N) >= FIRST_DMR_REGNO && (N) <= LAST_DMR_REGNO)
+
/* Alternate name for any vector register supporting floating point, no matter
which instruction set(s) are available. */
#define VFLOAT_REGNO_P(N) \
@@ -1087,6 +1103,7 @@ enum reg_class
FLOAT_REGS,
ALTIVEC_REGS,
VSX_REGS,
+ DM_REGS,
VRSAVE_REGS,
VSCR_REGS,
GEN_OR_FLOAT_REGS,
@@ -1116,6 +1133,7 @@ enum reg_class
"FLOAT_REGS", \
"ALTIVEC_REGS", \
"VSX_REGS", \
+ "DM_REGS", \
"VRSAVE_REGS", \
"VSCR_REGS", \
"GEN_OR_FLOAT_REGS", \
@@ -1150,6 +1168,8 @@ enum reg_class
{ 0x00000000, 0x00000000, 0xffffffff, 0x00000000 }, \
/* VSX_REGS. */ \
{ 0x00000000, 0xffffffff, 0xffffffff, 0x00000000 }, \
+ /* DM_REGS. */ \
+ { 0x00000000, 0x00000000, 0x00000000, 0x007f8000 }, \
/* VRSAVE_REGS. */ \
{ 0x00000000, 0x00000000, 0x00000000, 0x00001000 }, \
/* VSCR_REGS. */ \
@@ -1177,7 +1197,7 @@ enum reg_class
/* CA_REGS. */ \
{ 0x00000000, 0x00000000, 0x00000000, 0x00000004 }, \
/* ALL_REGS. */ \
- { 0xffffffff, 0xffffffff, 0xffffffff, 0x00007fff } \
+ { 0xffffffff, 0xffffffff, 0xffffffff, 0x007fffff } \
}
/* The same information, inverted:
@@ -1201,6 +1221,7 @@ enum r6000_reg_class_enum {
RS6000_CONSTRAINT_wr, /* GPR register if 64-bit */
RS6000_CONSTRAINT_wx, /* FPR register for STFIWX */
RS6000_CONSTRAINT_wA, /* BASE_REGS if 64-bit. */
+ RS6000_CONSTRAINT_wD, /* Accumulator regs if MMA/Dense Math. */
RS6000_CONSTRAINT_MAX
};
@@ -2077,7 +2098,16 @@ extern char rs6000_reg_names[][8]; /* register names (0 vs. %r0). */
&rs6000_reg_names[108][0], /* vrsave */ \
&rs6000_reg_names[109][0], /* vscr */ \
\
- &rs6000_reg_names[110][0] /* sfp */ \
+ &rs6000_reg_names[110][0], /* sfp */ \
+ \
+ &rs6000_reg_names[111][0], /* dmr0 */ \
+ &rs6000_reg_names[112][0], /* dmr1 */ \
+ &rs6000_reg_names[113][0], /* dmr2 */ \
+ &rs6000_reg_names[114][0], /* dmr3 */ \
+ &rs6000_reg_names[115][0], /* dmr4 */ \
+ &rs6000_reg_names[116][0], /* dmr5 */ \
+ &rs6000_reg_names[117][0], /* dmr6 */ \
+ &rs6000_reg_names[118][0], /* dmr7 */ \
}
/* Table of additional register names to use in user input. */
@@ -2131,6 +2161,8 @@ extern char rs6000_reg_names[][8]; /* register names (0 vs. %r0). */
{"vs52", 84}, {"vs53", 85}, {"vs54", 86}, {"vs55", 87}, \
{"vs56", 88}, {"vs57", 89}, {"vs58", 90}, {"vs59", 91}, \
{"vs60", 92}, {"vs61", 93}, {"vs62", 94}, {"vs63", 95}, \
+ {"dmr0", 111}, {"dmr1", 112}, {"dmr2", 113}, {"dmr3", 114}, \
+ {"dmr4", 115}, {"dmr5", 116}, {"dmr6", 117}, {"dmr7", 118}, \
}
/* This is how to output an element of a case-vector that is relative. */
diff --git a/gcc/config/rs6000/rs6000.md b/gcc/config/rs6000/rs6000.md
index bc8bc6ab060..99e6515ba1d 100644
--- a/gcc/config/rs6000/rs6000.md
+++ b/gcc/config/rs6000/rs6000.md
@@ -51,6 +51,8 @@
(VRSAVE_REGNO 108)
(VSCR_REGNO 109)
(FRAME_POINTER_REGNUM 110)
+ (FIRST_DMR_REGNO 111)
+ (LAST_DMR_REGNO 118)
])
;;
@@ -355,7 +357,7 @@
(const (symbol_ref "(enum attr_cpu) rs6000_tune")))
;; The ISA we implement.
-(define_attr "isa" "any,p5,p6,p7,p7v,p8v,p9,p9v,p9kf,p9tf,p10"
+(define_attr "isa" "any,p5,p6,p7,p7v,p8v,p9,p9v,p9kf,p9tf,p10,dm,not_dm"
(const_string "any"))
;; Is this alternative enabled for the current CPU/ISA/etc.?
@@ -403,6 +405,14 @@
(and (eq_attr "isa" "p10")
(match_test "TARGET_POWER10"))
(const_int 1)
+
+ (and (eq_attr "isa" "dm")
+ (match_test "TARGET_DENSE_MATH"))
+ (const_int 1)
+
+ (and (eq_attr "isa" "not_dm")
+ (match_test "!TARGET_DENSE_MATH"))
+ (const_int 1)
] (const_int 0)))
;; If this instruction is microcoded on the CELL processor
diff --git a/gcc/doc/md.texi b/gcc/doc/md.texi
index 7b7e6507754..facb281d8ef 100644
--- a/gcc/doc/md.texi
+++ b/gcc/doc/md.texi
@@ -3441,6 +3441,13 @@ Like @code{d}, if @option{-mpowerpc-gfxopt} is used; otherwise, @code{NO_REGS}.
@item wA
Like @code{b}, if @option{-mpowerpc64} is used; otherwise, @code{NO_REGS}.
+@item wD
+Accumulator register if @option{-mma} is used; otherwise,
+@code{NO_REGS}. If @option{-mdense-math} is used, the accumulator
+register will be in the dense match register set. If
+@option{-mno-dense-math} is used, the accumulator register will
+overlap with the VSX vector registers 0..31.
+
@item wB
Signed 5-bit constant integer that can be loaded into an Altivec register.
^ permalink raw reply [flat|nested] 4+ messages in thread
* [gcc(refs/users/meissner/heads/work161-dmf)] PowerPC: Add support for accumulators in DMR registers.
@ 2024-03-01 21:46 Michael Meissner
0 siblings, 0 replies; 4+ messages in thread
From: Michael Meissner @ 2024-03-01 21:46 UTC (permalink / raw)
To: gcc-cvs
https://gcc.gnu.org/g:c9e2570f107941abdd704fb6ac04875fd6bfc51e
commit c9e2570f107941abdd704fb6ac04875fd6bfc51e
Author: Michael Meissner <meissner@linux.ibm.com>
Date: Fri Mar 1 16:31:46 2024 -0500
PowerPC: Add support for accumulators in DMR registers.
This patch changes the MMA instructions to use either VSX registers 0..31
(-mcpu=power10) or DMRs (-mcpu=future). In this patch, the existing MMA
instruction names are used.
A macro (__PPC_DMR__) is defined if the MMA instructions use the DMRs.
The MMA subsystem added the notion of accumulator registers as an optional
feature of ISA 3.1 (power10). In ISA 3.1, these accumulators overlapped with
the VSX registers 0..31, but logically the accumulator registers were separate
from the FPR registers. In ISA 3.1, it was anticipated that in future systems,
the accumulator registers may no overlap with the FPR registers. This patch
adds the support for dense math registers as separate registers.
This patch adds a new constraint (wD). If MMA is selected but dense math is
not selected (i.e. -mcpu=power10), the wD constraint will allow access to
accumulators that overlap with VSX registers 0..31. If -mcpu=future is used,
the wD constraint will only allow dense math registers.
This patch modifies the existing %A output modifier. If MMA is selected but
we don't have dense math registers the %A output modifier converts the VSX
register number to the accumulator number, by dividing it by 4. If both MMA and
dense math are selected, then %A will map the separate DMR registers into 0..7.
The intention is that user code using extended asm can be modified to run on
both MMA without dense math and MMA with dense math:
1) If possible, don't use extended asm, but instead use the MMA built-in
functions;
2) If you do need to write extended asm, change the d constraints
targetting accumulators should now use wD;
3) Only use the built-in zero, assemble and disassemble functions create
move data between vector quad types and dense math accumulators.
I.e. do not use the xxmfacc, xxmtacc, and xxsetaccz directly in the
extended asm code. The reason is these instructions assume there is a
1-to-1 correspondence between 4 adjacent FPR registers and an
accumulator that overlaps with those instructions. With accumulators
now being separate registers, there no longer is a 1-to-1
correspondence.
It is possible that the mangling for DMRs and the GDB register numbers may
produce other changes in the future.
2024-03-01 Michael Meissner <meissner@linux.ibm.com>
gcc/
* config/rs6000/constraints.md (wD constraint): New constraint.
* config/rs6000/mma.md (UNSPEC_DM_ASSEMBLE): New unspec.
(movxo_nodm): Rename from movxo, and restrict usage if dense math
registers are not available.
(movxo_dm): Verson of movxo that supports separate dense math
accumulators.
(mma_assemble_acc): Add dense math support to define_expand.
(mma_assemble_acc_nodm): Rename from mma_assemble_acc, and restrict it
to non dense math systems.
(mma_assemble_acc_dm): Dense math version of mma_assemble_acc.
(mma_disassemble_acc): Add dense math support to define_expand.
(mma_disassemble_acc_nodm): Rename from mma_disassemble_acc, and
restrict it to non dense math systems.
(mma_disassemble_acc_dm): Dense math version of mma_disassemble_acc.
(mma_<acc>): Convert to a define_expand to handle
non dense math and dense math cases.
(mma_<acc>_nodm insn): Rename from mma_<acc>, and restrict to non dense
math.
(mma_xxsetaccz): Convert to define_expand to handle non dense math and
dense math cases.
(mma_xxsetaccz_vsx): Rename from mma_xxsetaccz and restrict usage to non
dense math.
(mma_xxsetaccz_dm): Dense math version of mma_xxsetaccz.
(mma_<vv>): Add support for dense math.
(mma_<avv>): Likewise.
(mma_<pv>): Likewise.
(mma_<apv>): Likewise.
(mma_<vvi4i4i8>): Likewise.
(mma_<avvi4i4i8>): Likewise.
(mma_<vvi4i4i2>): Likewise.
(mma_<avvi4i4i2>): Likewise.
(mma_<vvi4i4>): Likewise.
(mma_<avvi4i4>): Likewise.
(mma_<pvi4i2>): Likewise.
(mma_<apvi4i2>): Likewise.
(mma_<vvi4i4i4>): Likewise.
(mma_<avvi4i4i4>): Likewise.
* config/rs6000/predicates.md (dmr_operand): New predicate.
(accumulator_operand): Likewise.
* config/rs6000/rs6000-c.cc (rs6000_target_modify_macros): Define
__PPC_DMR__ if we have dense math instructions.
* config/rs6000/rs6000.cc (enum rs6000_reg_type): Add DMR_REG_TYPE.
(enum rs6000_reload_reg_type): Add RELOAD_REG_DMR.
(LAST_RELOAD_REG_CLASS): Add support for DMR registers and the wD
constraint.
(reload_reg_map): Likewise.
(rs6000_reg_names): Likewise.
(alt_reg_names): Likewise.
(rs6000_hard_regno_nregs_internal): Likewise.
(rs6000_hard_regno_mode_ok_uncached): Likewise.
(rs6000_debug_reg_global): Likewise.
(rs6000_setup_reg_addr_masks): Likewise.
(rs6000_init_hard_regno_mode_ok): Likewise.
(rs6000_secondary_reload_memory): Add support for DMR registers.
(rs6000_secondary_reload_simple_move): Likewise.
(rs6000_preferred_reload_class): Likewise.
(rs6000_secondary_reload_class): Likewise.
(print_operand): Make %A handle both FPRs and DMRs.
(rs6000_dmr_register_move_cost): New helper function.
(rs6000_register_move_cost): Add support for DMR registers.
(rs6000_memory_move_cost): Likewise.
(rs6000_compute_pressure_classes): Likewise.
(rs6000_debugger_regno): Likewise.
(rs6000_split_multireg_move): Add support for DMRs.
* config/rs6000/rs6000.h (TARGET_DENSE_MATH): New macro.
(TARGET_MMA_NO_DENSE_MATH): Likewise
(UNITS_PER_DMR_WORD): New macro.
(FIRST_PSEUDO_REGISTER): Update for DMRs.
(FIXED_REGISTERS): Add DMRs.
(CALL_REALLY_USED_REGISTERS): Likewise.
(REG_ALLOC_ORDER): Likewise.
(DMR_REGNO_P): New macro.
(enum reg_class): Add DM_REGS.
(REG_CLASS_NAMES): Likewise.
(REG_CLASS_CONTENTS): Likewise.
(enum r6000_reg_class_enum): Add RS6000_CONSTRAINT_wD.
(REGISTER_NAMES): Add DMR registers.
(ADDITIONAL_REGISTER_NAMES): Likewise.
* config/rs6000/rs6000.md (FIRST_DMR_REGNO): New constant.
(LAST_DMR_REGNO): Likewise.
(isa attribute): Add 'dm' and 'not_dm' attributes.
(enabled attribute): Support 'dm' and 'not_dm' attributes.
* doc/md.texi (PowerPC constraints): Document wD constraint.
Diff:
---
gcc/config/rs6000/constraints.md | 3 +
gcc/config/rs6000/mma.md | 366 +++++++++++++++++++++++++--------------
gcc/config/rs6000/predicates.md | 32 ++++
gcc/config/rs6000/rs6000-c.cc | 3 +
gcc/config/rs6000/rs6000.cc | 211 ++++++++++++++++++----
gcc/config/rs6000/rs6000.h | 45 ++++-
gcc/config/rs6000/rs6000.md | 12 +-
gcc/doc/md.texi | 7 +
8 files changed, 511 insertions(+), 168 deletions(-)
diff --git a/gcc/config/rs6000/constraints.md b/gcc/config/rs6000/constraints.md
index 369a7b75042..277a30a8245 100644
--- a/gcc/config/rs6000/constraints.md
+++ b/gcc/config/rs6000/constraints.md
@@ -107,6 +107,9 @@
(match_test "TARGET_P8_VECTOR")
(match_operand 0 "s5bit_cint_operand")))
+(define_register_constraint "wD" "rs6000_constraints[RS6000_CONSTRAINT_wD]"
+ "Accumulator register.")
+
(define_constraint "wE"
"@internal Vector constant that can be loaded with the XXSPLTIB instruction."
(match_test "xxspltib_constant_nosplit (op, mode)"))
diff --git a/gcc/config/rs6000/mma.md b/gcc/config/rs6000/mma.md
index 04e2d0066df..ac26de93143 100644
--- a/gcc/config/rs6000/mma.md
+++ b/gcc/config/rs6000/mma.md
@@ -91,6 +91,7 @@
UNSPEC_MMA_XVI8GER4SPP
UNSPEC_MMA_XXMFACC
UNSPEC_MMA_XXMTACC
+ UNSPEC_DM_ASSEMBLE
])
(define_c_enum "unspecv"
@@ -314,7 +315,9 @@
(set_attr "length" "*,*,8")])
\f
-;; Vector quad support. XOmode can only live in FPRs.
+;; Vector quad support. Under the original MMA, XOmode can only live in VSX
+;; registers 0..31. With dense math, XOmode can live in either VSX registers
+;; (0..63) or DMR registers.
(define_expand "movxo"
[(set (match_operand:XO 0 "nonimmediate_operand")
(match_operand:XO 1 "input_operand"))]
@@ -339,10 +342,10 @@
gcc_assert (false);
})
-(define_insn_and_split "*movxo"
+(define_insn_and_split "*movxo_nodm"
[(set (match_operand:XO 0 "nonimmediate_operand" "=d,ZwO,d")
(match_operand:XO 1 "input_operand" "ZwO,d,d"))]
- "TARGET_MMA
+ "TARGET_MMA_NO_DENSE_MATH
&& (gpc_reg_operand (operands[0], XOmode)
|| gpc_reg_operand (operands[1], XOmode))"
"@
@@ -359,6 +362,31 @@
(set_attr "length" "*,*,16")
(set_attr "max_prefixed_insns" "2,2,*")])
+(define_insn_and_split "*movxo_dm"
+ [(set (match_operand:XO 0 "nonimmediate_operand" "=wa,QwO,wa,wD,wD,wa")
+ (match_operand:XO 1 "input_operand" "QwO,wa, wa,wa,wD,wD"))]
+ "TARGET_DENSE_MATH
+ && (gpc_reg_operand (operands[0], XOmode)
+ || gpc_reg_operand (operands[1], XOmode))"
+ "@
+ #
+ #
+ #
+ dmxxinstdmr512 %0,%1,%Y1,0
+ dmmr %0,%1
+ dmxxextfdmr512 %0,%Y0,%1,0"
+ "&& reload_completed
+ && !dmr_operand (operands[0], XOmode)
+ && !dmr_operand (operands[1], XOmode)"
+ [(const_int 0)]
+{
+ rs6000_split_multireg_move (operands[0], operands[1]);
+ DONE;
+}
+ [(set_attr "type" "vecload,vecstore,veclogical,mma,mma,mma")
+ (set_attr "length" "*,*,16,*,*,*")
+ (set_attr "max_prefixed_insns" "2,2,*,*,*,*")])
+
(define_expand "vsx_assemble_pair"
[(match_operand:OO 0 "vsx_register_operand")
(match_operand:V16QI 1 "mma_assemble_input_operand")
@@ -426,25 +454,38 @@
})
(define_expand "mma_assemble_acc"
- [(match_operand:XO 0 "fpr_reg_operand")
+ [(match_operand:XO 0 "accumulator_operand")
(match_operand:V16QI 1 "mma_assemble_input_operand")
(match_operand:V16QI 2 "mma_assemble_input_operand")
(match_operand:V16QI 3 "mma_assemble_input_operand")
(match_operand:V16QI 4 "mma_assemble_input_operand")]
"TARGET_MMA"
{
- rtx src = gen_rtx_UNSPEC_VOLATILE (XOmode,
- gen_rtvec (4, operands[1], operands[2],
- operands[3], operands[4]),
- UNSPECV_MMA_ASSEMBLE);
- emit_move_insn (operands[0], src);
+ rtx op0 = operands[0];
+ rtx op1 = operands[1];
+ rtx op2 = operands[2];
+ rtx op3 = operands[3];
+ rtx op4 = operands[4];
+
+ if (TARGET_DENSE_MATH)
+ {
+ rtx vpair1 = gen_reg_rtx (OOmode);
+ rtx vpair2 = gen_reg_rtx (OOmode);
+ emit_insn (gen_vsx_assemble_pair (vpair1, op1, op2));
+ emit_insn (gen_vsx_assemble_pair (vpair2, op3, op4));
+ emit_insn (gen_mma_assemble_acc_dm (op0, vpair1, vpair2));
+ }
+
+ else
+ emit_insn (gen_mma_assemble_acc_nodm (op0, op1, op2, op3, op4));
+
DONE;
})
;; We cannot update the four output registers atomically, so mark the output
-;; as an early clobber so we don't accidentally clobber the input operands. */
+;; as an early clobber so we don't accidentally clobber the input operands.
-(define_insn_and_split "*mma_assemble_acc"
+(define_insn_and_split "mma_assemble_acc_nodm"
[(set (match_operand:XO 0 "fpr_reg_operand" "=&d")
(unspec_volatile:XO
[(match_operand:V16QI 1 "mma_assemble_input_operand" "mwa")
@@ -452,7 +493,7 @@
(match_operand:V16QI 3 "mma_assemble_input_operand" "mwa")
(match_operand:V16QI 4 "mma_assemble_input_operand" "mwa")]
UNSPECV_MMA_ASSEMBLE))]
- "TARGET_MMA
+ "TARGET_MMA_NO_DENSE_MATH
&& fpr_reg_operand (operands[0], XOmode)"
"#"
"&& reload_completed"
@@ -466,28 +507,31 @@
DONE;
})
+;; On a system with dense math, we build the accumulators from two vector
+;; pairs.
+
+(define_insn "mma_assemble_acc_dm"
+ [(set (match_operand:XO 0 "dmr_operand" "=wD")
+ (unspec:XO [(match_operand:OO 1 "vsx_register_operand" "wa")
+ (match_operand:OO 2 "vsx_register_operand" "wa")]
+ UNSPEC_DM_ASSEMBLE))]
+ "TARGET_DENSE_MATH"
+ "dmxxinstdmr512 %0,%1,%2,0"
+ [(set_attr "type" "mma")])
+
(define_expand "mma_disassemble_acc"
- [(match_operand:V16QI 0 "mma_disassemble_output_operand")
- (match_operand:XO 1 "fpr_reg_operand")
- (match_operand 2 "const_0_to_3_operand")]
- "TARGET_MMA"
-{
- rtx src;
- int regoff = INTVAL (operands[2]);
- src = gen_rtx_UNSPEC (V16QImode,
- gen_rtvec (2, operands[1], GEN_INT (regoff)),
- UNSPEC_MMA_EXTRACT);
- emit_move_insn (operands[0], src);
- DONE;
-})
+ [(set (match_operand:V16QI 0 "register_operand")
+ (unspec:V16QI [(match_operand:XO 1 "register_operand")
+ (match_operand 2 "const_0_to_3_operand")]
+ UNSPEC_MMA_EXTRACT))]
+ "TARGET_MMA")
-(define_insn_and_split "*mma_disassemble_acc"
+(define_insn_and_split "*mma_disassemble_acc_nodm"
[(set (match_operand:V16QI 0 "mma_disassemble_output_operand" "=mwa")
- (unspec:V16QI [(match_operand:XO 1 "fpr_reg_operand" "d")
- (match_operand 2 "const_0_to_3_operand")]
+ (unspec:V16QI [(match_operand:XO 1 "fpr_reg_operand" "d")
+ (match_operand 2 "const_0_to_3_operand")]
UNSPEC_MMA_EXTRACT))]
- "TARGET_MMA
- && fpr_reg_operand (operands[1], XOmode)"
+ "TARGET_MMA_NO_DENSE_MATH"
"#"
"&& reload_completed"
[(const_int 0)]
@@ -499,194 +543,258 @@
DONE;
})
-;; MMA instructions that do not use their accumulators as an input, still
-;; must not allow their vector operands to overlap the registers used by
-;; the accumulator. We enforce this by marking the output as early clobber.
+(define_insn "*mma_disassemble_acc_dm"
+ [(set (match_operand:V16QI 0 "vsx_register_operand" "=wa")
+ (unspec:V16QI [(match_operand:XO 1 "dmr_operand" "wD")
+ (match_operand 2 "const_0_to_3_operand")]
+ UNSPEC_MMA_EXTRACT))]
+ "TARGET_DENSE_MATH"
+ "dmxxextfdmr256 %0,%1,2"
+ [(set_attr "type" "mma")])
+
+;; MMA instructions that do not use their accumulators as an input, still must
+;; not allow their vector operands to overlap the registers used by the
+;; accumulator. We enforce this by marking the output as early clobber. If we
+;; have dense math, we don't need the whole prime/de-prime action, so just make
+;; these instructions be NOPs.
-(define_insn "mma_<acc>"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=&d")
- (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0")]
- MMA_ACC))]
+(define_expand "mma_<acc>"
+ [(set (match_operand:XO 0 "accumulator_operand")
+ (unspec:XO [(match_operand:XO 1 "accumulator_operand")]
+ MMA_ACC))]
"TARGET_MMA"
+{
+ if (TARGET_DENSE_MATH)
+ {
+ if (!rtx_equal_p (operands[0], operands[1]))
+ emit_move_insn (operands[0], operands[1]);
+ DONE;
+ }
+
+ /* Generate the prime/de-prime code. */
+})
+
+(define_insn "*mma_<acc>_nodm"
+ [(set (match_operand:XO 0 "accumulator_operand" "=&d")
+ (unspec:XO [(match_operand:XO 1 "accumulator_operand" "0")]
+ MMA_ACC))]
+ "TARGET_MMA_NO_DENSE_MATH"
"<acc> %A0"
[(set_attr "type" "mma")])
;; We can't have integer constants in XOmode so we wrap this in an
-;; UNSPEC_VOLATILE.
+;; UNSPEC_VOLATILE for the non-dense math case. For dense math, we don't need
+;; to disable optimization and we can do a normal UNSPEC.
-(define_insn "mma_xxsetaccz"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=d")
+(define_expand "mma_xxsetaccz"
+ [(set (match_operand:XO 0 "register_operand")
(unspec_volatile:XO [(const_int 0)]
UNSPECV_MMA_XXSETACCZ))]
"TARGET_MMA"
+{
+ if (TARGET_DENSE_MATH)
+ {
+ emit_insn (gen_mma_xxsetaccz_dm (operands[0]));
+ DONE;
+ }
+})
+
+(define_insn "*mma_xxsetaccz_nodm"
+ [(set (match_operand:XO 0 "fpr_reg_operand" "=d")
+ (unspec_volatile:XO [(const_int 0)]
+ UNSPECV_MMA_XXSETACCZ))]
+ "TARGET_MMA_NO_DENSE_MATH"
"xxsetaccz %A0"
[(set_attr "type" "mma")])
+
+(define_insn "mma_xxsetaccz_dm"
+ [(set (match_operand:XO 0 "dmr_operand" "=wD")
+ (unspec:XO [(const_int 0)]
+ UNSPECV_MMA_XXSETACCZ))]
+ "TARGET_DENSE_MATH"
+ "dmsetdmrz %0"
+ [(set_attr "type" "mma")])
+
(define_insn "mma_<vv>"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d")
- (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "v,?wa")
- (match_operand:V16QI 2 "vsx_register_operand" "v,?wa")]
+ [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d")
+ (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa")]
MMA_VV))]
"TARGET_MMA"
"<vv> %A0,%x1,%x2"
- [(set_attr "type" "mma")])
+ [(set_attr "type" "mma")
+ (set_attr "isa" "dm,not_dm,not_dm")])
(define_insn "mma_<avv>"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d")
- (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0,0")
- (match_operand:V16QI 2 "vsx_register_operand" "v,?wa")
- (match_operand:V16QI 3 "vsx_register_operand" "v,?wa")]
+ [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d")
+ (unspec:XO [(match_operand:XO 1 "accumulator_operand" "0,0,0")
+ (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:V16QI 3 "vsx_register_operand" "wa,v,?wa")]
MMA_AVV))]
"TARGET_MMA"
"<avv> %A0,%x2,%x3"
- [(set_attr "type" "mma")])
+ [(set_attr "type" "mma")
+ (set_attr "isa" "dm,not_dm,not_dm")])
(define_insn "mma_<pv>"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d")
- (unspec:XO [(match_operand:OO 1 "vsx_register_operand" "v,?wa")
- (match_operand:V16QI 2 "vsx_register_operand" "v,?wa")]
+ [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d")
+ (unspec:XO [(match_operand:OO 1 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa")]
MMA_PV))]
"TARGET_MMA"
"<pv> %A0,%x1,%x2"
- [(set_attr "type" "mma")])
+ [(set_attr "type" "mma")
+ (set_attr "isa" "dm,not_dm,not_dm")])
(define_insn "mma_<apv>"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d")
- (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0,0")
- (match_operand:OO 2 "vsx_register_operand" "v,?wa")
- (match_operand:V16QI 3 "vsx_register_operand" "v,?wa")]
+ [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d")
+ (unspec:XO [(match_operand:XO 1 "accumulator_operand" "0,0,0")
+ (match_operand:OO 2 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:V16QI 3 "vsx_register_operand" "wa,v,?wa")]
MMA_APV))]
"TARGET_MMA"
"<apv> %A0,%x2,%x3"
- [(set_attr "type" "mma")])
+ [(set_attr "type" "mma")
+ (set_attr "isa" "dm,not_dm,not_dm")])
(define_insn "mma_<vvi4i4i8>"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d")
- (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "v,?wa")
- (match_operand:V16QI 2 "vsx_register_operand" "v,?wa")
- (match_operand:SI 3 "const_0_to_15_operand" "n,n")
- (match_operand:SI 4 "const_0_to_15_operand" "n,n")
- (match_operand:SI 5 "u8bit_cint_operand" "n,n")]
+ [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d")
+ (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:SI 3 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 4 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 5 "u8bit_cint_operand" "n,n,n")]
MMA_VVI4I4I8))]
"TARGET_MMA"
"<vvi4i4i8> %A0,%x1,%x2,%3,%4,%5"
[(set_attr "type" "mma")
- (set_attr "prefixed" "yes")])
+ (set_attr "prefixed" "yes")
+ (set_attr "isa" "dm,not_dm,not_dm")])
(define_insn "mma_<avvi4i4i8>"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d")
- (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0,0")
- (match_operand:V16QI 2 "vsx_register_operand" "v,?wa")
- (match_operand:V16QI 3 "vsx_register_operand" "v,?wa")
- (match_operand:SI 4 "const_0_to_15_operand" "n,n")
- (match_operand:SI 5 "const_0_to_15_operand" "n,n")
- (match_operand:SI 6 "u8bit_cint_operand" "n,n")]
+ [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d")
+ (unspec:XO [(match_operand:XO 1 "accumulator_operand" "0,0,0")
+ (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:V16QI 3 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:SI 4 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 5 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 6 "u8bit_cint_operand" "n,n,n")]
MMA_AVVI4I4I8))]
"TARGET_MMA"
"<avvi4i4i8> %A0,%x2,%x3,%4,%5,%6"
[(set_attr "type" "mma")
- (set_attr "prefixed" "yes")])
+ (set_attr "prefixed" "yes")
+ (set_attr "isa" "dm,not_dm,not_dm")])
(define_insn "mma_<vvi4i4i2>"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d")
- (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "v,?wa")
- (match_operand:V16QI 2 "vsx_register_operand" "v,?wa")
- (match_operand:SI 3 "const_0_to_15_operand" "n,n")
- (match_operand:SI 4 "const_0_to_15_operand" "n,n")
- (match_operand:SI 5 "const_0_to_3_operand" "n,n")]
+ [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d")
+ (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:SI 3 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 4 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 5 "const_0_to_3_operand" "n,n,n")]
MMA_VVI4I4I2))]
"TARGET_MMA"
"<vvi4i4i2> %A0,%x1,%x2,%3,%4,%5"
[(set_attr "type" "mma")
- (set_attr "prefixed" "yes")])
+ (set_attr "prefixed" "yes")
+ (set_attr "isa" "dm,not_dm,not_dm")])
(define_insn "mma_<avvi4i4i2>"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d")
- (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0,0")
- (match_operand:V16QI 2 "vsx_register_operand" "v,?wa")
- (match_operand:V16QI 3 "vsx_register_operand" "v,?wa")
- (match_operand:SI 4 "const_0_to_15_operand" "n,n")
- (match_operand:SI 5 "const_0_to_15_operand" "n,n")
- (match_operand:SI 6 "const_0_to_3_operand" "n,n")]
+ [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d")
+ (unspec:XO [(match_operand:XO 1 "accumulator_operand" "0,0,0")
+ (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:V16QI 3 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:SI 4 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 5 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 6 "const_0_to_3_operand" "n,n,n")]
MMA_AVVI4I4I2))]
"TARGET_MMA"
"<avvi4i4i2> %A0,%x2,%x3,%4,%5,%6"
[(set_attr "type" "mma")
- (set_attr "prefixed" "yes")])
+ (set_attr "prefixed" "yes")
+ (set_attr "isa" "dm,not_dm,not_dm")])
(define_insn "mma_<vvi4i4>"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d")
- (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "v,?wa")
- (match_operand:V16QI 2 "vsx_register_operand" "v,?wa")
- (match_operand:SI 3 "const_0_to_15_operand" "n,n")
- (match_operand:SI 4 "const_0_to_15_operand" "n,n")]
+ [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d")
+ (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:SI 3 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 4 "const_0_to_15_operand" "n,n,n")]
MMA_VVI4I4))]
"TARGET_MMA"
"<vvi4i4> %A0,%x1,%x2,%3,%4"
[(set_attr "type" "mma")
- (set_attr "prefixed" "yes")])
+ (set_attr "prefixed" "yes")
+ (set_attr "isa" "dm,not_dm,not_dm")])
(define_insn "mma_<avvi4i4>"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d")
- (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0,0")
- (match_operand:V16QI 2 "vsx_register_operand" "v,?wa")
- (match_operand:V16QI 3 "vsx_register_operand" "v,?wa")
- (match_operand:SI 4 "const_0_to_15_operand" "n,n")
- (match_operand:SI 5 "const_0_to_15_operand" "n,n")]
+ [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d")
+ (unspec:XO [(match_operand:XO 1 "accumulator_operand" "0,0,0")
+ (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:V16QI 3 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:SI 4 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 5 "const_0_to_15_operand" "n,n,n")]
MMA_AVVI4I4))]
"TARGET_MMA"
"<avvi4i4> %A0,%x2,%x3,%4,%5"
[(set_attr "type" "mma")
- (set_attr "prefixed" "yes")])
+ (set_attr "prefixed" "yes")
+ (set_attr "isa" "dm,not_dm,not_dm")])
(define_insn "mma_<pvi4i2>"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d")
- (unspec:XO [(match_operand:OO 1 "vsx_register_operand" "v,?wa")
- (match_operand:V16QI 2 "vsx_register_operand" "v,?wa")
- (match_operand:SI 3 "const_0_to_15_operand" "n,n")
- (match_operand:SI 4 "const_0_to_3_operand" "n,n")]
+ [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d")
+ (unspec:XO [(match_operand:OO 1 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:SI 3 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 4 "const_0_to_3_operand" "n,n,n")]
MMA_PVI4I2))]
"TARGET_MMA"
"<pvi4i2> %A0,%x1,%x2,%3,%4"
[(set_attr "type" "mma")
- (set_attr "prefixed" "yes")])
+ (set_attr "prefixed" "yes")
+ (set_attr "isa" "dm,not_dm,not_dm")])
(define_insn "mma_<apvi4i2>"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d")
- (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0,0")
- (match_operand:OO 2 "vsx_register_operand" "v,?wa")
- (match_operand:V16QI 3 "vsx_register_operand" "v,?wa")
- (match_operand:SI 4 "const_0_to_15_operand" "n,n")
- (match_operand:SI 5 "const_0_to_3_operand" "n,n")]
+ [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d")
+ (unspec:XO [(match_operand:XO 1 "accumulator_operand" "0,0,0")
+ (match_operand:OO 2 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:V16QI 3 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:SI 4 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 5 "const_0_to_3_operand" "n,n,n")]
MMA_APVI4I2))]
"TARGET_MMA"
"<apvi4i2> %A0,%x2,%x3,%4,%5"
[(set_attr "type" "mma")
- (set_attr "prefixed" "yes")])
+ (set_attr "prefixed" "yes")
+ (set_attr "isa" "dm,not_dm,not_dm")])
(define_insn "mma_<vvi4i4i4>"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d")
- (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "v,?wa")
- (match_operand:V16QI 2 "vsx_register_operand" "v,?wa")
- (match_operand:SI 3 "const_0_to_15_operand" "n,n")
- (match_operand:SI 4 "const_0_to_15_operand" "n,n")
- (match_operand:SI 5 "const_0_to_15_operand" "n,n")]
+ [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d")
+ (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:SI 3 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 4 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 5 "const_0_to_15_operand" "n,n,n")]
MMA_VVI4I4I4))]
"TARGET_MMA"
"<vvi4i4i4> %A0,%x1,%x2,%3,%4,%5"
[(set_attr "type" "mma")
- (set_attr "prefixed" "yes")])
+ (set_attr "prefixed" "yes")
+ (set_attr "isa" "dm,not_dm,not_dm")])
(define_insn "mma_<avvi4i4i4>"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d")
- (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0,0")
- (match_operand:V16QI 2 "vsx_register_operand" "v,?wa")
- (match_operand:V16QI 3 "vsx_register_operand" "v,?wa")
- (match_operand:SI 4 "const_0_to_15_operand" "n,n")
- (match_operand:SI 5 "const_0_to_15_operand" "n,n")
- (match_operand:SI 6 "const_0_to_15_operand" "n,n")]
+ [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d")
+ (unspec:XO [(match_operand:XO 1 "accumulator_operand" "0,0,0")
+ (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:V16QI 3 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:SI 4 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 5 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 6 "const_0_to_15_operand" "n,n,n")]
MMA_AVVI4I4I4))]
"TARGET_MMA"
"<avvi4i4i4> %A0,%x2,%x3,%4,%5,%6"
[(set_attr "type" "mma")
- (set_attr "prefixed" "yes")])
+ (set_attr "prefixed" "yes")
+ (set_attr "isa" "dm,not_dm,not_dm")])
diff --git a/gcc/config/rs6000/predicates.md b/gcc/config/rs6000/predicates.md
index d23ce9a77a3..3040dcd50a3 100644
--- a/gcc/config/rs6000/predicates.md
+++ b/gcc/config/rs6000/predicates.md
@@ -186,6 +186,38 @@
return VLOGICAL_REGNO_P (REGNO (op));
})
+;; Return 1 if op is a DMR register
+(define_predicate "dmr_operand"
+ (match_operand 0 "register_operand")
+{
+ if (!REG_P (op))
+ return 0;
+
+ if (!HARD_REGISTER_P (op))
+ return 1;
+
+ return DMR_REGNO_P (REGNO (op));
+})
+
+;; Return 1 if op is an accumulator. On power10 systems, the accumulators
+;; overlap with the FPRs, while on systems with dense math, the accumulators
+;; are separate dense math registers and do not overlap with the FPR
+;; registers..
+(define_predicate "accumulator_operand"
+ (match_operand 0 "register_operand")
+{
+ if (!REG_P (op))
+ return 0;
+
+ if (!HARD_REGISTER_P (op))
+ return 1;
+
+ int r = REGNO (op);
+ return (TARGET_DENSE_MATH
+ ? DMR_REGNO_P (r)
+ : FP_REGNO_P (r) && (r & 3) == 0);
+})
+
;; Return 1 if op is the carry register.
(define_predicate "ca_operand"
(match_operand 0 "register_operand")
diff --git a/gcc/config/rs6000/rs6000-c.cc b/gcc/config/rs6000/rs6000-c.cc
index d15bb85743c..ef65ed8c259 100644
--- a/gcc/config/rs6000/rs6000-c.cc
+++ b/gcc/config/rs6000/rs6000-c.cc
@@ -602,6 +602,9 @@ rs6000_target_modify_macros (bool define_p, HOST_WIDE_INT flags)
/* Tell the user if we support the MMA instructions. */
if ((flags & OPTION_MASK_MMA) != 0)
rs6000_define_or_undefine_macro (define_p, "__MMA__");
+ /* Tell the user if we support the dense math instructions. */
+ if ((flags & DENSE_MATH_FLAGS) == DENSE_MATH_FLAGS)
+ rs6000_define_or_undefine_macro (define_p, "__PPC_DMR__");
/* Whether pc-relative code is being generated. */
if ((flags & OPTION_MASK_PCREL) != 0)
rs6000_define_or_undefine_macro (define_p, "__PCREL__");
diff --git a/gcc/config/rs6000/rs6000.cc b/gcc/config/rs6000/rs6000.cc
index b1ad49d3734..95ecf922143 100644
--- a/gcc/config/rs6000/rs6000.cc
+++ b/gcc/config/rs6000/rs6000.cc
@@ -292,7 +292,8 @@ enum rs6000_reg_type {
ALTIVEC_REG_TYPE,
FPR_REG_TYPE,
SPR_REG_TYPE,
- CR_REG_TYPE
+ CR_REG_TYPE,
+ DMR_REG_TYPE
};
/* Map register class to register type. */
@@ -306,22 +307,23 @@ static enum rs6000_reg_type reg_class_to_reg_type[N_REG_CLASSES];
/* Register classes we care about in secondary reload or go if legitimate
- address. We only need to worry about GPR, FPR, and Altivec registers here,
- along an ANY field that is the OR of the 3 register classes. */
+ address. We only need to worry about GPR, FPR, Altivec, and DMR registers
+ here, along an ANY field that is the OR of the 4 register classes. */
enum rs6000_reload_reg_type {
RELOAD_REG_GPR, /* General purpose registers. */
RELOAD_REG_FPR, /* Traditional floating point regs. */
RELOAD_REG_VMX, /* Altivec (VMX) registers. */
- RELOAD_REG_ANY, /* OR of GPR, FPR, Altivec masks. */
+ RELOAD_REG_DMR, /* DMR registers. */
+ RELOAD_REG_ANY, /* OR of GPR/FPR/VMX/DMR masks. */
N_RELOAD_REG
};
-/* For setting up register classes, loop through the 3 register classes mapping
+/* For setting up register classes, loop through the 4 register classes mapping
into real registers, and skip the ANY class, which is just an OR of the
bits. */
#define FIRST_RELOAD_REG_CLASS RELOAD_REG_GPR
-#define LAST_RELOAD_REG_CLASS RELOAD_REG_VMX
+#define LAST_RELOAD_REG_CLASS RELOAD_REG_DMR
/* Map reload register type to a register in the register class. */
struct reload_reg_map_type {
@@ -333,6 +335,7 @@ static const struct reload_reg_map_type reload_reg_map[N_RELOAD_REG] = {
{ "Gpr", FIRST_GPR_REGNO }, /* RELOAD_REG_GPR. */
{ "Fpr", FIRST_FPR_REGNO }, /* RELOAD_REG_FPR. */
{ "VMX", FIRST_ALTIVEC_REGNO }, /* RELOAD_REG_VMX. */
+ { "DMR", FIRST_DMR_REGNO }, /* RELOAD_REG_DMR. */
{ "Any", -1 }, /* RELOAD_REG_ANY. */
};
@@ -1226,6 +1229,8 @@ char rs6000_reg_names[][8] =
"0", "1", "2", "3", "4", "5", "6", "7",
/* vrsave vscr sfp */
"vrsave", "vscr", "sfp",
+ /* DMRs */
+ "0", "1", "2", "3", "4", "5", "6", "7",
};
#ifdef TARGET_REGNAMES
@@ -1252,6 +1257,8 @@ static const char alt_reg_names[][8] =
"%cr0", "%cr1", "%cr2", "%cr3", "%cr4", "%cr5", "%cr6", "%cr7",
/* vrsave vscr sfp */
"vrsave", "vscr", "sfp",
+ /* DMRs */
+ "%dmr0", "%dmr1", "%dmr2", "%dmr3", "%dmr4", "%dmr5", "%dmr6", "%dmr7",
};
#endif
@@ -1836,6 +1843,9 @@ rs6000_hard_regno_nregs_internal (int regno, machine_mode mode)
else if (ALTIVEC_REGNO_P (regno))
reg_size = UNITS_PER_ALTIVEC_WORD;
+ else if (DMR_REGNO_P (regno))
+ reg_size = UNITS_PER_DMR_WORD;
+
else
reg_size = UNITS_PER_WORD;
@@ -1857,9 +1867,36 @@ rs6000_hard_regno_mode_ok_uncached (int regno, machine_mode mode)
if (mode == OOmode)
return (TARGET_MMA && VSX_REGNO_P (regno) && (regno & 1) == 0);
- /* MMA accumulator modes need FPR registers divisible by 4. */
+ /* On ISA 3.1 (power10), MMA accumulator modes need FPR registers divisible
+ by 4.
+
+ If dense math is enabled, allow all VSX registers plus the DMR registers.
+ We need to make sure we don't cross between the boundary of FPRs and
+ traditional Altiviec registers. */
if (mode == XOmode)
- return (TARGET_MMA && FP_REGNO_P (regno) && (regno & 3) == 0);
+ {
+ if (TARGET_MMA && !TARGET_DENSE_MATH)
+ return (FP_REGNO_P (regno) && (regno & 3) == 0);
+
+ else if (TARGET_DENSE_MATH)
+ {
+ if (DMR_REGNO_P (regno))
+ return 1;
+
+ if (FP_REGNO_P (regno))
+ return ((regno & 1) == 0 && regno <= LAST_FPR_REGNO - 3);
+
+ if (ALTIVEC_REGNO_P (regno))
+ return ((regno & 1) == 0 && regno <= LAST_ALTIVEC_REGNO - 3);
+ }
+
+ else
+ return 0;
+ }
+
+ /* No other types other than XOmode can go in DMRs. */
+ if (DMR_REGNO_P (regno))
+ return 0;
/* PTImode can only go in GPRs. Quad word memory operations require even/odd
register combinations, and use PTImode where we need to deal with quad
@@ -2302,6 +2339,7 @@ rs6000_debug_reg_global (void)
rs6000_debug_reg_print (FIRST_ALTIVEC_REGNO,
LAST_ALTIVEC_REGNO,
"vs");
+ rs6000_debug_reg_print (FIRST_DMR_REGNO, LAST_DMR_REGNO, "dmr");
rs6000_debug_reg_print (LR_REGNO, LR_REGNO, "lr");
rs6000_debug_reg_print (CTR_REGNO, CTR_REGNO, "ctr");
rs6000_debug_reg_print (CR0_REGNO, CR7_REGNO, "cr");
@@ -2322,6 +2360,7 @@ rs6000_debug_reg_global (void)
"wr reg_class = %s\n"
"wx reg_class = %s\n"
"wA reg_class = %s\n"
+ "wD reg_class = %s\n"
"\n",
reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_d]],
reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_v]],
@@ -2329,7 +2368,8 @@ rs6000_debug_reg_global (void)
reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_we]],
reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_wr]],
reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_wx]],
- reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_wA]]);
+ reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_wA]],
+ reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_wD]]);
nl = "\n";
for (m = 0; m < NUM_MACHINE_MODES; ++m)
@@ -2626,6 +2666,21 @@ rs6000_setup_reg_addr_masks (void)
addr_mask = 0;
reg = reload_reg_map[rc].reg;
+ /* Special case DMR registers. */
+ if (rc == RELOAD_REG_DMR)
+ {
+ if (TARGET_DENSE_MATH && m2 == XOmode)
+ {
+ addr_mask = RELOAD_REG_VALID;
+ reg_addr[m].addr_mask[rc] = addr_mask;
+ any_addr_mask |= addr_mask;
+ }
+ else
+ reg_addr[m].addr_mask[rc] = 0;
+
+ continue;
+ }
+
/* Can mode values go in the GPR/FPR/Altivec registers? */
if (reg >= 0 && rs6000_hard_regno_mode_ok_p[m][reg])
{
@@ -2776,6 +2831,9 @@ rs6000_init_hard_regno_mode_ok (bool global_init_p)
for (r = CR1_REGNO; r <= CR7_REGNO; ++r)
rs6000_regno_regclass[r] = CR_REGS;
+ for (r = FIRST_DMR_REGNO; r <= LAST_DMR_REGNO; ++r)
+ rs6000_regno_regclass[r] = DM_REGS;
+
rs6000_regno_regclass[LR_REGNO] = LINK_REGS;
rs6000_regno_regclass[CTR_REGNO] = CTR_REGS;
rs6000_regno_regclass[CA_REGNO] = NO_REGS;
@@ -2800,6 +2858,7 @@ rs6000_init_hard_regno_mode_ok (bool global_init_p)
reg_class_to_reg_type[(int)LINK_OR_CTR_REGS] = SPR_REG_TYPE;
reg_class_to_reg_type[(int)CR_REGS] = CR_REG_TYPE;
reg_class_to_reg_type[(int)CR0_REGS] = CR_REG_TYPE;
+ reg_class_to_reg_type[(int)DM_REGS] = DMR_REG_TYPE;
if (TARGET_VSX)
{
@@ -2986,6 +3045,13 @@ rs6000_init_hard_regno_mode_ok (bool global_init_p)
if (TARGET_DIRECT_MOVE_128)
rs6000_constraints[RS6000_CONSTRAINT_we] = VSX_REGS;
+ /* Support for the accumulator registers, either FPR registers (aka original
+ mma) or DMR registers (dense math). */
+ if (TARGET_DENSE_MATH)
+ rs6000_constraints[RS6000_CONSTRAINT_wD] = DM_REGS;
+ else if (TARGET_MMA)
+ rs6000_constraints[RS6000_CONSTRAINT_wD] = FLOAT_REGS;
+
/* Set up the reload helper and direct move functions. */
if (TARGET_VSX || TARGET_ALTIVEC)
{
@@ -12311,6 +12377,11 @@ rs6000_secondary_reload_memory (rtx addr,
addr_mask = (reg_addr[mode].addr_mask[RELOAD_REG_VMX]
& ~RELOAD_REG_AND_M16);
+ /* DMR registers use VSX registers, and need to generate some extra
+ instructions. */
+ else if (rclass == DM_REGS)
+ return 2;
+
/* If the register allocator hasn't made up its mind yet on the register
class to use, settle on defaults to use. */
else if (rclass == NO_REGS)
@@ -12639,6 +12710,13 @@ rs6000_secondary_reload_simple_move (enum rs6000_reg_type to_type,
|| (to_type == SPR_REG_TYPE && from_type == GPR_REG_TYPE)))
return true;
+ /* We can transfer between VSX registers and DMR registers without needing
+ extra registers. */
+ if (TARGET_DENSE_MATH && mode == XOmode
+ && ((to_type == DMR_REG_TYPE && from_type == VSX_REG_TYPE)
+ || (to_type == VSX_REG_TYPE && from_type == DMR_REG_TYPE)))
+ return true;
+
return false;
}
@@ -13333,6 +13411,10 @@ rs6000_preferred_reload_class (rtx x, enum reg_class rclass)
machine_mode mode = GET_MODE (x);
bool is_constant = CONSTANT_P (x);
+ /* DMR registers can't be loaded or stored. */
+ if (rclass == DM_REGS)
+ return NO_REGS;
+
/* If a mode can't go in FPR/ALTIVEC/VSX registers, don't return a preferred
reload class for it. */
if ((rclass == ALTIVEC_REGS || rclass == VSX_REGS)
@@ -13429,7 +13511,7 @@ rs6000_preferred_reload_class (rtx x, enum reg_class rclass)
return VSX_REGS;
if (mode == XOmode)
- return FLOAT_REGS;
+ return TARGET_DENSE_MATH ? VSX_REGS : FLOAT_REGS;
if (GET_MODE_CLASS (mode) == MODE_INT)
return GENERAL_REGS;
@@ -13554,6 +13636,11 @@ rs6000_secondary_reload_class (enum reg_class rclass, machine_mode mode,
else
regno = -1;
+ /* DMR registers don't have loads or stores. We have to go through the VSX
+ registers to load XOmode (vector quad). */
+ if (TARGET_DENSE_MATH && rclass == DM_REGS)
+ return VSX_REGS;
+
/* If we have VSX register moves, prefer moving scalar values between
Altivec registers and GPR by going via an FPR (and then via memory)
instead of reloading the secondary memory address for Altivec moves. */
@@ -14067,8 +14154,19 @@ print_operand (FILE *file, rtx x, int code)
output_operand. */
case 'A':
- /* Write the MMA accumulator number associated with VSX register X. */
- if (!REG_P (x) || !FP_REGNO_P (REGNO (x)) || (REGNO (x) % 4) != 0)
+ /* Write the MMA accumulator number associated with VSX register X. On
+ dense math systems, only allow DMR accumulators, not accumulators
+ overlapping with the FPR registers. */
+ if (!REG_P (x))
+ output_operand_lossage ("invalid %%A value");
+ else if (TARGET_DENSE_MATH)
+ {
+ if (DMR_REGNO_P (REGNO (x)))
+ fprintf (file, "%d", REGNO (x) - FIRST_DMR_REGNO);
+ else
+ output_operand_lossage ("%%A operand is not a DMR");
+ }
+ else if (!FP_REGNO_P (REGNO (x)) || (REGNO (x) % 4) != 0)
output_operand_lossage ("invalid %%A value");
else
fprintf (file, "%d", (REGNO (x) - FIRST_FPR_REGNO) / 4);
@@ -22738,6 +22836,31 @@ rs6000_debug_address_cost (rtx x, machine_mode mode,
}
+/* Subroutine to determine the move cost of dense math registers. If we are
+ moving to/from VSX_REGISTER registers, the cost is either 1 move (for
+ 512-bit accumulators) or 2 moves (for 1,024 dmr registers). If we are
+ moving to anything else like GPR registers, make the cost very high. */
+
+static int
+rs6000_dmr_register_move_cost (machine_mode mode, reg_class_t rclass)
+{
+ const int reg_move_base = 2;
+ HARD_REG_SET vsx_set = (reg_class_contents[rclass]
+ & reg_class_contents[VSX_REGS]);
+
+ if (TARGET_DENSE_MATH && !hard_reg_set_empty_p (vsx_set))
+ {
+ /* __vector_quad (i.e. XOmode) is tranfered in 1 instruction. */
+ if (mode == XOmode)
+ return reg_move_base;
+
+ else
+ return reg_move_base * 2 * hard_regno_nregs (FIRST_DMR_REGNO, mode);
+ }
+
+ return 1000 * 2 * hard_regno_nregs (FIRST_DMR_REGNO, mode);
+}
+
/* A C expression returning the cost of moving data from a register of class
CLASS1 to one of CLASS2. */
@@ -22751,17 +22874,28 @@ rs6000_register_move_cost (machine_mode mode,
if (TARGET_DEBUG_COST)
dbg_cost_ctrl++;
+ HARD_REG_SET to_vsx, from_vsx;
+ to_vsx = reg_class_contents[to] & reg_class_contents[VSX_REGS];
+ from_vsx = reg_class_contents[from] & reg_class_contents[VSX_REGS];
+
+ /* Special case DMR registers, that can only move to/from VSX registers. */
+ if (from == DM_REGS && to == DM_REGS)
+ ret = 2 * hard_regno_nregs (FIRST_DMR_REGNO, mode);
+
+ else if (from == DM_REGS)
+ ret = rs6000_dmr_register_move_cost (mode, to);
+
+ else if (to == DM_REGS)
+ ret = rs6000_dmr_register_move_cost (mode, from);
+
/* If we have VSX, we can easily move between FPR or Altivec registers,
otherwise we can only easily move within classes.
Do this first so we give best-case answers for union classes
containing both gprs and vsx regs. */
- HARD_REG_SET to_vsx, from_vsx;
- to_vsx = reg_class_contents[to] & reg_class_contents[VSX_REGS];
- from_vsx = reg_class_contents[from] & reg_class_contents[VSX_REGS];
- if (!hard_reg_set_empty_p (to_vsx)
- && !hard_reg_set_empty_p (from_vsx)
- && (TARGET_VSX
- || hard_reg_set_intersect_p (to_vsx, from_vsx)))
+ else if (!hard_reg_set_empty_p (to_vsx)
+ && !hard_reg_set_empty_p (from_vsx)
+ && (TARGET_VSX
+ || hard_reg_set_intersect_p (to_vsx, from_vsx)))
{
int reg = FIRST_FPR_REGNO;
if (TARGET_VSX
@@ -22856,6 +22990,9 @@ rs6000_memory_move_cost (machine_mode mode, reg_class_t rclass,
ret = 4 * hard_regno_nregs (32, mode);
else if (reg_classes_intersect_p (rclass, ALTIVEC_REGS))
ret = 4 * hard_regno_nregs (FIRST_ALTIVEC_REGNO, mode);
+ else if (reg_classes_intersect_p (rclass, DM_REGS))
+ ret = (rs6000_dmr_register_move_cost (mode, VSX_REGS)
+ + rs6000_memory_move_cost (mode, VSX_REGS, false));
else
ret = 4 + rs6000_register_move_cost (mode, rclass, GENERAL_REGS);
@@ -24064,6 +24201,8 @@ rs6000_compute_pressure_classes (enum reg_class *pressure_classes)
if (TARGET_HARD_FLOAT)
pressure_classes[n++] = FLOAT_REGS;
}
+ if (TARGET_DENSE_MATH)
+ pressure_classes[n++] = DM_REGS;
pressure_classes[n++] = CR_REGS;
pressure_classes[n++] = SPECIAL_REGS;
@@ -24228,6 +24367,10 @@ rs6000_debugger_regno (unsigned int regno, unsigned int format)
return 67;
if (regno == 64)
return 64;
+ /* XXX: This is a guess. The GCC register number for FIRST_DMR_REGNO is 111,
+ but the frame pointer regnum uses that. */
+ if (DMR_REGNO_P (regno))
+ return regno - FIRST_DMR_REGNO + 112;
gcc_unreachable ();
}
@@ -27471,7 +27614,7 @@ rs6000_split_multireg_move (rtx dst, rtx src)
/* If we are reading an accumulator register, we have to
deprime it before we can access it. */
- if (TARGET_MMA
+ if (TARGET_MMA && !TARGET_DENSE_MATH
&& GET_MODE (src) == XOmode && FP_REGNO_P (REGNO (src)))
emit_insn (gen_mma_xxmfacc (src, src));
@@ -27503,9 +27646,9 @@ rs6000_split_multireg_move (rtx dst, rtx src)
emit_insn (gen_rtx_SET (dst2, src2));
}
- /* If we are writing an accumulator register, we have to
- prime it after we've written it. */
- if (TARGET_MMA
+ /* If we are writing an accumulator register that overlaps with the
+ FPR registers, we have to prime it after we've written it. */
+ if (TARGET_MMA && !TARGET_DENSE_MATH
&& GET_MODE (dst) == XOmode && FP_REGNO_P (REGNO (dst)))
emit_insn (gen_mma_xxmtacc (dst, dst));
@@ -27519,7 +27662,9 @@ rs6000_split_multireg_move (rtx dst, rtx src)
|| XINT (src, 1) == UNSPECV_MMA_ASSEMBLE);
gcc_assert (REG_P (dst));
if (GET_MODE (src) == XOmode)
- gcc_assert (FP_REGNO_P (REGNO (dst)));
+ gcc_assert ((TARGET_DENSE_MATH
+ ? VSX_REGNO_P (REGNO (dst))
+ : FP_REGNO_P (REGNO (dst))));
if (GET_MODE (src) == OOmode)
gcc_assert (VSX_REGNO_P (REGNO (dst)));
@@ -27572,9 +27717,9 @@ rs6000_split_multireg_move (rtx dst, rtx src)
emit_insn (gen_rtx_SET (dst_i, op));
}
- /* We are writing an accumulator register, so we have to
- prime it after we've written it. */
- if (GET_MODE (src) == XOmode)
+ /* On systems without dense math where accumulators overlap with the
+ vector registers, we have to prime it after we've written it. */
+ if (GET_MODE (src) == XOmode && !TARGET_DENSE_MATH)
emit_insn (gen_mma_xxmtacc (dst, dst));
return;
@@ -27585,9 +27730,9 @@ rs6000_split_multireg_move (rtx dst, rtx src)
if (REG_P (src) && REG_P (dst) && (REGNO (src) < REGNO (dst)))
{
- /* If we are reading an accumulator register, we have to
- deprime it before we can access it. */
- if (TARGET_MMA
+ /* If we are reading an accumulator register and we don't have dense
+ math, we have to deprime it before we can access it. */
+ if (TARGET_MMA && !TARGET_DENSE_MATH
&& GET_MODE (src) == XOmode && FP_REGNO_P (REGNO (src)))
emit_insn (gen_mma_xxmfacc (src, src));
@@ -27615,7 +27760,7 @@ rs6000_split_multireg_move (rtx dst, rtx src)
/* If we are writing an accumulator register, we have to
prime it after we've written it. */
- if (TARGET_MMA
+ if (TARGET_MMA && !TARGET_DENSE_MATH
&& GET_MODE (dst) == XOmode && FP_REGNO_P (REGNO (dst)))
emit_insn (gen_mma_xxmtacc (dst, dst));
}
@@ -27752,7 +27897,7 @@ rs6000_split_multireg_move (rtx dst, rtx src)
/* If we are reading an accumulator register, we have to
deprime it before we can access it. */
- if (TARGET_MMA && REG_P (src)
+ if (TARGET_MMA && !TARGET_DENSE_MATH && REG_P (src)
&& GET_MODE (src) == XOmode && FP_REGNO_P (REGNO (src)))
emit_insn (gen_mma_xxmfacc (src, src));
@@ -27784,7 +27929,7 @@ rs6000_split_multireg_move (rtx dst, rtx src)
/* If we are writing an accumulator register, we have to
prime it after we've written it. */
- if (TARGET_MMA && REG_P (dst)
+ if (TARGET_MMA && !TARGET_DENSE_MATH && REG_P (dst)
&& GET_MODE (dst) == XOmode && FP_REGNO_P (REGNO (dst)))
emit_insn (gen_mma_xxmtacc (dst, dst));
diff --git a/gcc/config/rs6000/rs6000.h b/gcc/config/rs6000/rs6000.h
index 79ce1a8cbf1..caec2b3bf62 100644
--- a/gcc/config/rs6000/rs6000.h
+++ b/gcc/config/rs6000/rs6000.h
@@ -562,6 +562,13 @@ extern int rs6000_vector_align[];
&& TARGET_P8_VECTOR \
&& TARGET_POWERPC64)
+/* Whether we have dense math support. */
+#define TARGET_DENSE_MATH (TARGET_MMA && TARGET_FUTURE)
+#define TARGET_MMA_NO_DENSE_MATH (TARGET_MMA && !TARGET_FUTURE)
+
+/* Dense math flags. */
+#define DENSE_MATH_FLAGS (OPTION_MASK_FUTURE & OPTION_MASK_MMA)
+
/* Inlining allows targets to define the meanings of bits in target_info
field of ipa_fn_summary by itself, the used bits for rs6000 are listed
below. */
@@ -659,6 +666,7 @@ extern unsigned char rs6000_recip_bits[];
#define UNITS_PER_FP_WORD 8
#define UNITS_PER_ALTIVEC_WORD 16
#define UNITS_PER_VSX_WORD 16
+#define UNITS_PER_DMR_WORD 128
/* Type used for ptrdiff_t, as a string used in a declaration. */
#define PTRDIFF_TYPE "int"
@@ -786,7 +794,7 @@ enum data_align { align_abi, align_opt, align_both };
Another pseudo (not included in DWARF_FRAME_REGISTERS) is soft frame
pointer, which is eventually eliminated in favor of SP or FP. */
-#define FIRST_PSEUDO_REGISTER 111
+#define FIRST_PSEUDO_REGISTER 119
/* Use standard DWARF numbering for DWARF debugging information. */
#define DEBUGGER_REGNO(REGNO) rs6000_debugger_regno ((REGNO), 0)
@@ -823,7 +831,9 @@ enum data_align { align_abi, align_opt, align_both };
/* cr0..cr7 */ \
0, 0, 0, 0, 0, 0, 0, 0, \
/* vrsave vscr sfp */ \
- 1, 1, 1 \
+ 1, 1, 1, \
+ /* DMR registers. */ \
+ 0, 0, 0, 0, 0, 0, 0, 0 \
}
/* Like `CALL_USED_REGISTERS' except this macro doesn't require that
@@ -847,7 +857,9 @@ enum data_align { align_abi, align_opt, align_both };
/* cr0..cr7 */ \
1, 1, 0, 0, 0, 1, 1, 1, \
/* vrsave vscr sfp */ \
- 0, 0, 0 \
+ 0, 0, 0, \
+ /* DMR registers. */ \
+ 0, 0, 0, 0, 0, 0, 0, 0 \
}
#define TOTAL_ALTIVEC_REGS (LAST_ALTIVEC_REGNO - FIRST_ALTIVEC_REGNO + 1)
@@ -884,6 +896,7 @@ enum data_align { align_abi, align_opt, align_both };
v2 (not saved; incoming vector arg reg; return value)
v19 - v14 (not saved or used for anything)
v31 - v20 (saved; order given to save least number)
+ dmr0 - dmr7 (not saved)
vrsave, vscr (fixed)
sfp (fixed)
*/
@@ -926,6 +939,9 @@ enum data_align { align_abi, align_opt, align_both };
66, \
83, 82, 81, 80, 79, 78, \
95, 94, 93, 92, 91, 90, 89, 88, 87, 86, 85, 84, \
+ /* DMR registers. */ \
+ 111, 112, 113, 114, 115, 116, 117, 118, \
+ /* Vrsave, vscr, sfp. */ \
108, 109, \
110 \
}
@@ -952,6 +968,9 @@ enum data_align { align_abi, align_opt, align_both };
/* True if register is a VSX register. */
#define VSX_REGNO_P(N) (FP_REGNO_P (N) || ALTIVEC_REGNO_P (N))
+/* True if register is a DMR register. */
+#define DMR_REGNO_P(N) ((N) >= FIRST_DMR_REGNO && (N) <= LAST_DMR_REGNO)
+
/* Alternate name for any vector register supporting floating point, no matter
which instruction set(s) are available. */
#define VFLOAT_REGNO_P(N) \
@@ -1087,6 +1106,7 @@ enum reg_class
FLOAT_REGS,
ALTIVEC_REGS,
VSX_REGS,
+ DM_REGS,
VRSAVE_REGS,
VSCR_REGS,
GEN_OR_FLOAT_REGS,
@@ -1116,6 +1136,7 @@ enum reg_class
"FLOAT_REGS", \
"ALTIVEC_REGS", \
"VSX_REGS", \
+ "DM_REGS", \
"VRSAVE_REGS", \
"VSCR_REGS", \
"GEN_OR_FLOAT_REGS", \
@@ -1150,6 +1171,8 @@ enum reg_class
{ 0x00000000, 0x00000000, 0xffffffff, 0x00000000 }, \
/* VSX_REGS. */ \
{ 0x00000000, 0xffffffff, 0xffffffff, 0x00000000 }, \
+ /* DM_REGS. */ \
+ { 0x00000000, 0x00000000, 0x00000000, 0x007f8000 }, \
/* VRSAVE_REGS. */ \
{ 0x00000000, 0x00000000, 0x00000000, 0x00001000 }, \
/* VSCR_REGS. */ \
@@ -1177,7 +1200,7 @@ enum reg_class
/* CA_REGS. */ \
{ 0x00000000, 0x00000000, 0x00000000, 0x00000004 }, \
/* ALL_REGS. */ \
- { 0xffffffff, 0xffffffff, 0xffffffff, 0x00007fff } \
+ { 0xffffffff, 0xffffffff, 0xffffffff, 0x007fffff } \
}
/* The same information, inverted:
@@ -1201,6 +1224,7 @@ enum r6000_reg_class_enum {
RS6000_CONSTRAINT_wr, /* GPR register if 64-bit */
RS6000_CONSTRAINT_wx, /* FPR register for STFIWX */
RS6000_CONSTRAINT_wA, /* BASE_REGS if 64-bit. */
+ RS6000_CONSTRAINT_wD, /* Accumulator regs if MMA/Dense Math. */
RS6000_CONSTRAINT_MAX
};
@@ -2077,7 +2101,16 @@ extern char rs6000_reg_names[][8]; /* register names (0 vs. %r0). */
&rs6000_reg_names[108][0], /* vrsave */ \
&rs6000_reg_names[109][0], /* vscr */ \
\
- &rs6000_reg_names[110][0] /* sfp */ \
+ &rs6000_reg_names[110][0], /* sfp */ \
+ \
+ &rs6000_reg_names[111][0], /* dmr0 */ \
+ &rs6000_reg_names[112][0], /* dmr1 */ \
+ &rs6000_reg_names[113][0], /* dmr2 */ \
+ &rs6000_reg_names[114][0], /* dmr3 */ \
+ &rs6000_reg_names[115][0], /* dmr4 */ \
+ &rs6000_reg_names[116][0], /* dmr5 */ \
+ &rs6000_reg_names[117][0], /* dmr6 */ \
+ &rs6000_reg_names[118][0], /* dmr7 */ \
}
/* Table of additional register names to use in user input. */
@@ -2131,6 +2164,8 @@ extern char rs6000_reg_names[][8]; /* register names (0 vs. %r0). */
{"vs52", 84}, {"vs53", 85}, {"vs54", 86}, {"vs55", 87}, \
{"vs56", 88}, {"vs57", 89}, {"vs58", 90}, {"vs59", 91}, \
{"vs60", 92}, {"vs61", 93}, {"vs62", 94}, {"vs63", 95}, \
+ {"dmr0", 111}, {"dmr1", 112}, {"dmr2", 113}, {"dmr3", 114}, \
+ {"dmr4", 115}, {"dmr5", 116}, {"dmr6", 117}, {"dmr7", 118}, \
}
/* This is how to output an element of a case-vector that is relative. */
diff --git a/gcc/config/rs6000/rs6000.md b/gcc/config/rs6000/rs6000.md
index bc8bc6ab060..99e6515ba1d 100644
--- a/gcc/config/rs6000/rs6000.md
+++ b/gcc/config/rs6000/rs6000.md
@@ -51,6 +51,8 @@
(VRSAVE_REGNO 108)
(VSCR_REGNO 109)
(FRAME_POINTER_REGNUM 110)
+ (FIRST_DMR_REGNO 111)
+ (LAST_DMR_REGNO 118)
])
;;
@@ -355,7 +357,7 @@
(const (symbol_ref "(enum attr_cpu) rs6000_tune")))
;; The ISA we implement.
-(define_attr "isa" "any,p5,p6,p7,p7v,p8v,p9,p9v,p9kf,p9tf,p10"
+(define_attr "isa" "any,p5,p6,p7,p7v,p8v,p9,p9v,p9kf,p9tf,p10,dm,not_dm"
(const_string "any"))
;; Is this alternative enabled for the current CPU/ISA/etc.?
@@ -403,6 +405,14 @@
(and (eq_attr "isa" "p10")
(match_test "TARGET_POWER10"))
(const_int 1)
+
+ (and (eq_attr "isa" "dm")
+ (match_test "TARGET_DENSE_MATH"))
+ (const_int 1)
+
+ (and (eq_attr "isa" "not_dm")
+ (match_test "!TARGET_DENSE_MATH"))
+ (const_int 1)
] (const_int 0)))
;; If this instruction is microcoded on the CELL processor
diff --git a/gcc/doc/md.texi b/gcc/doc/md.texi
index 7b7e6507754..facb281d8ef 100644
--- a/gcc/doc/md.texi
+++ b/gcc/doc/md.texi
@@ -3441,6 +3441,13 @@ Like @code{d}, if @option{-mpowerpc-gfxopt} is used; otherwise, @code{NO_REGS}.
@item wA
Like @code{b}, if @option{-mpowerpc64} is used; otherwise, @code{NO_REGS}.
+@item wD
+Accumulator register if @option{-mma} is used; otherwise,
+@code{NO_REGS}. If @option{-mdense-math} is used, the accumulator
+register will be in the dense match register set. If
+@option{-mno-dense-math} is used, the accumulator register will
+overlap with the VSX vector registers 0..31.
+
@item wB
Signed 5-bit constant integer that can be loaded into an Altivec register.
^ permalink raw reply [flat|nested] 4+ messages in thread
* [gcc(refs/users/meissner/heads/work161-dmf)] PowerPC: Add support for accumulators in DMR registers.
@ 2024-02-29 19:18 Michael Meissner
0 siblings, 0 replies; 4+ messages in thread
From: Michael Meissner @ 2024-02-29 19:18 UTC (permalink / raw)
To: gcc-cvs
https://gcc.gnu.org/g:419f8923b4faa644245a54be1c4e5242695ead39
commit 419f8923b4faa644245a54be1c4e5242695ead39
Author: Michael Meissner <meissner@linux.ibm.com>
Date: Thu Feb 29 13:50:16 2024 -0500
PowerPC: Add support for accumulators in DMR registers.
The MMA subsystem added the notion of accumulator registers as an optional
feature of ISA 3.1 (power10). In ISA 3.1, these accumulators overlapped with
the VSX registers 0..31, but logically the accumulator registers were separate
from the FPR registers. In ISA 3.1, it was anticipated that in future systems,
the accumulator registers may no overlap with the FPR registers. This patch
adds the support for dense math registers as separate registers.
This particular patch does not change the MMA support to use the accumulators
within the dense math registers. This patch just adds the basic support for
having separate DMRs. The next patch will switch the MMA support to use the
accumulators if -mcpu=future is used.
For testing purposes, I added an undocumented option '-mdense-math' to enable
or disable the dense math support.
This patch adds a new constraint (wD). If MMA is selected but dense math is
not selected (i.e. -mcpu=power10), the wD constraint will allow access to
accumulators that overlap with VSX registers 0..31. If both MMA and dense math
are selected (i.e. -mcpu=future), the wD constraint will only allow dense math
registers.
This patch modifies the existing %A output modifier. If MMA is selected but
dense math is not selected, then %A output modifier converts the VSX register
number to the accumulator number, by dividing it by 4. If both MMA and dense
math are selected, then %A will map the separate DMR registers into 0..7.
The intention is that user code using extended asm can be modified to run on
both MMA without dense math and MMA with dense math:
1) If possible, don't use extended asm, but instead use the MMA built-in
functions;
2) If you do need to write extended asm, change the d constraints
targetting accumulators should now use wD;
3) Only use the built-in zero, assemble and disassemble functions create
move data between vector quad types and dense math accumulators.
I.e. do not use the xxmfacc, xxmtacc, and xxsetaccz directly in the
extended asm code. The reason is these instructions assume there is a
1-to-1 correspondence between 4 adjacent FPR registers and an
accumulator that overlaps with those instructions. With accumulators
now being separate registers, there no longer is a 1-to-1
correspondence.
It is possible that the mangling for DMRs and the GDB register numbers may
produce other changes in the future.
2024-02-29 Michael Meissner <meissner@linux.ibm.com>
gcc/
* config/rs6000/constraints.md (wD constraint): New constraint.
* config/rs6000/mma.md (UNSPEC_DM_ASSEMBLE): New unspec.
(movxo): Convert into define_expand.
(movxo_nodm): Version of movxo where accumulators overlap with VSX vector
registers 0..31.
(movxo_dm): Verson of movxo that supports separate dense math
accumulators.
(mma_assemble_acc): Add dense math support to define_expand.
(mma_assemble_acc_nodm): Rename from mma_assemble_acc, and restrict it
to non dense math systems.
(mma_assemble_acc_dm): Dense math version of mma_assemble_acc.
(mma_disassemble_acc): Add dense math support to define_expand.
(mma_disassemble_acc_nodm): Rename from mma_disassemble_acc, and
restrict it to non dense math systems.
(mma_disassemble_acc_dm): Dense math version of mma_disassemble_acc.
* config/rs6000/predicates.md (dmr_operand): New predicate.
(accumulator_operand): Likewise.
* config/rs6000/rs6000.cc (enum rs6000_reg_type): Add DMR_REG_TYPE.
(enum rs6000_reload_reg_type): Add RELOAD_REG_DMR.
(LAST_RELOAD_REG_CLASS): Add support for DMR registers and the wD
constraint.
(reload_reg_map): Likewise.
(rs6000_reg_names): Likewise.
(alt_reg_names): Likewise.
(rs6000_hard_regno_nregs_internal): Likewise.
(rs6000_hard_regno_mode_ok_uncached): Likewise.
(rs6000_debug_reg_global): Likewise.
(rs6000_setup_reg_addr_masks): Likewise.
(rs6000_init_hard_regno_mode_ok): Likewise.
(rs6000_secondary_reload_memory): Add support for DMR registers.
(rs6000_secondary_reload_simple_move): Likewise.
(rs6000_preferred_reload_class): Likewise.
(rs6000_secondary_reload_class): Likewise.
(print_operand): Make %A handle both FPRs and DMRs.
(rs6000_dmr_register_move_cost): New helper function.
(rs6000_register_move_cost): Add support for DMR registers.
(rs6000_memory_move_cost): Likewise.
(rs6000_compute_pressure_classes): Likewise.
(rs6000_debugger_regno): Likewise.
(rs6000_split_multireg_move): Add support for DMRs.
* config/rs6000/rs6000.h (TARGET_DENSE_MATH): New macro.
(TARGET_MMA_NO_DENSE_MATH): Likewise
(UNITS_PER_DMR_WORD): New macro.
(FIRST_PSEUDO_REGISTER): Update for DMRs.
(FIXED_REGISTERS): Add DMRs.
(CALL_REALLY_USED_REGISTERS): Likewise.
(REG_ALLOC_ORDER): Likewise.
(DMR_REGNO_P): New macro.
(enum reg_class): Add DM_REGS.
(REG_CLASS_NAMES): Likewise.
(REG_CLASS_CONTENTS): Likewise.
(enum r6000_reg_class_enum): Add RS6000_CONSTRAINT_wD.
(REGISTER_NAMES): Add DMR registers.
(ADDITIONAL_REGISTER_NAMES): Likewise.
* config/rs6000/rs6000.md (FIRST_DMR_REGNO): New constant.
(LAST_DMR_REGNO): Likewise.
(isa attribute): Add 'dm' and 'not_dm' attributes.
(enabled attribute): Support 'dm' and 'not_dm' attributes.
* doc/md.texi (PowerPC constraints): Document wD constraint.
Diff:
---
gcc/config/rs6000/mma.md | 251 ++++++++++++++++++++++++++----------------
gcc/config/rs6000/rs6000-c.cc | 3 +
gcc/config/rs6000/rs6000.cc | 35 +++---
gcc/config/rs6000/rs6000.h | 3 +
4 files changed, 181 insertions(+), 111 deletions(-)
diff --git a/gcc/config/rs6000/mma.md b/gcc/config/rs6000/mma.md
index 9bc9895bf2d..ac26de93143 100644
--- a/gcc/config/rs6000/mma.md
+++ b/gcc/config/rs6000/mma.md
@@ -552,190 +552,249 @@
"dmxxextfdmr256 %0,%1,2"
[(set_attr "type" "mma")])
-(define_insn "mma_<acc>"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=&d")
- (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0")]
- MMA_ACC))]
+;; MMA instructions that do not use their accumulators as an input, still must
+;; not allow their vector operands to overlap the registers used by the
+;; accumulator. We enforce this by marking the output as early clobber. If we
+;; have dense math, we don't need the whole prime/de-prime action, so just make
+;; these instructions be NOPs.
+
+(define_expand "mma_<acc>"
+ [(set (match_operand:XO 0 "accumulator_operand")
+ (unspec:XO [(match_operand:XO 1 "accumulator_operand")]
+ MMA_ACC))]
"TARGET_MMA"
+{
+ if (TARGET_DENSE_MATH)
+ {
+ if (!rtx_equal_p (operands[0], operands[1]))
+ emit_move_insn (operands[0], operands[1]);
+ DONE;
+ }
+
+ /* Generate the prime/de-prime code. */
+})
+
+(define_insn "*mma_<acc>_nodm"
+ [(set (match_operand:XO 0 "accumulator_operand" "=&d")
+ (unspec:XO [(match_operand:XO 1 "accumulator_operand" "0")]
+ MMA_ACC))]
+ "TARGET_MMA_NO_DENSE_MATH"
"<acc> %A0"
[(set_attr "type" "mma")])
;; We can't have integer constants in XOmode so we wrap this in an
-;; UNSPEC_VOLATILE.
+;; UNSPEC_VOLATILE for the non-dense math case. For dense math, we don't need
+;; to disable optimization and we can do a normal UNSPEC.
-(define_insn "mma_xxsetaccz"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=d")
+(define_expand "mma_xxsetaccz"
+ [(set (match_operand:XO 0 "register_operand")
(unspec_volatile:XO [(const_int 0)]
UNSPECV_MMA_XXSETACCZ))]
"TARGET_MMA"
+{
+ if (TARGET_DENSE_MATH)
+ {
+ emit_insn (gen_mma_xxsetaccz_dm (operands[0]));
+ DONE;
+ }
+})
+
+(define_insn "*mma_xxsetaccz_nodm"
+ [(set (match_operand:XO 0 "fpr_reg_operand" "=d")
+ (unspec_volatile:XO [(const_int 0)]
+ UNSPECV_MMA_XXSETACCZ))]
+ "TARGET_MMA_NO_DENSE_MATH"
"xxsetaccz %A0"
[(set_attr "type" "mma")])
+
+(define_insn "mma_xxsetaccz_dm"
+ [(set (match_operand:XO 0 "dmr_operand" "=wD")
+ (unspec:XO [(const_int 0)]
+ UNSPECV_MMA_XXSETACCZ))]
+ "TARGET_DENSE_MATH"
+ "dmsetdmrz %0"
+ [(set_attr "type" "mma")])
+
(define_insn "mma_<vv>"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d")
- (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "v,?wa")
- (match_operand:V16QI 2 "vsx_register_operand" "v,?wa")]
+ [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d")
+ (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa")]
MMA_VV))]
"TARGET_MMA"
"<vv> %A0,%x1,%x2"
- [(set_attr "type" "mma")])
+ [(set_attr "type" "mma")
+ (set_attr "isa" "dm,not_dm,not_dm")])
(define_insn "mma_<avv>"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d")
- (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0,0")
- (match_operand:V16QI 2 "vsx_register_operand" "v,?wa")
- (match_operand:V16QI 3 "vsx_register_operand" "v,?wa")]
+ [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d")
+ (unspec:XO [(match_operand:XO 1 "accumulator_operand" "0,0,0")
+ (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:V16QI 3 "vsx_register_operand" "wa,v,?wa")]
MMA_AVV))]
"TARGET_MMA"
"<avv> %A0,%x2,%x3"
- [(set_attr "type" "mma")])
+ [(set_attr "type" "mma")
+ (set_attr "isa" "dm,not_dm,not_dm")])
(define_insn "mma_<pv>"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d")
- (unspec:XO [(match_operand:OO 1 "vsx_register_operand" "v,?wa")
- (match_operand:V16QI 2 "vsx_register_operand" "v,?wa")]
+ [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d")
+ (unspec:XO [(match_operand:OO 1 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa")]
MMA_PV))]
"TARGET_MMA"
"<pv> %A0,%x1,%x2"
- [(set_attr "type" "mma")])
+ [(set_attr "type" "mma")
+ (set_attr "isa" "dm,not_dm,not_dm")])
(define_insn "mma_<apv>"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d")
- (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0,0")
- (match_operand:OO 2 "vsx_register_operand" "v,?wa")
- (match_operand:V16QI 3 "vsx_register_operand" "v,?wa")]
+ [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d")
+ (unspec:XO [(match_operand:XO 1 "accumulator_operand" "0,0,0")
+ (match_operand:OO 2 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:V16QI 3 "vsx_register_operand" "wa,v,?wa")]
MMA_APV))]
"TARGET_MMA"
"<apv> %A0,%x2,%x3"
- [(set_attr "type" "mma")])
+ [(set_attr "type" "mma")
+ (set_attr "isa" "dm,not_dm,not_dm")])
(define_insn "mma_<vvi4i4i8>"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d")
- (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "v,?wa")
- (match_operand:V16QI 2 "vsx_register_operand" "v,?wa")
- (match_operand:SI 3 "const_0_to_15_operand" "n,n")
- (match_operand:SI 4 "const_0_to_15_operand" "n,n")
- (match_operand:SI 5 "u8bit_cint_operand" "n,n")]
+ [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d")
+ (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:SI 3 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 4 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 5 "u8bit_cint_operand" "n,n,n")]
MMA_VVI4I4I8))]
"TARGET_MMA"
"<vvi4i4i8> %A0,%x1,%x2,%3,%4,%5"
[(set_attr "type" "mma")
- (set_attr "prefixed" "yes")])
+ (set_attr "prefixed" "yes")
+ (set_attr "isa" "dm,not_dm,not_dm")])
(define_insn "mma_<avvi4i4i8>"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d")
- (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0,0")
- (match_operand:V16QI 2 "vsx_register_operand" "v,?wa")
- (match_operand:V16QI 3 "vsx_register_operand" "v,?wa")
- (match_operand:SI 4 "const_0_to_15_operand" "n,n")
- (match_operand:SI 5 "const_0_to_15_operand" "n,n")
- (match_operand:SI 6 "u8bit_cint_operand" "n,n")]
+ [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d")
+ (unspec:XO [(match_operand:XO 1 "accumulator_operand" "0,0,0")
+ (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:V16QI 3 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:SI 4 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 5 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 6 "u8bit_cint_operand" "n,n,n")]
MMA_AVVI4I4I8))]
"TARGET_MMA"
"<avvi4i4i8> %A0,%x2,%x3,%4,%5,%6"
[(set_attr "type" "mma")
- (set_attr "prefixed" "yes")])
+ (set_attr "prefixed" "yes")
+ (set_attr "isa" "dm,not_dm,not_dm")])
(define_insn "mma_<vvi4i4i2>"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d")
- (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "v,?wa")
- (match_operand:V16QI 2 "vsx_register_operand" "v,?wa")
- (match_operand:SI 3 "const_0_to_15_operand" "n,n")
- (match_operand:SI 4 "const_0_to_15_operand" "n,n")
- (match_operand:SI 5 "const_0_to_3_operand" "n,n")]
+ [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d")
+ (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:SI 3 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 4 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 5 "const_0_to_3_operand" "n,n,n")]
MMA_VVI4I4I2))]
"TARGET_MMA"
"<vvi4i4i2> %A0,%x1,%x2,%3,%4,%5"
[(set_attr "type" "mma")
- (set_attr "prefixed" "yes")])
+ (set_attr "prefixed" "yes")
+ (set_attr "isa" "dm,not_dm,not_dm")])
(define_insn "mma_<avvi4i4i2>"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d")
- (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0,0")
- (match_operand:V16QI 2 "vsx_register_operand" "v,?wa")
- (match_operand:V16QI 3 "vsx_register_operand" "v,?wa")
- (match_operand:SI 4 "const_0_to_15_operand" "n,n")
- (match_operand:SI 5 "const_0_to_15_operand" "n,n")
- (match_operand:SI 6 "const_0_to_3_operand" "n,n")]
+ [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d")
+ (unspec:XO [(match_operand:XO 1 "accumulator_operand" "0,0,0")
+ (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:V16QI 3 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:SI 4 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 5 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 6 "const_0_to_3_operand" "n,n,n")]
MMA_AVVI4I4I2))]
"TARGET_MMA"
"<avvi4i4i2> %A0,%x2,%x3,%4,%5,%6"
[(set_attr "type" "mma")
- (set_attr "prefixed" "yes")])
+ (set_attr "prefixed" "yes")
+ (set_attr "isa" "dm,not_dm,not_dm")])
(define_insn "mma_<vvi4i4>"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d")
- (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "v,?wa")
- (match_operand:V16QI 2 "vsx_register_operand" "v,?wa")
- (match_operand:SI 3 "const_0_to_15_operand" "n,n")
- (match_operand:SI 4 "const_0_to_15_operand" "n,n")]
+ [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d")
+ (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:SI 3 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 4 "const_0_to_15_operand" "n,n,n")]
MMA_VVI4I4))]
"TARGET_MMA"
"<vvi4i4> %A0,%x1,%x2,%3,%4"
[(set_attr "type" "mma")
- (set_attr "prefixed" "yes")])
+ (set_attr "prefixed" "yes")
+ (set_attr "isa" "dm,not_dm,not_dm")])
(define_insn "mma_<avvi4i4>"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d")
- (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0,0")
- (match_operand:V16QI 2 "vsx_register_operand" "v,?wa")
- (match_operand:V16QI 3 "vsx_register_operand" "v,?wa")
- (match_operand:SI 4 "const_0_to_15_operand" "n,n")
- (match_operand:SI 5 "const_0_to_15_operand" "n,n")]
+ [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d")
+ (unspec:XO [(match_operand:XO 1 "accumulator_operand" "0,0,0")
+ (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:V16QI 3 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:SI 4 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 5 "const_0_to_15_operand" "n,n,n")]
MMA_AVVI4I4))]
"TARGET_MMA"
"<avvi4i4> %A0,%x2,%x3,%4,%5"
[(set_attr "type" "mma")
- (set_attr "prefixed" "yes")])
+ (set_attr "prefixed" "yes")
+ (set_attr "isa" "dm,not_dm,not_dm")])
(define_insn "mma_<pvi4i2>"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d")
- (unspec:XO [(match_operand:OO 1 "vsx_register_operand" "v,?wa")
- (match_operand:V16QI 2 "vsx_register_operand" "v,?wa")
- (match_operand:SI 3 "const_0_to_15_operand" "n,n")
- (match_operand:SI 4 "const_0_to_3_operand" "n,n")]
+ [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d")
+ (unspec:XO [(match_operand:OO 1 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:SI 3 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 4 "const_0_to_3_operand" "n,n,n")]
MMA_PVI4I2))]
"TARGET_MMA"
"<pvi4i2> %A0,%x1,%x2,%3,%4"
[(set_attr "type" "mma")
- (set_attr "prefixed" "yes")])
+ (set_attr "prefixed" "yes")
+ (set_attr "isa" "dm,not_dm,not_dm")])
(define_insn "mma_<apvi4i2>"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d")
- (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0,0")
- (match_operand:OO 2 "vsx_register_operand" "v,?wa")
- (match_operand:V16QI 3 "vsx_register_operand" "v,?wa")
- (match_operand:SI 4 "const_0_to_15_operand" "n,n")
- (match_operand:SI 5 "const_0_to_3_operand" "n,n")]
+ [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d")
+ (unspec:XO [(match_operand:XO 1 "accumulator_operand" "0,0,0")
+ (match_operand:OO 2 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:V16QI 3 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:SI 4 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 5 "const_0_to_3_operand" "n,n,n")]
MMA_APVI4I2))]
"TARGET_MMA"
"<apvi4i2> %A0,%x2,%x3,%4,%5"
[(set_attr "type" "mma")
- (set_attr "prefixed" "yes")])
+ (set_attr "prefixed" "yes")
+ (set_attr "isa" "dm,not_dm,not_dm")])
(define_insn "mma_<vvi4i4i4>"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d")
- (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "v,?wa")
- (match_operand:V16QI 2 "vsx_register_operand" "v,?wa")
- (match_operand:SI 3 "const_0_to_15_operand" "n,n")
- (match_operand:SI 4 "const_0_to_15_operand" "n,n")
- (match_operand:SI 5 "const_0_to_15_operand" "n,n")]
+ [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d")
+ (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:SI 3 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 4 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 5 "const_0_to_15_operand" "n,n,n")]
MMA_VVI4I4I4))]
"TARGET_MMA"
"<vvi4i4i4> %A0,%x1,%x2,%3,%4,%5"
[(set_attr "type" "mma")
- (set_attr "prefixed" "yes")])
+ (set_attr "prefixed" "yes")
+ (set_attr "isa" "dm,not_dm,not_dm")])
(define_insn "mma_<avvi4i4i4>"
- [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d")
- (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0,0")
- (match_operand:V16QI 2 "vsx_register_operand" "v,?wa")
- (match_operand:V16QI 3 "vsx_register_operand" "v,?wa")
- (match_operand:SI 4 "const_0_to_15_operand" "n,n")
- (match_operand:SI 5 "const_0_to_15_operand" "n,n")
- (match_operand:SI 6 "const_0_to_15_operand" "n,n")]
+ [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d")
+ (unspec:XO [(match_operand:XO 1 "accumulator_operand" "0,0,0")
+ (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:V16QI 3 "vsx_register_operand" "wa,v,?wa")
+ (match_operand:SI 4 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 5 "const_0_to_15_operand" "n,n,n")
+ (match_operand:SI 6 "const_0_to_15_operand" "n,n,n")]
MMA_AVVI4I4I4))]
"TARGET_MMA"
"<avvi4i4i4> %A0,%x2,%x3,%4,%5,%6"
[(set_attr "type" "mma")
- (set_attr "prefixed" "yes")])
+ (set_attr "prefixed" "yes")
+ (set_attr "isa" "dm,not_dm,not_dm")])
diff --git a/gcc/config/rs6000/rs6000-c.cc b/gcc/config/rs6000/rs6000-c.cc
index d15bb85743c..ef65ed8c259 100644
--- a/gcc/config/rs6000/rs6000-c.cc
+++ b/gcc/config/rs6000/rs6000-c.cc
@@ -602,6 +602,9 @@ rs6000_target_modify_macros (bool define_p, HOST_WIDE_INT flags)
/* Tell the user if we support the MMA instructions. */
if ((flags & OPTION_MASK_MMA) != 0)
rs6000_define_or_undefine_macro (define_p, "__MMA__");
+ /* Tell the user if we support the dense math instructions. */
+ if ((flags & DENSE_MATH_FLAGS) == DENSE_MATH_FLAGS)
+ rs6000_define_or_undefine_macro (define_p, "__PPC_DMR__");
/* Whether pc-relative code is being generated. */
if ((flags & OPTION_MASK_PCREL) != 0)
rs6000_define_or_undefine_macro (define_p, "__PCREL__");
diff --git a/gcc/config/rs6000/rs6000.cc b/gcc/config/rs6000/rs6000.cc
index 190d3828085..95ecf922143 100644
--- a/gcc/config/rs6000/rs6000.cc
+++ b/gcc/config/rs6000/rs6000.cc
@@ -14159,8 +14159,13 @@ print_operand (FILE *file, rtx x, int code)
overlapping with the FPR registers. */
if (!REG_P (x))
output_operand_lossage ("invalid %%A value");
- else if (TARGET_DENSE_MATH && DMR_REGNO_P (REGNO (x)))
- fprintf (file, "%d", REGNO (x) - FIRST_DMR_REGNO);
+ else if (TARGET_DENSE_MATH)
+ {
+ if (DMR_REGNO_P (REGNO (x)))
+ fprintf (file, "%d", REGNO (x) - FIRST_DMR_REGNO);
+ else
+ output_operand_lossage ("%%A operand is not a DMR");
+ }
else if (!FP_REGNO_P (REGNO (x)) || (REGNO (x) % 4) != 0)
output_operand_lossage ("invalid %%A value");
else
@@ -27609,7 +27614,7 @@ rs6000_split_multireg_move (rtx dst, rtx src)
/* If we are reading an accumulator register, we have to
deprime it before we can access it. */
- if (TARGET_MMA
+ if (TARGET_MMA && !TARGET_DENSE_MATH
&& GET_MODE (src) == XOmode && FP_REGNO_P (REGNO (src)))
emit_insn (gen_mma_xxmfacc (src, src));
@@ -27641,9 +27646,9 @@ rs6000_split_multireg_move (rtx dst, rtx src)
emit_insn (gen_rtx_SET (dst2, src2));
}
- /* If we are writing an accumulator register, we have to
- prime it after we've written it. */
- if (TARGET_MMA
+ /* If we are writing an accumulator register that overlaps with the
+ FPR registers, we have to prime it after we've written it. */
+ if (TARGET_MMA && !TARGET_DENSE_MATH
&& GET_MODE (dst) == XOmode && FP_REGNO_P (REGNO (dst)))
emit_insn (gen_mma_xxmtacc (dst, dst));
@@ -27712,9 +27717,9 @@ rs6000_split_multireg_move (rtx dst, rtx src)
emit_insn (gen_rtx_SET (dst_i, op));
}
- /* We are writing an accumulator register, so we have to
- prime it after we've written it. */
- if (GET_MODE (src) == XOmode)
+ /* On systems without dense math where accumulators overlap with the
+ vector registers, we have to prime it after we've written it. */
+ if (GET_MODE (src) == XOmode && !TARGET_DENSE_MATH)
emit_insn (gen_mma_xxmtacc (dst, dst));
return;
@@ -27725,9 +27730,9 @@ rs6000_split_multireg_move (rtx dst, rtx src)
if (REG_P (src) && REG_P (dst) && (REGNO (src) < REGNO (dst)))
{
- /* If we are reading an accumulator register, we have to
- deprime it before we can access it. */
- if (TARGET_MMA
+ /* If we are reading an accumulator register and we don't have dense
+ math, we have to deprime it before we can access it. */
+ if (TARGET_MMA && !TARGET_DENSE_MATH
&& GET_MODE (src) == XOmode && FP_REGNO_P (REGNO (src)))
emit_insn (gen_mma_xxmfacc (src, src));
@@ -27755,7 +27760,7 @@ rs6000_split_multireg_move (rtx dst, rtx src)
/* If we are writing an accumulator register, we have to
prime it after we've written it. */
- if (TARGET_MMA
+ if (TARGET_MMA && !TARGET_DENSE_MATH
&& GET_MODE (dst) == XOmode && FP_REGNO_P (REGNO (dst)))
emit_insn (gen_mma_xxmtacc (dst, dst));
}
@@ -27892,7 +27897,7 @@ rs6000_split_multireg_move (rtx dst, rtx src)
/* If we are reading an accumulator register, we have to
deprime it before we can access it. */
- if (TARGET_MMA && REG_P (src)
+ if (TARGET_MMA && !TARGET_DENSE_MATH && REG_P (src)
&& GET_MODE (src) == XOmode && FP_REGNO_P (REGNO (src)))
emit_insn (gen_mma_xxmfacc (src, src));
@@ -27924,7 +27929,7 @@ rs6000_split_multireg_move (rtx dst, rtx src)
/* If we are writing an accumulator register, we have to
prime it after we've written it. */
- if (TARGET_MMA && REG_P (dst)
+ if (TARGET_MMA && !TARGET_DENSE_MATH && REG_P (dst)
&& GET_MODE (dst) == XOmode && FP_REGNO_P (REGNO (dst)))
emit_insn (gen_mma_xxmtacc (dst, dst));
diff --git a/gcc/config/rs6000/rs6000.h b/gcc/config/rs6000/rs6000.h
index bea5db90fab..caec2b3bf62 100644
--- a/gcc/config/rs6000/rs6000.h
+++ b/gcc/config/rs6000/rs6000.h
@@ -566,6 +566,9 @@ extern int rs6000_vector_align[];
#define TARGET_DENSE_MATH (TARGET_MMA && TARGET_FUTURE)
#define TARGET_MMA_NO_DENSE_MATH (TARGET_MMA && !TARGET_FUTURE)
+/* Dense math flags. */
+#define DENSE_MATH_FLAGS (OPTION_MASK_FUTURE & OPTION_MASK_MMA)
+
/* Inlining allows targets to define the meanings of bits in target_info
field of ipa_fn_summary by itself, the used bits for rs6000 are listed
below. */
^ permalink raw reply [flat|nested] 4+ messages in thread
* [gcc(refs/users/meissner/heads/work161-dmf)] PowerPC: Add support for accumulators in DMR registers.
@ 2024-02-29 19:18 Michael Meissner
0 siblings, 0 replies; 4+ messages in thread
From: Michael Meissner @ 2024-02-29 19:18 UTC (permalink / raw)
To: gcc-cvs
https://gcc.gnu.org/g:28c7a4cd3ffbe2975f31d391f6f50251422cdc75
commit 28c7a4cd3ffbe2975f31d391f6f50251422cdc75
Author: Michael Meissner <meissner@linux.ibm.com>
Date: Thu Feb 29 13:45:50 2024 -0500
PowerPC: Add support for accumulators in DMR registers.
The MMA subsystem added the notion of accumulator registers as an optional
feature of ISA 3.1 (power10). In ISA 3.1, these accumulators overlapped with
the VSX registers 0..31, but logically the accumulator registers were separate
from the FPR registers. In ISA 3.1, it was anticipated that in future systems,
the accumulator registers may no overlap with the FPR registers. This patch
adds the support for dense math registers as separate registers.
This particular patch does not change the MMA support to use the accumulators
within the dense math registers. This patch just adds the basic support for
having separate DMRs. The next patch will switch the MMA support to use the
accumulators if -mcpu=future is used.
For testing purposes, I added an undocumented option '-mdense-math' to enable
or disable the dense math support.
This patch adds a new constraint (wD). If MMA is selected but dense math is
not selected (i.e. -mcpu=power10), the wD constraint will allow access to
accumulators that overlap with VSX registers 0..31. If both MMA and dense math
are selected (i.e. -mcpu=future), the wD constraint will only allow dense math
registers.
This patch modifies the existing %A output modifier. If MMA is selected but
dense math is not selected, then %A output modifier converts the VSX register
number to the accumulator number, by dividing it by 4. If both MMA and dense
math are selected, then %A will map the separate DMR registers into 0..7.
The intention is that user code using extended asm can be modified to run on
both MMA without dense math and MMA with dense math:
1) If possible, don't use extended asm, but instead use the MMA built-in
functions;
2) If you do need to write extended asm, change the d constraints
targetting accumulators should now use wD;
3) Only use the built-in zero, assemble and disassemble functions create
move data between vector quad types and dense math accumulators.
I.e. do not use the xxmfacc, xxmtacc, and xxsetaccz directly in the
extended asm code. The reason is these instructions assume there is a
1-to-1 correspondence between 4 adjacent FPR registers and an
accumulator that overlaps with those instructions. With accumulators
now being separate registers, there no longer is a 1-to-1
correspondence.
It is possible that the mangling for DMRs and the GDB register numbers may
produce other changes in the future.
2024-02-29 Michael Meissner <meissner@linux.ibm.com>
gcc/
* config/rs6000/constraints.md (wD constraint): New constraint.
* config/rs6000/mma.md (UNSPEC_DM_ASSEMBLE): New unspec.
(movxo): Convert into define_expand.
(movxo_nodm): Version of movxo where accumulators overlap with VSX vector
registers 0..31.
(movxo_dm): Verson of movxo that supports separate dense math
accumulators.
(mma_assemble_acc): Add dense math support to define_expand.
(mma_assemble_acc_nodm): Rename from mma_assemble_acc, and restrict it
to non dense math systems.
(mma_assemble_acc_dm): Dense math version of mma_assemble_acc.
(mma_disassemble_acc): Add dense math support to define_expand.
(mma_disassemble_acc_nodm): Rename from mma_disassemble_acc, and
restrict it to non dense math systems.
(mma_disassemble_acc_dm): Dense math version of mma_disassemble_acc.
* config/rs6000/predicates.md (dmr_operand): New predicate.
(accumulator_operand): Likewise.
* config/rs6000/rs6000.cc (enum rs6000_reg_type): Add DMR_REG_TYPE.
(enum rs6000_reload_reg_type): Add RELOAD_REG_DMR.
(LAST_RELOAD_REG_CLASS): Add support for DMR registers and the wD
constraint.
(reload_reg_map): Likewise.
(rs6000_reg_names): Likewise.
(alt_reg_names): Likewise.
(rs6000_hard_regno_nregs_internal): Likewise.
(rs6000_hard_regno_mode_ok_uncached): Likewise.
(rs6000_debug_reg_global): Likewise.
(rs6000_setup_reg_addr_masks): Likewise.
(rs6000_init_hard_regno_mode_ok): Likewise.
(rs6000_secondary_reload_memory): Add support for DMR registers.
(rs6000_secondary_reload_simple_move): Likewise.
(rs6000_preferred_reload_class): Likewise.
(rs6000_secondary_reload_class): Likewise.
(print_operand): Make %A handle both FPRs and DMRs.
(rs6000_dmr_register_move_cost): New helper function.
(rs6000_register_move_cost): Add support for DMR registers.
(rs6000_memory_move_cost): Likewise.
(rs6000_compute_pressure_classes): Likewise.
(rs6000_debugger_regno): Likewise.
(rs6000_split_multireg_move): Add support for DMRs.
* config/rs6000/rs6000.h (TARGET_DENSE_MATH): New macro.
(TARGET_MMA_NO_DENSE_MATH): Likewise
(UNITS_PER_DMR_WORD): New macro.
(FIRST_PSEUDO_REGISTER): Update for DMRs.
(FIXED_REGISTERS): Add DMRs.
(CALL_REALLY_USED_REGISTERS): Likewise.
(REG_ALLOC_ORDER): Likewise.
(DMR_REGNO_P): New macro.
(enum reg_class): Add DM_REGS.
(REG_CLASS_NAMES): Likewise.
(REG_CLASS_CONTENTS): Likewise.
(enum r6000_reg_class_enum): Add RS6000_CONSTRAINT_wD.
(REGISTER_NAMES): Add DMR registers.
(ADDITIONAL_REGISTER_NAMES): Likewise.
* config/rs6000/rs6000.md (FIRST_DMR_REGNO): New constant.
(LAST_DMR_REGNO): Likewise.
(isa attribute): Add 'dm' and 'not_dm' attributes.
(enabled attribute): Support 'dm' and 'not_dm' attributes.
* doc/md.texi (PowerPC constraints): Document wD constraint.
Diff:
---
gcc/config/rs6000/constraints.md | 3 +
gcc/config/rs6000/mma.md | 115 ++++++++++++++++++-------
gcc/config/rs6000/predicates.md | 32 +++++++
gcc/config/rs6000/rs6000.cc | 180 ++++++++++++++++++++++++++++++++++-----
gcc/config/rs6000/rs6000.h | 42 +++++++--
gcc/config/rs6000/rs6000.md | 12 ++-
gcc/doc/md.texi | 7 ++
7 files changed, 332 insertions(+), 59 deletions(-)
diff --git a/gcc/config/rs6000/constraints.md b/gcc/config/rs6000/constraints.md
index 369a7b75042..277a30a8245 100644
--- a/gcc/config/rs6000/constraints.md
+++ b/gcc/config/rs6000/constraints.md
@@ -107,6 +107,9 @@
(match_test "TARGET_P8_VECTOR")
(match_operand 0 "s5bit_cint_operand")))
+(define_register_constraint "wD" "rs6000_constraints[RS6000_CONSTRAINT_wD]"
+ "Accumulator register.")
+
(define_constraint "wE"
"@internal Vector constant that can be loaded with the XXSPLTIB instruction."
(match_test "xxspltib_constant_nosplit (op, mode)"))
diff --git a/gcc/config/rs6000/mma.md b/gcc/config/rs6000/mma.md
index 04e2d0066df..9bc9895bf2d 100644
--- a/gcc/config/rs6000/mma.md
+++ b/gcc/config/rs6000/mma.md
@@ -91,6 +91,7 @@
UNSPEC_MMA_XVI8GER4SPP
UNSPEC_MMA_XXMFACC
UNSPEC_MMA_XXMTACC
+ UNSPEC_DM_ASSEMBLE
])
(define_c_enum "unspecv"
@@ -314,7 +315,9 @@
(set_attr "length" "*,*,8")])
\f
-;; Vector quad support. XOmode can only live in FPRs.
+;; Vector quad support. Under the original MMA, XOmode can only live in VSX
+;; registers 0..31. With dense math, XOmode can live in either VSX registers
+;; (0..63) or DMR registers.
(define_expand "movxo"
[(set (match_operand:XO 0 "nonimmediate_operand")
(match_operand:XO 1 "input_operand"))]
@@ -339,10 +342,10 @@
gcc_assert (false);
})
-(define_insn_and_split "*movxo"
+(define_insn_and_split "*movxo_nodm"
[(set (match_operand:XO 0 "nonimmediate_operand" "=d,ZwO,d")
(match_operand:XO 1 "input_operand" "ZwO,d,d"))]
- "TARGET_MMA
+ "TARGET_MMA_NO_DENSE_MATH
&& (gpc_reg_operand (operands[0], XOmode)
|| gpc_reg_operand (operands[1], XOmode))"
"@
@@ -359,6 +362,31 @@
(set_attr "length" "*,*,16")
(set_attr "max_prefixed_insns" "2,2,*")])
+(define_insn_and_split "*movxo_dm"
+ [(set (match_operand:XO 0 "nonimmediate_operand" "=wa,QwO,wa,wD,wD,wa")
+ (match_operand:XO 1 "input_operand" "QwO,wa, wa,wa,wD,wD"))]
+ "TARGET_DENSE_MATH
+ && (gpc_reg_operand (operands[0], XOmode)
+ || gpc_reg_operand (operands[1], XOmode))"
+ "@
+ #
+ #
+ #
+ dmxxinstdmr512 %0,%1,%Y1,0
+ dmmr %0,%1
+ dmxxextfdmr512 %0,%Y0,%1,0"
+ "&& reload_completed
+ && !dmr_operand (operands[0], XOmode)
+ && !dmr_operand (operands[1], XOmode)"
+ [(const_int 0)]
+{
+ rs6000_split_multireg_move (operands[0], operands[1]);
+ DONE;
+}
+ [(set_attr "type" "vecload,vecstore,veclogical,mma,mma,mma")
+ (set_attr "length" "*,*,16,*,*,*")
+ (set_attr "max_prefixed_insns" "2,2,*,*,*,*")])
+
(define_expand "vsx_assemble_pair"
[(match_operand:OO 0 "vsx_register_operand")
(match_operand:V16QI 1 "mma_assemble_input_operand")
@@ -426,25 +454,38 @@
})
(define_expand "mma_assemble_acc"
- [(match_operand:XO 0 "fpr_reg_operand")
+ [(match_operand:XO 0 "accumulator_operand")
(match_operand:V16QI 1 "mma_assemble_input_operand")
(match_operand:V16QI 2 "mma_assemble_input_operand")
(match_operand:V16QI 3 "mma_assemble_input_operand")
(match_operand:V16QI 4 "mma_assemble_input_operand")]
"TARGET_MMA"
{
- rtx src = gen_rtx_UNSPEC_VOLATILE (XOmode,
- gen_rtvec (4, operands[1], operands[2],
- operands[3], operands[4]),
- UNSPECV_MMA_ASSEMBLE);
- emit_move_insn (operands[0], src);
+ rtx op0 = operands[0];
+ rtx op1 = operands[1];
+ rtx op2 = operands[2];
+ rtx op3 = operands[3];
+ rtx op4 = operands[4];
+
+ if (TARGET_DENSE_MATH)
+ {
+ rtx vpair1 = gen_reg_rtx (OOmode);
+ rtx vpair2 = gen_reg_rtx (OOmode);
+ emit_insn (gen_vsx_assemble_pair (vpair1, op1, op2));
+ emit_insn (gen_vsx_assemble_pair (vpair2, op3, op4));
+ emit_insn (gen_mma_assemble_acc_dm (op0, vpair1, vpair2));
+ }
+
+ else
+ emit_insn (gen_mma_assemble_acc_nodm (op0, op1, op2, op3, op4));
+
DONE;
})
;; We cannot update the four output registers atomically, so mark the output
-;; as an early clobber so we don't accidentally clobber the input operands. */
+;; as an early clobber so we don't accidentally clobber the input operands.
-(define_insn_and_split "*mma_assemble_acc"
+(define_insn_and_split "mma_assemble_acc_nodm"
[(set (match_operand:XO 0 "fpr_reg_operand" "=&d")
(unspec_volatile:XO
[(match_operand:V16QI 1 "mma_assemble_input_operand" "mwa")
@@ -452,7 +493,7 @@
(match_operand:V16QI 3 "mma_assemble_input_operand" "mwa")
(match_operand:V16QI 4 "mma_assemble_input_operand" "mwa")]
UNSPECV_MMA_ASSEMBLE))]
- "TARGET_MMA
+ "TARGET_MMA_NO_DENSE_MATH
&& fpr_reg_operand (operands[0], XOmode)"
"#"
"&& reload_completed"
@@ -466,28 +507,31 @@
DONE;
})
+;; On a system with dense math, we build the accumulators from two vector
+;; pairs.
+
+(define_insn "mma_assemble_acc_dm"
+ [(set (match_operand:XO 0 "dmr_operand" "=wD")
+ (unspec:XO [(match_operand:OO 1 "vsx_register_operand" "wa")
+ (match_operand:OO 2 "vsx_register_operand" "wa")]
+ UNSPEC_DM_ASSEMBLE))]
+ "TARGET_DENSE_MATH"
+ "dmxxinstdmr512 %0,%1,%2,0"
+ [(set_attr "type" "mma")])
+
(define_expand "mma_disassemble_acc"
- [(match_operand:V16QI 0 "mma_disassemble_output_operand")
- (match_operand:XO 1 "fpr_reg_operand")
- (match_operand 2 "const_0_to_3_operand")]
- "TARGET_MMA"
-{
- rtx src;
- int regoff = INTVAL (operands[2]);
- src = gen_rtx_UNSPEC (V16QImode,
- gen_rtvec (2, operands[1], GEN_INT (regoff)),
- UNSPEC_MMA_EXTRACT);
- emit_move_insn (operands[0], src);
- DONE;
-})
+ [(set (match_operand:V16QI 0 "register_operand")
+ (unspec:V16QI [(match_operand:XO 1 "register_operand")
+ (match_operand 2 "const_0_to_3_operand")]
+ UNSPEC_MMA_EXTRACT))]
+ "TARGET_MMA")
-(define_insn_and_split "*mma_disassemble_acc"
+(define_insn_and_split "*mma_disassemble_acc_nodm"
[(set (match_operand:V16QI 0 "mma_disassemble_output_operand" "=mwa")
- (unspec:V16QI [(match_operand:XO 1 "fpr_reg_operand" "d")
- (match_operand 2 "const_0_to_3_operand")]
+ (unspec:V16QI [(match_operand:XO 1 "fpr_reg_operand" "d")
+ (match_operand 2 "const_0_to_3_operand")]
UNSPEC_MMA_EXTRACT))]
- "TARGET_MMA
- && fpr_reg_operand (operands[1], XOmode)"
+ "TARGET_MMA_NO_DENSE_MATH"
"#"
"&& reload_completed"
[(const_int 0)]
@@ -499,9 +543,14 @@
DONE;
})
-;; MMA instructions that do not use their accumulators as an input, still
-;; must not allow their vector operands to overlap the registers used by
-;; the accumulator. We enforce this by marking the output as early clobber.
+(define_insn "*mma_disassemble_acc_dm"
+ [(set (match_operand:V16QI 0 "vsx_register_operand" "=wa")
+ (unspec:V16QI [(match_operand:XO 1 "dmr_operand" "wD")
+ (match_operand 2 "const_0_to_3_operand")]
+ UNSPEC_MMA_EXTRACT))]
+ "TARGET_DENSE_MATH"
+ "dmxxextfdmr256 %0,%1,2"
+ [(set_attr "type" "mma")])
(define_insn "mma_<acc>"
[(set (match_operand:XO 0 "fpr_reg_operand" "=&d")
diff --git a/gcc/config/rs6000/predicates.md b/gcc/config/rs6000/predicates.md
index d23ce9a77a3..3040dcd50a3 100644
--- a/gcc/config/rs6000/predicates.md
+++ b/gcc/config/rs6000/predicates.md
@@ -186,6 +186,38 @@
return VLOGICAL_REGNO_P (REGNO (op));
})
+;; Return 1 if op is a DMR register
+(define_predicate "dmr_operand"
+ (match_operand 0 "register_operand")
+{
+ if (!REG_P (op))
+ return 0;
+
+ if (!HARD_REGISTER_P (op))
+ return 1;
+
+ return DMR_REGNO_P (REGNO (op));
+})
+
+;; Return 1 if op is an accumulator. On power10 systems, the accumulators
+;; overlap with the FPRs, while on systems with dense math, the accumulators
+;; are separate dense math registers and do not overlap with the FPR
+;; registers..
+(define_predicate "accumulator_operand"
+ (match_operand 0 "register_operand")
+{
+ if (!REG_P (op))
+ return 0;
+
+ if (!HARD_REGISTER_P (op))
+ return 1;
+
+ int r = REGNO (op);
+ return (TARGET_DENSE_MATH
+ ? DMR_REGNO_P (r)
+ : FP_REGNO_P (r) && (r & 3) == 0);
+})
+
;; Return 1 if op is the carry register.
(define_predicate "ca_operand"
(match_operand 0 "register_operand")
diff --git a/gcc/config/rs6000/rs6000.cc b/gcc/config/rs6000/rs6000.cc
index b1ad49d3734..190d3828085 100644
--- a/gcc/config/rs6000/rs6000.cc
+++ b/gcc/config/rs6000/rs6000.cc
@@ -292,7 +292,8 @@ enum rs6000_reg_type {
ALTIVEC_REG_TYPE,
FPR_REG_TYPE,
SPR_REG_TYPE,
- CR_REG_TYPE
+ CR_REG_TYPE,
+ DMR_REG_TYPE
};
/* Map register class to register type. */
@@ -306,22 +307,23 @@ static enum rs6000_reg_type reg_class_to_reg_type[N_REG_CLASSES];
/* Register classes we care about in secondary reload or go if legitimate
- address. We only need to worry about GPR, FPR, and Altivec registers here,
- along an ANY field that is the OR of the 3 register classes. */
+ address. We only need to worry about GPR, FPR, Altivec, and DMR registers
+ here, along an ANY field that is the OR of the 4 register classes. */
enum rs6000_reload_reg_type {
RELOAD_REG_GPR, /* General purpose registers. */
RELOAD_REG_FPR, /* Traditional floating point regs. */
RELOAD_REG_VMX, /* Altivec (VMX) registers. */
- RELOAD_REG_ANY, /* OR of GPR, FPR, Altivec masks. */
+ RELOAD_REG_DMR, /* DMR registers. */
+ RELOAD_REG_ANY, /* OR of GPR/FPR/VMX/DMR masks. */
N_RELOAD_REG
};
-/* For setting up register classes, loop through the 3 register classes mapping
+/* For setting up register classes, loop through the 4 register classes mapping
into real registers, and skip the ANY class, which is just an OR of the
bits. */
#define FIRST_RELOAD_REG_CLASS RELOAD_REG_GPR
-#define LAST_RELOAD_REG_CLASS RELOAD_REG_VMX
+#define LAST_RELOAD_REG_CLASS RELOAD_REG_DMR
/* Map reload register type to a register in the register class. */
struct reload_reg_map_type {
@@ -333,6 +335,7 @@ static const struct reload_reg_map_type reload_reg_map[N_RELOAD_REG] = {
{ "Gpr", FIRST_GPR_REGNO }, /* RELOAD_REG_GPR. */
{ "Fpr", FIRST_FPR_REGNO }, /* RELOAD_REG_FPR. */
{ "VMX", FIRST_ALTIVEC_REGNO }, /* RELOAD_REG_VMX. */
+ { "DMR", FIRST_DMR_REGNO }, /* RELOAD_REG_DMR. */
{ "Any", -1 }, /* RELOAD_REG_ANY. */
};
@@ -1226,6 +1229,8 @@ char rs6000_reg_names[][8] =
"0", "1", "2", "3", "4", "5", "6", "7",
/* vrsave vscr sfp */
"vrsave", "vscr", "sfp",
+ /* DMRs */
+ "0", "1", "2", "3", "4", "5", "6", "7",
};
#ifdef TARGET_REGNAMES
@@ -1252,6 +1257,8 @@ static const char alt_reg_names[][8] =
"%cr0", "%cr1", "%cr2", "%cr3", "%cr4", "%cr5", "%cr6", "%cr7",
/* vrsave vscr sfp */
"vrsave", "vscr", "sfp",
+ /* DMRs */
+ "%dmr0", "%dmr1", "%dmr2", "%dmr3", "%dmr4", "%dmr5", "%dmr6", "%dmr7",
};
#endif
@@ -1836,6 +1843,9 @@ rs6000_hard_regno_nregs_internal (int regno, machine_mode mode)
else if (ALTIVEC_REGNO_P (regno))
reg_size = UNITS_PER_ALTIVEC_WORD;
+ else if (DMR_REGNO_P (regno))
+ reg_size = UNITS_PER_DMR_WORD;
+
else
reg_size = UNITS_PER_WORD;
@@ -1857,9 +1867,36 @@ rs6000_hard_regno_mode_ok_uncached (int regno, machine_mode mode)
if (mode == OOmode)
return (TARGET_MMA && VSX_REGNO_P (regno) && (regno & 1) == 0);
- /* MMA accumulator modes need FPR registers divisible by 4. */
+ /* On ISA 3.1 (power10), MMA accumulator modes need FPR registers divisible
+ by 4.
+
+ If dense math is enabled, allow all VSX registers plus the DMR registers.
+ We need to make sure we don't cross between the boundary of FPRs and
+ traditional Altiviec registers. */
if (mode == XOmode)
- return (TARGET_MMA && FP_REGNO_P (regno) && (regno & 3) == 0);
+ {
+ if (TARGET_MMA && !TARGET_DENSE_MATH)
+ return (FP_REGNO_P (regno) && (regno & 3) == 0);
+
+ else if (TARGET_DENSE_MATH)
+ {
+ if (DMR_REGNO_P (regno))
+ return 1;
+
+ if (FP_REGNO_P (regno))
+ return ((regno & 1) == 0 && regno <= LAST_FPR_REGNO - 3);
+
+ if (ALTIVEC_REGNO_P (regno))
+ return ((regno & 1) == 0 && regno <= LAST_ALTIVEC_REGNO - 3);
+ }
+
+ else
+ return 0;
+ }
+
+ /* No other types other than XOmode can go in DMRs. */
+ if (DMR_REGNO_P (regno))
+ return 0;
/* PTImode can only go in GPRs. Quad word memory operations require even/odd
register combinations, and use PTImode where we need to deal with quad
@@ -2302,6 +2339,7 @@ rs6000_debug_reg_global (void)
rs6000_debug_reg_print (FIRST_ALTIVEC_REGNO,
LAST_ALTIVEC_REGNO,
"vs");
+ rs6000_debug_reg_print (FIRST_DMR_REGNO, LAST_DMR_REGNO, "dmr");
rs6000_debug_reg_print (LR_REGNO, LR_REGNO, "lr");
rs6000_debug_reg_print (CTR_REGNO, CTR_REGNO, "ctr");
rs6000_debug_reg_print (CR0_REGNO, CR7_REGNO, "cr");
@@ -2322,6 +2360,7 @@ rs6000_debug_reg_global (void)
"wr reg_class = %s\n"
"wx reg_class = %s\n"
"wA reg_class = %s\n"
+ "wD reg_class = %s\n"
"\n",
reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_d]],
reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_v]],
@@ -2329,7 +2368,8 @@ rs6000_debug_reg_global (void)
reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_we]],
reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_wr]],
reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_wx]],
- reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_wA]]);
+ reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_wA]],
+ reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_wD]]);
nl = "\n";
for (m = 0; m < NUM_MACHINE_MODES; ++m)
@@ -2626,6 +2666,21 @@ rs6000_setup_reg_addr_masks (void)
addr_mask = 0;
reg = reload_reg_map[rc].reg;
+ /* Special case DMR registers. */
+ if (rc == RELOAD_REG_DMR)
+ {
+ if (TARGET_DENSE_MATH && m2 == XOmode)
+ {
+ addr_mask = RELOAD_REG_VALID;
+ reg_addr[m].addr_mask[rc] = addr_mask;
+ any_addr_mask |= addr_mask;
+ }
+ else
+ reg_addr[m].addr_mask[rc] = 0;
+
+ continue;
+ }
+
/* Can mode values go in the GPR/FPR/Altivec registers? */
if (reg >= 0 && rs6000_hard_regno_mode_ok_p[m][reg])
{
@@ -2776,6 +2831,9 @@ rs6000_init_hard_regno_mode_ok (bool global_init_p)
for (r = CR1_REGNO; r <= CR7_REGNO; ++r)
rs6000_regno_regclass[r] = CR_REGS;
+ for (r = FIRST_DMR_REGNO; r <= LAST_DMR_REGNO; ++r)
+ rs6000_regno_regclass[r] = DM_REGS;
+
rs6000_regno_regclass[LR_REGNO] = LINK_REGS;
rs6000_regno_regclass[CTR_REGNO] = CTR_REGS;
rs6000_regno_regclass[CA_REGNO] = NO_REGS;
@@ -2800,6 +2858,7 @@ rs6000_init_hard_regno_mode_ok (bool global_init_p)
reg_class_to_reg_type[(int)LINK_OR_CTR_REGS] = SPR_REG_TYPE;
reg_class_to_reg_type[(int)CR_REGS] = CR_REG_TYPE;
reg_class_to_reg_type[(int)CR0_REGS] = CR_REG_TYPE;
+ reg_class_to_reg_type[(int)DM_REGS] = DMR_REG_TYPE;
if (TARGET_VSX)
{
@@ -2986,6 +3045,13 @@ rs6000_init_hard_regno_mode_ok (bool global_init_p)
if (TARGET_DIRECT_MOVE_128)
rs6000_constraints[RS6000_CONSTRAINT_we] = VSX_REGS;
+ /* Support for the accumulator registers, either FPR registers (aka original
+ mma) or DMR registers (dense math). */
+ if (TARGET_DENSE_MATH)
+ rs6000_constraints[RS6000_CONSTRAINT_wD] = DM_REGS;
+ else if (TARGET_MMA)
+ rs6000_constraints[RS6000_CONSTRAINT_wD] = FLOAT_REGS;
+
/* Set up the reload helper and direct move functions. */
if (TARGET_VSX || TARGET_ALTIVEC)
{
@@ -12311,6 +12377,11 @@ rs6000_secondary_reload_memory (rtx addr,
addr_mask = (reg_addr[mode].addr_mask[RELOAD_REG_VMX]
& ~RELOAD_REG_AND_M16);
+ /* DMR registers use VSX registers, and need to generate some extra
+ instructions. */
+ else if (rclass == DM_REGS)
+ return 2;
+
/* If the register allocator hasn't made up its mind yet on the register
class to use, settle on defaults to use. */
else if (rclass == NO_REGS)
@@ -12639,6 +12710,13 @@ rs6000_secondary_reload_simple_move (enum rs6000_reg_type to_type,
|| (to_type == SPR_REG_TYPE && from_type == GPR_REG_TYPE)))
return true;
+ /* We can transfer between VSX registers and DMR registers without needing
+ extra registers. */
+ if (TARGET_DENSE_MATH && mode == XOmode
+ && ((to_type == DMR_REG_TYPE && from_type == VSX_REG_TYPE)
+ || (to_type == VSX_REG_TYPE && from_type == DMR_REG_TYPE)))
+ return true;
+
return false;
}
@@ -13333,6 +13411,10 @@ rs6000_preferred_reload_class (rtx x, enum reg_class rclass)
machine_mode mode = GET_MODE (x);
bool is_constant = CONSTANT_P (x);
+ /* DMR registers can't be loaded or stored. */
+ if (rclass == DM_REGS)
+ return NO_REGS;
+
/* If a mode can't go in FPR/ALTIVEC/VSX registers, don't return a preferred
reload class for it. */
if ((rclass == ALTIVEC_REGS || rclass == VSX_REGS)
@@ -13429,7 +13511,7 @@ rs6000_preferred_reload_class (rtx x, enum reg_class rclass)
return VSX_REGS;
if (mode == XOmode)
- return FLOAT_REGS;
+ return TARGET_DENSE_MATH ? VSX_REGS : FLOAT_REGS;
if (GET_MODE_CLASS (mode) == MODE_INT)
return GENERAL_REGS;
@@ -13554,6 +13636,11 @@ rs6000_secondary_reload_class (enum reg_class rclass, machine_mode mode,
else
regno = -1;
+ /* DMR registers don't have loads or stores. We have to go through the VSX
+ registers to load XOmode (vector quad). */
+ if (TARGET_DENSE_MATH && rclass == DM_REGS)
+ return VSX_REGS;
+
/* If we have VSX register moves, prefer moving scalar values between
Altivec registers and GPR by going via an FPR (and then via memory)
instead of reloading the secondary memory address for Altivec moves. */
@@ -14067,8 +14154,14 @@ print_operand (FILE *file, rtx x, int code)
output_operand. */
case 'A':
- /* Write the MMA accumulator number associated with VSX register X. */
- if (!REG_P (x) || !FP_REGNO_P (REGNO (x)) || (REGNO (x) % 4) != 0)
+ /* Write the MMA accumulator number associated with VSX register X. On
+ dense math systems, only allow DMR accumulators, not accumulators
+ overlapping with the FPR registers. */
+ if (!REG_P (x))
+ output_operand_lossage ("invalid %%A value");
+ else if (TARGET_DENSE_MATH && DMR_REGNO_P (REGNO (x)))
+ fprintf (file, "%d", REGNO (x) - FIRST_DMR_REGNO);
+ else if (!FP_REGNO_P (REGNO (x)) || (REGNO (x) % 4) != 0)
output_operand_lossage ("invalid %%A value");
else
fprintf (file, "%d", (REGNO (x) - FIRST_FPR_REGNO) / 4);
@@ -22738,6 +22831,31 @@ rs6000_debug_address_cost (rtx x, machine_mode mode,
}
+/* Subroutine to determine the move cost of dense math registers. If we are
+ moving to/from VSX_REGISTER registers, the cost is either 1 move (for
+ 512-bit accumulators) or 2 moves (for 1,024 dmr registers). If we are
+ moving to anything else like GPR registers, make the cost very high. */
+
+static int
+rs6000_dmr_register_move_cost (machine_mode mode, reg_class_t rclass)
+{
+ const int reg_move_base = 2;
+ HARD_REG_SET vsx_set = (reg_class_contents[rclass]
+ & reg_class_contents[VSX_REGS]);
+
+ if (TARGET_DENSE_MATH && !hard_reg_set_empty_p (vsx_set))
+ {
+ /* __vector_quad (i.e. XOmode) is tranfered in 1 instruction. */
+ if (mode == XOmode)
+ return reg_move_base;
+
+ else
+ return reg_move_base * 2 * hard_regno_nregs (FIRST_DMR_REGNO, mode);
+ }
+
+ return 1000 * 2 * hard_regno_nregs (FIRST_DMR_REGNO, mode);
+}
+
/* A C expression returning the cost of moving data from a register of class
CLASS1 to one of CLASS2. */
@@ -22751,17 +22869,28 @@ rs6000_register_move_cost (machine_mode mode,
if (TARGET_DEBUG_COST)
dbg_cost_ctrl++;
+ HARD_REG_SET to_vsx, from_vsx;
+ to_vsx = reg_class_contents[to] & reg_class_contents[VSX_REGS];
+ from_vsx = reg_class_contents[from] & reg_class_contents[VSX_REGS];
+
+ /* Special case DMR registers, that can only move to/from VSX registers. */
+ if (from == DM_REGS && to == DM_REGS)
+ ret = 2 * hard_regno_nregs (FIRST_DMR_REGNO, mode);
+
+ else if (from == DM_REGS)
+ ret = rs6000_dmr_register_move_cost (mode, to);
+
+ else if (to == DM_REGS)
+ ret = rs6000_dmr_register_move_cost (mode, from);
+
/* If we have VSX, we can easily move between FPR or Altivec registers,
otherwise we can only easily move within classes.
Do this first so we give best-case answers for union classes
containing both gprs and vsx regs. */
- HARD_REG_SET to_vsx, from_vsx;
- to_vsx = reg_class_contents[to] & reg_class_contents[VSX_REGS];
- from_vsx = reg_class_contents[from] & reg_class_contents[VSX_REGS];
- if (!hard_reg_set_empty_p (to_vsx)
- && !hard_reg_set_empty_p (from_vsx)
- && (TARGET_VSX
- || hard_reg_set_intersect_p (to_vsx, from_vsx)))
+ else if (!hard_reg_set_empty_p (to_vsx)
+ && !hard_reg_set_empty_p (from_vsx)
+ && (TARGET_VSX
+ || hard_reg_set_intersect_p (to_vsx, from_vsx)))
{
int reg = FIRST_FPR_REGNO;
if (TARGET_VSX
@@ -22856,6 +22985,9 @@ rs6000_memory_move_cost (machine_mode mode, reg_class_t rclass,
ret = 4 * hard_regno_nregs (32, mode);
else if (reg_classes_intersect_p (rclass, ALTIVEC_REGS))
ret = 4 * hard_regno_nregs (FIRST_ALTIVEC_REGNO, mode);
+ else if (reg_classes_intersect_p (rclass, DM_REGS))
+ ret = (rs6000_dmr_register_move_cost (mode, VSX_REGS)
+ + rs6000_memory_move_cost (mode, VSX_REGS, false));
else
ret = 4 + rs6000_register_move_cost (mode, rclass, GENERAL_REGS);
@@ -24064,6 +24196,8 @@ rs6000_compute_pressure_classes (enum reg_class *pressure_classes)
if (TARGET_HARD_FLOAT)
pressure_classes[n++] = FLOAT_REGS;
}
+ if (TARGET_DENSE_MATH)
+ pressure_classes[n++] = DM_REGS;
pressure_classes[n++] = CR_REGS;
pressure_classes[n++] = SPECIAL_REGS;
@@ -24228,6 +24362,10 @@ rs6000_debugger_regno (unsigned int regno, unsigned int format)
return 67;
if (regno == 64)
return 64;
+ /* XXX: This is a guess. The GCC register number for FIRST_DMR_REGNO is 111,
+ but the frame pointer regnum uses that. */
+ if (DMR_REGNO_P (regno))
+ return regno - FIRST_DMR_REGNO + 112;
gcc_unreachable ();
}
@@ -27519,7 +27657,9 @@ rs6000_split_multireg_move (rtx dst, rtx src)
|| XINT (src, 1) == UNSPECV_MMA_ASSEMBLE);
gcc_assert (REG_P (dst));
if (GET_MODE (src) == XOmode)
- gcc_assert (FP_REGNO_P (REGNO (dst)));
+ gcc_assert ((TARGET_DENSE_MATH
+ ? VSX_REGNO_P (REGNO (dst))
+ : FP_REGNO_P (REGNO (dst))));
if (GET_MODE (src) == OOmode)
gcc_assert (VSX_REGNO_P (REGNO (dst)));
diff --git a/gcc/config/rs6000/rs6000.h b/gcc/config/rs6000/rs6000.h
index 79ce1a8cbf1..bea5db90fab 100644
--- a/gcc/config/rs6000/rs6000.h
+++ b/gcc/config/rs6000/rs6000.h
@@ -562,6 +562,10 @@ extern int rs6000_vector_align[];
&& TARGET_P8_VECTOR \
&& TARGET_POWERPC64)
+/* Whether we have dense math support. */
+#define TARGET_DENSE_MATH (TARGET_MMA && TARGET_FUTURE)
+#define TARGET_MMA_NO_DENSE_MATH (TARGET_MMA && !TARGET_FUTURE)
+
/* Inlining allows targets to define the meanings of bits in target_info
field of ipa_fn_summary by itself, the used bits for rs6000 are listed
below. */
@@ -659,6 +663,7 @@ extern unsigned char rs6000_recip_bits[];
#define UNITS_PER_FP_WORD 8
#define UNITS_PER_ALTIVEC_WORD 16
#define UNITS_PER_VSX_WORD 16
+#define UNITS_PER_DMR_WORD 128
/* Type used for ptrdiff_t, as a string used in a declaration. */
#define PTRDIFF_TYPE "int"
@@ -786,7 +791,7 @@ enum data_align { align_abi, align_opt, align_both };
Another pseudo (not included in DWARF_FRAME_REGISTERS) is soft frame
pointer, which is eventually eliminated in favor of SP or FP. */
-#define FIRST_PSEUDO_REGISTER 111
+#define FIRST_PSEUDO_REGISTER 119
/* Use standard DWARF numbering for DWARF debugging information. */
#define DEBUGGER_REGNO(REGNO) rs6000_debugger_regno ((REGNO), 0)
@@ -823,7 +828,9 @@ enum data_align { align_abi, align_opt, align_both };
/* cr0..cr7 */ \
0, 0, 0, 0, 0, 0, 0, 0, \
/* vrsave vscr sfp */ \
- 1, 1, 1 \
+ 1, 1, 1, \
+ /* DMR registers. */ \
+ 0, 0, 0, 0, 0, 0, 0, 0 \
}
/* Like `CALL_USED_REGISTERS' except this macro doesn't require that
@@ -847,7 +854,9 @@ enum data_align { align_abi, align_opt, align_both };
/* cr0..cr7 */ \
1, 1, 0, 0, 0, 1, 1, 1, \
/* vrsave vscr sfp */ \
- 0, 0, 0 \
+ 0, 0, 0, \
+ /* DMR registers. */ \
+ 0, 0, 0, 0, 0, 0, 0, 0 \
}
#define TOTAL_ALTIVEC_REGS (LAST_ALTIVEC_REGNO - FIRST_ALTIVEC_REGNO + 1)
@@ -884,6 +893,7 @@ enum data_align { align_abi, align_opt, align_both };
v2 (not saved; incoming vector arg reg; return value)
v19 - v14 (not saved or used for anything)
v31 - v20 (saved; order given to save least number)
+ dmr0 - dmr7 (not saved)
vrsave, vscr (fixed)
sfp (fixed)
*/
@@ -926,6 +936,9 @@ enum data_align { align_abi, align_opt, align_both };
66, \
83, 82, 81, 80, 79, 78, \
95, 94, 93, 92, 91, 90, 89, 88, 87, 86, 85, 84, \
+ /* DMR registers. */ \
+ 111, 112, 113, 114, 115, 116, 117, 118, \
+ /* Vrsave, vscr, sfp. */ \
108, 109, \
110 \
}
@@ -952,6 +965,9 @@ enum data_align { align_abi, align_opt, align_both };
/* True if register is a VSX register. */
#define VSX_REGNO_P(N) (FP_REGNO_P (N) || ALTIVEC_REGNO_P (N))
+/* True if register is a DMR register. */
+#define DMR_REGNO_P(N) ((N) >= FIRST_DMR_REGNO && (N) <= LAST_DMR_REGNO)
+
/* Alternate name for any vector register supporting floating point, no matter
which instruction set(s) are available. */
#define VFLOAT_REGNO_P(N) \
@@ -1087,6 +1103,7 @@ enum reg_class
FLOAT_REGS,
ALTIVEC_REGS,
VSX_REGS,
+ DM_REGS,
VRSAVE_REGS,
VSCR_REGS,
GEN_OR_FLOAT_REGS,
@@ -1116,6 +1133,7 @@ enum reg_class
"FLOAT_REGS", \
"ALTIVEC_REGS", \
"VSX_REGS", \
+ "DM_REGS", \
"VRSAVE_REGS", \
"VSCR_REGS", \
"GEN_OR_FLOAT_REGS", \
@@ -1150,6 +1168,8 @@ enum reg_class
{ 0x00000000, 0x00000000, 0xffffffff, 0x00000000 }, \
/* VSX_REGS. */ \
{ 0x00000000, 0xffffffff, 0xffffffff, 0x00000000 }, \
+ /* DM_REGS. */ \
+ { 0x00000000, 0x00000000, 0x00000000, 0x007f8000 }, \
/* VRSAVE_REGS. */ \
{ 0x00000000, 0x00000000, 0x00000000, 0x00001000 }, \
/* VSCR_REGS. */ \
@@ -1177,7 +1197,7 @@ enum reg_class
/* CA_REGS. */ \
{ 0x00000000, 0x00000000, 0x00000000, 0x00000004 }, \
/* ALL_REGS. */ \
- { 0xffffffff, 0xffffffff, 0xffffffff, 0x00007fff } \
+ { 0xffffffff, 0xffffffff, 0xffffffff, 0x007fffff } \
}
/* The same information, inverted:
@@ -1201,6 +1221,7 @@ enum r6000_reg_class_enum {
RS6000_CONSTRAINT_wr, /* GPR register if 64-bit */
RS6000_CONSTRAINT_wx, /* FPR register for STFIWX */
RS6000_CONSTRAINT_wA, /* BASE_REGS if 64-bit. */
+ RS6000_CONSTRAINT_wD, /* Accumulator regs if MMA/Dense Math. */
RS6000_CONSTRAINT_MAX
};
@@ -2077,7 +2098,16 @@ extern char rs6000_reg_names[][8]; /* register names (0 vs. %r0). */
&rs6000_reg_names[108][0], /* vrsave */ \
&rs6000_reg_names[109][0], /* vscr */ \
\
- &rs6000_reg_names[110][0] /* sfp */ \
+ &rs6000_reg_names[110][0], /* sfp */ \
+ \
+ &rs6000_reg_names[111][0], /* dmr0 */ \
+ &rs6000_reg_names[112][0], /* dmr1 */ \
+ &rs6000_reg_names[113][0], /* dmr2 */ \
+ &rs6000_reg_names[114][0], /* dmr3 */ \
+ &rs6000_reg_names[115][0], /* dmr4 */ \
+ &rs6000_reg_names[116][0], /* dmr5 */ \
+ &rs6000_reg_names[117][0], /* dmr6 */ \
+ &rs6000_reg_names[118][0], /* dmr7 */ \
}
/* Table of additional register names to use in user input. */
@@ -2131,6 +2161,8 @@ extern char rs6000_reg_names[][8]; /* register names (0 vs. %r0). */
{"vs52", 84}, {"vs53", 85}, {"vs54", 86}, {"vs55", 87}, \
{"vs56", 88}, {"vs57", 89}, {"vs58", 90}, {"vs59", 91}, \
{"vs60", 92}, {"vs61", 93}, {"vs62", 94}, {"vs63", 95}, \
+ {"dmr0", 111}, {"dmr1", 112}, {"dmr2", 113}, {"dmr3", 114}, \
+ {"dmr4", 115}, {"dmr5", 116}, {"dmr6", 117}, {"dmr7", 118}, \
}
/* This is how to output an element of a case-vector that is relative. */
diff --git a/gcc/config/rs6000/rs6000.md b/gcc/config/rs6000/rs6000.md
index bc8bc6ab060..99e6515ba1d 100644
--- a/gcc/config/rs6000/rs6000.md
+++ b/gcc/config/rs6000/rs6000.md
@@ -51,6 +51,8 @@
(VRSAVE_REGNO 108)
(VSCR_REGNO 109)
(FRAME_POINTER_REGNUM 110)
+ (FIRST_DMR_REGNO 111)
+ (LAST_DMR_REGNO 118)
])
;;
@@ -355,7 +357,7 @@
(const (symbol_ref "(enum attr_cpu) rs6000_tune")))
;; The ISA we implement.
-(define_attr "isa" "any,p5,p6,p7,p7v,p8v,p9,p9v,p9kf,p9tf,p10"
+(define_attr "isa" "any,p5,p6,p7,p7v,p8v,p9,p9v,p9kf,p9tf,p10,dm,not_dm"
(const_string "any"))
;; Is this alternative enabled for the current CPU/ISA/etc.?
@@ -403,6 +405,14 @@
(and (eq_attr "isa" "p10")
(match_test "TARGET_POWER10"))
(const_int 1)
+
+ (and (eq_attr "isa" "dm")
+ (match_test "TARGET_DENSE_MATH"))
+ (const_int 1)
+
+ (and (eq_attr "isa" "not_dm")
+ (match_test "!TARGET_DENSE_MATH"))
+ (const_int 1)
] (const_int 0)))
;; If this instruction is microcoded on the CELL processor
diff --git a/gcc/doc/md.texi b/gcc/doc/md.texi
index 7b7e6507754..facb281d8ef 100644
--- a/gcc/doc/md.texi
+++ b/gcc/doc/md.texi
@@ -3441,6 +3441,13 @@ Like @code{d}, if @option{-mpowerpc-gfxopt} is used; otherwise, @code{NO_REGS}.
@item wA
Like @code{b}, if @option{-mpowerpc64} is used; otherwise, @code{NO_REGS}.
+@item wD
+Accumulator register if @option{-mma} is used; otherwise,
+@code{NO_REGS}. If @option{-mdense-math} is used, the accumulator
+register will be in the dense match register set. If
+@option{-mno-dense-math} is used, the accumulator register will
+overlap with the VSX vector registers 0..31.
+
@item wB
Signed 5-bit constant integer that can be loaded into an Altivec register.
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2024-03-01 21:46 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-02-29 19:28 [gcc(refs/users/meissner/heads/work161-dmf)] PowerPC: Add support for accumulators in DMR registers Michael Meissner
-- strict thread matches above, loose matches on Subject: below --
2024-03-01 21:46 Michael Meissner
2024-02-29 19:18 Michael Meissner
2024-02-29 19:18 Michael Meissner
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).