From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: by sourceware.org (Postfix, from userid 1005) id CDBE63858431; Thu, 2 Feb 2023 20:26:43 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org CDBE63858431 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1675369603; bh=LfoMOHkDHCVgg87jl2JUwNkWdVu321VLq6nu/cg8L1Q=; h=From:To:Subject:Date:From; b=peQfgHf0Gd82z0Aq7AwYm3bjpbUHsW/t5yx5bZSWAIGocoEsDf8a/GnrVJBvwEXxQ 4wfTcv528d9bWEEIOGvNBHgEPK/5Ldxe1YYuBYRmdG13rCz/kAHxosocPPiA0JTEcu NTBTV3G0Udn4wkyCzCYuuiGRGHeD0jDRkG40wtLQ= Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: Michael Meissner To: gcc-cvs@gcc.gnu.org Subject: [gcc(refs/users/meissner/heads/dmf008)] PowerPC: Make MMA insns support DMR registers. X-Act-Checkin: gcc X-Git-Author: Michael Meissner X-Git-Refname: refs/users/meissner/heads/dmf008 X-Git-Oldrev: 254c9b4bb2e58c4c05ac8fdbaf80d9f794aad6e1 X-Git-Newrev: f44378486d1d62e6885e0143fa4c82389a284cee Message-Id: <20230202202643.CDBE63858431@sourceware.org> Date: Thu, 2 Feb 2023 20:26:43 +0000 (GMT) List-Id: https://gcc.gnu.org/g:f44378486d1d62e6885e0143fa4c82389a284cee commit f44378486d1d62e6885e0143fa4c82389a284cee Author: Michael Meissner Date: Thu Feb 2 15:09:51 2023 -0500 PowerPC: Make MMA insns support DMR registers. This patch changes the MMA instructions to use either FPR registers (-mcpu=power10) or DMRs (-mcpu=future). In this patch, the existing MMA instruction names are used. A macro (__PPC_DMR__) is defined if the MMA instructions use the DMRs. The patches have been tested on the following platforms. I added the patches for PR target/107299 that I submitted on November 2nd before doing the builds so that GCC would build on systems using IEEE 128-bit long double. * https://gcc.gnu.org/pipermail/gcc-patches/2022-November/604834.html There were no regressions with doing bootstrap builds and running the regression tests: 1) Power10 LE using --with-cpu=power10 --with-long-double-format=ieee; 2) Power10 LE using --with-cpu=power10 --with-long-double-format=ibm; 3) Power9 LE using --with-cpu=power9 --with-long-double-format=ibm; and 4) Power8 BE using --with-cpu=power8 (both 32-bit & 64-bit tested). Can I check this patch into the GCC 13 master branch? 2023-02-02 Michael Meissner gcc/ * config/rs6000/mma.md (mma_): New define_expand to handle mma_ for dense math and non dense math. (mma_ insn): Restrict to non dense math. (mma_xxsetaccz): Convert to define_expand to handle non dense math and dense math. (mma_xxsetaccz_vsx): Rename from mma_xxsetaccz and restrict usage to non dense math. (mma_xxsetaccz_dm): Dense math version of mma_xxsetaccz. (mma_): Add support for dense math. (mma_): Likewise. (mma_): Likewise. (mma_): Likewise. (mma_): Likewise. (mma_): Likewise. (mma_): Likewise. (mma_): Likewise. (mma_): Likewise. (mma_): Likewise. (mma_): Likewise. (mma_): Likewise. (mma_): Likewise. (mma_): Likewise. * config/rs6000/rs6000-c.cc (rs6000_target_modify_macros): Define __PPC_DMR__ if we have dense math instructions. * config/rs6000/rs6000.cc (print_operand): Make %A handle only DMRs if dense math and only FPRs if not dense math. (rs6000_split_multireg_move): Do not generate the xxmtacc instruction to prime the DMR registers or the xxmfacc instruction to de-prime instructions if we have dense math register support. Diff: --- gcc/config/rs6000/mma.md | 247 ++++++++++++++++++++++++++---------------- gcc/config/rs6000/rs6000-c.cc | 3 + gcc/config/rs6000/rs6000.cc | 35 +++--- 3 files changed, 176 insertions(+), 109 deletions(-) diff --git a/gcc/config/rs6000/mma.md b/gcc/config/rs6000/mma.md index 59ca6835f7c..9e3feb3ea54 100644 --- a/gcc/config/rs6000/mma.md +++ b/gcc/config/rs6000/mma.md @@ -552,190 +552,249 @@ "dmxxextfdmr256 %0,%1,2" [(set_attr "type" "mma")]) -(define_insn "mma_" +;; MMA instructions that do not use their accumulators as an input, still must +;; not allow their vector operands to overlap the registers used by the +;; accumulator. We enforce this by marking the output as early clobber. If we +;; have dense math, we don't need the whole prime/de-prime action, so just make +;; thse instructions be NOPs. + +(define_expand "mma_" + [(set (match_operand:XO 0 "register_operand") + (unspec:XO [(match_operand:XO 1 "register_operand")] + MMA_ACC))] + "TARGET_MMA" +{ + if (TARGET_DENSE_MATH) + { + if (!rtx_equal_p (operands[0], operands[1])) + emit_move_insn (operands[0], operands[1]); + DONE; + } + + /* Generate the prime/de-prime code. */ +}) + +(define_insn "*mma_" [(set (match_operand:XO 0 "fpr_reg_operand" "=&d") (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0")] MMA_ACC))] - "TARGET_MMA" + "TARGET_MMA && !TARGET_DENSE_MATH" " %A0" [(set_attr "type" "mma")]) ;; We can't have integer constants in XOmode so we wrap this in an -;; UNSPEC_VOLATILE. +;; UNSPEC_VOLATILE for the non-dense math case. For dense math, we don't need +;; to disable optimization and we can do a normal UNSPEC. -(define_insn "mma_xxsetaccz" - [(set (match_operand:XO 0 "fpr_reg_operand" "=d") +(define_expand "mma_xxsetaccz" + [(set (match_operand:XO 0 "register_operand") (unspec_volatile:XO [(const_int 0)] UNSPECV_MMA_XXSETACCZ))] "TARGET_MMA" +{ + if (TARGET_DENSE_MATH) + { + emit_insn (gen_mma_xxsetaccz_dm (operands[0])); + DONE; + } +}) + +(define_insn "*mma_xxsetaccz_vsx" + [(set (match_operand:XO 0 "fpr_reg_operand" "=d") + (unspec_volatile:XO [(const_int 0)] + UNSPECV_MMA_XXSETACCZ))] + "TARGET_MMA && !TARGET_DENSE_MATH" "xxsetaccz %A0" [(set_attr "type" "mma")]) + +(define_insn "mma_xxsetaccz_dm" + [(set (match_operand:XO 0 "dmr_operand" "=wD") + (unspec:XO [(const_int 0)] + UNSPECV_MMA_XXSETACCZ))] + "TARGET_DENSE_MATH" + "dmsetdmrz %0" + [(set_attr "type" "mma")]) + (define_insn "mma_" - [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d") - (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "v,?wa") - (match_operand:V16QI 2 "vsx_register_operand" "v,?wa")] + [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d") + (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "wa,v,?wa") + (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa")] MMA_VV))] "TARGET_MMA" " %A0,%x1,%x2" - [(set_attr "type" "mma")]) + [(set_attr "type" "mma") + (set_attr "isa" "dm,not_dm,not_dm")]) (define_insn "mma_" - [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d") - (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0,0") - (match_operand:V16QI 2 "vsx_register_operand" "v,?wa") - (match_operand:V16QI 3 "vsx_register_operand" "v,?wa")] + [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d") + (unspec:XO [(match_operand:XO 1 "accumulator_operand" "0,0,0") + (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa") + (match_operand:V16QI 3 "vsx_register_operand" "wa,v,?wa")] MMA_AVV))] "TARGET_MMA" " %A0,%x2,%x3" - [(set_attr "type" "mma")]) + [(set_attr "type" "mma") + (set_attr "isa" "dm,not_dm,not_dm")]) (define_insn "mma_" - [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d") - (unspec:XO [(match_operand:OO 1 "vsx_register_operand" "v,?wa") - (match_operand:V16QI 2 "vsx_register_operand" "v,?wa")] + [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d") + (unspec:XO [(match_operand:OO 1 "vsx_register_operand" "wa,v,?wa") + (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa")] MMA_PV))] "TARGET_MMA" " %A0,%x1,%x2" - [(set_attr "type" "mma")]) + [(set_attr "type" "mma") + (set_attr "isa" "dm,not_dm,not_dm")]) (define_insn "mma_" - [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d") - (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0,0") - (match_operand:OO 2 "vsx_register_operand" "v,?wa") - (match_operand:V16QI 3 "vsx_register_operand" "v,?wa")] + [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d") + (unspec:XO [(match_operand:XO 1 "accumulator_operand" "0,0,0") + (match_operand:OO 2 "vsx_register_operand" "wa,v,?wa") + (match_operand:V16QI 3 "vsx_register_operand" "wa,v,?wa")] MMA_APV))] "TARGET_MMA" " %A0,%x2,%x3" - [(set_attr "type" "mma")]) + [(set_attr "type" "mma") + (set_attr "isa" "dm,not_dm,not_dm")]) (define_insn "mma_" - [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d") - (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "v,?wa") - (match_operand:V16QI 2 "vsx_register_operand" "v,?wa") - (match_operand:SI 3 "const_0_to_15_operand" "n,n") - (match_operand:SI 4 "const_0_to_15_operand" "n,n") - (match_operand:SI 5 "u8bit_cint_operand" "n,n")] + [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d") + (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "wa,v,?wa") + (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa") + (match_operand:SI 3 "const_0_to_15_operand" "n,n,n") + (match_operand:SI 4 "const_0_to_15_operand" "n,n,n") + (match_operand:SI 5 "u8bit_cint_operand" "n,n,n")] MMA_VVI4I4I8))] "TARGET_MMA" " %A0,%x1,%x2,%3,%4,%5" [(set_attr "type" "mma") - (set_attr "prefixed" "yes")]) + (set_attr "prefixed" "yes") + (set_attr "isa" "dm,not_dm,not_dm")]) (define_insn "mma_" - [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d") - (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0,0") - (match_operand:V16QI 2 "vsx_register_operand" "v,?wa") - (match_operand:V16QI 3 "vsx_register_operand" "v,?wa") - (match_operand:SI 4 "const_0_to_15_operand" "n,n") - (match_operand:SI 5 "const_0_to_15_operand" "n,n") - (match_operand:SI 6 "u8bit_cint_operand" "n,n")] + [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d") + (unspec:XO [(match_operand:XO 1 "accumulator_operand" "0,0,0") + (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa") + (match_operand:V16QI 3 "vsx_register_operand" "wa,v,?wa") + (match_operand:SI 4 "const_0_to_15_operand" "n,n,n") + (match_operand:SI 5 "const_0_to_15_operand" "n,n,n") + (match_operand:SI 6 "u8bit_cint_operand" "n,n,n")] MMA_AVVI4I4I8))] "TARGET_MMA" " %A0,%x2,%x3,%4,%5,%6" [(set_attr "type" "mma") - (set_attr "prefixed" "yes")]) + (set_attr "prefixed" "yes") + (set_attr "isa" "dm,not_dm,not_dm")]) (define_insn "mma_" - [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d") - (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "v,?wa") - (match_operand:V16QI 2 "vsx_register_operand" "v,?wa") - (match_operand:SI 3 "const_0_to_15_operand" "n,n") - (match_operand:SI 4 "const_0_to_15_operand" "n,n") - (match_operand:SI 5 "const_0_to_3_operand" "n,n")] + [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d") + (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "wa,v,?wa") + (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa") + (match_operand:SI 3 "const_0_to_15_operand" "n,n,n") + (match_operand:SI 4 "const_0_to_15_operand" "n,n,n") + (match_operand:SI 5 "const_0_to_3_operand" "n,n,n")] MMA_VVI4I4I2))] "TARGET_MMA" " %A0,%x1,%x2,%3,%4,%5" [(set_attr "type" "mma") - (set_attr "prefixed" "yes")]) + (set_attr "prefixed" "yes") + (set_attr "isa" "dm,not_dm,not_dm")]) (define_insn "mma_" - [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d") - (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0,0") - (match_operand:V16QI 2 "vsx_register_operand" "v,?wa") - (match_operand:V16QI 3 "vsx_register_operand" "v,?wa") - (match_operand:SI 4 "const_0_to_15_operand" "n,n") - (match_operand:SI 5 "const_0_to_15_operand" "n,n") - (match_operand:SI 6 "const_0_to_3_operand" "n,n")] + [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d") + (unspec:XO [(match_operand:XO 1 "accumulator_operand" "0,0,0") + (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa") + (match_operand:V16QI 3 "vsx_register_operand" "wa,v,?wa") + (match_operand:SI 4 "const_0_to_15_operand" "n,n,n") + (match_operand:SI 5 "const_0_to_15_operand" "n,n,n") + (match_operand:SI 6 "const_0_to_3_operand" "n,n,n")] MMA_AVVI4I4I2))] "TARGET_MMA" " %A0,%x2,%x3,%4,%5,%6" [(set_attr "type" "mma") - (set_attr "prefixed" "yes")]) + (set_attr "prefixed" "yes") + (set_attr "isa" "dm,not_dm,not_dm")]) (define_insn "mma_" - [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d") - (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "v,?wa") - (match_operand:V16QI 2 "vsx_register_operand" "v,?wa") - (match_operand:SI 3 "const_0_to_15_operand" "n,n") - (match_operand:SI 4 "const_0_to_15_operand" "n,n")] + [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d") + (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "wa,v,?wa") + (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa") + (match_operand:SI 3 "const_0_to_15_operand" "n,n,n") + (match_operand:SI 4 "const_0_to_15_operand" "n,n,n")] MMA_VVI4I4))] "TARGET_MMA" " %A0,%x1,%x2,%3,%4" [(set_attr "type" "mma") - (set_attr "prefixed" "yes")]) + (set_attr "prefixed" "yes") + (set_attr "isa" "dm,not_dm,not_dm")]) (define_insn "mma_" - [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d") - (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0,0") - (match_operand:V16QI 2 "vsx_register_operand" "v,?wa") - (match_operand:V16QI 3 "vsx_register_operand" "v,?wa") - (match_operand:SI 4 "const_0_to_15_operand" "n,n") - (match_operand:SI 5 "const_0_to_15_operand" "n,n")] + [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d") + (unspec:XO [(match_operand:XO 1 "accumulator_operand" "0,0,0") + (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa") + (match_operand:V16QI 3 "vsx_register_operand" "wa,v,?wa") + (match_operand:SI 4 "const_0_to_15_operand" "n,n,n") + (match_operand:SI 5 "const_0_to_15_operand" "n,n,n")] MMA_AVVI4I4))] "TARGET_MMA" " %A0,%x2,%x3,%4,%5" [(set_attr "type" "mma") - (set_attr "prefixed" "yes")]) + (set_attr "prefixed" "yes") + (set_attr "isa" "dm,not_dm,not_dm")]) (define_insn "mma_" - [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d") - (unspec:XO [(match_operand:OO 1 "vsx_register_operand" "v,?wa") - (match_operand:V16QI 2 "vsx_register_operand" "v,?wa") - (match_operand:SI 3 "const_0_to_15_operand" "n,n") - (match_operand:SI 4 "const_0_to_3_operand" "n,n")] + [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d") + (unspec:XO [(match_operand:OO 1 "vsx_register_operand" "wa,v,?wa") + (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa") + (match_operand:SI 3 "const_0_to_15_operand" "n,n,n") + (match_operand:SI 4 "const_0_to_3_operand" "n,n,n")] MMA_PVI4I2))] "TARGET_MMA" " %A0,%x1,%x2,%3,%4" [(set_attr "type" "mma") - (set_attr "prefixed" "yes")]) + (set_attr "prefixed" "yes") + (set_attr "isa" "dm,not_dm,not_dm")]) (define_insn "mma_" - [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d") - (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0,0") - (match_operand:OO 2 "vsx_register_operand" "v,?wa") - (match_operand:V16QI 3 "vsx_register_operand" "v,?wa") - (match_operand:SI 4 "const_0_to_15_operand" "n,n") - (match_operand:SI 5 "const_0_to_3_operand" "n,n")] + [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d") + (unspec:XO [(match_operand:XO 1 "accumulator_operand" "0,0,0") + (match_operand:OO 2 "vsx_register_operand" "wa,v,?wa") + (match_operand:V16QI 3 "vsx_register_operand" "wa,v,?wa") + (match_operand:SI 4 "const_0_to_15_operand" "n,n,n") + (match_operand:SI 5 "const_0_to_3_operand" "n,n,n")] MMA_APVI4I2))] "TARGET_MMA" " %A0,%x2,%x3,%4,%5" [(set_attr "type" "mma") - (set_attr "prefixed" "yes")]) + (set_attr "prefixed" "yes") + (set_attr "isa" "dm,not_dm,not_dm")]) (define_insn "mma_" - [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d") - (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "v,?wa") - (match_operand:V16QI 2 "vsx_register_operand" "v,?wa") - (match_operand:SI 3 "const_0_to_15_operand" "n,n") - (match_operand:SI 4 "const_0_to_15_operand" "n,n") - (match_operand:SI 5 "const_0_to_15_operand" "n,n")] + [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d") + (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "wa,v,?wa") + (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa") + (match_operand:SI 3 "const_0_to_15_operand" "n,n,n") + (match_operand:SI 4 "const_0_to_15_operand" "n,n,n") + (match_operand:SI 5 "const_0_to_15_operand" "n,n,n")] MMA_VVI4I4I4))] "TARGET_MMA" " %A0,%x1,%x2,%3,%4,%5" [(set_attr "type" "mma") - (set_attr "prefixed" "yes")]) + (set_attr "prefixed" "yes") + (set_attr "isa" "dm,not_dm,not_dm")]) (define_insn "mma_" - [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d") - (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0,0") - (match_operand:V16QI 2 "vsx_register_operand" "v,?wa") - (match_operand:V16QI 3 "vsx_register_operand" "v,?wa") - (match_operand:SI 4 "const_0_to_15_operand" "n,n") - (match_operand:SI 5 "const_0_to_15_operand" "n,n") - (match_operand:SI 6 "const_0_to_15_operand" "n,n")] + [(set (match_operand:XO 0 "accumulator_operand" "=wD,&d,&d") + (unspec:XO [(match_operand:XO 1 "accumulator_operand" "0,0,0") + (match_operand:V16QI 2 "vsx_register_operand" "wa,v,?wa") + (match_operand:V16QI 3 "vsx_register_operand" "wa,v,?wa") + (match_operand:SI 4 "const_0_to_15_operand" "n,n,n") + (match_operand:SI 5 "const_0_to_15_operand" "n,n,n") + (match_operand:SI 6 "const_0_to_15_operand" "n,n,n")] MMA_AVVI4I4I4))] "TARGET_MMA" " %A0,%x2,%x3,%4,%5,%6" [(set_attr "type" "mma") - (set_attr "prefixed" "yes")]) + (set_attr "prefixed" "yes") + (set_attr "isa" "dm,not_dm,not_dm")]) diff --git a/gcc/config/rs6000/rs6000-c.cc b/gcc/config/rs6000/rs6000-c.cc index 2803014f2b6..baf1f4dc92b 100644 --- a/gcc/config/rs6000/rs6000-c.cc +++ b/gcc/config/rs6000/rs6000-c.cc @@ -600,6 +600,9 @@ rs6000_target_modify_macros (bool define_p, HOST_WIDE_INT flags) /* Tell the user if we support the MMA instructions. */ if ((flags & OPTION_MASK_MMA) != 0) rs6000_define_or_undefine_macro (define_p, "__MMA__"); + /* Tell the user if we support the dense math instructions. */ + if ((flags & OPTION_MASK_DENSE_MATH) != 0) + rs6000_define_or_undefine_macro (define_p, "__PPC_DMR__"); /* Whether pc-relative code is being generated. */ if ((flags & OPTION_MASK_PCREL) != 0) rs6000_define_or_undefine_macro (define_p, "__PCREL__"); diff --git a/gcc/config/rs6000/rs6000.cc b/gcc/config/rs6000/rs6000.cc index 8ecb3021ff9..c8f05f6f2d7 100644 --- a/gcc/config/rs6000/rs6000.cc +++ b/gcc/config/rs6000/rs6000.cc @@ -13910,8 +13910,13 @@ print_operand (FILE *file, rtx x, int code) overlapping with the FPR registers. */ if (!REG_P (x)) output_operand_lossage ("invalid %%A value"); - else if (TARGET_DENSE_MATH && DMR_REGNO_P (REGNO (x))) - fprintf (file, "%d", REGNO (x) - FIRST_DMR_REGNO); + else if (TARGET_DENSE_MATH) + { + if (DMR_REGNO_P (REGNO (x))) + fprintf (file, "%d", REGNO (x) - FIRST_DMR_REGNO); + else + output_operand_lossage ("%%A operand is not a DMR"); + } else if (!FP_REGNO_P (REGNO (x)) || (REGNO (x) % 4) != 0) output_operand_lossage ("invalid %%A value"); else @@ -27356,7 +27361,7 @@ rs6000_split_multireg_move (rtx dst, rtx src) /* If we are reading an accumulator register, we have to deprime it before we can access it. */ - if (TARGET_MMA + if (TARGET_MMA && !TARGET_DENSE_MATH && GET_MODE (src) == XOmode && FP_REGNO_P (REGNO (src))) emit_insn (gen_mma_xxmfacc (src, src)); @@ -27388,9 +27393,9 @@ rs6000_split_multireg_move (rtx dst, rtx src) emit_insn (gen_rtx_SET (dst2, src2)); } - /* If we are writing an accumulator register, we have to - prime it after we've written it. */ - if (TARGET_MMA + /* If we are writing an accumulator register that overlaps with the + FPR registers, we have to prime it after we've written it. */ + if (TARGET_MMA && !TARGET_DENSE_MATH && GET_MODE (dst) == XOmode && FP_REGNO_P (REGNO (dst))) emit_insn (gen_mma_xxmtacc (dst, dst)); @@ -27459,9 +27464,9 @@ rs6000_split_multireg_move (rtx dst, rtx src) emit_insn (gen_rtx_SET (dst_i, op)); } - /* We are writing an accumulator register, so we have to - prime it after we've written it. */ - if (GET_MODE (src) == XOmode) + /* On systems without dense math where accumulators overlap with the + vector registers, we have to prime it after we've written it. */ + if (GET_MODE (src) == XOmode && !TARGET_DENSE_MATH) emit_insn (gen_mma_xxmtacc (dst, dst)); return; @@ -27472,9 +27477,9 @@ rs6000_split_multireg_move (rtx dst, rtx src) if (REG_P (src) && REG_P (dst) && (REGNO (src) < REGNO (dst))) { - /* If we are reading an accumulator register, we have to - deprime it before we can access it. */ - if (TARGET_MMA + /* If we are reading an accumulator register and we don't have dense + math, we have to deprime it before we can access it. */ + if (TARGET_MMA && !TARGET_DENSE_MATH && GET_MODE (src) == XOmode && FP_REGNO_P (REGNO (src))) emit_insn (gen_mma_xxmfacc (src, src)); @@ -27502,7 +27507,7 @@ rs6000_split_multireg_move (rtx dst, rtx src) /* If we are writing an accumulator register, we have to prime it after we've written it. */ - if (TARGET_MMA + if (TARGET_MMA && !TARGET_DENSE_MATH && GET_MODE (dst) == XOmode && FP_REGNO_P (REGNO (dst))) emit_insn (gen_mma_xxmtacc (dst, dst)); } @@ -27639,7 +27644,7 @@ rs6000_split_multireg_move (rtx dst, rtx src) /* If we are reading an accumulator register, we have to deprime it before we can access it. */ - if (TARGET_MMA && REG_P (src) + if (TARGET_MMA && !TARGET_DENSE_MATH && REG_P (src) && GET_MODE (src) == XOmode && FP_REGNO_P (REGNO (src))) emit_insn (gen_mma_xxmfacc (src, src)); @@ -27671,7 +27676,7 @@ rs6000_split_multireg_move (rtx dst, rtx src) /* If we are writing an accumulator register, we have to prime it after we've written it. */ - if (TARGET_MMA && REG_P (dst) + if (TARGET_MMA && !TARGET_DENSE_MATH && REG_P (dst) && GET_MODE (dst) == XOmode && FP_REGNO_P (REGNO (dst))) emit_insn (gen_mma_xxmtacc (dst, dst));