* [PATCH] RISC-V: Refactor and cleanup fma patterns
@ 2023-09-18 11:37 Lehua Ding
2023-09-19 3:10 ` juzhe.zhong
0 siblings, 1 reply; 7+ messages in thread
From: Lehua Ding @ 2023-09-18 11:37 UTC (permalink / raw)
To: gcc-patches
Cc: juzhe.zhong, kito.cheng, rdapp.gcc, palmer, jeffreyalaw, lehua.ding
At present, FMA autovec's patterns do not fully use the corresponding pattern
in vector.md. The previous reason is that the merge operand of pattern in
vector.md cannot be VUNDEF. Now allowing it to be VUNDEF, reunify insn used for
reload pass into vector.md, and the corresponding vlmax pattern in autovec.md
is used for combine. This patch also refactors the corresponding combine
pattern inside autovec-opt.md and removes the unused ones.
gcc/ChangeLog:
* config/riscv/autovec-opt.md (*<optab>_fma<mode>):
Removed old combine patterns.
(*single_<optab>mult_plus<mode>): Ditto.
(*double_<optab>mult_plus<mode>): Ditto.
(*sign_zero_extend_fma): Ditto.
(*zero_sign_extend_fma): Ditto.
(*double_widen_fma<mode>): Ditto.
(*single_widen_fma<mode>): Ditto.
(*double_widen_fnma<mode>): Ditto.
(*single_widen_fnma<mode>): Ditto.
(*double_widen_fms<mode>): Ditto.
(*single_widen_fms<mode>): Ditto.
(*double_widen_fnms<mode>): Ditto.
(*single_widen_fnms<mode>): Ditto.
(*reduc_plus_scal_<mode>): Adjust name.
(*widen_reduc_plus_scal_<mode>): Adjust name.
(*dual_widen_fma<mode>): New combine pattern.
(*dual_widen_fmasu<mode>): Ditto.
(*dual_widen_fmaus<mode>): Ditto.
(*dual_fma<mode>): Ditto.
(*single_fma<mode>): Ditto.
(*dual_fnma<mode>): Ditto.
(*single_fnma<mode>): Ditto.
(*dual_fms<mode>): Ditto.
(*single_fms<mode>): Ditto.
(*dual_fnms<mode>): Ditto.
(*single_fnms<mode>): Ditto.
* config/riscv/autovec.md (fma<mode>4):
Reafctor fma pattern.
(*fma<VI:mode><P:mode>): Removed.
(fnma<mode>4): Reafctor.
(*fnma<VI:mode><P:mode>): Removed.
(*fma<VF:mode><P:mode>): Removed.
(*fnma<VF:mode><P:mode>): Removed.
(fms<mode>4): Reafctor.
(*fms<VF:mode><P:mode>): Removed.
(fnms<mode>4): Reafctor.
(*fnms<VF:mode><P:mode>): Removed.
* config/riscv/riscv-protos.h (prepare_ternary_operands):
Adjust prototype.
* config/riscv/riscv-v.cc (prepare_ternary_operands): Refactor.
* config/riscv/vector.md (*pred_mul_plus<mode>_undef): New pattern.
(*pred_mul_plus<mode>): Removed.
(*pred_mul_plus<mode>_scalar): Removed.
(*pred_mul_plus<mode>_extended_scalar): Removed.
(*pred_minus_mul<mode>_undef): New pattern.
(*pred_minus_mul<mode>): Removed.
(*pred_minus_mul<mode>_scalar): Removed.
(*pred_minus_mul<mode>_extended_scalar): Removed.
(*pred_mul_<optab><mode>_undef): New pattern.
(*pred_mul_<optab><mode>): Removed.
(*pred_mul_<optab><mode>_scalar): Removed.
(*pred_mul_neg_<optab><mode>_undef): New pattern.
(*pred_mul_neg_<optab><mode>): Removed.
(*pred_mul_neg_<optab><mode>_scalar): Removed.
---
gcc/config/riscv/autovec-opt.md | 736 ++++++++++++++------------------
gcc/config/riscv/autovec.md | 301 ++++---------
gcc/config/riscv/riscv-protos.h | 2 +-
gcc/config/riscv/riscv-v.cc | 14 +-
gcc/config/riscv/vector.md | 439 ++++++-------------
5 files changed, 528 insertions(+), 964 deletions(-)
diff --git a/gcc/config/riscv/autovec-opt.md b/gcc/config/riscv/autovec-opt.md
index df516849527..c94cd0ae087 100644
--- a/gcc/config/riscv/autovec-opt.md
+++ b/gcc/config/riscv/autovec-opt.md
@@ -110,166 +110,6 @@
[(set_attr "type" "vmalu")
(set_attr "mode" "<MODE>")])
-;; =========================================================================
-;; == Widening Ternary arithmetic
-;; =========================================================================
-
-;; -------------------------------------------------------------------------
-;; ---- [INT] VWMACC
-;; -------------------------------------------------------------------------
-;; Includes:
-;; - vwmacc.vv
-;; - vwmaccu.vv
-;; -------------------------------------------------------------------------
-
-;; Combine ext + ext + fma ===> widen fma.
-;; Most of circumstantces, LoopVectorizer will generate the following IR:
-;; vect__8.64_40 = (vector([4,4]) int) vect__7.63_41;
-;; vect__11.68_35 = (vector([4,4]) int) vect__10.67_36;
-;; vect__13.70_33 = .FMA (vect__11.68_35, vect__8.64_40, vect__4.60_45);
-(define_insn_and_split "*<optab>_fma<mode>"
- [(set (match_operand:VWEXTI 0 "register_operand")
- (plus:VWEXTI
- (mult:VWEXTI
- (any_extend:VWEXTI
- (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
- (any_extend:VWEXTI
- (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
- (match_operand:VWEXTI 1 "register_operand")))]
- "TARGET_VECTOR && can_create_pseudo_p ()"
- "#"
- "&& 1"
- [(const_int 0)]
- {
- riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_plus (<CODE>, <MODE>mode),
- riscv_vector::WIDEN_TERNARY_OP, operands);
- DONE;
- }
- [(set_attr "type" "viwmuladd")
- (set_attr "mode" "<V_DOUBLE_TRUNC>")])
-
-;; This helps to match ext + fma.
-(define_insn_and_split "*single_<optab>mult_plus<mode>"
- [(set (match_operand:VWEXTI 0 "register_operand")
- (plus:VWEXTI
- (mult:VWEXTI
- (any_extend:VWEXTI
- (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
- (match_operand:VWEXTI 3 "register_operand"))
- (match_operand:VWEXTI 1 "register_operand")))]
- "TARGET_VECTOR && can_create_pseudo_p ()"
- "#"
- "&& 1"
- [(const_int 0)]
- {
- insn_code icode = code_for_pred_vf2 (<CODE>, <MODE>mode);
- rtx tmp = gen_reg_rtx (<MODE>mode);
- rtx ext_ops[] = {tmp, operands[2]};
- riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ext_ops);
-
- rtx dst = expand_ternary_op (<MODE>mode, fma_optab, tmp, operands[3],
- operands[1], operands[0], 0);
- emit_move_insn (operands[0], dst);
- DONE;
- }
- [(set_attr "type" "viwmuladd")
- (set_attr "mode" "<V_DOUBLE_TRUNC>")])
-
-;; Combine ext + ext + mult + plus ===> widen fma.
-;; We have some special cases generated by LoopVectorizer:
-;; vect__8.18_46 = (vector([8,8]) signed short) vect__7.17_47;
-;; vect__11.22_41 = (vector([8,8]) signed short) vect__10.21_42;
-;; vect__12.23_40 = vect__11.22_41 * vect__8.18_46;
-;; vect__14.25_38 = vect__13.24_39 + vect__5.14_51;
-;; This situation doesn't generate FMA IR.
-(define_insn_and_split "*double_<optab>mult_plus<mode>"
- [(set (match_operand:VWEXTI 0 "register_operand")
- (if_then_else:VWEXTI
- (unspec:<VM>
- [(match_operand:<VM> 1 "vector_mask_operand")
- (match_operand 6 "vector_length_operand")
- (match_operand 7 "const_int_operand")
- (match_operand 8 "const_int_operand")
- (match_operand 9 "const_int_operand")
- (reg:SI VL_REGNUM)
- (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
- (plus:VWEXTI
- (if_then_else:VWEXTI
- (unspec:<VM>
- [(match_dup 1)
- (match_dup 6)
- (match_dup 7)
- (match_dup 8)
- (match_dup 9)
- (reg:SI VL_REGNUM)
- (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
- (mult:VWEXTI
- (any_extend:VWEXTI
- (match_operand:<V_DOUBLE_TRUNC> 4 "register_operand"))
- (any_extend:VWEXTI
- (match_operand:<V_DOUBLE_TRUNC> 5 "register_operand")))
- (match_operand:VWEXTI 2 "vector_undef_operand"))
- (match_operand:VWEXTI 3 "register_operand"))
- (match_dup 2)))]
- "TARGET_VECTOR && can_create_pseudo_p ()"
- "#"
- "&& 1"
- [(const_int 0)]
- {
- emit_insn (gen_pred_widen_mul_plus (<CODE>, <MODE>mode, operands[0],
- operands[1], operands[3], operands[4],
- operands[5], operands[6], operands[7],
- operands[8], operands[9]));
- DONE;
- }
- [(set_attr "type" "viwmuladd")
- (set_attr "mode" "<V_DOUBLE_TRUNC>")])
-
-;; Combine sign_extend + zero_extend + fma ===> widen fma (su).
-(define_insn_and_split "*sign_zero_extend_fma"
- [(set (match_operand:VWEXTI 0 "register_operand")
- (plus:VWEXTI
- (mult:VWEXTI
- (sign_extend:VWEXTI
- (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
- (zero_extend:VWEXTI
- (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
- (match_operand:VWEXTI 1 "register_operand")))]
- "TARGET_VECTOR && can_create_pseudo_p ()"
- "#"
- "&& 1"
- [(const_int 0)]
- {
- riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_plussu (<MODE>mode),
- riscv_vector::WIDEN_TERNARY_OP, operands);
- DONE;
- }
- [(set_attr "type" "viwmuladd")
- (set_attr "mode" "<V_DOUBLE_TRUNC>")])
-
-;; This helps to match zero_extend + sign_extend + fma.
-(define_insn_and_split "*zero_sign_extend_fma"
- [(set (match_operand:VWEXTI 0 "register_operand")
- (plus:VWEXTI
- (mult:VWEXTI
- (zero_extend:VWEXTI
- (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
- (sign_extend:VWEXTI
- (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
- (match_operand:VWEXTI 1 "register_operand")))]
- "TARGET_VECTOR && can_create_pseudo_p ()"
- "#"
- "&& 1"
- [(const_int 0)]
- {
- rtx ops[] = {operands[0], operands[1], operands[3], operands[2]};
- riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_plussu (<MODE>mode),
- riscv_vector::WIDEN_TERNARY_OP, ops);
- DONE;
- }
- [(set_attr "type" "viwmuladd")
- (set_attr "mode" "<V_DOUBLE_TRUNC>")])
-
;; -------------------------------------------------------------------------
;; ---- [INT] Binary narrow shifts.
;; -------------------------------------------------------------------------
@@ -345,269 +185,9 @@
[(set_attr "type" "vimovvx")
(set_attr "mode" "<MODE>")])
-;; -------------------------------------------------------------------------
-;; ---- [FP] VFWMACC
-;; -------------------------------------------------------------------------
-;; Includes:
-;; - vfwmacc.vv
-;; -------------------------------------------------------------------------
-
-;; Combine ext + ext + fma ===> widen fma.
-;; Most of circumstantces, LoopVectorizer will generate the following IR:
-;; vect__8.176_40 = (vector([2,2]) double) vect__7.175_41;
-;; vect__11.180_35 = (vector([2,2]) double) vect__10.179_36;
-;; vect__13.182_33 = .FMA (vect__11.180_35, vect__8.176_40, vect__4.172_45);
-(define_insn_and_split "*double_widen_fma<mode>"
- [(set (match_operand:VWEXTF 0 "register_operand")
- (unspec:VWEXTF
- [(fma:VWEXTF
- (float_extend:VWEXTF
- (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
- (float_extend:VWEXTF
- (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand"))
- (match_operand:VWEXTF 1 "register_operand"))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
- "TARGET_VECTOR && can_create_pseudo_p ()"
- "#"
- "&& 1"
- [(const_int 0)]
- {
- riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul (PLUS, <MODE>mode),
- riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, operands);
- DONE;
- }
- [(set_attr "type" "vfwmuladd")
- (set_attr "mode" "<V_DOUBLE_TRUNC>")
- (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
-
-;; This helps to match ext + fma.
-(define_insn_and_split "*single_widen_fma<mode>"
- [(set (match_operand:VWEXTF 0 "register_operand")
- (unspec:VWEXTF
- [(fma:VWEXTF
- (float_extend:VWEXTF
- (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
- (match_operand:VWEXTF 3 "register_operand")
- (match_operand:VWEXTF 1 "register_operand"))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
- "TARGET_VECTOR && can_create_pseudo_p ()"
- "#"
- "&& 1"
- [(const_int 0)]
- {
- insn_code icode = code_for_pred_extend (<MODE>mode);
- rtx tmp = gen_reg_rtx (<MODE>mode);
- rtx ext_ops[] = {tmp, operands[2]};
- riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ext_ops);
-
- rtx dst = expand_ternary_op (<MODE>mode, fma_optab, tmp, operands[3],
- operands[1], operands[0], 0);
- emit_move_insn (operands[0], dst);
- DONE;
- }
- [(set_attr "type" "vfwmuladd")
- (set_attr "mode" "<V_DOUBLE_TRUNC>")
- (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
-
-;; -------------------------------------------------------------------------
-;; ---- [FP] VFWNMSAC
-;; -------------------------------------------------------------------------
-;; Includes:
-;; - vfwnmsac.vv
-;; -------------------------------------------------------------------------
-
-;; Combine ext + ext + fnma ===> widen fnma.
-;; Most of circumstantces, LoopVectorizer will generate the following IR:
-;; vect__8.176_40 = (vector([2,2]) double) vect__7.175_41;
-;; vect__11.180_35 = (vector([2,2]) double) vect__10.179_36;
-;; vect__13.182_33 = .FNMA (vect__11.180_35, vect__8.176_40, vect__4.172_45);
-(define_insn_and_split "*double_widen_fnma<mode>"
- [(set (match_operand:VWEXTF 0 "register_operand")
- (unspec:VWEXTF
- [(fma:VWEXTF
- (neg:VWEXTF
- (float_extend:VWEXTF
- (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand")))
- (float_extend:VWEXTF
- (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand"))
- (match_operand:VWEXTF 1 "register_operand"))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
- "TARGET_VECTOR && can_create_pseudo_p ()"
- "#"
- "&& 1"
- [(const_int 0)]
- {
- riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_neg (PLUS, <MODE>mode),
- riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, operands);
- DONE;
- }
- [(set_attr "type" "vfwmuladd")
- (set_attr "mode" "<V_DOUBLE_TRUNC>")
- (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
-
-;; This helps to match ext + fnma.
-(define_insn_and_split "*single_widen_fnma<mode>"
- [(set (match_operand:VWEXTF 0 "register_operand")
- (unspec:VWEXTF
- [(fma:VWEXTF
- (neg:VWEXTF
- (float_extend:VWEXTF
- (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand")))
- (match_operand:VWEXTF 3 "register_operand")
- (match_operand:VWEXTF 1 "register_operand"))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
- "TARGET_VECTOR && can_create_pseudo_p ()"
- "#"
- "&& 1"
- [(const_int 0)]
- {
- insn_code icode = code_for_pred_extend (<MODE>mode);
- rtx tmp = gen_reg_rtx (<MODE>mode);
- rtx ext_ops[] = {tmp, operands[2]};
- riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ext_ops);
-
- rtx dst = expand_ternary_op (<MODE>mode, fnma_optab, tmp, operands[3],
- operands[1], operands[0], 0);
- emit_move_insn (operands[0], dst);
- DONE;
- }
- [(set_attr "type" "vfwmuladd")
- (set_attr "mode" "<V_DOUBLE_TRUNC>")
- (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
-
-;; -------------------------------------------------------------------------
-;; ---- [FP] VFWMSAC
-;; -------------------------------------------------------------------------
-;; Includes:
-;; - vfwmsac.vv
-;; -------------------------------------------------------------------------
-
-;; Combine ext + ext + fms ===> widen fms.
-;; Most of circumstantces, LoopVectorizer will generate the following IR:
-;; vect__8.176_40 = (vector([2,2]) double) vect__7.175_41;
-;; vect__11.180_35 = (vector([2,2]) double) vect__10.179_36;
-;; vect__13.182_33 = .FMS (vect__11.180_35, vect__8.176_40, vect__4.172_45);
-(define_insn_and_split "*double_widen_fms<mode>"
- [(set (match_operand:VWEXTF 0 "register_operand")
- (unspec:VWEXTF
- [(fma:VWEXTF
- (float_extend:VWEXTF
- (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
- (float_extend:VWEXTF
- (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand"))
- (neg:VWEXTF
- (match_operand:VWEXTF 1 "register_operand")))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
- "TARGET_VECTOR && can_create_pseudo_p ()"
- "#"
- "&& 1"
- [(const_int 0)]
- {
- riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul (MINUS, <MODE>mode),
- riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, operands);
- DONE;
- }
- [(set_attr "type" "vfwmuladd")
- (set_attr "mode" "<V_DOUBLE_TRUNC>")
- (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
-
-;; This helps to match ext + fms.
-(define_insn_and_split "*single_widen_fms<mode>"
- [(set (match_operand:VWEXTF 0 "register_operand")
- (unspec:VWEXTF
- [(fma:VWEXTF
- (float_extend:VWEXTF
- (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
- (match_operand:VWEXTF 3 "register_operand")
- (neg:VWEXTF
- (match_operand:VWEXTF 1 "register_operand")))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
- "TARGET_VECTOR && can_create_pseudo_p ()"
- "#"
- "&& 1"
- [(const_int 0)]
- {
- insn_code icode = code_for_pred_extend (<MODE>mode);
- rtx tmp = gen_reg_rtx (<MODE>mode);
- rtx ext_ops[] = {tmp, operands[2]};
- riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ext_ops);
-
- rtx dst = expand_ternary_op (<MODE>mode, fms_optab, tmp, operands[3],
- operands[1], operands[0], 0);
- emit_move_insn (operands[0], dst);
- DONE;
- }
- [(set_attr "type" "vfwmuladd")
- (set_attr "mode" "<V_DOUBLE_TRUNC>")
- (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
-
-;; -------------------------------------------------------------------------
-;; ---- [FP] VFWNMACC
-;; -------------------------------------------------------------------------
-;; Includes:
-;; - vfwnmacc.vv
-;; -------------------------------------------------------------------------
-
-;; Combine ext + ext + fnms ===> widen fnms.
-;; Most of circumstantces, LoopVectorizer will generate the following IR:
-;; vect__8.176_40 = (vector([2,2]) double) vect__7.175_41;
-;; vect__11.180_35 = (vector([2,2]) double) vect__10.179_36;
-;; vect__13.182_33 = .FNMS (vect__11.180_35, vect__8.176_40, vect__4.172_45);
-(define_insn_and_split "*double_widen_fnms<mode>"
- [(set (match_operand:VWEXTF 0 "register_operand")
- (unspec:VWEXTF
- [(fma:VWEXTF
- (neg:VWEXTF
- (float_extend:VWEXTF
- (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand")))
- (float_extend:VWEXTF
- (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand"))
- (neg:VWEXTF
- (match_operand:VWEXTF 1 "register_operand")))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
- "TARGET_VECTOR && can_create_pseudo_p ()"
- "#"
- "&& 1"
- [(const_int 0)]
- {
- riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_neg (MINUS, <MODE>mode),
- riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, operands);
- DONE;
- }
- [(set_attr "type" "vfwmuladd")
- (set_attr "mode" "<V_DOUBLE_TRUNC>")
- (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
-
-;; This helps to match ext + fnms.
-(define_insn_and_split "*single_widen_fnms<mode>"
- [(set (match_operand:VWEXTF 0 "register_operand")
- (unspec:VWEXTF
- [(fma:VWEXTF
- (neg:VWEXTF
- (float_extend:VWEXTF
- (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand")))
- (match_operand:VWEXTF 3 "register_operand")
- (neg:VWEXTF
- (match_operand:VWEXTF 1 "register_operand")))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
- "TARGET_VECTOR && can_create_pseudo_p ()"
- "#"
- "&& 1"
- [(const_int 0)]
- {
- insn_code icode = code_for_pred_extend (<MODE>mode);
- rtx tmp = gen_reg_rtx (<MODE>mode);
- rtx ext_ops[] = {tmp, operands[2]};
- riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ext_ops);
-
- rtx dst = expand_ternary_op (<MODE>mode, fnms_optab, tmp, operands[3],
- operands[1], operands[0], 0);
- emit_move_insn (operands[0], dst);
- DONE;
- }
- [(set_attr "type" "vfwmuladd")
- (set_attr "mode" "<V_DOUBLE_TRUNC>")
- (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
+;; =============================================================================
+;; All combine patterns for combine pass.
+;; =============================================================================
;; =============================================================================
;; Combine op + vmerge to cond_op
@@ -1197,7 +777,7 @@
[(set_attr "type" "vfwmul")])
;; Combine extend + vredsum to vwredsum[u]
-(define_insn_and_split "*reduc_plus_scal_<mode>"
+(define_insn_and_split "*widen_reduc_plus_scal_<mode>"
[(set (match_operand:<V_DOUBLE_EXTEND_VEL> 0 "register_operand")
(unspec:<V_DOUBLE_EXTEND_VEL> [
(any_extend:<V_DOUBLE_EXTEND>
@@ -1215,7 +795,7 @@
[(set_attr "type" "vector")])
;; Combine extend + vfredusum to vfwredusum
-(define_insn_and_split "*reduc_plus_scal_<mode>"
+(define_insn_and_split "*widen_reduc_plus_scal_<mode>"
[(set (match_operand:<V_DOUBLE_EXTEND_VEL> 0 "register_operand")
(unspec:<V_DOUBLE_EXTEND_VEL> [
(float_extend:<V_DOUBLE_EXTEND>
@@ -1278,6 +858,312 @@
}
[(set_attr "type" "vector")])
+;; =============================================================================
+;; Combine extend + ternop to widen_ternop
+;; =============================================================================
+
+;; Combine ext + fma(vmacc,vmadd) to widen_fma (vwmacc)
+(define_insn_and_split "*dual_widen_fma<mode>"
+ [(set (match_operand:VWEXTI 0 "register_operand")
+ (plus:VWEXTI
+ (mult:VWEXTI
+ (any_extend:VWEXTI
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
+ (any_extend:VWEXTI
+ (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
+ (match_operand:VWEXTI 1 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+ {
+ rtx ops[] = {operands[0], operands[1], operands[2], operands[3]};
+ riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_plus (<CODE>, <MODE>mode),
+ riscv_vector::WIDEN_TERNARY_OP, ops);
+ DONE;
+ }
+ [(set_attr "type" "viwmuladd")])
+
+;; Combine sign_extend + zero_extend + fma to widen_fma (vwmaccsu)
+(define_insn_and_split "*dual_widen_fmasu<mode>"
+ [(set (match_operand:VWEXTI 0 "register_operand")
+ (plus:VWEXTI
+ (mult:VWEXTI
+ (sign_extend:VWEXTI
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
+ (zero_extend:VWEXTI
+ (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
+ (match_operand:VWEXTI 1 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+ {
+ rtx ops[] = {operands[0], operands[1], operands[2], operands[3]};
+ riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_plussu (<MODE>mode),
+ riscv_vector::WIDEN_TERNARY_OP, ops);
+ DONE;
+ }
+ [(set_attr "type" "viwmuladd")])
+
+;; Combine zero_extend + sign_extend + fma to widen_fma (vwmaccsu)
+(define_insn_and_split "*dual_widen_fmaus<mode>"
+ [(set (match_operand:VWEXTI 0 "register_operand")
+ (plus:VWEXTI
+ (mult:VWEXTI
+ (zero_extend:VWEXTI
+ (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand"))
+ (sign_extend:VWEXTI
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand")))
+ (match_operand:VWEXTI 1 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+ {
+ rtx ops[] = {operands[0], operands[1], operands[2], operands[3]};
+ riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_plussu (<MODE>mode),
+ riscv_vector::WIDEN_TERNARY_OP, ops);
+ DONE;
+ }
+ [(set_attr "type" "viwmuladd")])
+
+;; This combine pattern does not correspond to an single instruction.
+;; This is a temporary pattern produced by a combine pass and if there
+;; is no further combine into widen pattern, then fall back to extend
+;; pattern and non-widen fma pattern.
+(define_insn_and_split "*single_widen_fma<mode>"
+ [(set (match_operand:VWEXTI 0 "register_operand")
+ (plus:VWEXTI
+ (mult:VWEXTI
+ (any_extend:VWEXTI
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
+ (match_operand:VWEXTI 3 "register_operand"))
+ (match_operand:VWEXTI 1 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+ {
+ insn_code extend_icode = code_for_pred_vf2 (<CODE>, <MODE>mode);
+ rtx tmp = gen_reg_rtx (<MODE>mode);
+ rtx extend_ops[] = {tmp, operands[2]};
+ riscv_vector::emit_vlmax_insn (extend_icode, riscv_vector::UNARY_OP,
+ extend_ops);
+
+ rtx ops[] = {operands[0], tmp, operands[3], operands[1]};
+ riscv_vector::emit_vlmax_insn (code_for_pred_mul_plus (<MODE>mode),
+ riscv_vector::TERNARY_OP, ops);
+ DONE;
+ }
+ [(set_attr "type" "viwmuladd")])
+
+;; Combine extend + fma to widen_fma (vfwmacc)
+(define_insn_and_split "*dual_fma<mode>"
+ [(set (match_operand:VWEXTF 0 "register_operand")
+ (plus:VWEXTF
+ (mult:VWEXTF
+ (float_extend:VWEXTF
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
+ (float_extend:VWEXTF
+ (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
+ (match_operand:VWEXTF 1 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+ {
+ rtx ops[] = {operands[0], operands[1], operands[2], operands[3]};
+ riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul (PLUS, <MODE>mode),
+ riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, ops);
+ DONE;
+ }
+ [(set_attr "type" "vfwmuladd")])
+
+;; This combine pattern does not correspond to an single instruction.
+;; This is a temporary pattern produced by a combine pass and if there
+;; is no further combine into widen pattern, then fall back to extend
+;; pattern and non-widen fma pattern.
+(define_insn_and_split "*single_fma<mode>"
+ [(set (match_operand:VWEXTF 0 "register_operand")
+ (plus:VWEXTF
+ (mult:VWEXTF
+ (float_extend:VWEXTF
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
+ (match_operand:VWEXTF 3 "register_operand"))
+ (match_operand:VWEXTF 1 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+ {
+ insn_code icode = code_for_pred_extend (<MODE>mode);
+ rtx tmp = gen_reg_rtx (<MODE>mode);
+ rtx ext_ops[] = {tmp, operands[2]};
+ riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ext_ops);
+
+ rtx ops[] = {operands[0], tmp, operands[3], operands[1]};
+ riscv_vector::emit_vlmax_insn (code_for_pred_mul (PLUS, <MODE>mode),
+ riscv_vector::TERNARY_OP_FRM_DYN, ops);
+ DONE;
+ }
+ [(set_attr "type" "vfwmuladd")])
+
+;; Combine extend + fnma to widen_fnma (vfwnmsac)
+(define_insn_and_split "*dual_fnma<mode>"
+ [(set (match_operand:VWEXTF 0 "register_operand")
+ (minus:VWEXTF
+ (match_operand:VWEXTF 1 "register_operand")
+ (mult:VWEXTF
+ (float_extend:VWEXTF
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
+ (float_extend:VWEXTF
+ (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+ {
+ rtx ops[] = {operands[0], operands[1], operands[2], operands[3]};
+ riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_neg (PLUS, <MODE>mode),
+ riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, ops);
+ DONE;
+ }
+ [(set_attr "type" "vfwmuladd")])
+
+;; This combine pattern does not correspond to an single instruction.
+;; This is a temporary pattern produced by a combine pass and if there
+;; is no further combine into widen pattern, then fall back to extend
+;; pattern and non-widen fnma pattern.
+(define_insn_and_split "*single_fnma<mode>"
+ [(set (match_operand:VWEXTF 0 "register_operand")
+ (minus:VWEXTF
+ (match_operand:VWEXTF 1 "register_operand")
+ (mult:VWEXTF
+ (float_extend:VWEXTF
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
+ (match_operand:VWEXTF 3 "register_operand"))))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+ {
+ insn_code icode = code_for_pred_extend (<MODE>mode);
+ rtx tmp = gen_reg_rtx (<MODE>mode);
+ rtx ext_ops[] = {tmp, operands[2]};
+ riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ext_ops);
+
+ rtx ops[] = {operands[0], tmp, operands[3], operands[1]};
+ riscv_vector::emit_vlmax_insn (code_for_pred_mul_neg (PLUS, <MODE>mode),
+ riscv_vector::TERNARY_OP_FRM_DYN, ops);
+ DONE;
+ }
+ [(set_attr "type" "vfwmuladd")])
+
+;; Combine extend + fms to widen_fms (vfwmsac)
+(define_insn_and_split "*dual_fms<mode>"
+ [(set (match_operand:VWEXTF 0 "register_operand")
+ (minus:VWEXTF
+ (mult:VWEXTF
+ (float_extend:VWEXTF
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
+ (float_extend:VWEXTF
+ (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
+ (match_operand:VWEXTF 1 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+ {
+ rtx ops[] = {operands[0], operands[1], operands[2], operands[3]};
+ riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul (MINUS, <MODE>mode),
+ riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, ops);
+ DONE;
+ }
+ [(set_attr "type" "vfwmuladd")])
+
+;; This combine pattern does not correspond to an single instruction.
+;; This is a temporary pattern produced by a combine pass and if there
+;; is no further combine into widen pattern, then fall back to extend
+;; pattern and non-widen fms pattern.
+(define_insn_and_split "*single_fms<mode>"
+ [(set (match_operand:VWEXTF 0 "register_operand")
+ (minus:VWEXTF
+ (mult:VWEXTF
+ (float_extend:VWEXTF
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
+ (match_operand:VWEXTF 3 "register_operand"))
+ (match_operand:VWEXTF 1 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+ {
+ insn_code icode = code_for_pred_extend (<MODE>mode);
+ rtx tmp = gen_reg_rtx (<MODE>mode);
+ rtx ext_ops[] = {tmp, operands[2]};
+ riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ext_ops);
+
+ rtx ops[] = {operands[0], tmp, operands[3], operands[1]};
+ riscv_vector::emit_vlmax_insn (code_for_pred_mul (MINUS, <MODE>mode),
+ riscv_vector::TERNARY_OP_FRM_DYN, ops);
+ DONE;
+ }
+ [(set_attr "type" "vfwmuladd")])
+
+;; Combine extend + fnms to widen_fnms (vfwnmacc)
+(define_insn_and_split "*dual_fnms<mode>"
+ [(set (match_operand:VWEXTF 0 "register_operand")
+ (minus:VWEXTF
+ (mult:VWEXTF
+ (neg:VWEXTF
+ (float_extend:VWEXTF
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand")))
+ (float_extend:VWEXTF
+ (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
+ (match_operand:VWEXTF 1 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+ {
+ rtx ops[] = {operands[0], operands[1], operands[2], operands[3]};
+ riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_neg (MINUS, <MODE>mode),
+ riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, ops);
+ DONE;
+ }
+ [(set_attr "type" "vfwmuladd")])
+
+;; This combine pattern does not correspond to an single instruction.
+;; This is a temporary pattern produced by a combine pass and if there
+;; is no further combine into widen pattern, then fall back to extend
+;; pattern and non-widen fnms pattern.
+(define_insn_and_split "*single_fnms<mode>"
+ [(set (match_operand:VWEXTF 0 "register_operand")
+ (minus:VWEXTF
+ (mult:VWEXTF
+ (neg:VWEXTF
+ (match_operand:VWEXTF 3 "register_operand"))
+ (float_extend:VWEXTF
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand")))
+ (match_operand:VWEXTF 1 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+ {
+ insn_code icode = code_for_pred_extend (<MODE>mode);
+ rtx tmp = gen_reg_rtx (<MODE>mode);
+ rtx ext_ops[] = {tmp, operands[2]};
+ riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ext_ops);
+
+ rtx ops[] = {operands[0], tmp, operands[3], operands[1]};
+ riscv_vector::emit_vlmax_insn (code_for_pred_mul_neg (MINUS, <MODE>mode),
+ riscv_vector::TERNARY_OP_FRM_DYN, ops);
+ DONE;
+ }
+ [(set_attr "type" "vfwmuladd")])
+
;; =============================================================================
;; Misc combine patterns
;; =============================================================================
diff --git a/gcc/config/riscv/autovec.md b/gcc/config/riscv/autovec.md
index 8537b9d41f6..96aaf869f88 100644
--- a/gcc/config/riscv/autovec.md
+++ b/gcc/config/riscv/autovec.md
@@ -1078,57 +1078,25 @@
;; - vmadd
;; -------------------------------------------------------------------------
-;; We can't expand FMA for the following reasons:
-;; 1. Before RA, we don't know which multiply-add instruction is the ideal one.
-;; The vmacc is the ideal instruction when operands[3] overlaps operands[0].
-;; The vmadd is the ideal instruction when operands[1|2] overlaps operands[0].
-;; 2. According to vector.md, the multiply-add patterns has 'merge' operand which
-;; is the operands[5]. Since operands[5] should overlap operands[0], this operand
-;; should be allocated the same regno as operands[1|2|3].
-;; 3. The 'merge' operand is always a real merge operand and we don't allow undefined
-;; operand.
-;; 4. The operation of FMA pattern needs VLMAX vsetlvi which needs a VL operand.
-;;
-;; In this situation, we design the codegen of FMA as follows:
-;; 1. clobber a scratch in the expand pattern of FMA.
-;; 2. Let's RA decide which input operand (operands[1|2|3]) overlap operands[0].
-;; 3. Generate instructions (vmacc or vmadd) according to the register allocation
-;; result after reload_completed.
-(define_expand "fma<mode>4"
- [(parallel
- [(set (match_operand:VI 0 "register_operand")
- (plus:VI
- (mult:VI
- (match_operand:VI 1 "register_operand")
- (match_operand:VI 2 "register_operand"))
- (match_operand:VI 3 "register_operand")))
- (clobber (match_dup 4))])]
- "TARGET_VECTOR"
- {
- operands[4] = gen_reg_rtx (Pmode);
- })
-
-(define_insn_and_split "*fma<VI:mode><P:mode>"
- [(set (match_operand:VI 0 "register_operand" "=vr, vr, ?&vr")
+(define_insn_and_split "fma<mode>4"
+ [(set (match_operand:VI 0 "register_operand")
(plus:VI
(mult:VI
- (match_operand:VI 1 "register_operand" " %0, vr, vr")
- (match_operand:VI 2 "register_operand" " vr, vr, vr"))
- (match_operand:VI 3 "register_operand" " vr, 0, vr")))
- (clobber (match_operand:P 4 "register_operand" "=r,r,r"))]
- "TARGET_VECTOR"
+ (match_operand:VI 1 "register_operand")
+ (match_operand:VI 2 "register_operand"))
+ (match_operand:VI 3 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
"#"
- "&& reload_completed"
+ "&& 1"
[(const_int 0)]
{
- riscv_vector::emit_vlmax_vsetvl (<VI:MODE>mode, operands[4]);
- rtx ops[] = {operands[0], operands[1], operands[2], operands[3], operands[0]};
- riscv_vector::emit_vlmax_insn_lra (code_for_pred_mul_plus (<VI:MODE>mode),
- riscv_vector::TERNARY_OP, ops, operands[4]);
+ rtx ops[] = {operands[0], operands[1], operands[2], operands[3],
+ RVV_VUNDEF(<MODE>mode)};
+ riscv_vector::emit_vlmax_insn (code_for_pred_mul_plus (<MODE>mode),
+ riscv_vector::TERNARY_OP, ops);
DONE;
}
- [(set_attr "type" "vimuladd")
- (set_attr "mode" "<VI:MODE>")])
+ [(set_attr "type" "vector")])
;; -------------------------------------------------------------------------
;; ---- [INT] VNMSAC and VNMSUB
@@ -1138,41 +1106,25 @@
;; - vnmsub
;; -------------------------------------------------------------------------
-(define_expand "fnma<mode>4"
- [(parallel
- [(set (match_operand:VI 0 "register_operand")
- (minus:VI
- (match_operand:VI 3 "register_operand")
- (mult:VI
- (match_operand:VI 1 "register_operand")
- (match_operand:VI 2 "register_operand"))))
- (clobber (match_dup 4))])]
- "TARGET_VECTOR"
- {
- operands[4] = gen_reg_rtx (Pmode);
- })
-
-(define_insn_and_split "*fnma<VI:mode><P:mode>"
- [(set (match_operand:VI 0 "register_operand" "=vr, vr, ?&vr")
- (minus:VI
- (match_operand:VI 3 "register_operand" " vr, 0, vr")
- (mult:VI
- (match_operand:VI 1 "register_operand" " %0, vr, vr")
- (match_operand:VI 2 "register_operand" " vr, vr, vr"))))
- (clobber (match_operand:P 4 "register_operand" "=r,r,r"))]
- "TARGET_VECTOR"
+(define_insn_and_split "fnma<mode>4"
+ [(set (match_operand:VI 0 "register_operand")
+ (minus:VI
+ (match_operand:VI 3 "register_operand")
+ (mult:VI
+ (match_operand:VI 1 "register_operand")
+ (match_operand:VI 2 "register_operand"))))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
"#"
- "&& reload_completed"
+ "&& 1"
[(const_int 0)]
{
- riscv_vector::emit_vlmax_vsetvl (<VI:MODE>mode, operands[4]);
- rtx ops[] = {operands[0], operands[1], operands[2], operands[3], operands[0]};
- riscv_vector::emit_vlmax_insn_lra (code_for_pred_minus_mul (<VI:MODE>mode),
- riscv_vector::TERNARY_OP, ops, operands[4]);
+ rtx ops[] = {operands[0], operands[1], operands[2], operands[3],
+ RVV_VUNDEF(<MODE>mode)};
+ riscv_vector::emit_vlmax_insn (code_for_pred_minus_mul (<MODE>mode),
+ riscv_vector::TERNARY_OP, ops);
DONE;
}
- [(set_attr "type" "vimuladd")
- (set_attr "mode" "<VI:MODE>")])
+ [(set_attr "type" "vector")])
;; -------------------------------------------------------------------------
;; ---- [FP] VFMACC and VFMADD
@@ -1182,45 +1134,25 @@
;; - vfmadd
;; -------------------------------------------------------------------------
-(define_expand "fma<mode>4"
- [(parallel
- [(set (match_operand:VF 0 "register_operand")
- (unspec:VF
- [(fma:VF
- (match_operand:VF 1 "register_operand")
- (match_operand:VF 2 "register_operand")
- (match_operand:VF 3 "register_operand"))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
- (clobber (match_dup 4))])]
- "TARGET_VECTOR"
- {
- operands[4] = gen_reg_rtx (Pmode);
- }
- [(set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
-
-(define_insn_and_split "*fma<VF:mode><P:mode>"
- [(set (match_operand:VF 0 "register_operand" "=vr, vr, ?&vr")
- (unspec:VF
- [(fma:VF
- (match_operand:VF 1 "register_operand" " %0, vr, vr")
- (match_operand:VF 2 "register_operand" " vr, vr, vr")
- (match_operand:VF 3 "register_operand" " vr, 0, vr"))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
- (clobber (match_operand:P 4 "register_operand" "=r,r,r"))]
- "TARGET_VECTOR"
+(define_insn_and_split "fma<mode>4"
+ [(set (match_operand:VF 0 "register_operand")
+ (plus:VF
+ (mult:VF
+ (match_operand:VF 1 "register_operand")
+ (match_operand:VF 2 "register_operand"))
+ (match_operand:VF 3 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
"#"
- "&& reload_completed"
+ "&& 1"
[(const_int 0)]
{
- riscv_vector::emit_vlmax_vsetvl (<VF:MODE>mode, operands[4]);
- rtx ops[] = {operands[0], operands[1], operands[2], operands[3], operands[0]};
- riscv_vector::emit_vlmax_insn_lra (code_for_pred_mul (PLUS, <VF:MODE>mode),
- riscv_vector::TERNARY_OP_FRM_DYN, ops, operands[4]);
+ rtx ops[] = {operands[0], operands[1], operands[2], operands[3],
+ RVV_VUNDEF(<MODE>mode)};
+ riscv_vector::emit_vlmax_insn (code_for_pred_mul (PLUS, <MODE>mode),
+ riscv_vector::TERNARY_OP_FRM_DYN, ops);
DONE;
}
- [(set_attr "type" "vfmuladd")
- (set_attr "mode" "<VF:MODE>")
- (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
+ [(set_attr "type" "vector")])
;; -------------------------------------------------------------------------
;; ---- [FP] VFNMSAC and VFNMSUB
@@ -1230,47 +1162,25 @@
;; - vfnmsub
;; -------------------------------------------------------------------------
-(define_expand "fnma<mode>4"
- [(parallel
- [(set (match_operand:VF 0 "register_operand")
- (unspec:VF
- [(fma:VF
- (neg:VF
- (match_operand:VF 1 "register_operand"))
- (match_operand:VF 2 "register_operand")
- (match_operand:VF 3 "register_operand"))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
- (clobber (match_dup 4))])]
- "TARGET_VECTOR"
- {
- operands[4] = gen_reg_rtx (Pmode);
- }
- [(set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
-
-(define_insn_and_split "*fnma<VF:mode><P:mode>"
- [(set (match_operand:VF 0 "register_operand" "=vr, vr, ?&vr")
- (unspec:VF
- [(fma:VF
- (neg:VF
- (match_operand:VF 1 "register_operand" " %0, vr, vr"))
- (match_operand:VF 2 "register_operand" " vr, vr, vr")
- (match_operand:VF 3 "register_operand" " vr, 0, vr"))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
- (clobber (match_operand:P 4 "register_operand" "=r,r,r"))]
- "TARGET_VECTOR"
+(define_insn_and_split "fnma<mode>4"
+ [(set (match_operand:VF 0 "register_operand")
+ (minus:VF
+ (match_operand:VF 3 "register_operand")
+ (mult:VF
+ (match_operand:VF 1 "register_operand")
+ (match_operand:VF 2 "register_operand"))))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
"#"
- "&& reload_completed"
+ "&& 1"
[(const_int 0)]
{
- riscv_vector::emit_vlmax_vsetvl (<VF:MODE>mode, operands[4]);
- rtx ops[] = {operands[0], operands[1], operands[2], operands[3], operands[0]};
- riscv_vector::emit_vlmax_insn_lra (code_for_pred_mul_neg (PLUS, <VF:MODE>mode),
- riscv_vector::TERNARY_OP_FRM_DYN, ops, operands[4]);
+ rtx ops[] = {operands[0], operands[1], operands[2], operands[3],
+ RVV_VUNDEF(<MODE>mode)};
+ riscv_vector::emit_vlmax_insn (code_for_pred_mul_neg (PLUS, <MODE>mode),
+ riscv_vector::TERNARY_OP_FRM_DYN, ops);
DONE;
}
- [(set_attr "type" "vfmuladd")
- (set_attr "mode" "<VF:MODE>")
- (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
+ [(set_attr "type" "vector")])
;; -------------------------------------------------------------------------
;; ---- [FP] VFMSAC and VFMSUB
@@ -1280,47 +1190,25 @@
;; - vfmsub
;; -------------------------------------------------------------------------
-(define_expand "fms<mode>4"
- [(parallel
- [(set (match_operand:VF 0 "register_operand")
- (unspec:VF
- [(fma:VF
- (match_operand:VF 1 "register_operand")
- (match_operand:VF 2 "register_operand")
- (neg:VF
- (match_operand:VF 3 "register_operand")))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
- (clobber (match_dup 4))])]
- "TARGET_VECTOR"
- {
- operands[4] = gen_reg_rtx (Pmode);
- }
- [(set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
-
-(define_insn_and_split "*fms<VF:mode><P:mode>"
- [(set (match_operand:VF 0 "register_operand" "=vr, vr, ?&vr")
- (unspec:VF
- [(fma:VF
- (match_operand:VF 1 "register_operand" " %0, vr, vr")
- (match_operand:VF 2 "register_operand" " vr, vr, vr")
- (neg:VF
- (match_operand:VF 3 "register_operand" " vr, 0, vr")))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
- (clobber (match_operand:P 4 "register_operand" "=r,r,r"))]
- "TARGET_VECTOR"
+(define_insn_and_split "fms<mode>4"
+ [(set (match_operand:VF 0 "register_operand")
+ (minus:VF
+ (mult:VF
+ (match_operand:VF 1 "register_operand")
+ (match_operand:VF 2 "register_operand"))
+ (match_operand:VF 3 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
"#"
- "&& reload_completed"
+ "&& 1"
[(const_int 0)]
{
- riscv_vector::emit_vlmax_vsetvl (<VF:MODE>mode, operands[4]);
- rtx ops[] = {operands[0], operands[1], operands[2], operands[3], operands[0]};
- riscv_vector::emit_vlmax_insn_lra (code_for_pred_mul (MINUS, <VF:MODE>mode),
- riscv_vector::TERNARY_OP_FRM_DYN, ops, operands[4]);
+ rtx ops[] = {operands[0], operands[1], operands[2], operands[3],
+ RVV_VUNDEF(<MODE>mode)};
+ riscv_vector::emit_vlmax_insn (code_for_pred_mul (MINUS, <MODE>mode),
+ riscv_vector::TERNARY_OP_FRM_DYN, ops);
DONE;
}
- [(set_attr "type" "vfmuladd")
- (set_attr "mode" "<VF:MODE>")
- (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
+ [(set_attr "type" "vector")])
;; -------------------------------------------------------------------------
;; ---- [FP] VFNMACC and VFNMADD
@@ -1330,49 +1218,26 @@
;; - vfnmadd
;; -------------------------------------------------------------------------
-(define_expand "fnms<mode>4"
- [(parallel
- [(set (match_operand:VF 0 "register_operand")
- (unspec:VF
- [(fma:VF
- (neg:VF
- (match_operand:VF 1 "register_operand"))
- (match_operand:VF 2 "register_operand")
- (neg:VF
- (match_operand:VF 3 "register_operand")))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
- (clobber (match_dup 4))])]
- "TARGET_VECTOR"
- {
- operands[4] = gen_reg_rtx (Pmode);
- }
- [(set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
-
-(define_insn_and_split "*fnms<VF:mode><P:mode>"
- [(set (match_operand:VF 0 "register_operand" "=vr, vr, ?&vr")
- (unspec:VF
- [(fma:VF
- (neg:VF
- (match_operand:VF 1 "register_operand" " %0, vr, vr"))
- (match_operand:VF 2 "register_operand" " vr, vr, vr")
- (neg:VF
- (match_operand:VF 3 "register_operand" " vr, 0, vr")))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
- (clobber (match_operand:P 4 "register_operand" "=r,r,r"))]
- "TARGET_VECTOR"
+(define_insn_and_split "fnms<mode>4"
+ [(set (match_operand:VF 0 "register_operand")
+ (minus:VF
+ (neg:VF
+ (mult:VF
+ (match_operand:VF 1 "register_operand")
+ (match_operand:VF 2 "register_operand")))
+ (match_operand:VF 3 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
"#"
- "&& reload_completed"
+ "&& 1"
[(const_int 0)]
{
- riscv_vector::emit_vlmax_vsetvl (<VF:MODE>mode, operands[4]);
- rtx ops[] = {operands[0], operands[1], operands[2], operands[3], operands[0]};
- riscv_vector::emit_vlmax_insn_lra (code_for_pred_mul_neg (MINUS, <VF:MODE>mode),
- riscv_vector::TERNARY_OP_FRM_DYN, ops, operands[4]);
+ rtx ops[] = {operands[0], operands[1], operands[2], operands[3],
+ RVV_VUNDEF(<MODE>mode)};
+ riscv_vector::emit_vlmax_insn (code_for_pred_mul_neg (MINUS, <MODE>mode),
+ riscv_vector::TERNARY_OP_FRM_DYN, ops);
DONE;
}
- [(set_attr "type" "vfmuladd")
- (set_attr "mode" "<VF:MODE>")
- (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
+ [(set_attr "type" "vector")])
;; =========================================================================
;; == SELECT_VL
diff --git a/gcc/config/riscv/riscv-protos.h b/gcc/config/riscv/riscv-protos.h
index 44fa36c32ab..46fc8f39974 100644
--- a/gcc/config/riscv/riscv-protos.h
+++ b/gcc/config/riscv/riscv-protos.h
@@ -462,7 +462,7 @@ void expand_select_vl (rtx *);
void expand_load_store (rtx *, bool);
void expand_gather_scatter (rtx *, bool);
void expand_cond_len_ternop (unsigned, rtx *);
-void prepare_ternary_operands (rtx *, bool = false);
+void prepare_ternary_operands (rtx *);
void expand_lanes_load_store (rtx *, bool);
void expand_fold_extract_last (rtx *);
diff --git a/gcc/config/riscv/riscv-v.cc b/gcc/config/riscv/riscv-v.cc
index 668594b65ed..f51986aebde 100644
--- a/gcc/config/riscv/riscv-v.cc
+++ b/gcc/config/riscv/riscv-v.cc
@@ -3253,15 +3253,15 @@ expand_reduction (unsigned unspec, rtx *ops, rtx init, reduction_type type)
/* Prepare ops for ternary operations.
It can be called before or after RA. */
void
-prepare_ternary_operands (rtx *ops, bool split_p)
+prepare_ternary_operands (rtx *ops)
{
machine_mode mode = GET_MODE (ops[0]);
- if (split_p
- || (!rtx_equal_p (ops[2], ops[5])
- && !rtx_equal_p (ops[3], ops[5])
- && !rtx_equal_p (ops[4], ops[5])
- && riscv_get_v_regno_alignment (mode) == 8))
+ if (!rtx_equal_p (ops[5], RVV_VUNDEF (mode))
+ && (VECTOR_MODE_P (GET_MODE (ops[2]))
+ && !rtx_equal_p (ops[2], ops[5]))
+ && !rtx_equal_p (ops[3], ops[5])
+ && !rtx_equal_p (ops[4], ops[5]))
{
/* RA will fail to find vector REG and report ICE, so we pre-merge
the ops for LMUL = 8. */
@@ -3286,6 +3286,8 @@ prepare_ternary_operands (rtx *ops, bool split_p)
/* TODO: ??? Maybe we could support splitting FMA (a, 4, b)
into PLUS (ASHIFT (a, 2), b) according to uarchs. */
}
+ gcc_assert (rtx_equal_p (ops[5], RVV_VUNDEF (mode))
+ || rtx_equal_p (ops[5], ops[2]) || rtx_equal_p (ops[5], ops[4]));
}
/* Expand VEC_MASK_LEN_{LOAD_LANES,STORE_LANES}. */
diff --git a/gcc/config/riscv/vector.md b/gcc/config/riscv/vector.md
index fb0c8abd995..e174322a459 100644
--- a/gcc/config/riscv/vector.md
+++ b/gcc/config/riscv/vector.md
@@ -5002,12 +5002,40 @@
(match_operand:VI 2 "register_operand")
(match_operand:VI 3 "register_operand"))
(match_operand:VI 4 "register_operand"))
- (match_operand:VI 5 "register_operand")))]
+ (match_operand:VI 5 "vector_merge_operand")))]
"TARGET_VECTOR"
{
riscv_vector::prepare_ternary_operands (operands);
})
+(define_insn "*pred_mul_plus<mode>_undef"
+ [(set (match_operand:VI 0 "register_operand" "=vd, vd,?&vd, vr, vr,?&vr")
+ (if_then_else:VI
+ (unspec:<VM>
+ [(match_operand:<VM> 1 "vector_mask_operand" " vm, vm, vm,Wc1,Wc1, Wc1")
+ (match_operand 6 "vector_length_operand" " rK, rK, rK, rK, rK, rK")
+ (match_operand 7 "const_int_operand" " i, i, i, i, i, i")
+ (match_operand 8 "const_int_operand" " i, i, i, i, i, i")
+ (match_operand 9 "const_int_operand" " i, i, i, i, i, i")
+ (reg:SI VL_REGNUM)
+ (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
+ (plus:VI
+ (mult:VI
+ (match_operand:VI 3 "register_operand" " 0, vr, vr, 0, vr, vr")
+ (match_operand:VI 4 "register_operand" " vr, vr, vr, vr, vr, vr"))
+ (match_operand:VI 5 "register_operand" " vr, 0, vr, vr, 0, vr"))
+ (match_operand:VI 2 "vector_undef_operand")))]
+ "TARGET_VECTOR"
+ "@
+ vmadd.vv\t%0,%4,%5%p1
+ vmacc.vv\t%0,%3,%4%p1
+ vmv.v.v\t%0,%4\;vmacc.vv\t%0,%3,%4%p1
+ vmadd.vv\t%0,%4,%5%p1
+ vmacc.vv\t%0,%3,%4%p1
+ vmv.v.v\t%0,%5\;vmacc.vv\t%0,%3,%4%p1"
+ [(set_attr "type" "vimuladd")
+ (set_attr "mode" "<MODE>")])
+
(define_insn "*pred_madd<mode>"
[(set (match_operand:VI 0 "register_operand" "=vd,?&vd, vr,?&vr")
(if_then_else:VI
@@ -5070,35 +5098,6 @@
(set (attr "ma") (symbol_ref "riscv_vector::get_ma(operands[7])"))
(set (attr "avl_type") (symbol_ref "INTVAL (operands[8])"))])
-(define_insn_and_rewrite "*pred_mul_plus<mode>"
- [(set (match_operand:VI 0 "register_operand" "=&vr")
- (if_then_else:VI
- (unspec:<VM>
- [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
- (match_operand 6 "vector_length_operand" " rK")
- (match_operand 7 "const_int_operand" " i")
- (match_operand 8 "const_int_operand" " i")
- (match_operand 9 "const_int_operand" " i")
- (reg:SI VL_REGNUM)
- (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
- (plus:VI
- (mult:VI
- (match_operand:VI 2 "register_operand" " vr")
- (match_operand:VI 3 "register_operand" " vr"))
- (match_operand:VI 4 "register_operand" " vr"))
- (match_operand:VI 5 "register_operand" " vr")))]
- "TARGET_VECTOR
- && !rtx_equal_p (operands[2], operands[5])
- && !rtx_equal_p (operands[3], operands[5])
- && !rtx_equal_p (operands[4], operands[5])"
- "#"
- "&& reload_completed"
- {
- riscv_vector::prepare_ternary_operands (operands, true);
- }
- [(set_attr "type" "vimuladd")
- (set_attr "mode" "<MODE>")])
-
(define_expand "@pred_mul_plus<mode>_scalar"
[(set (match_operand:VI_QHS 0 "register_operand")
(if_then_else:VI_QHS
@@ -5184,35 +5183,6 @@
(set (attr "ma") (symbol_ref "riscv_vector::get_ma(operands[7])"))
(set (attr "avl_type") (symbol_ref "INTVAL (operands[8])"))])
-(define_insn_and_rewrite "*pred_mul_plus<mode>_scalar"
- [(set (match_operand:VI 0 "register_operand" "=&vr")
- (if_then_else:VI
- (unspec:<VM>
- [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
- (match_operand 6 "vector_length_operand" " rK")
- (match_operand 7 "const_int_operand" " i")
- (match_operand 8 "const_int_operand" " i")
- (match_operand 9 "const_int_operand" " i")
- (reg:SI VL_REGNUM)
- (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
- (plus:VI
- (mult:VI
- (vec_duplicate:VI
- (match_operand:<VEL> 2 "register_operand" " r"))
- (match_operand:VI 3 "register_operand" " vr"))
- (match_operand:VI 4 "vector_arith_operand" " vr"))
- (match_operand:VI 5 "register_operand" " vr")))]
- "TARGET_VECTOR
- && !rtx_equal_p (operands[3], operands[5])
- && !rtx_equal_p (operands[4], operands[5])"
- "#"
- "&& reload_completed"
- {
- riscv_vector::prepare_ternary_operands (operands, true);
- }
- [(set_attr "type" "vimuladd")
- (set_attr "mode" "<MODE>")])
-
(define_expand "@pred_mul_plus<mode>_scalar"
[(set (match_operand:VI_D 0 "register_operand")
(if_then_else:VI_D
@@ -5313,36 +5283,6 @@
(set (attr "ma") (symbol_ref "riscv_vector::get_ma(operands[7])"))
(set (attr "avl_type") (symbol_ref "INTVAL (operands[8])"))])
-(define_insn_and_rewrite "*pred_mul_plus<mode>_extended_scalar"
- [(set (match_operand:VI_D 0 "register_operand" "=&vr")
- (if_then_else:VI_D
- (unspec:<VM>
- [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
- (match_operand 6 "vector_length_operand" " rK")
- (match_operand 7 "const_int_operand" " i")
- (match_operand 8 "const_int_operand" " i")
- (match_operand 9 "const_int_operand" " i")
- (reg:SI VL_REGNUM)
- (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
- (plus:VI_D
- (mult:VI_D
- (vec_duplicate:VI_D
- (sign_extend:<VEL>
- (match_operand:<VSUBEL> 2 "register_operand" " r")))
- (match_operand:VI_D 3 "register_operand" " vr"))
- (match_operand:VI_D 4 "register_operand" " vr"))
- (match_operand:VI_D 5 "register_operand" " vr")))]
- "TARGET_VECTOR
- && !rtx_equal_p (operands[3], operands[5])
- && !rtx_equal_p (operands[4], operands[5])"
- "#"
- "&& reload_completed"
- {
- riscv_vector::prepare_ternary_operands (operands, true);
- }
- [(set_attr "type" "vimuladd")
- (set_attr "mode" "<MODE>")])
-
(define_expand "@pred_minus_mul<mode>"
[(set (match_operand:VI 0 "register_operand")
(if_then_else:VI
@@ -5359,12 +5299,40 @@
(mult:VI
(match_operand:VI 2 "register_operand")
(match_operand:VI 3 "register_operand")))
- (match_operand:VI 5 "register_operand")))]
+ (match_operand:VI 5 "vector_merge_operand")))]
"TARGET_VECTOR"
{
riscv_vector::prepare_ternary_operands (operands);
})
+(define_insn "*pred_minus_mul<mode>_undef"
+ [(set (match_operand:VI 0 "register_operand" "=vd, vd,?&vd, vr, vr,?&vr")
+ (if_then_else:VI
+ (unspec:<VM>
+ [(match_operand:<VM> 1 "vector_mask_operand" " vm, vm, vm,Wc1,Wc1, Wc1")
+ (match_operand 6 "vector_length_operand" " rK, rK, rK, rK, rK, rK")
+ (match_operand 7 "const_int_operand" " i, i, i, i, i, i")
+ (match_operand 8 "const_int_operand" " i, i, i, i, i, i")
+ (match_operand 9 "const_int_operand" " i, i, i, i, i, i")
+ (reg:SI VL_REGNUM)
+ (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
+ (minus:VI
+ (match_operand:VI 5 "register_operand" " vr, 0, vr, vr, 0, vr")
+ (mult:VI
+ (match_operand:VI 3 "register_operand" " 0, vr, vr, 0, vr, vr")
+ (match_operand:VI 4 "register_operand" " vr, vr, vr, vr, vr, vr")))
+ (match_operand:VI 2 "vector_undef_operand")))]
+ "TARGET_VECTOR"
+ "@
+ vnmsub.vv\t%0,%4,%5%p1
+ vnmsac.vv\t%0,%3,%4%p1
+ vmv.v.v\t%0,%3\;vnmsub.vv\t%0,%4,%5%p1
+ vnmsub.vv\t%0,%4,%5%p1
+ vnmsac.vv\t%0,%3,%4%p1
+ vmv.v.v\t%0,%3\;vnmsub.vv\t%0,%4,%5%p1"
+ [(set_attr "type" "vimuladd")
+ (set_attr "mode" "<MODE>")])
+
(define_insn "*pred_nmsub<mode>"
[(set (match_operand:VI 0 "register_operand" "=vd,?&vd, vr,?&vr")
(if_then_else:VI
@@ -5427,35 +5395,6 @@
(set (attr "ma") (symbol_ref "riscv_vector::get_ma(operands[7])"))
(set (attr "avl_type") (symbol_ref "INTVAL (operands[8])"))])
-(define_insn_and_rewrite "*pred_minus_mul<mode>"
- [(set (match_operand:VI 0 "register_operand" "=&vr")
- (if_then_else:VI
- (unspec:<VM>
- [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
- (match_operand 6 "vector_length_operand" " rK")
- (match_operand 7 "const_int_operand" " i")
- (match_operand 8 "const_int_operand" " i")
- (match_operand 9 "const_int_operand" " i")
- (reg:SI VL_REGNUM)
- (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
- (minus:VI
- (match_operand:VI 4 "vector_arith_operand" " vr")
- (mult:VI
- (match_operand:VI 2 "register_operand" " vr")
- (match_operand:VI 3 "register_operand" " vr")))
- (match_operand:VI 5 "register_operand" " vr")))]
- "TARGET_VECTOR
- && !rtx_equal_p (operands[2], operands[5])
- && !rtx_equal_p (operands[3], operands[5])
- && !rtx_equal_p (operands[4], operands[5])"
- "#"
- "&& reload_completed"
- {
- riscv_vector::prepare_ternary_operands (operands, true);
- }
- [(set_attr "type" "vimuladd")
- (set_attr "mode" "<MODE>")])
-
(define_expand "@pred_minus_mul<mode>_scalar"
[(set (match_operand:VI_QHS 0 "register_operand")
(if_then_else:VI_QHS
@@ -5541,35 +5480,6 @@
(set (attr "ma") (symbol_ref "riscv_vector::get_ma(operands[7])"))
(set (attr "avl_type") (symbol_ref "INTVAL (operands[8])"))])
-(define_insn_and_rewrite "*pred_minus_mul<mode>_scalar"
- [(set (match_operand:VI 0 "register_operand" "=&vr")
- (if_then_else:VI
- (unspec:<VM>
- [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
- (match_operand 6 "vector_length_operand" " rK")
- (match_operand 7 "const_int_operand" " i")
- (match_operand 8 "const_int_operand" " i")
- (match_operand 9 "const_int_operand" " i")
- (reg:SI VL_REGNUM)
- (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
- (minus:VI
- (match_operand:VI 4 "vector_arith_operand" " vr")
- (mult:VI
- (vec_duplicate:VI
- (match_operand:<VEL> 2 "register_operand" " r"))
- (match_operand:VI 3 "register_operand" " vr")))
- (match_operand:VI 5 "register_operand" " vr")))]
- "TARGET_VECTOR
- && !rtx_equal_p (operands[3], operands[5])
- && !rtx_equal_p (operands[4], operands[5])"
- "#"
- "&& reload_completed"
- {
- riscv_vector::prepare_ternary_operands (operands, true);
- }
- [(set_attr "type" "vimuladd")
- (set_attr "mode" "<MODE>")])
-
(define_expand "@pred_minus_mul<mode>_scalar"
[(set (match_operand:VI_D 0 "register_operand")
(if_then_else:VI_D
@@ -5670,36 +5580,6 @@
(set (attr "ma") (symbol_ref "riscv_vector::get_ma(operands[7])"))
(set (attr "avl_type") (symbol_ref "INTVAL (operands[8])"))])
-(define_insn_and_rewrite "*pred_minus_mul<mode>_extended_scalar"
- [(set (match_operand:VI_D 0 "register_operand" "=&vr")
- (if_then_else:VI_D
- (unspec:<VM>
- [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
- (match_operand 6 "vector_length_operand" " rK")
- (match_operand 7 "const_int_operand" " i")
- (match_operand 8 "const_int_operand" " i")
- (match_operand 9 "const_int_operand" " i")
- (reg:SI VL_REGNUM)
- (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
- (minus:VI_D
- (match_operand:VI_D 4 "vector_arith_operand" " vr")
- (mult:VI_D
- (vec_duplicate:VI_D
- (sign_extend:<VEL>
- (match_operand:<VSUBEL> 2 "register_operand" " r")))
- (match_operand:VI_D 3 "register_operand" " vr")))
- (match_operand:VI_D 5 "register_operand" " vr")))]
- "TARGET_VECTOR
- && !rtx_equal_p (operands[3], operands[5])
- && !rtx_equal_p (operands[4], operands[5])"
- "#"
- "&& reload_completed"
- {
- riscv_vector::prepare_ternary_operands (operands, true);
- }
- [(set_attr "type" "vimuladd")
- (set_attr "mode" "<MODE>")])
-
;; -------------------------------------------------------------------------------
;; ---- Predicated widen integer ternary operations
;; -------------------------------------------------------------------------------
@@ -6276,12 +6156,44 @@
(match_operand:VF 2 "register_operand")
(match_operand:VF 3 "register_operand"))
(match_operand:VF 4 "register_operand"))
- (match_operand:VF 5 "register_operand")))]
+ (match_operand:VF 5 "vector_merge_operand")))]
"TARGET_VECTOR"
{
riscv_vector::prepare_ternary_operands (operands);
})
+(define_insn "*pred_mul_<optab><mode>_undef"
+ [(set (match_operand:VF 0 "register_operand" "=vd,vd,?&vd, vr, vr,?&vr")
+ (if_then_else:VF
+ (unspec:<VM>
+ [(match_operand:<VM> 1 "vector_mask_operand" " vm,vm, vm,Wc1,Wc1, Wc1")
+ (match_operand 6 "vector_length_operand" " rK,rK, rK, rK, rK, rK")
+ (match_operand 7 "const_int_operand" " i, i, i, i, i, i")
+ (match_operand 8 "const_int_operand" " i, i, i, i, i, i")
+ (match_operand 9 "const_int_operand" " i, i, i, i, i, i")
+ (match_operand 10 "const_int_operand" " i, i, i, i, i, i")
+ (reg:SI VL_REGNUM)
+ (reg:SI VTYPE_REGNUM)
+ (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE)
+ (plus_minus:VF
+ (mult:VF
+ (match_operand:VF 3 "register_operand" " 0,vr, vr, 0, vr, vr")
+ (match_operand:VF 4 "register_operand" " vr,vr, vr, vr, vr, vr"))
+ (match_operand:VF 5 "register_operand" " vr, 0, vr, vr, 0, vr"))
+ (match_operand:VF 2 "vector_undef_operand")))]
+ "TARGET_VECTOR"
+ "@
+ vf<madd_msub>.vv\t%0,%4,%5%p1
+ vf<macc_msac>.vv\t%0,%3,%4%p1
+ vmv.v.v\t%0,%3\;vf<madd_msub>.vv\t%0,%4,%5%p1
+ vf<madd_msub>.vv\t%0,%4,%5%p1
+ vf<macc_msac>.vv\t%0,%3,%4%p1
+ vmv.v.v\t%0,%3\;vf<madd_msub>.vv\t%0,%4,%5%p1"
+ [(set_attr "type" "vfmuladd")
+ (set_attr "mode" "<MODE>")
+ (set (attr "frm_mode")
+ (symbol_ref "riscv_vector::get_frm_mode (operands[10])"))])
+
(define_insn "*pred_<madd_msub><mode>"
[(set (match_operand:VF 0 "register_operand" "=vd, ?&vd, vr, ?&vr")
(if_then_else:VF
@@ -6352,39 +6264,6 @@
(set (attr "frm_mode")
(symbol_ref "riscv_vector::get_frm_mode (operands[9])"))])
-(define_insn_and_rewrite "*pred_mul_<optab><mode>"
- [(set (match_operand:VF 0 "register_operand" "=&vr")
- (if_then_else:VF
- (unspec:<VM>
- [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
- (match_operand 6 "vector_length_operand" " rK")
- (match_operand 7 "const_int_operand" " i")
- (match_operand 8 "const_int_operand" " i")
- (match_operand 9 "const_int_operand" " i")
- (match_operand 10 "const_int_operand" " i")
- (reg:SI VL_REGNUM)
- (reg:SI VTYPE_REGNUM)
- (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE)
- (plus_minus:VF
- (mult:VF
- (match_operand:VF 2 "register_operand" " vr")
- (match_operand:VF 3 "register_operand" " vr"))
- (match_operand:VF 4 "register_operand" " vr"))
- (match_operand:VF 5 "register_operand" " vr")))]
- "TARGET_VECTOR
- && !rtx_equal_p (operands[2], operands[5])
- && !rtx_equal_p (operands[3], operands[5])
- && !rtx_equal_p (operands[4], operands[5])"
- "#"
- "&& reload_completed"
- {
- riscv_vector::prepare_ternary_operands (operands, true);
- }
- [(set_attr "type" "vfmuladd")
- (set_attr "mode" "<MODE>")
- (set (attr "frm_mode")
- (symbol_ref "riscv_vector::get_frm_mode (operands[10])"))])
-
(define_expand "@pred_mul_<optab><mode>_scalar"
[(set (match_operand:VF 0 "register_operand")
(if_then_else:VF
@@ -6480,39 +6359,6 @@
(set (attr "frm_mode")
(symbol_ref "riscv_vector::get_frm_mode (operands[9])"))])
-(define_insn_and_rewrite "*pred_mul_<optab><mode>_scalar"
- [(set (match_operand:VF 0 "register_operand" "=&vr")
- (if_then_else:VF
- (unspec:<VM>
- [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
- (match_operand 6 "vector_length_operand" " rK")
- (match_operand 7 "const_int_operand" " i")
- (match_operand 8 "const_int_operand" " i")
- (match_operand 9 "const_int_operand" " i")
- (match_operand 10 "const_int_operand" " i")
- (reg:SI VL_REGNUM)
- (reg:SI VTYPE_REGNUM)
- (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE)
- (plus_minus:VF
- (mult:VF
- (vec_duplicate:VF
- (match_operand:<VEL> 2 "register_operand" " f"))
- (match_operand:VF 3 "register_operand" " vr"))
- (match_operand:VF 4 "vector_arith_operand" " vr"))
- (match_operand:VF 5 "register_operand" " vr")))]
- "TARGET_VECTOR
- && !rtx_equal_p (operands[3], operands[5])
- && !rtx_equal_p (operands[4], operands[5])"
- "#"
- "&& reload_completed"
- {
- riscv_vector::prepare_ternary_operands (operands, true);
- }
- [(set_attr "type" "vfmuladd")
- (set_attr "mode" "<MODE>")
- (set (attr "frm_mode")
- (symbol_ref "riscv_vector::get_frm_mode (operands[10])"))])
-
(define_expand "@pred_mul_neg_<optab><mode>"
[(set (match_operand:VF 0 "register_operand")
(if_then_else:VF
@@ -6532,12 +6378,45 @@
(match_operand:VF 2 "register_operand")
(match_operand:VF 3 "register_operand")))
(match_operand:VF 4 "register_operand"))
- (match_operand:VF 5 "register_operand")))]
+ (match_operand:VF 5 "vector_merge_operand")))]
"TARGET_VECTOR"
{
riscv_vector::prepare_ternary_operands (operands);
})
+(define_insn "*pred_mul_neg_<optab><mode>_undef"
+ [(set (match_operand:VF 0 "register_operand" "=vd,vd,?&vd, vr, vr,?&vr")
+ (if_then_else:VF
+ (unspec:<VM>
+ [(match_operand:<VM> 1 "vector_mask_operand" " vm,vm, vm,Wc1,Wc1, Wc1")
+ (match_operand 6 "vector_length_operand" " rK,rK, rK, rK, rK, rK")
+ (match_operand 7 "const_int_operand" " i, i, i, i, i, i")
+ (match_operand 8 "const_int_operand" " i, i, i, i, i, i")
+ (match_operand 9 "const_int_operand" " i, i, i, i, i, i")
+ (match_operand 10 "const_int_operand" " i, i, i, i, i, i")
+ (reg:SI VL_REGNUM)
+ (reg:SI VTYPE_REGNUM)
+ (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE)
+ (plus_minus:VF
+ (neg:VF
+ (mult:VF
+ (match_operand:VF 3 "register_operand" " 0,vr, vr, 0, vr, vr")
+ (match_operand:VF 4 "register_operand" " vr,vr, vr, vr, vr, vr")))
+ (match_operand:VF 5 "register_operand" " vr, 0, vr, vr, 0, vr"))
+ (match_operand:VF 2 "vector_undef_operand")))]
+ "TARGET_VECTOR"
+ "@
+ vf<nmsub_nmadd>.vv\t%0,%4,%5%p1
+ vf<nmsac_nmacc>.vv\t%0,%3,%4%p1
+ vmv.v.v\t%0,%3\;vf<nmsub_nmadd>.vv\t%0,%4,%5%p1
+ vf<nmsub_nmadd>.vv\t%0,%4,%5%p1
+ vf<nmsac_nmacc>.vv\t%0,%3,%4%p1
+ vmv.v.v\t%0,%3\;vf<nmsub_nmadd>.vv\t%0,%4,%5%p1"
+ [(set_attr "type" "vfmuladd")
+ (set_attr "mode" "<MODE>")
+ (set (attr "frm_mode")
+ (symbol_ref "riscv_vector::get_frm_mode (operands[10])"))])
+
(define_insn "*pred_<nmsub_nmadd><mode>"
[(set (match_operand:VF 0 "register_operand" "=vd, ?&vd, vr, ?&vr")
(if_then_else:VF
@@ -6610,40 +6489,6 @@
(set (attr "frm_mode")
(symbol_ref "riscv_vector::get_frm_mode (operands[9])"))])
-(define_insn_and_rewrite "*pred_mul_neg_<optab><mode>"
- [(set (match_operand:VF 0 "register_operand" "=&vr")
- (if_then_else:VF
- (unspec:<VM>
- [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
- (match_operand 6 "vector_length_operand" " rK")
- (match_operand 7 "const_int_operand" " i")
- (match_operand 8 "const_int_operand" " i")
- (match_operand 9 "const_int_operand" " i")
- (match_operand 10 "const_int_operand" " i")
- (reg:SI VL_REGNUM)
- (reg:SI VTYPE_REGNUM)
- (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE)
- (plus_minus:VF
- (neg:VF
- (mult:VF
- (match_operand:VF 2 "register_operand" " vr")
- (match_operand:VF 3 "register_operand" " vr")))
- (match_operand:VF 4 "vector_arith_operand" " vr"))
- (match_operand:VF 5 "register_operand" " vr")))]
- "TARGET_VECTOR
- && !rtx_equal_p (operands[2], operands[5])
- && !rtx_equal_p (operands[3], operands[5])
- && !rtx_equal_p (operands[4], operands[5])"
- "#"
- "&& reload_completed"
- {
- riscv_vector::prepare_ternary_operands (operands, true);
- }
- [(set_attr "type" "vfmuladd")
- (set_attr "mode" "<MODE>")
- (set (attr "frm_mode")
- (symbol_ref "riscv_vector::get_frm_mode (operands[10])"))])
-
(define_expand "@pred_mul_neg_<optab><mode>_scalar"
[(set (match_operand:VF 0 "register_operand")
(if_then_else:VF
@@ -6742,40 +6587,6 @@
(set (attr "frm_mode")
(symbol_ref "riscv_vector::get_frm_mode (operands[9])"))])
-(define_insn_and_rewrite "*pred_mul_neg_<optab><mode>_scalar"
- [(set (match_operand:VF 0 "register_operand" "=&vr")
- (if_then_else:VF
- (unspec:<VM>
- [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
- (match_operand 6 "vector_length_operand" " rK")
- (match_operand 7 "const_int_operand" " i")
- (match_operand 8 "const_int_operand" " i")
- (match_operand 9 "const_int_operand" " i")
- (match_operand 10 "const_int_operand" " i")
- (reg:SI VL_REGNUM)
- (reg:SI VTYPE_REGNUM)
- (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE)
- (plus_minus:VF
- (neg:VF
- (mult:VF
- (vec_duplicate:VF
- (match_operand:<VEL> 2 "register_operand" " f"))
- (match_operand:VF 3 "register_operand" " vr")))
- (match_operand:VF 4 "vector_arith_operand" " vr"))
- (match_operand:VF 5 "register_operand" " vr")))]
- "TARGET_VECTOR
- && !rtx_equal_p (operands[3], operands[5])
- && !rtx_equal_p (operands[4], operands[5])"
- "#"
- "&& reload_completed"
- {
- riscv_vector::prepare_ternary_operands (operands, true);
- }
- [(set_attr "type" "vfmuladd")
- (set_attr "mode" "<MODE>")
- (set (attr "frm_mode")
- (symbol_ref "riscv_vector::get_frm_mode (operands[10])"))])
-
;; -------------------------------------------------------------------------------
;; ---- Predicated floating-point unary operations
;; -------------------------------------------------------------------------------
--
2.36.3
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] RISC-V: Refactor and cleanup fma patterns
2023-09-18 11:37 [PATCH] RISC-V: Refactor and cleanup fma patterns Lehua Ding
@ 2023-09-19 3:10 ` juzhe.zhong
2023-09-19 3:23 ` Lehua Ding
0 siblings, 1 reply; 7+ messages in thread
From: juzhe.zhong @ 2023-09-19 3:10 UTC (permalink / raw)
To: 丁乐华, gcc-patches
Cc: kito.cheng, Robin Dapp, palmer, jeffreyalaw, 丁乐华
[-- Attachment #1: Type: text/plain, Size: 72788 bytes --]
Thanks for the refactoring.
This patch is needed in VLS fma support and undefined value enabling support.
LGTM.
juzhe.zhong@rivai.ai
From: Lehua Ding
Date: 2023-09-18 19:37
To: gcc-patches
CC: juzhe.zhong; kito.cheng; rdapp.gcc; palmer; jeffreyalaw; lehua.ding
Subject: [PATCH] RISC-V: Refactor and cleanup fma patterns
At present, FMA autovec's patterns do not fully use the corresponding pattern
in vector.md. The previous reason is that the merge operand of pattern in
vector.md cannot be VUNDEF. Now allowing it to be VUNDEF, reunify insn used for
reload pass into vector.md, and the corresponding vlmax pattern in autovec.md
is used for combine. This patch also refactors the corresponding combine
pattern inside autovec-opt.md and removes the unused ones.
gcc/ChangeLog:
* config/riscv/autovec-opt.md (*<optab>_fma<mode>):
Removed old combine patterns.
(*single_<optab>mult_plus<mode>): Ditto.
(*double_<optab>mult_plus<mode>): Ditto.
(*sign_zero_extend_fma): Ditto.
(*zero_sign_extend_fma): Ditto.
(*double_widen_fma<mode>): Ditto.
(*single_widen_fma<mode>): Ditto.
(*double_widen_fnma<mode>): Ditto.
(*single_widen_fnma<mode>): Ditto.
(*double_widen_fms<mode>): Ditto.
(*single_widen_fms<mode>): Ditto.
(*double_widen_fnms<mode>): Ditto.
(*single_widen_fnms<mode>): Ditto.
(*reduc_plus_scal_<mode>): Adjust name.
(*widen_reduc_plus_scal_<mode>): Adjust name.
(*dual_widen_fma<mode>): New combine pattern.
(*dual_widen_fmasu<mode>): Ditto.
(*dual_widen_fmaus<mode>): Ditto.
(*dual_fma<mode>): Ditto.
(*single_fma<mode>): Ditto.
(*dual_fnma<mode>): Ditto.
(*single_fnma<mode>): Ditto.
(*dual_fms<mode>): Ditto.
(*single_fms<mode>): Ditto.
(*dual_fnms<mode>): Ditto.
(*single_fnms<mode>): Ditto.
* config/riscv/autovec.md (fma<mode>4):
Reafctor fma pattern.
(*fma<VI:mode><P:mode>): Removed.
(fnma<mode>4): Reafctor.
(*fnma<VI:mode><P:mode>): Removed.
(*fma<VF:mode><P:mode>): Removed.
(*fnma<VF:mode><P:mode>): Removed.
(fms<mode>4): Reafctor.
(*fms<VF:mode><P:mode>): Removed.
(fnms<mode>4): Reafctor.
(*fnms<VF:mode><P:mode>): Removed.
* config/riscv/riscv-protos.h (prepare_ternary_operands):
Adjust prototype.
* config/riscv/riscv-v.cc (prepare_ternary_operands): Refactor.
* config/riscv/vector.md (*pred_mul_plus<mode>_undef): New pattern.
(*pred_mul_plus<mode>): Removed.
(*pred_mul_plus<mode>_scalar): Removed.
(*pred_mul_plus<mode>_extended_scalar): Removed.
(*pred_minus_mul<mode>_undef): New pattern.
(*pred_minus_mul<mode>): Removed.
(*pred_minus_mul<mode>_scalar): Removed.
(*pred_minus_mul<mode>_extended_scalar): Removed.
(*pred_mul_<optab><mode>_undef): New pattern.
(*pred_mul_<optab><mode>): Removed.
(*pred_mul_<optab><mode>_scalar): Removed.
(*pred_mul_neg_<optab><mode>_undef): New pattern.
(*pred_mul_neg_<optab><mode>): Removed.
(*pred_mul_neg_<optab><mode>_scalar): Removed.
---
gcc/config/riscv/autovec-opt.md | 736 ++++++++++++++------------------
gcc/config/riscv/autovec.md | 301 ++++---------
gcc/config/riscv/riscv-protos.h | 2 +-
gcc/config/riscv/riscv-v.cc | 14 +-
gcc/config/riscv/vector.md | 439 ++++++-------------
5 files changed, 528 insertions(+), 964 deletions(-)
diff --git a/gcc/config/riscv/autovec-opt.md b/gcc/config/riscv/autovec-opt.md
index df516849527..c94cd0ae087 100644
--- a/gcc/config/riscv/autovec-opt.md
+++ b/gcc/config/riscv/autovec-opt.md
@@ -110,166 +110,6 @@
[(set_attr "type" "vmalu")
(set_attr "mode" "<MODE>")])
-;; =========================================================================
-;; == Widening Ternary arithmetic
-;; =========================================================================
-
-;; -------------------------------------------------------------------------
-;; ---- [INT] VWMACC
-;; -------------------------------------------------------------------------
-;; Includes:
-;; - vwmacc.vv
-;; - vwmaccu.vv
-;; -------------------------------------------------------------------------
-
-;; Combine ext + ext + fma ===> widen fma.
-;; Most of circumstantces, LoopVectorizer will generate the following IR:
-;; vect__8.64_40 = (vector([4,4]) int) vect__7.63_41;
-;; vect__11.68_35 = (vector([4,4]) int) vect__10.67_36;
-;; vect__13.70_33 = .FMA (vect__11.68_35, vect__8.64_40, vect__4.60_45);
-(define_insn_and_split "*<optab>_fma<mode>"
- [(set (match_operand:VWEXTI 0 "register_operand")
- (plus:VWEXTI
- (mult:VWEXTI
- (any_extend:VWEXTI
- (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
- (any_extend:VWEXTI
- (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
- (match_operand:VWEXTI 1 "register_operand")))]
- "TARGET_VECTOR && can_create_pseudo_p ()"
- "#"
- "&& 1"
- [(const_int 0)]
- {
- riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_plus (<CODE>, <MODE>mode),
- riscv_vector::WIDEN_TERNARY_OP, operands);
- DONE;
- }
- [(set_attr "type" "viwmuladd")
- (set_attr "mode" "<V_DOUBLE_TRUNC>")])
-
-;; This helps to match ext + fma.
-(define_insn_and_split "*single_<optab>mult_plus<mode>"
- [(set (match_operand:VWEXTI 0 "register_operand")
- (plus:VWEXTI
- (mult:VWEXTI
- (any_extend:VWEXTI
- (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
- (match_operand:VWEXTI 3 "register_operand"))
- (match_operand:VWEXTI 1 "register_operand")))]
- "TARGET_VECTOR && can_create_pseudo_p ()"
- "#"
- "&& 1"
- [(const_int 0)]
- {
- insn_code icode = code_for_pred_vf2 (<CODE>, <MODE>mode);
- rtx tmp = gen_reg_rtx (<MODE>mode);
- rtx ext_ops[] = {tmp, operands[2]};
- riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ext_ops);
-
- rtx dst = expand_ternary_op (<MODE>mode, fma_optab, tmp, operands[3],
- operands[1], operands[0], 0);
- emit_move_insn (operands[0], dst);
- DONE;
- }
- [(set_attr "type" "viwmuladd")
- (set_attr "mode" "<V_DOUBLE_TRUNC>")])
-
-;; Combine ext + ext + mult + plus ===> widen fma.
-;; We have some special cases generated by LoopVectorizer:
-;; vect__8.18_46 = (vector([8,8]) signed short) vect__7.17_47;
-;; vect__11.22_41 = (vector([8,8]) signed short) vect__10.21_42;
-;; vect__12.23_40 = vect__11.22_41 * vect__8.18_46;
-;; vect__14.25_38 = vect__13.24_39 + vect__5.14_51;
-;; This situation doesn't generate FMA IR.
-(define_insn_and_split "*double_<optab>mult_plus<mode>"
- [(set (match_operand:VWEXTI 0 "register_operand")
- (if_then_else:VWEXTI
- (unspec:<VM>
- [(match_operand:<VM> 1 "vector_mask_operand")
- (match_operand 6 "vector_length_operand")
- (match_operand 7 "const_int_operand")
- (match_operand 8 "const_int_operand")
- (match_operand 9 "const_int_operand")
- (reg:SI VL_REGNUM)
- (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
- (plus:VWEXTI
- (if_then_else:VWEXTI
- (unspec:<VM>
- [(match_dup 1)
- (match_dup 6)
- (match_dup 7)
- (match_dup 8)
- (match_dup 9)
- (reg:SI VL_REGNUM)
- (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
- (mult:VWEXTI
- (any_extend:VWEXTI
- (match_operand:<V_DOUBLE_TRUNC> 4 "register_operand"))
- (any_extend:VWEXTI
- (match_operand:<V_DOUBLE_TRUNC> 5 "register_operand")))
- (match_operand:VWEXTI 2 "vector_undef_operand"))
- (match_operand:VWEXTI 3 "register_operand"))
- (match_dup 2)))]
- "TARGET_VECTOR && can_create_pseudo_p ()"
- "#"
- "&& 1"
- [(const_int 0)]
- {
- emit_insn (gen_pred_widen_mul_plus (<CODE>, <MODE>mode, operands[0],
- operands[1], operands[3], operands[4],
- operands[5], operands[6], operands[7],
- operands[8], operands[9]));
- DONE;
- }
- [(set_attr "type" "viwmuladd")
- (set_attr "mode" "<V_DOUBLE_TRUNC>")])
-
-;; Combine sign_extend + zero_extend + fma ===> widen fma (su).
-(define_insn_and_split "*sign_zero_extend_fma"
- [(set (match_operand:VWEXTI 0 "register_operand")
- (plus:VWEXTI
- (mult:VWEXTI
- (sign_extend:VWEXTI
- (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
- (zero_extend:VWEXTI
- (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
- (match_operand:VWEXTI 1 "register_operand")))]
- "TARGET_VECTOR && can_create_pseudo_p ()"
- "#"
- "&& 1"
- [(const_int 0)]
- {
- riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_plussu (<MODE>mode),
- riscv_vector::WIDEN_TERNARY_OP, operands);
- DONE;
- }
- [(set_attr "type" "viwmuladd")
- (set_attr "mode" "<V_DOUBLE_TRUNC>")])
-
-;; This helps to match zero_extend + sign_extend + fma.
-(define_insn_and_split "*zero_sign_extend_fma"
- [(set (match_operand:VWEXTI 0 "register_operand")
- (plus:VWEXTI
- (mult:VWEXTI
- (zero_extend:VWEXTI
- (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
- (sign_extend:VWEXTI
- (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
- (match_operand:VWEXTI 1 "register_operand")))]
- "TARGET_VECTOR && can_create_pseudo_p ()"
- "#"
- "&& 1"
- [(const_int 0)]
- {
- rtx ops[] = {operands[0], operands[1], operands[3], operands[2]};
- riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_plussu (<MODE>mode),
- riscv_vector::WIDEN_TERNARY_OP, ops);
- DONE;
- }
- [(set_attr "type" "viwmuladd")
- (set_attr "mode" "<V_DOUBLE_TRUNC>")])
-
;; -------------------------------------------------------------------------
;; ---- [INT] Binary narrow shifts.
;; -------------------------------------------------------------------------
@@ -345,269 +185,9 @@
[(set_attr "type" "vimovvx")
(set_attr "mode" "<MODE>")])
-;; -------------------------------------------------------------------------
-;; ---- [FP] VFWMACC
-;; -------------------------------------------------------------------------
-;; Includes:
-;; - vfwmacc.vv
-;; -------------------------------------------------------------------------
-
-;; Combine ext + ext + fma ===> widen fma.
-;; Most of circumstantces, LoopVectorizer will generate the following IR:
-;; vect__8.176_40 = (vector([2,2]) double) vect__7.175_41;
-;; vect__11.180_35 = (vector([2,2]) double) vect__10.179_36;
-;; vect__13.182_33 = .FMA (vect__11.180_35, vect__8.176_40, vect__4.172_45);
-(define_insn_and_split "*double_widen_fma<mode>"
- [(set (match_operand:VWEXTF 0 "register_operand")
- (unspec:VWEXTF
- [(fma:VWEXTF
- (float_extend:VWEXTF
- (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
- (float_extend:VWEXTF
- (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand"))
- (match_operand:VWEXTF 1 "register_operand"))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
- "TARGET_VECTOR && can_create_pseudo_p ()"
- "#"
- "&& 1"
- [(const_int 0)]
- {
- riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul (PLUS, <MODE>mode),
- riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, operands);
- DONE;
- }
- [(set_attr "type" "vfwmuladd")
- (set_attr "mode" "<V_DOUBLE_TRUNC>")
- (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
-
-;; This helps to match ext + fma.
-(define_insn_and_split "*single_widen_fma<mode>"
- [(set (match_operand:VWEXTF 0 "register_operand")
- (unspec:VWEXTF
- [(fma:VWEXTF
- (float_extend:VWEXTF
- (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
- (match_operand:VWEXTF 3 "register_operand")
- (match_operand:VWEXTF 1 "register_operand"))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
- "TARGET_VECTOR && can_create_pseudo_p ()"
- "#"
- "&& 1"
- [(const_int 0)]
- {
- insn_code icode = code_for_pred_extend (<MODE>mode);
- rtx tmp = gen_reg_rtx (<MODE>mode);
- rtx ext_ops[] = {tmp, operands[2]};
- riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ext_ops);
-
- rtx dst = expand_ternary_op (<MODE>mode, fma_optab, tmp, operands[3],
- operands[1], operands[0], 0);
- emit_move_insn (operands[0], dst);
- DONE;
- }
- [(set_attr "type" "vfwmuladd")
- (set_attr "mode" "<V_DOUBLE_TRUNC>")
- (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
-
-;; -------------------------------------------------------------------------
-;; ---- [FP] VFWNMSAC
-;; -------------------------------------------------------------------------
-;; Includes:
-;; - vfwnmsac.vv
-;; -------------------------------------------------------------------------
-
-;; Combine ext + ext + fnma ===> widen fnma.
-;; Most of circumstantces, LoopVectorizer will generate the following IR:
-;; vect__8.176_40 = (vector([2,2]) double) vect__7.175_41;
-;; vect__11.180_35 = (vector([2,2]) double) vect__10.179_36;
-;; vect__13.182_33 = .FNMA (vect__11.180_35, vect__8.176_40, vect__4.172_45);
-(define_insn_and_split "*double_widen_fnma<mode>"
- [(set (match_operand:VWEXTF 0 "register_operand")
- (unspec:VWEXTF
- [(fma:VWEXTF
- (neg:VWEXTF
- (float_extend:VWEXTF
- (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand")))
- (float_extend:VWEXTF
- (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand"))
- (match_operand:VWEXTF 1 "register_operand"))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
- "TARGET_VECTOR && can_create_pseudo_p ()"
- "#"
- "&& 1"
- [(const_int 0)]
- {
- riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_neg (PLUS, <MODE>mode),
- riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, operands);
- DONE;
- }
- [(set_attr "type" "vfwmuladd")
- (set_attr "mode" "<V_DOUBLE_TRUNC>")
- (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
-
-;; This helps to match ext + fnma.
-(define_insn_and_split "*single_widen_fnma<mode>"
- [(set (match_operand:VWEXTF 0 "register_operand")
- (unspec:VWEXTF
- [(fma:VWEXTF
- (neg:VWEXTF
- (float_extend:VWEXTF
- (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand")))
- (match_operand:VWEXTF 3 "register_operand")
- (match_operand:VWEXTF 1 "register_operand"))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
- "TARGET_VECTOR && can_create_pseudo_p ()"
- "#"
- "&& 1"
- [(const_int 0)]
- {
- insn_code icode = code_for_pred_extend (<MODE>mode);
- rtx tmp = gen_reg_rtx (<MODE>mode);
- rtx ext_ops[] = {tmp, operands[2]};
- riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ext_ops);
-
- rtx dst = expand_ternary_op (<MODE>mode, fnma_optab, tmp, operands[3],
- operands[1], operands[0], 0);
- emit_move_insn (operands[0], dst);
- DONE;
- }
- [(set_attr "type" "vfwmuladd")
- (set_attr "mode" "<V_DOUBLE_TRUNC>")
- (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
-
-;; -------------------------------------------------------------------------
-;; ---- [FP] VFWMSAC
-;; -------------------------------------------------------------------------
-;; Includes:
-;; - vfwmsac.vv
-;; -------------------------------------------------------------------------
-
-;; Combine ext + ext + fms ===> widen fms.
-;; Most of circumstantces, LoopVectorizer will generate the following IR:
-;; vect__8.176_40 = (vector([2,2]) double) vect__7.175_41;
-;; vect__11.180_35 = (vector([2,2]) double) vect__10.179_36;
-;; vect__13.182_33 = .FMS (vect__11.180_35, vect__8.176_40, vect__4.172_45);
-(define_insn_and_split "*double_widen_fms<mode>"
- [(set (match_operand:VWEXTF 0 "register_operand")
- (unspec:VWEXTF
- [(fma:VWEXTF
- (float_extend:VWEXTF
- (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
- (float_extend:VWEXTF
- (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand"))
- (neg:VWEXTF
- (match_operand:VWEXTF 1 "register_operand")))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
- "TARGET_VECTOR && can_create_pseudo_p ()"
- "#"
- "&& 1"
- [(const_int 0)]
- {
- riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul (MINUS, <MODE>mode),
- riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, operands);
- DONE;
- }
- [(set_attr "type" "vfwmuladd")
- (set_attr "mode" "<V_DOUBLE_TRUNC>")
- (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
-
-;; This helps to match ext + fms.
-(define_insn_and_split "*single_widen_fms<mode>"
- [(set (match_operand:VWEXTF 0 "register_operand")
- (unspec:VWEXTF
- [(fma:VWEXTF
- (float_extend:VWEXTF
- (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
- (match_operand:VWEXTF 3 "register_operand")
- (neg:VWEXTF
- (match_operand:VWEXTF 1 "register_operand")))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
- "TARGET_VECTOR && can_create_pseudo_p ()"
- "#"
- "&& 1"
- [(const_int 0)]
- {
- insn_code icode = code_for_pred_extend (<MODE>mode);
- rtx tmp = gen_reg_rtx (<MODE>mode);
- rtx ext_ops[] = {tmp, operands[2]};
- riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ext_ops);
-
- rtx dst = expand_ternary_op (<MODE>mode, fms_optab, tmp, operands[3],
- operands[1], operands[0], 0);
- emit_move_insn (operands[0], dst);
- DONE;
- }
- [(set_attr "type" "vfwmuladd")
- (set_attr "mode" "<V_DOUBLE_TRUNC>")
- (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
-
-;; -------------------------------------------------------------------------
-;; ---- [FP] VFWNMACC
-;; -------------------------------------------------------------------------
-;; Includes:
-;; - vfwnmacc.vv
-;; -------------------------------------------------------------------------
-
-;; Combine ext + ext + fnms ===> widen fnms.
-;; Most of circumstantces, LoopVectorizer will generate the following IR:
-;; vect__8.176_40 = (vector([2,2]) double) vect__7.175_41;
-;; vect__11.180_35 = (vector([2,2]) double) vect__10.179_36;
-;; vect__13.182_33 = .FNMS (vect__11.180_35, vect__8.176_40, vect__4.172_45);
-(define_insn_and_split "*double_widen_fnms<mode>"
- [(set (match_operand:VWEXTF 0 "register_operand")
- (unspec:VWEXTF
- [(fma:VWEXTF
- (neg:VWEXTF
- (float_extend:VWEXTF
- (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand")))
- (float_extend:VWEXTF
- (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand"))
- (neg:VWEXTF
- (match_operand:VWEXTF 1 "register_operand")))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
- "TARGET_VECTOR && can_create_pseudo_p ()"
- "#"
- "&& 1"
- [(const_int 0)]
- {
- riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_neg (MINUS, <MODE>mode),
- riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, operands);
- DONE;
- }
- [(set_attr "type" "vfwmuladd")
- (set_attr "mode" "<V_DOUBLE_TRUNC>")
- (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
-
-;; This helps to match ext + fnms.
-(define_insn_and_split "*single_widen_fnms<mode>"
- [(set (match_operand:VWEXTF 0 "register_operand")
- (unspec:VWEXTF
- [(fma:VWEXTF
- (neg:VWEXTF
- (float_extend:VWEXTF
- (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand")))
- (match_operand:VWEXTF 3 "register_operand")
- (neg:VWEXTF
- (match_operand:VWEXTF 1 "register_operand")))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
- "TARGET_VECTOR && can_create_pseudo_p ()"
- "#"
- "&& 1"
- [(const_int 0)]
- {
- insn_code icode = code_for_pred_extend (<MODE>mode);
- rtx tmp = gen_reg_rtx (<MODE>mode);
- rtx ext_ops[] = {tmp, operands[2]};
- riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ext_ops);
-
- rtx dst = expand_ternary_op (<MODE>mode, fnms_optab, tmp, operands[3],
- operands[1], operands[0], 0);
- emit_move_insn (operands[0], dst);
- DONE;
- }
- [(set_attr "type" "vfwmuladd")
- (set_attr "mode" "<V_DOUBLE_TRUNC>")
- (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
+;; =============================================================================
+;; All combine patterns for combine pass.
+;; =============================================================================
;; =============================================================================
;; Combine op + vmerge to cond_op
@@ -1197,7 +777,7 @@
[(set_attr "type" "vfwmul")])
;; Combine extend + vredsum to vwredsum[u]
-(define_insn_and_split "*reduc_plus_scal_<mode>"
+(define_insn_and_split "*widen_reduc_plus_scal_<mode>"
[(set (match_operand:<V_DOUBLE_EXTEND_VEL> 0 "register_operand")
(unspec:<V_DOUBLE_EXTEND_VEL> [
(any_extend:<V_DOUBLE_EXTEND>
@@ -1215,7 +795,7 @@
[(set_attr "type" "vector")])
;; Combine extend + vfredusum to vfwredusum
-(define_insn_and_split "*reduc_plus_scal_<mode>"
+(define_insn_and_split "*widen_reduc_plus_scal_<mode>"
[(set (match_operand:<V_DOUBLE_EXTEND_VEL> 0 "register_operand")
(unspec:<V_DOUBLE_EXTEND_VEL> [
(float_extend:<V_DOUBLE_EXTEND>
@@ -1278,6 +858,312 @@
}
[(set_attr "type" "vector")])
+;; =============================================================================
+;; Combine extend + ternop to widen_ternop
+;; =============================================================================
+
+;; Combine ext + fma(vmacc,vmadd) to widen_fma (vwmacc)
+(define_insn_and_split "*dual_widen_fma<mode>"
+ [(set (match_operand:VWEXTI 0 "register_operand")
+ (plus:VWEXTI
+ (mult:VWEXTI
+ (any_extend:VWEXTI
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
+ (any_extend:VWEXTI
+ (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
+ (match_operand:VWEXTI 1 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+ {
+ rtx ops[] = {operands[0], operands[1], operands[2], operands[3]};
+ riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_plus (<CODE>, <MODE>mode),
+ riscv_vector::WIDEN_TERNARY_OP, ops);
+ DONE;
+ }
+ [(set_attr "type" "viwmuladd")])
+
+;; Combine sign_extend + zero_extend + fma to widen_fma (vwmaccsu)
+(define_insn_and_split "*dual_widen_fmasu<mode>"
+ [(set (match_operand:VWEXTI 0 "register_operand")
+ (plus:VWEXTI
+ (mult:VWEXTI
+ (sign_extend:VWEXTI
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
+ (zero_extend:VWEXTI
+ (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
+ (match_operand:VWEXTI 1 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+ {
+ rtx ops[] = {operands[0], operands[1], operands[2], operands[3]};
+ riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_plussu (<MODE>mode),
+ riscv_vector::WIDEN_TERNARY_OP, ops);
+ DONE;
+ }
+ [(set_attr "type" "viwmuladd")])
+
+;; Combine zero_extend + sign_extend + fma to widen_fma (vwmaccsu)
+(define_insn_and_split "*dual_widen_fmaus<mode>"
+ [(set (match_operand:VWEXTI 0 "register_operand")
+ (plus:VWEXTI
+ (mult:VWEXTI
+ (zero_extend:VWEXTI
+ (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand"))
+ (sign_extend:VWEXTI
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand")))
+ (match_operand:VWEXTI 1 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+ {
+ rtx ops[] = {operands[0], operands[1], operands[2], operands[3]};
+ riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_plussu (<MODE>mode),
+ riscv_vector::WIDEN_TERNARY_OP, ops);
+ DONE;
+ }
+ [(set_attr "type" "viwmuladd")])
+
+;; This combine pattern does not correspond to an single instruction.
+;; This is a temporary pattern produced by a combine pass and if there
+;; is no further combine into widen pattern, then fall back to extend
+;; pattern and non-widen fma pattern.
+(define_insn_and_split "*single_widen_fma<mode>"
+ [(set (match_operand:VWEXTI 0 "register_operand")
+ (plus:VWEXTI
+ (mult:VWEXTI
+ (any_extend:VWEXTI
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
+ (match_operand:VWEXTI 3 "register_operand"))
+ (match_operand:VWEXTI 1 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+ {
+ insn_code extend_icode = code_for_pred_vf2 (<CODE>, <MODE>mode);
+ rtx tmp = gen_reg_rtx (<MODE>mode);
+ rtx extend_ops[] = {tmp, operands[2]};
+ riscv_vector::emit_vlmax_insn (extend_icode, riscv_vector::UNARY_OP,
+ extend_ops);
+
+ rtx ops[] = {operands[0], tmp, operands[3], operands[1]};
+ riscv_vector::emit_vlmax_insn (code_for_pred_mul_plus (<MODE>mode),
+ riscv_vector::TERNARY_OP, ops);
+ DONE;
+ }
+ [(set_attr "type" "viwmuladd")])
+
+;; Combine extend + fma to widen_fma (vfwmacc)
+(define_insn_and_split "*dual_fma<mode>"
+ [(set (match_operand:VWEXTF 0 "register_operand")
+ (plus:VWEXTF
+ (mult:VWEXTF
+ (float_extend:VWEXTF
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
+ (float_extend:VWEXTF
+ (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
+ (match_operand:VWEXTF 1 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+ {
+ rtx ops[] = {operands[0], operands[1], operands[2], operands[3]};
+ riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul (PLUS, <MODE>mode),
+ riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, ops);
+ DONE;
+ }
+ [(set_attr "type" "vfwmuladd")])
+
+;; This combine pattern does not correspond to an single instruction.
+;; This is a temporary pattern produced by a combine pass and if there
+;; is no further combine into widen pattern, then fall back to extend
+;; pattern and non-widen fma pattern.
+(define_insn_and_split "*single_fma<mode>"
+ [(set (match_operand:VWEXTF 0 "register_operand")
+ (plus:VWEXTF
+ (mult:VWEXTF
+ (float_extend:VWEXTF
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
+ (match_operand:VWEXTF 3 "register_operand"))
+ (match_operand:VWEXTF 1 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+ {
+ insn_code icode = code_for_pred_extend (<MODE>mode);
+ rtx tmp = gen_reg_rtx (<MODE>mode);
+ rtx ext_ops[] = {tmp, operands[2]};
+ riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ext_ops);
+
+ rtx ops[] = {operands[0], tmp, operands[3], operands[1]};
+ riscv_vector::emit_vlmax_insn (code_for_pred_mul (PLUS, <MODE>mode),
+ riscv_vector::TERNARY_OP_FRM_DYN, ops);
+ DONE;
+ }
+ [(set_attr "type" "vfwmuladd")])
+
+;; Combine extend + fnma to widen_fnma (vfwnmsac)
+(define_insn_and_split "*dual_fnma<mode>"
+ [(set (match_operand:VWEXTF 0 "register_operand")
+ (minus:VWEXTF
+ (match_operand:VWEXTF 1 "register_operand")
+ (mult:VWEXTF
+ (float_extend:VWEXTF
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
+ (float_extend:VWEXTF
+ (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+ {
+ rtx ops[] = {operands[0], operands[1], operands[2], operands[3]};
+ riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_neg (PLUS, <MODE>mode),
+ riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, ops);
+ DONE;
+ }
+ [(set_attr "type" "vfwmuladd")])
+
+;; This combine pattern does not correspond to an single instruction.
+;; This is a temporary pattern produced by a combine pass and if there
+;; is no further combine into widen pattern, then fall back to extend
+;; pattern and non-widen fnma pattern.
+(define_insn_and_split "*single_fnma<mode>"
+ [(set (match_operand:VWEXTF 0 "register_operand")
+ (minus:VWEXTF
+ (match_operand:VWEXTF 1 "register_operand")
+ (mult:VWEXTF
+ (float_extend:VWEXTF
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
+ (match_operand:VWEXTF 3 "register_operand"))))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+ {
+ insn_code icode = code_for_pred_extend (<MODE>mode);
+ rtx tmp = gen_reg_rtx (<MODE>mode);
+ rtx ext_ops[] = {tmp, operands[2]};
+ riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ext_ops);
+
+ rtx ops[] = {operands[0], tmp, operands[3], operands[1]};
+ riscv_vector::emit_vlmax_insn (code_for_pred_mul_neg (PLUS, <MODE>mode),
+ riscv_vector::TERNARY_OP_FRM_DYN, ops);
+ DONE;
+ }
+ [(set_attr "type" "vfwmuladd")])
+
+;; Combine extend + fms to widen_fms (vfwmsac)
+(define_insn_and_split "*dual_fms<mode>"
+ [(set (match_operand:VWEXTF 0 "register_operand")
+ (minus:VWEXTF
+ (mult:VWEXTF
+ (float_extend:VWEXTF
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
+ (float_extend:VWEXTF
+ (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
+ (match_operand:VWEXTF 1 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+ {
+ rtx ops[] = {operands[0], operands[1], operands[2], operands[3]};
+ riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul (MINUS, <MODE>mode),
+ riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, ops);
+ DONE;
+ }
+ [(set_attr "type" "vfwmuladd")])
+
+;; This combine pattern does not correspond to an single instruction.
+;; This is a temporary pattern produced by a combine pass and if there
+;; is no further combine into widen pattern, then fall back to extend
+;; pattern and non-widen fms pattern.
+(define_insn_and_split "*single_fms<mode>"
+ [(set (match_operand:VWEXTF 0 "register_operand")
+ (minus:VWEXTF
+ (mult:VWEXTF
+ (float_extend:VWEXTF
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
+ (match_operand:VWEXTF 3 "register_operand"))
+ (match_operand:VWEXTF 1 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+ {
+ insn_code icode = code_for_pred_extend (<MODE>mode);
+ rtx tmp = gen_reg_rtx (<MODE>mode);
+ rtx ext_ops[] = {tmp, operands[2]};
+ riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ext_ops);
+
+ rtx ops[] = {operands[0], tmp, operands[3], operands[1]};
+ riscv_vector::emit_vlmax_insn (code_for_pred_mul (MINUS, <MODE>mode),
+ riscv_vector::TERNARY_OP_FRM_DYN, ops);
+ DONE;
+ }
+ [(set_attr "type" "vfwmuladd")])
+
+;; Combine extend + fnms to widen_fnms (vfwnmacc)
+(define_insn_and_split "*dual_fnms<mode>"
+ [(set (match_operand:VWEXTF 0 "register_operand")
+ (minus:VWEXTF
+ (mult:VWEXTF
+ (neg:VWEXTF
+ (float_extend:VWEXTF
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand")))
+ (float_extend:VWEXTF
+ (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
+ (match_operand:VWEXTF 1 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+ {
+ rtx ops[] = {operands[0], operands[1], operands[2], operands[3]};
+ riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_neg (MINUS, <MODE>mode),
+ riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, ops);
+ DONE;
+ }
+ [(set_attr "type" "vfwmuladd")])
+
+;; This combine pattern does not correspond to an single instruction.
+;; This is a temporary pattern produced by a combine pass and if there
+;; is no further combine into widen pattern, then fall back to extend
+;; pattern and non-widen fnms pattern.
+(define_insn_and_split "*single_fnms<mode>"
+ [(set (match_operand:VWEXTF 0 "register_operand")
+ (minus:VWEXTF
+ (mult:VWEXTF
+ (neg:VWEXTF
+ (match_operand:VWEXTF 3 "register_operand"))
+ (float_extend:VWEXTF
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand")))
+ (match_operand:VWEXTF 1 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+ {
+ insn_code icode = code_for_pred_extend (<MODE>mode);
+ rtx tmp = gen_reg_rtx (<MODE>mode);
+ rtx ext_ops[] = {tmp, operands[2]};
+ riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ext_ops);
+
+ rtx ops[] = {operands[0], tmp, operands[3], operands[1]};
+ riscv_vector::emit_vlmax_insn (code_for_pred_mul_neg (MINUS, <MODE>mode),
+ riscv_vector::TERNARY_OP_FRM_DYN, ops);
+ DONE;
+ }
+ [(set_attr "type" "vfwmuladd")])
+
;; =============================================================================
;; Misc combine patterns
;; =============================================================================
diff --git a/gcc/config/riscv/autovec.md b/gcc/config/riscv/autovec.md
index 8537b9d41f6..96aaf869f88 100644
--- a/gcc/config/riscv/autovec.md
+++ b/gcc/config/riscv/autovec.md
@@ -1078,57 +1078,25 @@
;; - vmadd
;; -------------------------------------------------------------------------
-;; We can't expand FMA for the following reasons:
-;; 1. Before RA, we don't know which multiply-add instruction is the ideal one.
-;; The vmacc is the ideal instruction when operands[3] overlaps operands[0].
-;; The vmadd is the ideal instruction when operands[1|2] overlaps operands[0].
-;; 2. According to vector.md, the multiply-add patterns has 'merge' operand which
-;; is the operands[5]. Since operands[5] should overlap operands[0], this operand
-;; should be allocated the same regno as operands[1|2|3].
-;; 3. The 'merge' operand is always a real merge operand and we don't allow undefined
-;; operand.
-;; 4. The operation of FMA pattern needs VLMAX vsetlvi which needs a VL operand.
-;;
-;; In this situation, we design the codegen of FMA as follows:
-;; 1. clobber a scratch in the expand pattern of FMA.
-;; 2. Let's RA decide which input operand (operands[1|2|3]) overlap operands[0].
-;; 3. Generate instructions (vmacc or vmadd) according to the register allocation
-;; result after reload_completed.
-(define_expand "fma<mode>4"
- [(parallel
- [(set (match_operand:VI 0 "register_operand")
- (plus:VI
- (mult:VI
- (match_operand:VI 1 "register_operand")
- (match_operand:VI 2 "register_operand"))
- (match_operand:VI 3 "register_operand")))
- (clobber (match_dup 4))])]
- "TARGET_VECTOR"
- {
- operands[4] = gen_reg_rtx (Pmode);
- })
-
-(define_insn_and_split "*fma<VI:mode><P:mode>"
- [(set (match_operand:VI 0 "register_operand" "=vr, vr, ?&vr")
+(define_insn_and_split "fma<mode>4"
+ [(set (match_operand:VI 0 "register_operand")
(plus:VI
(mult:VI
- (match_operand:VI 1 "register_operand" " %0, vr, vr")
- (match_operand:VI 2 "register_operand" " vr, vr, vr"))
- (match_operand:VI 3 "register_operand" " vr, 0, vr")))
- (clobber (match_operand:P 4 "register_operand" "=r,r,r"))]
- "TARGET_VECTOR"
+ (match_operand:VI 1 "register_operand")
+ (match_operand:VI 2 "register_operand"))
+ (match_operand:VI 3 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
"#"
- "&& reload_completed"
+ "&& 1"
[(const_int 0)]
{
- riscv_vector::emit_vlmax_vsetvl (<VI:MODE>mode, operands[4]);
- rtx ops[] = {operands[0], operands[1], operands[2], operands[3], operands[0]};
- riscv_vector::emit_vlmax_insn_lra (code_for_pred_mul_plus (<VI:MODE>mode),
- riscv_vector::TERNARY_OP, ops, operands[4]);
+ rtx ops[] = {operands[0], operands[1], operands[2], operands[3],
+ RVV_VUNDEF(<MODE>mode)};
+ riscv_vector::emit_vlmax_insn (code_for_pred_mul_plus (<MODE>mode),
+ riscv_vector::TERNARY_OP, ops);
DONE;
}
- [(set_attr "type" "vimuladd")
- (set_attr "mode" "<VI:MODE>")])
+ [(set_attr "type" "vector")])
;; -------------------------------------------------------------------------
;; ---- [INT] VNMSAC and VNMSUB
@@ -1138,41 +1106,25 @@
;; - vnmsub
;; -------------------------------------------------------------------------
-(define_expand "fnma<mode>4"
- [(parallel
- [(set (match_operand:VI 0 "register_operand")
- (minus:VI
- (match_operand:VI 3 "register_operand")
- (mult:VI
- (match_operand:VI 1 "register_operand")
- (match_operand:VI 2 "register_operand"))))
- (clobber (match_dup 4))])]
- "TARGET_VECTOR"
- {
- operands[4] = gen_reg_rtx (Pmode);
- })
-
-(define_insn_and_split "*fnma<VI:mode><P:mode>"
- [(set (match_operand:VI 0 "register_operand" "=vr, vr, ?&vr")
- (minus:VI
- (match_operand:VI 3 "register_operand" " vr, 0, vr")
- (mult:VI
- (match_operand:VI 1 "register_operand" " %0, vr, vr")
- (match_operand:VI 2 "register_operand" " vr, vr, vr"))))
- (clobber (match_operand:P 4 "register_operand" "=r,r,r"))]
- "TARGET_VECTOR"
+(define_insn_and_split "fnma<mode>4"
+ [(set (match_operand:VI 0 "register_operand")
+ (minus:VI
+ (match_operand:VI 3 "register_operand")
+ (mult:VI
+ (match_operand:VI 1 "register_operand")
+ (match_operand:VI 2 "register_operand"))))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
"#"
- "&& reload_completed"
+ "&& 1"
[(const_int 0)]
{
- riscv_vector::emit_vlmax_vsetvl (<VI:MODE>mode, operands[4]);
- rtx ops[] = {operands[0], operands[1], operands[2], operands[3], operands[0]};
- riscv_vector::emit_vlmax_insn_lra (code_for_pred_minus_mul (<VI:MODE>mode),
- riscv_vector::TERNARY_OP, ops, operands[4]);
+ rtx ops[] = {operands[0], operands[1], operands[2], operands[3],
+ RVV_VUNDEF(<MODE>mode)};
+ riscv_vector::emit_vlmax_insn (code_for_pred_minus_mul (<MODE>mode),
+ riscv_vector::TERNARY_OP, ops);
DONE;
}
- [(set_attr "type" "vimuladd")
- (set_attr "mode" "<VI:MODE>")])
+ [(set_attr "type" "vector")])
;; -------------------------------------------------------------------------
;; ---- [FP] VFMACC and VFMADD
@@ -1182,45 +1134,25 @@
;; - vfmadd
;; -------------------------------------------------------------------------
-(define_expand "fma<mode>4"
- [(parallel
- [(set (match_operand:VF 0 "register_operand")
- (unspec:VF
- [(fma:VF
- (match_operand:VF 1 "register_operand")
- (match_operand:VF 2 "register_operand")
- (match_operand:VF 3 "register_operand"))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
- (clobber (match_dup 4))])]
- "TARGET_VECTOR"
- {
- operands[4] = gen_reg_rtx (Pmode);
- }
- [(set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
-
-(define_insn_and_split "*fma<VF:mode><P:mode>"
- [(set (match_operand:VF 0 "register_operand" "=vr, vr, ?&vr")
- (unspec:VF
- [(fma:VF
- (match_operand:VF 1 "register_operand" " %0, vr, vr")
- (match_operand:VF 2 "register_operand" " vr, vr, vr")
- (match_operand:VF 3 "register_operand" " vr, 0, vr"))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
- (clobber (match_operand:P 4 "register_operand" "=r,r,r"))]
- "TARGET_VECTOR"
+(define_insn_and_split "fma<mode>4"
+ [(set (match_operand:VF 0 "register_operand")
+ (plus:VF
+ (mult:VF
+ (match_operand:VF 1 "register_operand")
+ (match_operand:VF 2 "register_operand"))
+ (match_operand:VF 3 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
"#"
- "&& reload_completed"
+ "&& 1"
[(const_int 0)]
{
- riscv_vector::emit_vlmax_vsetvl (<VF:MODE>mode, operands[4]);
- rtx ops[] = {operands[0], operands[1], operands[2], operands[3], operands[0]};
- riscv_vector::emit_vlmax_insn_lra (code_for_pred_mul (PLUS, <VF:MODE>mode),
- riscv_vector::TERNARY_OP_FRM_DYN, ops, operands[4]);
+ rtx ops[] = {operands[0], operands[1], operands[2], operands[3],
+ RVV_VUNDEF(<MODE>mode)};
+ riscv_vector::emit_vlmax_insn (code_for_pred_mul (PLUS, <MODE>mode),
+ riscv_vector::TERNARY_OP_FRM_DYN, ops);
DONE;
}
- [(set_attr "type" "vfmuladd")
- (set_attr "mode" "<VF:MODE>")
- (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
+ [(set_attr "type" "vector")])
;; -------------------------------------------------------------------------
;; ---- [FP] VFNMSAC and VFNMSUB
@@ -1230,47 +1162,25 @@
;; - vfnmsub
;; -------------------------------------------------------------------------
-(define_expand "fnma<mode>4"
- [(parallel
- [(set (match_operand:VF 0 "register_operand")
- (unspec:VF
- [(fma:VF
- (neg:VF
- (match_operand:VF 1 "register_operand"))
- (match_operand:VF 2 "register_operand")
- (match_operand:VF 3 "register_operand"))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
- (clobber (match_dup 4))])]
- "TARGET_VECTOR"
- {
- operands[4] = gen_reg_rtx (Pmode);
- }
- [(set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
-
-(define_insn_and_split "*fnma<VF:mode><P:mode>"
- [(set (match_operand:VF 0 "register_operand" "=vr, vr, ?&vr")
- (unspec:VF
- [(fma:VF
- (neg:VF
- (match_operand:VF 1 "register_operand" " %0, vr, vr"))
- (match_operand:VF 2 "register_operand" " vr, vr, vr")
- (match_operand:VF 3 "register_operand" " vr, 0, vr"))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
- (clobber (match_operand:P 4 "register_operand" "=r,r,r"))]
- "TARGET_VECTOR"
+(define_insn_and_split "fnma<mode>4"
+ [(set (match_operand:VF 0 "register_operand")
+ (minus:VF
+ (match_operand:VF 3 "register_operand")
+ (mult:VF
+ (match_operand:VF 1 "register_operand")
+ (match_operand:VF 2 "register_operand"))))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
"#"
- "&& reload_completed"
+ "&& 1"
[(const_int 0)]
{
- riscv_vector::emit_vlmax_vsetvl (<VF:MODE>mode, operands[4]);
- rtx ops[] = {operands[0], operands[1], operands[2], operands[3], operands[0]};
- riscv_vector::emit_vlmax_insn_lra (code_for_pred_mul_neg (PLUS, <VF:MODE>mode),
- riscv_vector::TERNARY_OP_FRM_DYN, ops, operands[4]);
+ rtx ops[] = {operands[0], operands[1], operands[2], operands[3],
+ RVV_VUNDEF(<MODE>mode)};
+ riscv_vector::emit_vlmax_insn (code_for_pred_mul_neg (PLUS, <MODE>mode),
+ riscv_vector::TERNARY_OP_FRM_DYN, ops);
DONE;
}
- [(set_attr "type" "vfmuladd")
- (set_attr "mode" "<VF:MODE>")
- (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
+ [(set_attr "type" "vector")])
;; -------------------------------------------------------------------------
;; ---- [FP] VFMSAC and VFMSUB
@@ -1280,47 +1190,25 @@
;; - vfmsub
;; -------------------------------------------------------------------------
-(define_expand "fms<mode>4"
- [(parallel
- [(set (match_operand:VF 0 "register_operand")
- (unspec:VF
- [(fma:VF
- (match_operand:VF 1 "register_operand")
- (match_operand:VF 2 "register_operand")
- (neg:VF
- (match_operand:VF 3 "register_operand")))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
- (clobber (match_dup 4))])]
- "TARGET_VECTOR"
- {
- operands[4] = gen_reg_rtx (Pmode);
- }
- [(set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
-
-(define_insn_and_split "*fms<VF:mode><P:mode>"
- [(set (match_operand:VF 0 "register_operand" "=vr, vr, ?&vr")
- (unspec:VF
- [(fma:VF
- (match_operand:VF 1 "register_operand" " %0, vr, vr")
- (match_operand:VF 2 "register_operand" " vr, vr, vr")
- (neg:VF
- (match_operand:VF 3 "register_operand" " vr, 0, vr")))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
- (clobber (match_operand:P 4 "register_operand" "=r,r,r"))]
- "TARGET_VECTOR"
+(define_insn_and_split "fms<mode>4"
+ [(set (match_operand:VF 0 "register_operand")
+ (minus:VF
+ (mult:VF
+ (match_operand:VF 1 "register_operand")
+ (match_operand:VF 2 "register_operand"))
+ (match_operand:VF 3 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
"#"
- "&& reload_completed"
+ "&& 1"
[(const_int 0)]
{
- riscv_vector::emit_vlmax_vsetvl (<VF:MODE>mode, operands[4]);
- rtx ops[] = {operands[0], operands[1], operands[2], operands[3], operands[0]};
- riscv_vector::emit_vlmax_insn_lra (code_for_pred_mul (MINUS, <VF:MODE>mode),
- riscv_vector::TERNARY_OP_FRM_DYN, ops, operands[4]);
+ rtx ops[] = {operands[0], operands[1], operands[2], operands[3],
+ RVV_VUNDEF(<MODE>mode)};
+ riscv_vector::emit_vlmax_insn (code_for_pred_mul (MINUS, <MODE>mode),
+ riscv_vector::TERNARY_OP_FRM_DYN, ops);
DONE;
}
- [(set_attr "type" "vfmuladd")
- (set_attr "mode" "<VF:MODE>")
- (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
+ [(set_attr "type" "vector")])
;; -------------------------------------------------------------------------
;; ---- [FP] VFNMACC and VFNMADD
@@ -1330,49 +1218,26 @@
;; - vfnmadd
;; -------------------------------------------------------------------------
-(define_expand "fnms<mode>4"
- [(parallel
- [(set (match_operand:VF 0 "register_operand")
- (unspec:VF
- [(fma:VF
- (neg:VF
- (match_operand:VF 1 "register_operand"))
- (match_operand:VF 2 "register_operand")
- (neg:VF
- (match_operand:VF 3 "register_operand")))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
- (clobber (match_dup 4))])]
- "TARGET_VECTOR"
- {
- operands[4] = gen_reg_rtx (Pmode);
- }
- [(set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
-
-(define_insn_and_split "*fnms<VF:mode><P:mode>"
- [(set (match_operand:VF 0 "register_operand" "=vr, vr, ?&vr")
- (unspec:VF
- [(fma:VF
- (neg:VF
- (match_operand:VF 1 "register_operand" " %0, vr, vr"))
- (match_operand:VF 2 "register_operand" " vr, vr, vr")
- (neg:VF
- (match_operand:VF 3 "register_operand" " vr, 0, vr")))
- (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
- (clobber (match_operand:P 4 "register_operand" "=r,r,r"))]
- "TARGET_VECTOR"
+(define_insn_and_split "fnms<mode>4"
+ [(set (match_operand:VF 0 "register_operand")
+ (minus:VF
+ (neg:VF
+ (mult:VF
+ (match_operand:VF 1 "register_operand")
+ (match_operand:VF 2 "register_operand")))
+ (match_operand:VF 3 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
"#"
- "&& reload_completed"
+ "&& 1"
[(const_int 0)]
{
- riscv_vector::emit_vlmax_vsetvl (<VF:MODE>mode, operands[4]);
- rtx ops[] = {operands[0], operands[1], operands[2], operands[3], operands[0]};
- riscv_vector::emit_vlmax_insn_lra (code_for_pred_mul_neg (MINUS, <VF:MODE>mode),
- riscv_vector::TERNARY_OP_FRM_DYN, ops, operands[4]);
+ rtx ops[] = {operands[0], operands[1], operands[2], operands[3],
+ RVV_VUNDEF(<MODE>mode)};
+ riscv_vector::emit_vlmax_insn (code_for_pred_mul_neg (MINUS, <MODE>mode),
+ riscv_vector::TERNARY_OP_FRM_DYN, ops);
DONE;
}
- [(set_attr "type" "vfmuladd")
- (set_attr "mode" "<VF:MODE>")
- (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
+ [(set_attr "type" "vector")])
;; =========================================================================
;; == SELECT_VL
diff --git a/gcc/config/riscv/riscv-protos.h b/gcc/config/riscv/riscv-protos.h
index 44fa36c32ab..46fc8f39974 100644
--- a/gcc/config/riscv/riscv-protos.h
+++ b/gcc/config/riscv/riscv-protos.h
@@ -462,7 +462,7 @@ void expand_select_vl (rtx *);
void expand_load_store (rtx *, bool);
void expand_gather_scatter (rtx *, bool);
void expand_cond_len_ternop (unsigned, rtx *);
-void prepare_ternary_operands (rtx *, bool = false);
+void prepare_ternary_operands (rtx *);
void expand_lanes_load_store (rtx *, bool);
void expand_fold_extract_last (rtx *);
diff --git a/gcc/config/riscv/riscv-v.cc b/gcc/config/riscv/riscv-v.cc
index 668594b65ed..f51986aebde 100644
--- a/gcc/config/riscv/riscv-v.cc
+++ b/gcc/config/riscv/riscv-v.cc
@@ -3253,15 +3253,15 @@ expand_reduction (unsigned unspec, rtx *ops, rtx init, reduction_type type)
/* Prepare ops for ternary operations.
It can be called before or after RA. */
void
-prepare_ternary_operands (rtx *ops, bool split_p)
+prepare_ternary_operands (rtx *ops)
{
machine_mode mode = GET_MODE (ops[0]);
- if (split_p
- || (!rtx_equal_p (ops[2], ops[5])
- && !rtx_equal_p (ops[3], ops[5])
- && !rtx_equal_p (ops[4], ops[5])
- && riscv_get_v_regno_alignment (mode) == 8))
+ if (!rtx_equal_p (ops[5], RVV_VUNDEF (mode))
+ && (VECTOR_MODE_P (GET_MODE (ops[2]))
+ && !rtx_equal_p (ops[2], ops[5]))
+ && !rtx_equal_p (ops[3], ops[5])
+ && !rtx_equal_p (ops[4], ops[5]))
{
/* RA will fail to find vector REG and report ICE, so we pre-merge
the ops for LMUL = 8. */
@@ -3286,6 +3286,8 @@ prepare_ternary_operands (rtx *ops, bool split_p)
/* TODO: ??? Maybe we could support splitting FMA (a, 4, b)
into PLUS (ASHIFT (a, 2), b) according to uarchs. */
}
+ gcc_assert (rtx_equal_p (ops[5], RVV_VUNDEF (mode))
+ || rtx_equal_p (ops[5], ops[2]) || rtx_equal_p (ops[5], ops[4]));
}
/* Expand VEC_MASK_LEN_{LOAD_LANES,STORE_LANES}. */
diff --git a/gcc/config/riscv/vector.md b/gcc/config/riscv/vector.md
index fb0c8abd995..e174322a459 100644
--- a/gcc/config/riscv/vector.md
+++ b/gcc/config/riscv/vector.md
@@ -5002,12 +5002,40 @@
(match_operand:VI 2 "register_operand")
(match_operand:VI 3 "register_operand"))
(match_operand:VI 4 "register_operand"))
- (match_operand:VI 5 "register_operand")))]
+ (match_operand:VI 5 "vector_merge_operand")))]
"TARGET_VECTOR"
{
riscv_vector::prepare_ternary_operands (operands);
})
+(define_insn "*pred_mul_plus<mode>_undef"
+ [(set (match_operand:VI 0 "register_operand" "=vd, vd,?&vd, vr, vr,?&vr")
+ (if_then_else:VI
+ (unspec:<VM>
+ [(match_operand:<VM> 1 "vector_mask_operand" " vm, vm, vm,Wc1,Wc1, Wc1")
+ (match_operand 6 "vector_length_operand" " rK, rK, rK, rK, rK, rK")
+ (match_operand 7 "const_int_operand" " i, i, i, i, i, i")
+ (match_operand 8 "const_int_operand" " i, i, i, i, i, i")
+ (match_operand 9 "const_int_operand" " i, i, i, i, i, i")
+ (reg:SI VL_REGNUM)
+ (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
+ (plus:VI
+ (mult:VI
+ (match_operand:VI 3 "register_operand" " 0, vr, vr, 0, vr, vr")
+ (match_operand:VI 4 "register_operand" " vr, vr, vr, vr, vr, vr"))
+ (match_operand:VI 5 "register_operand" " vr, 0, vr, vr, 0, vr"))
+ (match_operand:VI 2 "vector_undef_operand")))]
+ "TARGET_VECTOR"
+ "@
+ vmadd.vv\t%0,%4,%5%p1
+ vmacc.vv\t%0,%3,%4%p1
+ vmv.v.v\t%0,%4\;vmacc.vv\t%0,%3,%4%p1
+ vmadd.vv\t%0,%4,%5%p1
+ vmacc.vv\t%0,%3,%4%p1
+ vmv.v.v\t%0,%5\;vmacc.vv\t%0,%3,%4%p1"
+ [(set_attr "type" "vimuladd")
+ (set_attr "mode" "<MODE>")])
+
(define_insn "*pred_madd<mode>"
[(set (match_operand:VI 0 "register_operand" "=vd,?&vd, vr,?&vr")
(if_then_else:VI
@@ -5070,35 +5098,6 @@
(set (attr "ma") (symbol_ref "riscv_vector::get_ma(operands[7])"))
(set (attr "avl_type") (symbol_ref "INTVAL (operands[8])"))])
-(define_insn_and_rewrite "*pred_mul_plus<mode>"
- [(set (match_operand:VI 0 "register_operand" "=&vr")
- (if_then_else:VI
- (unspec:<VM>
- [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
- (match_operand 6 "vector_length_operand" " rK")
- (match_operand 7 "const_int_operand" " i")
- (match_operand 8 "const_int_operand" " i")
- (match_operand 9 "const_int_operand" " i")
- (reg:SI VL_REGNUM)
- (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
- (plus:VI
- (mult:VI
- (match_operand:VI 2 "register_operand" " vr")
- (match_operand:VI 3 "register_operand" " vr"))
- (match_operand:VI 4 "register_operand" " vr"))
- (match_operand:VI 5 "register_operand" " vr")))]
- "TARGET_VECTOR
- && !rtx_equal_p (operands[2], operands[5])
- && !rtx_equal_p (operands[3], operands[5])
- && !rtx_equal_p (operands[4], operands[5])"
- "#"
- "&& reload_completed"
- {
- riscv_vector::prepare_ternary_operands (operands, true);
- }
- [(set_attr "type" "vimuladd")
- (set_attr "mode" "<MODE>")])
-
(define_expand "@pred_mul_plus<mode>_scalar"
[(set (match_operand:VI_QHS 0 "register_operand")
(if_then_else:VI_QHS
@@ -5184,35 +5183,6 @@
(set (attr "ma") (symbol_ref "riscv_vector::get_ma(operands[7])"))
(set (attr "avl_type") (symbol_ref "INTVAL (operands[8])"))])
-(define_insn_and_rewrite "*pred_mul_plus<mode>_scalar"
- [(set (match_operand:VI 0 "register_operand" "=&vr")
- (if_then_else:VI
- (unspec:<VM>
- [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
- (match_operand 6 "vector_length_operand" " rK")
- (match_operand 7 "const_int_operand" " i")
- (match_operand 8 "const_int_operand" " i")
- (match_operand 9 "const_int_operand" " i")
- (reg:SI VL_REGNUM)
- (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
- (plus:VI
- (mult:VI
- (vec_duplicate:VI
- (match_operand:<VEL> 2 "register_operand" " r"))
- (match_operand:VI 3 "register_operand" " vr"))
- (match_operand:VI 4 "vector_arith_operand" " vr"))
- (match_operand:VI 5 "register_operand" " vr")))]
- "TARGET_VECTOR
- && !rtx_equal_p (operands[3], operands[5])
- && !rtx_equal_p (operands[4], operands[5])"
- "#"
- "&& reload_completed"
- {
- riscv_vector::prepare_ternary_operands (operands, true);
- }
- [(set_attr "type" "vimuladd")
- (set_attr "mode" "<MODE>")])
-
(define_expand "@pred_mul_plus<mode>_scalar"
[(set (match_operand:VI_D 0 "register_operand")
(if_then_else:VI_D
@@ -5313,36 +5283,6 @@
(set (attr "ma") (symbol_ref "riscv_vector::get_ma(operands[7])"))
(set (attr "avl_type") (symbol_ref "INTVAL (operands[8])"))])
-(define_insn_and_rewrite "*pred_mul_plus<mode>_extended_scalar"
- [(set (match_operand:VI_D 0 "register_operand" "=&vr")
- (if_then_else:VI_D
- (unspec:<VM>
- [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
- (match_operand 6 "vector_length_operand" " rK")
- (match_operand 7 "const_int_operand" " i")
- (match_operand 8 "const_int_operand" " i")
- (match_operand 9 "const_int_operand" " i")
- (reg:SI VL_REGNUM)
- (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
- (plus:VI_D
- (mult:VI_D
- (vec_duplicate:VI_D
- (sign_extend:<VEL>
- (match_operand:<VSUBEL> 2 "register_operand" " r")))
- (match_operand:VI_D 3 "register_operand" " vr"))
- (match_operand:VI_D 4 "register_operand" " vr"))
- (match_operand:VI_D 5 "register_operand" " vr")))]
- "TARGET_VECTOR
- && !rtx_equal_p (operands[3], operands[5])
- && !rtx_equal_p (operands[4], operands[5])"
- "#"
- "&& reload_completed"
- {
- riscv_vector::prepare_ternary_operands (operands, true);
- }
- [(set_attr "type" "vimuladd")
- (set_attr "mode" "<MODE>")])
-
(define_expand "@pred_minus_mul<mode>"
[(set (match_operand:VI 0 "register_operand")
(if_then_else:VI
@@ -5359,12 +5299,40 @@
(mult:VI
(match_operand:VI 2 "register_operand")
(match_operand:VI 3 "register_operand")))
- (match_operand:VI 5 "register_operand")))]
+ (match_operand:VI 5 "vector_merge_operand")))]
"TARGET_VECTOR"
{
riscv_vector::prepare_ternary_operands (operands);
})
+(define_insn "*pred_minus_mul<mode>_undef"
+ [(set (match_operand:VI 0 "register_operand" "=vd, vd,?&vd, vr, vr,?&vr")
+ (if_then_else:VI
+ (unspec:<VM>
+ [(match_operand:<VM> 1 "vector_mask_operand" " vm, vm, vm,Wc1,Wc1, Wc1")
+ (match_operand 6 "vector_length_operand" " rK, rK, rK, rK, rK, rK")
+ (match_operand 7 "const_int_operand" " i, i, i, i, i, i")
+ (match_operand 8 "const_int_operand" " i, i, i, i, i, i")
+ (match_operand 9 "const_int_operand" " i, i, i, i, i, i")
+ (reg:SI VL_REGNUM)
+ (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
+ (minus:VI
+ (match_operand:VI 5 "register_operand" " vr, 0, vr, vr, 0, vr")
+ (mult:VI
+ (match_operand:VI 3 "register_operand" " 0, vr, vr, 0, vr, vr")
+ (match_operand:VI 4 "register_operand" " vr, vr, vr, vr, vr, vr")))
+ (match_operand:VI 2 "vector_undef_operand")))]
+ "TARGET_VECTOR"
+ "@
+ vnmsub.vv\t%0,%4,%5%p1
+ vnmsac.vv\t%0,%3,%4%p1
+ vmv.v.v\t%0,%3\;vnmsub.vv\t%0,%4,%5%p1
+ vnmsub.vv\t%0,%4,%5%p1
+ vnmsac.vv\t%0,%3,%4%p1
+ vmv.v.v\t%0,%3\;vnmsub.vv\t%0,%4,%5%p1"
+ [(set_attr "type" "vimuladd")
+ (set_attr "mode" "<MODE>")])
+
(define_insn "*pred_nmsub<mode>"
[(set (match_operand:VI 0 "register_operand" "=vd,?&vd, vr,?&vr")
(if_then_else:VI
@@ -5427,35 +5395,6 @@
(set (attr "ma") (symbol_ref "riscv_vector::get_ma(operands[7])"))
(set (attr "avl_type") (symbol_ref "INTVAL (operands[8])"))])
-(define_insn_and_rewrite "*pred_minus_mul<mode>"
- [(set (match_operand:VI 0 "register_operand" "=&vr")
- (if_then_else:VI
- (unspec:<VM>
- [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
- (match_operand 6 "vector_length_operand" " rK")
- (match_operand 7 "const_int_operand" " i")
- (match_operand 8 "const_int_operand" " i")
- (match_operand 9 "const_int_operand" " i")
- (reg:SI VL_REGNUM)
- (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
- (minus:VI
- (match_operand:VI 4 "vector_arith_operand" " vr")
- (mult:VI
- (match_operand:VI 2 "register_operand" " vr")
- (match_operand:VI 3 "register_operand" " vr")))
- (match_operand:VI 5 "register_operand" " vr")))]
- "TARGET_VECTOR
- && !rtx_equal_p (operands[2], operands[5])
- && !rtx_equal_p (operands[3], operands[5])
- && !rtx_equal_p (operands[4], operands[5])"
- "#"
- "&& reload_completed"
- {
- riscv_vector::prepare_ternary_operands (operands, true);
- }
- [(set_attr "type" "vimuladd")
- (set_attr "mode" "<MODE>")])
-
(define_expand "@pred_minus_mul<mode>_scalar"
[(set (match_operand:VI_QHS 0 "register_operand")
(if_then_else:VI_QHS
@@ -5541,35 +5480,6 @@
(set (attr "ma") (symbol_ref "riscv_vector::get_ma(operands[7])"))
(set (attr "avl_type") (symbol_ref "INTVAL (operands[8])"))])
-(define_insn_and_rewrite "*pred_minus_mul<mode>_scalar"
- [(set (match_operand:VI 0 "register_operand" "=&vr")
- (if_then_else:VI
- (unspec:<VM>
- [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
- (match_operand 6 "vector_length_operand" " rK")
- (match_operand 7 "const_int_operand" " i")
- (match_operand 8 "const_int_operand" " i")
- (match_operand 9 "const_int_operand" " i")
- (reg:SI VL_REGNUM)
- (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
- (minus:VI
- (match_operand:VI 4 "vector_arith_operand" " vr")
- (mult:VI
- (vec_duplicate:VI
- (match_operand:<VEL> 2 "register_operand" " r"))
- (match_operand:VI 3 "register_operand" " vr")))
- (match_operand:VI 5 "register_operand" " vr")))]
- "TARGET_VECTOR
- && !rtx_equal_p (operands[3], operands[5])
- && !rtx_equal_p (operands[4], operands[5])"
- "#"
- "&& reload_completed"
- {
- riscv_vector::prepare_ternary_operands (operands, true);
- }
- [(set_attr "type" "vimuladd")
- (set_attr "mode" "<MODE>")])
-
(define_expand "@pred_minus_mul<mode>_scalar"
[(set (match_operand:VI_D 0 "register_operand")
(if_then_else:VI_D
@@ -5670,36 +5580,6 @@
(set (attr "ma") (symbol_ref "riscv_vector::get_ma(operands[7])"))
(set (attr "avl_type") (symbol_ref "INTVAL (operands[8])"))])
-(define_insn_and_rewrite "*pred_minus_mul<mode>_extended_scalar"
- [(set (match_operand:VI_D 0 "register_operand" "=&vr")
- (if_then_else:VI_D
- (unspec:<VM>
- [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
- (match_operand 6 "vector_length_operand" " rK")
- (match_operand 7 "const_int_operand" " i")
- (match_operand 8 "const_int_operand" " i")
- (match_operand 9 "const_int_operand" " i")
- (reg:SI VL_REGNUM)
- (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
- (minus:VI_D
- (match_operand:VI_D 4 "vector_arith_operand" " vr")
- (mult:VI_D
- (vec_duplicate:VI_D
- (sign_extend:<VEL>
- (match_operand:<VSUBEL> 2 "register_operand" " r")))
- (match_operand:VI_D 3 "register_operand" " vr")))
- (match_operand:VI_D 5 "register_operand" " vr")))]
- "TARGET_VECTOR
- && !rtx_equal_p (operands[3], operands[5])
- && !rtx_equal_p (operands[4], operands[5])"
- "#"
- "&& reload_completed"
- {
- riscv_vector::prepare_ternary_operands (operands, true);
- }
- [(set_attr "type" "vimuladd")
- (set_attr "mode" "<MODE>")])
-
;; -------------------------------------------------------------------------------
;; ---- Predicated widen integer ternary operations
;; -------------------------------------------------------------------------------
@@ -6276,12 +6156,44 @@
(match_operand:VF 2 "register_operand")
(match_operand:VF 3 "register_operand"))
(match_operand:VF 4 "register_operand"))
- (match_operand:VF 5 "register_operand")))]
+ (match_operand:VF 5 "vector_merge_operand")))]
"TARGET_VECTOR"
{
riscv_vector::prepare_ternary_operands (operands);
})
+(define_insn "*pred_mul_<optab><mode>_undef"
+ [(set (match_operand:VF 0 "register_operand" "=vd,vd,?&vd, vr, vr,?&vr")
+ (if_then_else:VF
+ (unspec:<VM>
+ [(match_operand:<VM> 1 "vector_mask_operand" " vm,vm, vm,Wc1,Wc1, Wc1")
+ (match_operand 6 "vector_length_operand" " rK,rK, rK, rK, rK, rK")
+ (match_operand 7 "const_int_operand" " i, i, i, i, i, i")
+ (match_operand 8 "const_int_operand" " i, i, i, i, i, i")
+ (match_operand 9 "const_int_operand" " i, i, i, i, i, i")
+ (match_operand 10 "const_int_operand" " i, i, i, i, i, i")
+ (reg:SI VL_REGNUM)
+ (reg:SI VTYPE_REGNUM)
+ (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE)
+ (plus_minus:VF
+ (mult:VF
+ (match_operand:VF 3 "register_operand" " 0,vr, vr, 0, vr, vr")
+ (match_operand:VF 4 "register_operand" " vr,vr, vr, vr, vr, vr"))
+ (match_operand:VF 5 "register_operand" " vr, 0, vr, vr, 0, vr"))
+ (match_operand:VF 2 "vector_undef_operand")))]
+ "TARGET_VECTOR"
+ "@
+ vf<madd_msub>.vv\t%0,%4,%5%p1
+ vf<macc_msac>.vv\t%0,%3,%4%p1
+ vmv.v.v\t%0,%3\;vf<madd_msub>.vv\t%0,%4,%5%p1
+ vf<madd_msub>.vv\t%0,%4,%5%p1
+ vf<macc_msac>.vv\t%0,%3,%4%p1
+ vmv.v.v\t%0,%3\;vf<madd_msub>.vv\t%0,%4,%5%p1"
+ [(set_attr "type" "vfmuladd")
+ (set_attr "mode" "<MODE>")
+ (set (attr "frm_mode")
+ (symbol_ref "riscv_vector::get_frm_mode (operands[10])"))])
+
(define_insn "*pred_<madd_msub><mode>"
[(set (match_operand:VF 0 "register_operand" "=vd, ?&vd, vr, ?&vr")
(if_then_else:VF
@@ -6352,39 +6264,6 @@
(set (attr "frm_mode")
(symbol_ref "riscv_vector::get_frm_mode (operands[9])"))])
-(define_insn_and_rewrite "*pred_mul_<optab><mode>"
- [(set (match_operand:VF 0 "register_operand" "=&vr")
- (if_then_else:VF
- (unspec:<VM>
- [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
- (match_operand 6 "vector_length_operand" " rK")
- (match_operand 7 "const_int_operand" " i")
- (match_operand 8 "const_int_operand" " i")
- (match_operand 9 "const_int_operand" " i")
- (match_operand 10 "const_int_operand" " i")
- (reg:SI VL_REGNUM)
- (reg:SI VTYPE_REGNUM)
- (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE)
- (plus_minus:VF
- (mult:VF
- (match_operand:VF 2 "register_operand" " vr")
- (match_operand:VF 3 "register_operand" " vr"))
- (match_operand:VF 4 "register_operand" " vr"))
- (match_operand:VF 5 "register_operand" " vr")))]
- "TARGET_VECTOR
- && !rtx_equal_p (operands[2], operands[5])
- && !rtx_equal_p (operands[3], operands[5])
- && !rtx_equal_p (operands[4], operands[5])"
- "#"
- "&& reload_completed"
- {
- riscv_vector::prepare_ternary_operands (operands, true);
- }
- [(set_attr "type" "vfmuladd")
- (set_attr "mode" "<MODE>")
- (set (attr "frm_mode")
- (symbol_ref "riscv_vector::get_frm_mode (operands[10])"))])
-
(define_expand "@pred_mul_<optab><mode>_scalar"
[(set (match_operand:VF 0 "register_operand")
(if_then_else:VF
@@ -6480,39 +6359,6 @@
(set (attr "frm_mode")
(symbol_ref "riscv_vector::get_frm_mode (operands[9])"))])
-(define_insn_and_rewrite "*pred_mul_<optab><mode>_scalar"
- [(set (match_operand:VF 0 "register_operand" "=&vr")
- (if_then_else:VF
- (unspec:<VM>
- [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
- (match_operand 6 "vector_length_operand" " rK")
- (match_operand 7 "const_int_operand" " i")
- (match_operand 8 "const_int_operand" " i")
- (match_operand 9 "const_int_operand" " i")
- (match_operand 10 "const_int_operand" " i")
- (reg:SI VL_REGNUM)
- (reg:SI VTYPE_REGNUM)
- (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE)
- (plus_minus:VF
- (mult:VF
- (vec_duplicate:VF
- (match_operand:<VEL> 2 "register_operand" " f"))
- (match_operand:VF 3 "register_operand" " vr"))
- (match_operand:VF 4 "vector_arith_operand" " vr"))
- (match_operand:VF 5 "register_operand" " vr")))]
- "TARGET_VECTOR
- && !rtx_equal_p (operands[3], operands[5])
- && !rtx_equal_p (operands[4], operands[5])"
- "#"
- "&& reload_completed"
- {
- riscv_vector::prepare_ternary_operands (operands, true);
- }
- [(set_attr "type" "vfmuladd")
- (set_attr "mode" "<MODE>")
- (set (attr "frm_mode")
- (symbol_ref "riscv_vector::get_frm_mode (operands[10])"))])
-
(define_expand "@pred_mul_neg_<optab><mode>"
[(set (match_operand:VF 0 "register_operand")
(if_then_else:VF
@@ -6532,12 +6378,45 @@
(match_operand:VF 2 "register_operand")
(match_operand:VF 3 "register_operand")))
(match_operand:VF 4 "register_operand"))
- (match_operand:VF 5 "register_operand")))]
+ (match_operand:VF 5 "vector_merge_operand")))]
"TARGET_VECTOR"
{
riscv_vector::prepare_ternary_operands (operands);
})
+(define_insn "*pred_mul_neg_<optab><mode>_undef"
+ [(set (match_operand:VF 0 "register_operand" "=vd,vd,?&vd, vr, vr,?&vr")
+ (if_then_else:VF
+ (unspec:<VM>
+ [(match_operand:<VM> 1 "vector_mask_operand" " vm,vm, vm,Wc1,Wc1, Wc1")
+ (match_operand 6 "vector_length_operand" " rK,rK, rK, rK, rK, rK")
+ (match_operand 7 "const_int_operand" " i, i, i, i, i, i")
+ (match_operand 8 "const_int_operand" " i, i, i, i, i, i")
+ (match_operand 9 "const_int_operand" " i, i, i, i, i, i")
+ (match_operand 10 "const_int_operand" " i, i, i, i, i, i")
+ (reg:SI VL_REGNUM)
+ (reg:SI VTYPE_REGNUM)
+ (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE)
+ (plus_minus:VF
+ (neg:VF
+ (mult:VF
+ (match_operand:VF 3 "register_operand" " 0,vr, vr, 0, vr, vr")
+ (match_operand:VF 4 "register_operand" " vr,vr, vr, vr, vr, vr")))
+ (match_operand:VF 5 "register_operand" " vr, 0, vr, vr, 0, vr"))
+ (match_operand:VF 2 "vector_undef_operand")))]
+ "TARGET_VECTOR"
+ "@
+ vf<nmsub_nmadd>.vv\t%0,%4,%5%p1
+ vf<nmsac_nmacc>.vv\t%0,%3,%4%p1
+ vmv.v.v\t%0,%3\;vf<nmsub_nmadd>.vv\t%0,%4,%5%p1
+ vf<nmsub_nmadd>.vv\t%0,%4,%5%p1
+ vf<nmsac_nmacc>.vv\t%0,%3,%4%p1
+ vmv.v.v\t%0,%3\;vf<nmsub_nmadd>.vv\t%0,%4,%5%p1"
+ [(set_attr "type" "vfmuladd")
+ (set_attr "mode" "<MODE>")
+ (set (attr "frm_mode")
+ (symbol_ref "riscv_vector::get_frm_mode (operands[10])"))])
+
(define_insn "*pred_<nmsub_nmadd><mode>"
[(set (match_operand:VF 0 "register_operand" "=vd, ?&vd, vr, ?&vr")
(if_then_else:VF
@@ -6610,40 +6489,6 @@
(set (attr "frm_mode")
(symbol_ref "riscv_vector::get_frm_mode (operands[9])"))])
-(define_insn_and_rewrite "*pred_mul_neg_<optab><mode>"
- [(set (match_operand:VF 0 "register_operand" "=&vr")
- (if_then_else:VF
- (unspec:<VM>
- [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
- (match_operand 6 "vector_length_operand" " rK")
- (match_operand 7 "const_int_operand" " i")
- (match_operand 8 "const_int_operand" " i")
- (match_operand 9 "const_int_operand" " i")
- (match_operand 10 "const_int_operand" " i")
- (reg:SI VL_REGNUM)
- (reg:SI VTYPE_REGNUM)
- (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE)
- (plus_minus:VF
- (neg:VF
- (mult:VF
- (match_operand:VF 2 "register_operand" " vr")
- (match_operand:VF 3 "register_operand" " vr")))
- (match_operand:VF 4 "vector_arith_operand" " vr"))
- (match_operand:VF 5 "register_operand" " vr")))]
- "TARGET_VECTOR
- && !rtx_equal_p (operands[2], operands[5])
- && !rtx_equal_p (operands[3], operands[5])
- && !rtx_equal_p (operands[4], operands[5])"
- "#"
- "&& reload_completed"
- {
- riscv_vector::prepare_ternary_operands (operands, true);
- }
- [(set_attr "type" "vfmuladd")
- (set_attr "mode" "<MODE>")
- (set (attr "frm_mode")
- (symbol_ref "riscv_vector::get_frm_mode (operands[10])"))])
-
(define_expand "@pred_mul_neg_<optab><mode>_scalar"
[(set (match_operand:VF 0 "register_operand")
(if_then_else:VF
@@ -6742,40 +6587,6 @@
(set (attr "frm_mode")
(symbol_ref "riscv_vector::get_frm_mode (operands[9])"))])
-(define_insn_and_rewrite "*pred_mul_neg_<optab><mode>_scalar"
- [(set (match_operand:VF 0 "register_operand" "=&vr")
- (if_then_else:VF
- (unspec:<VM>
- [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
- (match_operand 6 "vector_length_operand" " rK")
- (match_operand 7 "const_int_operand" " i")
- (match_operand 8 "const_int_operand" " i")
- (match_operand 9 "const_int_operand" " i")
- (match_operand 10 "const_int_operand" " i")
- (reg:SI VL_REGNUM)
- (reg:SI VTYPE_REGNUM)
- (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE)
- (plus_minus:VF
- (neg:VF
- (mult:VF
- (vec_duplicate:VF
- (match_operand:<VEL> 2 "register_operand" " f"))
- (match_operand:VF 3 "register_operand" " vr")))
- (match_operand:VF 4 "vector_arith_operand" " vr"))
- (match_operand:VF 5 "register_operand" " vr")))]
- "TARGET_VECTOR
- && !rtx_equal_p (operands[3], operands[5])
- && !rtx_equal_p (operands[4], operands[5])"
- "#"
- "&& reload_completed"
- {
- riscv_vector::prepare_ternary_operands (operands, true);
- }
- [(set_attr "type" "vfmuladd")
- (set_attr "mode" "<MODE>")
- (set (attr "frm_mode")
- (symbol_ref "riscv_vector::get_frm_mode (operands[10])"))])
-
;; -------------------------------------------------------------------------------
;; ---- Predicated floating-point unary operations
;; -------------------------------------------------------------------------------
--
2.36.3
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] RISC-V: Refactor and cleanup fma patterns
2023-09-19 3:10 ` juzhe.zhong
@ 2023-09-19 3:23 ` Lehua Ding
2023-09-19 20:32 ` Patrick O'Neill
0 siblings, 1 reply; 7+ messages in thread
From: Lehua Ding @ 2023-09-19 3:23 UTC (permalink / raw)
To: juzhe.zhong, gcc-patches; +Cc: kito.cheng, Robin Dapp, palmer, jeffreyalaw
Committed, thanks Juzhe.
On 2023/9/19 11:10, juzhe.zhong@rivai.ai wrote:
> Thanks for the refactoring.
>
> This patch is needed in VLS fma support and undefined value enabling
> support.
>
> LGTM.
>
> ------------------------------------------------------------------------
> juzhe.zhong@rivai.ai
>
> *From:* Lehua Ding <mailto:lehua.ding@rivai.ai>
> *Date:* 2023-09-18 19:37
> *To:* gcc-patches <mailto:gcc-patches@gcc.gnu.org>
> *CC:* juzhe.zhong <mailto:juzhe.zhong@rivai.ai>; kito.cheng
> <mailto:kito.cheng@gmail.com>; rdapp.gcc
> <mailto:rdapp.gcc@gmail.com>; palmer <mailto:palmer@rivosinc.com>;
> jeffreyalaw <mailto:jeffreyalaw@gmail.com>; lehua.ding
> <mailto:lehua.ding@rivai.ai>
> *Subject:* [PATCH] RISC-V: Refactor and cleanup fma patterns
> At present, FMA autovec's patterns do not fully use the
> corresponding pattern
> in vector.md. The previous reason is that the merge operand of
> pattern in
> vector.md cannot be VUNDEF. Now allowing it to be VUNDEF, reunify
> insn used for
> reload pass into vector.md, and the corresponding vlmax pattern in
> autovec.md
> is used for combine. This patch also refactors the corresponding combine
> pattern inside autovec-opt.md and removes the unused ones.
> gcc/ChangeLog:
> * config/riscv/autovec-opt.md (*<optab>_fma<mode>):
> Removed old combine patterns.
> (*single_<optab>mult_plus<mode>): Ditto.
> (*double_<optab>mult_plus<mode>): Ditto.
> (*sign_zero_extend_fma): Ditto.
> (*zero_sign_extend_fma): Ditto.
> (*double_widen_fma<mode>): Ditto.
> (*single_widen_fma<mode>): Ditto.
> (*double_widen_fnma<mode>): Ditto.
> (*single_widen_fnma<mode>): Ditto.
> (*double_widen_fms<mode>): Ditto.
> (*single_widen_fms<mode>): Ditto.
> (*double_widen_fnms<mode>): Ditto.
> (*single_widen_fnms<mode>): Ditto.
> (*reduc_plus_scal_<mode>): Adjust name.
> (*widen_reduc_plus_scal_<mode>): Adjust name.
> (*dual_widen_fma<mode>): New combine pattern.
> (*dual_widen_fmasu<mode>): Ditto.
> (*dual_widen_fmaus<mode>): Ditto.
> (*dual_fma<mode>): Ditto.
> (*single_fma<mode>): Ditto.
> (*dual_fnma<mode>): Ditto.
> (*single_fnma<mode>): Ditto.
> (*dual_fms<mode>): Ditto.
> (*single_fms<mode>): Ditto.
> (*dual_fnms<mode>): Ditto.
> (*single_fnms<mode>): Ditto.
> * config/riscv/autovec.md (fma<mode>4):
> Reafctor fma pattern.
> (*fma<VI:mode><P:mode>): Removed.
> (fnma<mode>4): Reafctor.
> (*fnma<VI:mode><P:mode>): Removed.
> (*fma<VF:mode><P:mode>): Removed.
> (*fnma<VF:mode><P:mode>): Removed.
> (fms<mode>4): Reafctor.
> (*fms<VF:mode><P:mode>): Removed.
> (fnms<mode>4): Reafctor.
> (*fnms<VF:mode><P:mode>): Removed.
> * config/riscv/riscv-protos.h (prepare_ternary_operands):
> Adjust prototype.
> * config/riscv/riscv-v.cc (prepare_ternary_operands): Refactor.
> * config/riscv/vector.md (*pred_mul_plus<mode>_undef): New pattern.
> (*pred_mul_plus<mode>): Removed.
> (*pred_mul_plus<mode>_scalar): Removed.
> (*pred_mul_plus<mode>_extended_scalar): Removed.
> (*pred_minus_mul<mode>_undef): New pattern.
> (*pred_minus_mul<mode>): Removed.
> (*pred_minus_mul<mode>_scalar): Removed.
> (*pred_minus_mul<mode>_extended_scalar): Removed.
> (*pred_mul_<optab><mode>_undef): New pattern.
> (*pred_mul_<optab><mode>): Removed.
> (*pred_mul_<optab><mode>_scalar): Removed.
> (*pred_mul_neg_<optab><mode>_undef): New pattern.
> (*pred_mul_neg_<optab><mode>): Removed.
> (*pred_mul_neg_<optab><mode>_scalar): Removed.
> ---
> gcc/config/riscv/autovec-opt.md | 736 ++++++++++++++------------------
> gcc/config/riscv/autovec.md | 301 ++++---------
> gcc/config/riscv/riscv-protos.h | 2 +-
> gcc/config/riscv/riscv-v.cc | 14 +-
> gcc/config/riscv/vector.md | 439 ++++++-------------
> 5 files changed, 528 insertions(+), 964 deletions(-)
> diff --git a/gcc/config/riscv/autovec-opt.md
> b/gcc/config/riscv/autovec-opt.md
> index df516849527..c94cd0ae087 100644
> --- a/gcc/config/riscv/autovec-opt.md
> +++ b/gcc/config/riscv/autovec-opt.md
> @@ -110,166 +110,6 @@
> [(set_attr "type" "vmalu")
> (set_attr "mode" "<MODE>")])
> -;;
> =========================================================================
> -;; == Widening Ternary arithmetic
> -;;
> =========================================================================
> -
> -;;
> -------------------------------------------------------------------------
> -;; ---- [INT] VWMACC
> -;;
> -------------------------------------------------------------------------
> -;; Includes:
> -;; - vwmacc.vv
> -;; - vwmaccu.vv
> -;;
> -------------------------------------------------------------------------
> -
> -;; Combine ext + ext + fma ===> widen fma.
> -;; Most of circumstantces, LoopVectorizer will generate the
> following IR:
> -;; vect__8.64_40 = (vector([4,4]) int) vect__7.63_41;
> -;; vect__11.68_35 = (vector([4,4]) int) vect__10.67_36;
> -;; vect__13.70_33 = .FMA (vect__11.68_35, vect__8.64_40,
> vect__4.60_45);
> -(define_insn_and_split "*<optab>_fma<mode>"
> - [(set (match_operand:VWEXTI 0 "register_operand")
> - (plus:VWEXTI
> - (mult:VWEXTI
> - (any_extend:VWEXTI
> - (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
> - (any_extend:VWEXTI
> - (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
> - (match_operand:VWEXTI 1 "register_operand")))]
> - "TARGET_VECTOR && can_create_pseudo_p ()"
> - "#"
> - "&& 1"
> - [(const_int 0)]
> - {
> - riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_plus
> (<CODE>, <MODE>mode),
> - riscv_vector::WIDEN_TERNARY_OP, operands);
> - DONE;
> - }
> - [(set_attr "type" "viwmuladd")
> - (set_attr "mode" "<V_DOUBLE_TRUNC>")])
> -
> -;; This helps to match ext + fma.
> -(define_insn_and_split "*single_<optab>mult_plus<mode>"
> - [(set (match_operand:VWEXTI 0 "register_operand")
> - (plus:VWEXTI
> - (mult:VWEXTI
> - (any_extend:VWEXTI
> - (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
> - (match_operand:VWEXTI 3 "register_operand"))
> - (match_operand:VWEXTI 1 "register_operand")))]
> - "TARGET_VECTOR && can_create_pseudo_p ()"
> - "#"
> - "&& 1"
> - [(const_int 0)]
> - {
> - insn_code icode = code_for_pred_vf2 (<CODE>, <MODE>mode);
> - rtx tmp = gen_reg_rtx (<MODE>mode);
> - rtx ext_ops[] = {tmp, operands[2]};
> - riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP,
> ext_ops);
> -
> - rtx dst = expand_ternary_op (<MODE>mode, fma_optab, tmp,
> operands[3],
> - operands[1], operands[0], 0);
> - emit_move_insn (operands[0], dst);
> - DONE;
> - }
> - [(set_attr "type" "viwmuladd")
> - (set_attr "mode" "<V_DOUBLE_TRUNC>")])
> -
> -;; Combine ext + ext + mult + plus ===> widen fma.
> -;; We have some special cases generated by LoopVectorizer:
> -;; vect__8.18_46 = (vector([8,8]) signed short) vect__7.17_47;
> -;; vect__11.22_41 = (vector([8,8]) signed short) vect__10.21_42;
> -;; vect__12.23_40 = vect__11.22_41 * vect__8.18_46;
> -;; vect__14.25_38 = vect__13.24_39 + vect__5.14_51;
> -;; This situation doesn't generate FMA IR.
> -(define_insn_and_split "*double_<optab>mult_plus<mode>"
> - [(set (match_operand:VWEXTI 0 "register_operand")
> - (if_then_else:VWEXTI
> - (unspec:<VM>
> - [(match_operand:<VM> 1 "vector_mask_operand")
> - (match_operand 6 "vector_length_operand")
> - (match_operand 7 "const_int_operand")
> - (match_operand 8 "const_int_operand")
> - (match_operand 9 "const_int_operand")
> - (reg:SI VL_REGNUM)
> - (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
> - (plus:VWEXTI
> - (if_then_else:VWEXTI
> - (unspec:<VM>
> - [(match_dup 1)
> - (match_dup 6)
> - (match_dup 7)
> - (match_dup 8)
> - (match_dup 9)
> - (reg:SI VL_REGNUM)
> - (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
> - (mult:VWEXTI
> - (any_extend:VWEXTI
> - (match_operand:<V_DOUBLE_TRUNC> 4 "register_operand"))
> - (any_extend:VWEXTI
> - (match_operand:<V_DOUBLE_TRUNC> 5 "register_operand")))
> - (match_operand:VWEXTI 2 "vector_undef_operand"))
> - (match_operand:VWEXTI 3 "register_operand"))
> - (match_dup 2)))]
> - "TARGET_VECTOR && can_create_pseudo_p ()"
> - "#"
> - "&& 1"
> - [(const_int 0)]
> - {
> - emit_insn (gen_pred_widen_mul_plus (<CODE>, <MODE>mode,
> operands[0],
> - operands[1], operands[3], operands[4],
> - operands[5], operands[6], operands[7],
> - operands[8], operands[9]));
> - DONE;
> - }
> - [(set_attr "type" "viwmuladd")
> - (set_attr "mode" "<V_DOUBLE_TRUNC>")])
> -
> -;; Combine sign_extend + zero_extend + fma ===> widen fma (su).
> -(define_insn_and_split "*sign_zero_extend_fma"
> - [(set (match_operand:VWEXTI 0 "register_operand")
> - (plus:VWEXTI
> - (mult:VWEXTI
> - (sign_extend:VWEXTI
> - (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
> - (zero_extend:VWEXTI
> - (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
> - (match_operand:VWEXTI 1 "register_operand")))]
> - "TARGET_VECTOR && can_create_pseudo_p ()"
> - "#"
> - "&& 1"
> - [(const_int 0)]
> - {
> - riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_plussu
> (<MODE>mode),
> - riscv_vector::WIDEN_TERNARY_OP, operands);
> - DONE;
> - }
> - [(set_attr "type" "viwmuladd")
> - (set_attr "mode" "<V_DOUBLE_TRUNC>")])
> -
> -;; This helps to match zero_extend + sign_extend + fma.
> -(define_insn_and_split "*zero_sign_extend_fma"
> - [(set (match_operand:VWEXTI 0 "register_operand")
> - (plus:VWEXTI
> - (mult:VWEXTI
> - (zero_extend:VWEXTI
> - (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
> - (sign_extend:VWEXTI
> - (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
> - (match_operand:VWEXTI 1 "register_operand")))]
> - "TARGET_VECTOR && can_create_pseudo_p ()"
> - "#"
> - "&& 1"
> - [(const_int 0)]
> - {
> - rtx ops[] = {operands[0], operands[1], operands[3], operands[2]};
> - riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_plussu
> (<MODE>mode),
> - riscv_vector::WIDEN_TERNARY_OP, ops);
> - DONE;
> - }
> - [(set_attr "type" "viwmuladd")
> - (set_attr "mode" "<V_DOUBLE_TRUNC>")])
> -
> ;;
> -------------------------------------------------------------------------
> ;; ---- [INT] Binary narrow shifts.
> ;;
> -------------------------------------------------------------------------
> @@ -345,269 +185,9 @@
> [(set_attr "type" "vimovvx")
> (set_attr "mode" "<MODE>")])
> -;;
> -------------------------------------------------------------------------
> -;; ---- [FP] VFWMACC
> -;;
> -------------------------------------------------------------------------
> -;; Includes:
> -;; - vfwmacc.vv
> -;;
> -------------------------------------------------------------------------
> -
> -;; Combine ext + ext + fma ===> widen fma.
> -;; Most of circumstantces, LoopVectorizer will generate the
> following IR:
> -;; vect__8.176_40 = (vector([2,2]) double) vect__7.175_41;
> -;; vect__11.180_35 = (vector([2,2]) double) vect__10.179_36;
> -;; vect__13.182_33 = .FMA (vect__11.180_35, vect__8.176_40,
> vect__4.172_45);
> -(define_insn_and_split "*double_widen_fma<mode>"
> - [(set (match_operand:VWEXTF 0 "register_operand")
> - (unspec:VWEXTF
> - [(fma:VWEXTF
> - (float_extend:VWEXTF
> - (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
> - (float_extend:VWEXTF
> - (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand"))
> - (match_operand:VWEXTF 1 "register_operand"))
> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
> - "TARGET_VECTOR && can_create_pseudo_p ()"
> - "#"
> - "&& 1"
> - [(const_int 0)]
> - {
> - riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul (PLUS,
> <MODE>mode),
> - riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, operands);
> - DONE;
> - }
> - [(set_attr "type" "vfwmuladd")
> - (set_attr "mode" "<V_DOUBLE_TRUNC>")
> - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
> -
> -;; This helps to match ext + fma.
> -(define_insn_and_split "*single_widen_fma<mode>"
> - [(set (match_operand:VWEXTF 0 "register_operand")
> - (unspec:VWEXTF
> - [(fma:VWEXTF
> - (float_extend:VWEXTF
> - (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
> - (match_operand:VWEXTF 3 "register_operand")
> - (match_operand:VWEXTF 1 "register_operand"))
> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
> - "TARGET_VECTOR && can_create_pseudo_p ()"
> - "#"
> - "&& 1"
> - [(const_int 0)]
> - {
> - insn_code icode = code_for_pred_extend (<MODE>mode);
> - rtx tmp = gen_reg_rtx (<MODE>mode);
> - rtx ext_ops[] = {tmp, operands[2]};
> - riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP,
> ext_ops);
> -
> - rtx dst = expand_ternary_op (<MODE>mode, fma_optab, tmp,
> operands[3],
> - operands[1], operands[0], 0);
> - emit_move_insn (operands[0], dst);
> - DONE;
> - }
> - [(set_attr "type" "vfwmuladd")
> - (set_attr "mode" "<V_DOUBLE_TRUNC>")
> - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
> -
> -;;
> -------------------------------------------------------------------------
> -;; ---- [FP] VFWNMSAC
> -;;
> -------------------------------------------------------------------------
> -;; Includes:
> -;; - vfwnmsac.vv
> -;;
> -------------------------------------------------------------------------
> -
> -;; Combine ext + ext + fnma ===> widen fnma.
> -;; Most of circumstantces, LoopVectorizer will generate the
> following IR:
> -;; vect__8.176_40 = (vector([2,2]) double) vect__7.175_41;
> -;; vect__11.180_35 = (vector([2,2]) double) vect__10.179_36;
> -;; vect__13.182_33 = .FNMA (vect__11.180_35, vect__8.176_40,
> vect__4.172_45);
> -(define_insn_and_split "*double_widen_fnma<mode>"
> - [(set (match_operand:VWEXTF 0 "register_operand")
> - (unspec:VWEXTF
> - [(fma:VWEXTF
> - (neg:VWEXTF
> - (float_extend:VWEXTF
> - (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand")))
> - (float_extend:VWEXTF
> - (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand"))
> - (match_operand:VWEXTF 1 "register_operand"))
> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
> - "TARGET_VECTOR && can_create_pseudo_p ()"
> - "#"
> - "&& 1"
> - [(const_int 0)]
> - {
> - riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_neg
> (PLUS, <MODE>mode),
> - riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, operands);
> - DONE;
> - }
> - [(set_attr "type" "vfwmuladd")
> - (set_attr "mode" "<V_DOUBLE_TRUNC>")
> - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
> -
> -;; This helps to match ext + fnma.
> -(define_insn_and_split "*single_widen_fnma<mode>"
> - [(set (match_operand:VWEXTF 0 "register_operand")
> - (unspec:VWEXTF
> - [(fma:VWEXTF
> - (neg:VWEXTF
> - (float_extend:VWEXTF
> - (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand")))
> - (match_operand:VWEXTF 3 "register_operand")
> - (match_operand:VWEXTF 1 "register_operand"))
> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
> - "TARGET_VECTOR && can_create_pseudo_p ()"
> - "#"
> - "&& 1"
> - [(const_int 0)]
> - {
> - insn_code icode = code_for_pred_extend (<MODE>mode);
> - rtx tmp = gen_reg_rtx (<MODE>mode);
> - rtx ext_ops[] = {tmp, operands[2]};
> - riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP,
> ext_ops);
> -
> - rtx dst = expand_ternary_op (<MODE>mode, fnma_optab, tmp,
> operands[3],
> - operands[1], operands[0], 0);
> - emit_move_insn (operands[0], dst);
> - DONE;
> - }
> - [(set_attr "type" "vfwmuladd")
> - (set_attr "mode" "<V_DOUBLE_TRUNC>")
> - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
> -
> -;;
> -------------------------------------------------------------------------
> -;; ---- [FP] VFWMSAC
> -;;
> -------------------------------------------------------------------------
> -;; Includes:
> -;; - vfwmsac.vv
> -;;
> -------------------------------------------------------------------------
> -
> -;; Combine ext + ext + fms ===> widen fms.
> -;; Most of circumstantces, LoopVectorizer will generate the
> following IR:
> -;; vect__8.176_40 = (vector([2,2]) double) vect__7.175_41;
> -;; vect__11.180_35 = (vector([2,2]) double) vect__10.179_36;
> -;; vect__13.182_33 = .FMS (vect__11.180_35, vect__8.176_40,
> vect__4.172_45);
> -(define_insn_and_split "*double_widen_fms<mode>"
> - [(set (match_operand:VWEXTF 0 "register_operand")
> - (unspec:VWEXTF
> - [(fma:VWEXTF
> - (float_extend:VWEXTF
> - (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
> - (float_extend:VWEXTF
> - (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand"))
> - (neg:VWEXTF
> - (match_operand:VWEXTF 1 "register_operand")))
> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
> - "TARGET_VECTOR && can_create_pseudo_p ()"
> - "#"
> - "&& 1"
> - [(const_int 0)]
> - {
> - riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul (MINUS,
> <MODE>mode),
> - riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, operands);
> - DONE;
> - }
> - [(set_attr "type" "vfwmuladd")
> - (set_attr "mode" "<V_DOUBLE_TRUNC>")
> - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
> -
> -;; This helps to match ext + fms.
> -(define_insn_and_split "*single_widen_fms<mode>"
> - [(set (match_operand:VWEXTF 0 "register_operand")
> - (unspec:VWEXTF
> - [(fma:VWEXTF
> - (float_extend:VWEXTF
> - (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
> - (match_operand:VWEXTF 3 "register_operand")
> - (neg:VWEXTF
> - (match_operand:VWEXTF 1 "register_operand")))
> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
> - "TARGET_VECTOR && can_create_pseudo_p ()"
> - "#"
> - "&& 1"
> - [(const_int 0)]
> - {
> - insn_code icode = code_for_pred_extend (<MODE>mode);
> - rtx tmp = gen_reg_rtx (<MODE>mode);
> - rtx ext_ops[] = {tmp, operands[2]};
> - riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP,
> ext_ops);
> -
> - rtx dst = expand_ternary_op (<MODE>mode, fms_optab, tmp,
> operands[3],
> - operands[1], operands[0], 0);
> - emit_move_insn (operands[0], dst);
> - DONE;
> - }
> - [(set_attr "type" "vfwmuladd")
> - (set_attr "mode" "<V_DOUBLE_TRUNC>")
> - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
> -
> -;;
> -------------------------------------------------------------------------
> -;; ---- [FP] VFWNMACC
> -;;
> -------------------------------------------------------------------------
> -;; Includes:
> -;; - vfwnmacc.vv
> -;;
> -------------------------------------------------------------------------
> -
> -;; Combine ext + ext + fnms ===> widen fnms.
> -;; Most of circumstantces, LoopVectorizer will generate the
> following IR:
> -;; vect__8.176_40 = (vector([2,2]) double) vect__7.175_41;
> -;; vect__11.180_35 = (vector([2,2]) double) vect__10.179_36;
> -;; vect__13.182_33 = .FNMS (vect__11.180_35, vect__8.176_40,
> vect__4.172_45);
> -(define_insn_and_split "*double_widen_fnms<mode>"
> - [(set (match_operand:VWEXTF 0 "register_operand")
> - (unspec:VWEXTF
> - [(fma:VWEXTF
> - (neg:VWEXTF
> - (float_extend:VWEXTF
> - (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand")))
> - (float_extend:VWEXTF
> - (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand"))
> - (neg:VWEXTF
> - (match_operand:VWEXTF 1 "register_operand")))
> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
> - "TARGET_VECTOR && can_create_pseudo_p ()"
> - "#"
> - "&& 1"
> - [(const_int 0)]
> - {
> - riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_neg
> (MINUS, <MODE>mode),
> - riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, operands);
> - DONE;
> - }
> - [(set_attr "type" "vfwmuladd")
> - (set_attr "mode" "<V_DOUBLE_TRUNC>")
> - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
> -
> -;; This helps to match ext + fnms.
> -(define_insn_and_split "*single_widen_fnms<mode>"
> - [(set (match_operand:VWEXTF 0 "register_operand")
> - (unspec:VWEXTF
> - [(fma:VWEXTF
> - (neg:VWEXTF
> - (float_extend:VWEXTF
> - (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand")))
> - (match_operand:VWEXTF 3 "register_operand")
> - (neg:VWEXTF
> - (match_operand:VWEXTF 1 "register_operand")))
> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
> - "TARGET_VECTOR && can_create_pseudo_p ()"
> - "#"
> - "&& 1"
> - [(const_int 0)]
> - {
> - insn_code icode = code_for_pred_extend (<MODE>mode);
> - rtx tmp = gen_reg_rtx (<MODE>mode);
> - rtx ext_ops[] = {tmp, operands[2]};
> - riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP,
> ext_ops);
> -
> - rtx dst = expand_ternary_op (<MODE>mode, fnms_optab, tmp,
> operands[3],
> - operands[1], operands[0], 0);
> - emit_move_insn (operands[0], dst);
> - DONE;
> - }
> - [(set_attr "type" "vfwmuladd")
> - (set_attr "mode" "<V_DOUBLE_TRUNC>")
> - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
> +;;
> =============================================================================
> +;; All combine patterns for combine pass.
> +;;
> =============================================================================
> ;;
> =============================================================================
> ;; Combine op + vmerge to cond_op
> @@ -1197,7 +777,7 @@
> [(set_attr "type" "vfwmul")])
> ;; Combine extend + vredsum to vwredsum[u]
> -(define_insn_and_split "*reduc_plus_scal_<mode>"
> +(define_insn_and_split "*widen_reduc_plus_scal_<mode>"
> [(set (match_operand:<V_DOUBLE_EXTEND_VEL> 0 "register_operand")
> (unspec:<V_DOUBLE_EXTEND_VEL> [
> (any_extend:<V_DOUBLE_EXTEND>
> @@ -1215,7 +795,7 @@
> [(set_attr "type" "vector")])
> ;; Combine extend + vfredusum to vfwredusum
> -(define_insn_and_split "*reduc_plus_scal_<mode>"
> +(define_insn_and_split "*widen_reduc_plus_scal_<mode>"
> [(set (match_operand:<V_DOUBLE_EXTEND_VEL> 0 "register_operand")
> (unspec:<V_DOUBLE_EXTEND_VEL> [
> (float_extend:<V_DOUBLE_EXTEND>
> @@ -1278,6 +858,312 @@
> }
> [(set_attr "type" "vector")])
> +;;
> =============================================================================
> +;; Combine extend + ternop to widen_ternop
> +;;
> =============================================================================
> +
> +;; Combine ext + fma(vmacc,vmadd) to widen_fma (vwmacc)
> +(define_insn_and_split "*dual_widen_fma<mode>"
> + [(set (match_operand:VWEXTI 0 "register_operand")
> + (plus:VWEXTI
> + (mult:VWEXTI
> + (any_extend:VWEXTI
> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
> + (any_extend:VWEXTI
> + (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
> + (match_operand:VWEXTI 1 "register_operand")))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> + "#"
> + "&& 1"
> + [(const_int 0)]
> + {
> + rtx ops[] = {operands[0], operands[1], operands[2], operands[3]};
> + riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_plus
> (<CODE>, <MODE>mode),
> + riscv_vector::WIDEN_TERNARY_OP, ops);
> + DONE;
> + }
> + [(set_attr "type" "viwmuladd")])
> +
> +;; Combine sign_extend + zero_extend + fma to widen_fma (vwmaccsu)
> +(define_insn_and_split "*dual_widen_fmasu<mode>"
> + [(set (match_operand:VWEXTI 0 "register_operand")
> + (plus:VWEXTI
> + (mult:VWEXTI
> + (sign_extend:VWEXTI
> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
> + (zero_extend:VWEXTI
> + (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
> + (match_operand:VWEXTI 1 "register_operand")))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> + "#"
> + "&& 1"
> + [(const_int 0)]
> + {
> + rtx ops[] = {operands[0], operands[1], operands[2], operands[3]};
> + riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_plussu
> (<MODE>mode),
> + riscv_vector::WIDEN_TERNARY_OP, ops);
> + DONE;
> + }
> + [(set_attr "type" "viwmuladd")])
> +
> +;; Combine zero_extend + sign_extend + fma to widen_fma (vwmaccsu)
> +(define_insn_and_split "*dual_widen_fmaus<mode>"
> + [(set (match_operand:VWEXTI 0 "register_operand")
> + (plus:VWEXTI
> + (mult:VWEXTI
> + (zero_extend:VWEXTI
> + (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand"))
> + (sign_extend:VWEXTI
> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand")))
> + (match_operand:VWEXTI 1 "register_operand")))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> + "#"
> + "&& 1"
> + [(const_int 0)]
> + {
> + rtx ops[] = {operands[0], operands[1], operands[2], operands[3]};
> + riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_plussu
> (<MODE>mode),
> + riscv_vector::WIDEN_TERNARY_OP, ops);
> + DONE;
> + }
> + [(set_attr "type" "viwmuladd")])
> +
> +;; This combine pattern does not correspond to an single instruction.
> +;; This is a temporary pattern produced by a combine pass and if there
> +;; is no further combine into widen pattern, then fall back to extend
> +;; pattern and non-widen fma pattern.
> +(define_insn_and_split "*single_widen_fma<mode>"
> + [(set (match_operand:VWEXTI 0 "register_operand")
> + (plus:VWEXTI
> + (mult:VWEXTI
> + (any_extend:VWEXTI
> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
> + (match_operand:VWEXTI 3 "register_operand"))
> + (match_operand:VWEXTI 1 "register_operand")))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> + "#"
> + "&& 1"
> + [(const_int 0)]
> + {
> + insn_code extend_icode = code_for_pred_vf2 (<CODE>, <MODE>mode);
> + rtx tmp = gen_reg_rtx (<MODE>mode);
> + rtx extend_ops[] = {tmp, operands[2]};
> + riscv_vector::emit_vlmax_insn (extend_icode,
> riscv_vector::UNARY_OP,
> + extend_ops);
> +
> + rtx ops[] = {operands[0], tmp, operands[3], operands[1]};
> + riscv_vector::emit_vlmax_insn (code_for_pred_mul_plus (<MODE>mode),
> + riscv_vector::TERNARY_OP, ops);
> + DONE;
> + }
> + [(set_attr "type" "viwmuladd")])
> +
> +;; Combine extend + fma to widen_fma (vfwmacc)
> +(define_insn_and_split "*dual_fma<mode>"
> + [(set (match_operand:VWEXTF 0 "register_operand")
> + (plus:VWEXTF
> + (mult:VWEXTF
> + (float_extend:VWEXTF
> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
> + (float_extend:VWEXTF
> + (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
> + (match_operand:VWEXTF 1 "register_operand")))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> + "#"
> + "&& 1"
> + [(const_int 0)]
> + {
> + rtx ops[] = {operands[0], operands[1], operands[2], operands[3]};
> + riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul (PLUS,
> <MODE>mode),
> + riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, ops);
> + DONE;
> + }
> + [(set_attr "type" "vfwmuladd")])
> +
> +;; This combine pattern does not correspond to an single instruction.
> +;; This is a temporary pattern produced by a combine pass and if there
> +;; is no further combine into widen pattern, then fall back to extend
> +;; pattern and non-widen fma pattern.
> +(define_insn_and_split "*single_fma<mode>"
> + [(set (match_operand:VWEXTF 0 "register_operand")
> + (plus:VWEXTF
> + (mult:VWEXTF
> + (float_extend:VWEXTF
> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
> + (match_operand:VWEXTF 3 "register_operand"))
> + (match_operand:VWEXTF 1 "register_operand")))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> + "#"
> + "&& 1"
> + [(const_int 0)]
> + {
> + insn_code icode = code_for_pred_extend (<MODE>mode);
> + rtx tmp = gen_reg_rtx (<MODE>mode);
> + rtx ext_ops[] = {tmp, operands[2]};
> + riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP,
> ext_ops);
> +
> + rtx ops[] = {operands[0], tmp, operands[3], operands[1]};
> + riscv_vector::emit_vlmax_insn (code_for_pred_mul (PLUS,
> <MODE>mode),
> + riscv_vector::TERNARY_OP_FRM_DYN, ops);
> + DONE;
> + }
> + [(set_attr "type" "vfwmuladd")])
> +
> +;; Combine extend + fnma to widen_fnma (vfwnmsac)
> +(define_insn_and_split "*dual_fnma<mode>"
> + [(set (match_operand:VWEXTF 0 "register_operand")
> + (minus:VWEXTF
> + (match_operand:VWEXTF 1 "register_operand")
> + (mult:VWEXTF
> + (float_extend:VWEXTF
> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
> + (float_extend:VWEXTF
> + (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> + "#"
> + "&& 1"
> + [(const_int 0)]
> + {
> + rtx ops[] = {operands[0], operands[1], operands[2], operands[3]};
> + riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_neg
> (PLUS, <MODE>mode),
> + riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, ops);
> + DONE;
> + }
> + [(set_attr "type" "vfwmuladd")])
> +
> +;; This combine pattern does not correspond to an single instruction.
> +;; This is a temporary pattern produced by a combine pass and if there
> +;; is no further combine into widen pattern, then fall back to extend
> +;; pattern and non-widen fnma pattern.
> +(define_insn_and_split "*single_fnma<mode>"
> + [(set (match_operand:VWEXTF 0 "register_operand")
> + (minus:VWEXTF
> + (match_operand:VWEXTF 1 "register_operand")
> + (mult:VWEXTF
> + (float_extend:VWEXTF
> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
> + (match_operand:VWEXTF 3 "register_operand"))))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> + "#"
> + "&& 1"
> + [(const_int 0)]
> + {
> + insn_code icode = code_for_pred_extend (<MODE>mode);
> + rtx tmp = gen_reg_rtx (<MODE>mode);
> + rtx ext_ops[] = {tmp, operands[2]};
> + riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP,
> ext_ops);
> +
> + rtx ops[] = {operands[0], tmp, operands[3], operands[1]};
> + riscv_vector::emit_vlmax_insn (code_for_pred_mul_neg (PLUS,
> <MODE>mode),
> + riscv_vector::TERNARY_OP_FRM_DYN, ops);
> + DONE;
> + }
> + [(set_attr "type" "vfwmuladd")])
> +
> +;; Combine extend + fms to widen_fms (vfwmsac)
> +(define_insn_and_split "*dual_fms<mode>"
> + [(set (match_operand:VWEXTF 0 "register_operand")
> + (minus:VWEXTF
> + (mult:VWEXTF
> + (float_extend:VWEXTF
> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
> + (float_extend:VWEXTF
> + (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
> + (match_operand:VWEXTF 1 "register_operand")))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> + "#"
> + "&& 1"
> + [(const_int 0)]
> + {
> + rtx ops[] = {operands[0], operands[1], operands[2], operands[3]};
> + riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul (MINUS,
> <MODE>mode),
> + riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, ops);
> + DONE;
> + }
> + [(set_attr "type" "vfwmuladd")])
> +
> +;; This combine pattern does not correspond to an single instruction.
> +;; This is a temporary pattern produced by a combine pass and if there
> +;; is no further combine into widen pattern, then fall back to extend
> +;; pattern and non-widen fms pattern.
> +(define_insn_and_split "*single_fms<mode>"
> + [(set (match_operand:VWEXTF 0 "register_operand")
> + (minus:VWEXTF
> + (mult:VWEXTF
> + (float_extend:VWEXTF
> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
> + (match_operand:VWEXTF 3 "register_operand"))
> + (match_operand:VWEXTF 1 "register_operand")))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> + "#"
> + "&& 1"
> + [(const_int 0)]
> + {
> + insn_code icode = code_for_pred_extend (<MODE>mode);
> + rtx tmp = gen_reg_rtx (<MODE>mode);
> + rtx ext_ops[] = {tmp, operands[2]};
> + riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP,
> ext_ops);
> +
> + rtx ops[] = {operands[0], tmp, operands[3], operands[1]};
> + riscv_vector::emit_vlmax_insn (code_for_pred_mul (MINUS,
> <MODE>mode),
> + riscv_vector::TERNARY_OP_FRM_DYN, ops);
> + DONE;
> + }
> + [(set_attr "type" "vfwmuladd")])
> +
> +;; Combine extend + fnms to widen_fnms (vfwnmacc)
> +(define_insn_and_split "*dual_fnms<mode>"
> + [(set (match_operand:VWEXTF 0 "register_operand")
> + (minus:VWEXTF
> + (mult:VWEXTF
> + (neg:VWEXTF
> + (float_extend:VWEXTF
> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand")))
> + (float_extend:VWEXTF
> + (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
> + (match_operand:VWEXTF 1 "register_operand")))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> + "#"
> + "&& 1"
> + [(const_int 0)]
> + {
> + rtx ops[] = {operands[0], operands[1], operands[2], operands[3]};
> + riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_neg
> (MINUS, <MODE>mode),
> + riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, ops);
> + DONE;
> + }
> + [(set_attr "type" "vfwmuladd")])
> +
> +;; This combine pattern does not correspond to an single instruction.
> +;; This is a temporary pattern produced by a combine pass and if there
> +;; is no further combine into widen pattern, then fall back to extend
> +;; pattern and non-widen fnms pattern.
> +(define_insn_and_split "*single_fnms<mode>"
> + [(set (match_operand:VWEXTF 0 "register_operand")
> + (minus:VWEXTF
> + (mult:VWEXTF
> + (neg:VWEXTF
> + (match_operand:VWEXTF 3 "register_operand"))
> + (float_extend:VWEXTF
> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand")))
> + (match_operand:VWEXTF 1 "register_operand")))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> + "#"
> + "&& 1"
> + [(const_int 0)]
> + {
> + insn_code icode = code_for_pred_extend (<MODE>mode);
> + rtx tmp = gen_reg_rtx (<MODE>mode);
> + rtx ext_ops[] = {tmp, operands[2]};
> + riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP,
> ext_ops);
> +
> + rtx ops[] = {operands[0], tmp, operands[3], operands[1]};
> + riscv_vector::emit_vlmax_insn (code_for_pred_mul_neg (MINUS,
> <MODE>mode),
> + riscv_vector::TERNARY_OP_FRM_DYN, ops);
> + DONE;
> + }
> + [(set_attr "type" "vfwmuladd")])
> +
> ;;
> =============================================================================
> ;; Misc combine patterns
> ;;
> =============================================================================
> diff --git a/gcc/config/riscv/autovec.md b/gcc/config/riscv/autovec.md
> index 8537b9d41f6..96aaf869f88 100644
> --- a/gcc/config/riscv/autovec.md
> +++ b/gcc/config/riscv/autovec.md
> @@ -1078,57 +1078,25 @@
> ;; - vmadd
> ;;
> -------------------------------------------------------------------------
> -;; We can't expand FMA for the following reasons:
> -;; 1. Before RA, we don't know which multiply-add instruction is
> the ideal one.
> -;; The vmacc is the ideal instruction when operands[3] overlaps
> operands[0].
> -;; The vmadd is the ideal instruction when operands[1|2]
> overlaps operands[0].
> -;; 2. According to vector.md, the multiply-add patterns has 'merge'
> operand which
> -;; is the operands[5]. Since operands[5] should overlap
> operands[0], this operand
> -;; should be allocated the same regno as operands[1|2|3].
> -;; 3. The 'merge' operand is always a real merge operand and we
> don't allow undefined
> -;; operand.
> -;; 4. The operation of FMA pattern needs VLMAX vsetlvi which needs
> a VL operand.
> -;;
> -;; In this situation, we design the codegen of FMA as follows:
> -;; 1. clobber a scratch in the expand pattern of FMA.
> -;; 2. Let's RA decide which input operand (operands[1|2|3]) overlap
> operands[0].
> -;; 3. Generate instructions (vmacc or vmadd) according to the
> register allocation
> -;; result after reload_completed.
> -(define_expand "fma<mode>4"
> - [(parallel
> - [(set (match_operand:VI 0 "register_operand")
> - (plus:VI
> - (mult:VI
> - (match_operand:VI 1 "register_operand")
> - (match_operand:VI 2 "register_operand"))
> - (match_operand:VI 3 "register_operand")))
> - (clobber (match_dup 4))])]
> - "TARGET_VECTOR"
> - {
> - operands[4] = gen_reg_rtx (Pmode);
> - })
> -
> -(define_insn_and_split "*fma<VI:mode><P:mode>"
> - [(set (match_operand:VI 0 "register_operand" "=vr, vr, ?&vr")
> +(define_insn_and_split "fma<mode>4"
> + [(set (match_operand:VI 0 "register_operand")
> (plus:VI
> (mult:VI
> - (match_operand:VI 1 "register_operand" " %0, vr, vr")
> - (match_operand:VI 2 "register_operand" " vr, vr, vr"))
> - (match_operand:VI 3 "register_operand" " vr, 0, vr")))
> - (clobber (match_operand:P 4 "register_operand" "=r,r,r"))]
> - "TARGET_VECTOR"
> + (match_operand:VI 1 "register_operand")
> + (match_operand:VI 2 "register_operand"))
> + (match_operand:VI 3 "register_operand")))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> "#"
> - "&& reload_completed"
> + "&& 1"
> [(const_int 0)]
> {
> - riscv_vector::emit_vlmax_vsetvl (<VI:MODE>mode, operands[4]);
> - rtx ops[] = {operands[0], operands[1], operands[2],
> operands[3], operands[0]};
> - riscv_vector::emit_vlmax_insn_lra (code_for_pred_mul_plus
> (<VI:MODE>mode),
> - riscv_vector::TERNARY_OP, ops, operands[4]);
> + rtx ops[] = {operands[0], operands[1], operands[2], operands[3],
> + RVV_VUNDEF(<MODE>mode)};
> + riscv_vector::emit_vlmax_insn (code_for_pred_mul_plus (<MODE>mode),
> + riscv_vector::TERNARY_OP, ops);
> DONE;
> }
> - [(set_attr "type" "vimuladd")
> - (set_attr "mode" "<VI:MODE>")])
> + [(set_attr "type" "vector")])
> ;;
> -------------------------------------------------------------------------
> ;; ---- [INT] VNMSAC and VNMSUB
> @@ -1138,41 +1106,25 @@
> ;; - vnmsub
> ;;
> -------------------------------------------------------------------------
> -(define_expand "fnma<mode>4"
> - [(parallel
> - [(set (match_operand:VI 0 "register_operand")
> - (minus:VI
> - (match_operand:VI 3 "register_operand")
> - (mult:VI
> - (match_operand:VI 1 "register_operand")
> - (match_operand:VI 2 "register_operand"))))
> - (clobber (match_dup 4))])]
> - "TARGET_VECTOR"
> - {
> - operands[4] = gen_reg_rtx (Pmode);
> - })
> -
> -(define_insn_and_split "*fnma<VI:mode><P:mode>"
> - [(set (match_operand:VI 0 "register_operand" "=vr, vr, ?&vr")
> - (minus:VI
> - (match_operand:VI 3 "register_operand" " vr, 0, vr")
> - (mult:VI
> - (match_operand:VI 1 "register_operand" " %0, vr, vr")
> - (match_operand:VI 2 "register_operand" " vr, vr, vr"))))
> - (clobber (match_operand:P 4 "register_operand" "=r,r,r"))]
> - "TARGET_VECTOR"
> +(define_insn_and_split "fnma<mode>4"
> + [(set (match_operand:VI 0 "register_operand")
> + (minus:VI
> + (match_operand:VI 3 "register_operand")
> + (mult:VI
> + (match_operand:VI 1 "register_operand")
> + (match_operand:VI 2 "register_operand"))))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> "#"
> - "&& reload_completed"
> + "&& 1"
> [(const_int 0)]
> {
> - riscv_vector::emit_vlmax_vsetvl (<VI:MODE>mode, operands[4]);
> - rtx ops[] = {operands[0], operands[1], operands[2],
> operands[3], operands[0]};
> - riscv_vector::emit_vlmax_insn_lra (code_for_pred_minus_mul
> (<VI:MODE>mode),
> - riscv_vector::TERNARY_OP,
> ops, operands[4]);
> + rtx ops[] = {operands[0], operands[1], operands[2], operands[3],
> + RVV_VUNDEF(<MODE>mode)};
> + riscv_vector::emit_vlmax_insn (code_for_pred_minus_mul
> (<MODE>mode),
> + riscv_vector::TERNARY_OP, ops);
> DONE;
> }
> - [(set_attr "type" "vimuladd")
> - (set_attr "mode" "<VI:MODE>")])
> + [(set_attr "type" "vector")])
> ;;
> -------------------------------------------------------------------------
> ;; ---- [FP] VFMACC and VFMADD
> @@ -1182,45 +1134,25 @@
> ;; - vfmadd
> ;;
> -------------------------------------------------------------------------
> -(define_expand "fma<mode>4"
> - [(parallel
> - [(set (match_operand:VF 0 "register_operand")
> - (unspec:VF
> - [(fma:VF
> - (match_operand:VF 1 "register_operand")
> - (match_operand:VF 2 "register_operand")
> - (match_operand:VF 3 "register_operand"))
> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
> - (clobber (match_dup 4))])]
> - "TARGET_VECTOR"
> - {
> - operands[4] = gen_reg_rtx (Pmode);
> - }
> - [(set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
> -
> -(define_insn_and_split "*fma<VF:mode><P:mode>"
> - [(set (match_operand:VF 0 "register_operand" "=vr, vr, ?&vr")
> - (unspec:VF
> - [(fma:VF
> - (match_operand:VF 1 "register_operand" " %0, vr, vr")
> - (match_operand:VF 2 "register_operand" " vr, vr, vr")
> - (match_operand:VF 3 "register_operand" " vr, 0, vr"))
> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
> - (clobber (match_operand:P 4 "register_operand" "=r,r,r"))]
> - "TARGET_VECTOR"
> +(define_insn_and_split "fma<mode>4"
> + [(set (match_operand:VF 0 "register_operand")
> + (plus:VF
> + (mult:VF
> + (match_operand:VF 1 "register_operand")
> + (match_operand:VF 2 "register_operand"))
> + (match_operand:VF 3 "register_operand")))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> "#"
> - "&& reload_completed"
> + "&& 1"
> [(const_int 0)]
> {
> - riscv_vector::emit_vlmax_vsetvl (<VF:MODE>mode, operands[4]);
> - rtx ops[] = {operands[0], operands[1], operands[2],
> operands[3], operands[0]};
> - riscv_vector::emit_vlmax_insn_lra (code_for_pred_mul (PLUS,
> <VF:MODE>mode),
> - riscv_vector::TERNARY_OP_FRM_DYN, ops, operands[4]);
> + rtx ops[] = {operands[0], operands[1], operands[2], operands[3],
> + RVV_VUNDEF(<MODE>mode)};
> + riscv_vector::emit_vlmax_insn (code_for_pred_mul (PLUS,
> <MODE>mode),
> + riscv_vector::TERNARY_OP_FRM_DYN, ops);
> DONE;
> }
> - [(set_attr "type" "vfmuladd")
> - (set_attr "mode" "<VF:MODE>")
> - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
> + [(set_attr "type" "vector")])
> ;;
> -------------------------------------------------------------------------
> ;; ---- [FP] VFNMSAC and VFNMSUB
> @@ -1230,47 +1162,25 @@
> ;; - vfnmsub
> ;;
> -------------------------------------------------------------------------
> -(define_expand "fnma<mode>4"
> - [(parallel
> - [(set (match_operand:VF 0 "register_operand")
> - (unspec:VF
> - [(fma:VF
> - (neg:VF
> - (match_operand:VF 1 "register_operand"))
> - (match_operand:VF 2 "register_operand")
> - (match_operand:VF 3 "register_operand"))
> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
> - (clobber (match_dup 4))])]
> - "TARGET_VECTOR"
> - {
> - operands[4] = gen_reg_rtx (Pmode);
> - }
> - [(set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
> -
> -(define_insn_and_split "*fnma<VF:mode><P:mode>"
> - [(set (match_operand:VF 0 "register_operand" "=vr, vr, ?&vr")
> - (unspec:VF
> - [(fma:VF
> - (neg:VF
> - (match_operand:VF 1 "register_operand" " %0, vr, vr"))
> - (match_operand:VF 2 "register_operand" " vr, vr, vr")
> - (match_operand:VF 3 "register_operand" " vr, 0, vr"))
> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
> - (clobber (match_operand:P 4 "register_operand" "=r,r,r"))]
> - "TARGET_VECTOR"
> +(define_insn_and_split "fnma<mode>4"
> + [(set (match_operand:VF 0 "register_operand")
> + (minus:VF
> + (match_operand:VF 3 "register_operand")
> + (mult:VF
> + (match_operand:VF 1 "register_operand")
> + (match_operand:VF 2 "register_operand"))))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> "#"
> - "&& reload_completed"
> + "&& 1"
> [(const_int 0)]
> {
> - riscv_vector::emit_vlmax_vsetvl (<VF:MODE>mode, operands[4]);
> - rtx ops[] = {operands[0], operands[1], operands[2],
> operands[3], operands[0]};
> - riscv_vector::emit_vlmax_insn_lra (code_for_pred_mul_neg (PLUS,
> <VF:MODE>mode),
> - riscv_vector::TERNARY_OP_FRM_DYN, ops, operands[4]);
> + rtx ops[] = {operands[0], operands[1], operands[2], operands[3],
> + RVV_VUNDEF(<MODE>mode)};
> + riscv_vector::emit_vlmax_insn (code_for_pred_mul_neg (PLUS,
> <MODE>mode),
> + riscv_vector::TERNARY_OP_FRM_DYN, ops);
> DONE;
> }
> - [(set_attr "type" "vfmuladd")
> - (set_attr "mode" "<VF:MODE>")
> - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
> + [(set_attr "type" "vector")])
> ;;
> -------------------------------------------------------------------------
> ;; ---- [FP] VFMSAC and VFMSUB
> @@ -1280,47 +1190,25 @@
> ;; - vfmsub
> ;;
> -------------------------------------------------------------------------
> -(define_expand "fms<mode>4"
> - [(parallel
> - [(set (match_operand:VF 0 "register_operand")
> - (unspec:VF
> - [(fma:VF
> - (match_operand:VF 1 "register_operand")
> - (match_operand:VF 2 "register_operand")
> - (neg:VF
> - (match_operand:VF 3 "register_operand")))
> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
> - (clobber (match_dup 4))])]
> - "TARGET_VECTOR"
> - {
> - operands[4] = gen_reg_rtx (Pmode);
> - }
> - [(set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
> -
> -(define_insn_and_split "*fms<VF:mode><P:mode>"
> - [(set (match_operand:VF 0 "register_operand" "=vr, vr, ?&vr")
> - (unspec:VF
> - [(fma:VF
> - (match_operand:VF 1 "register_operand" " %0, vr, vr")
> - (match_operand:VF 2 "register_operand" " vr, vr, vr")
> - (neg:VF
> - (match_operand:VF 3 "register_operand" " vr, 0, vr")))
> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
> - (clobber (match_operand:P 4 "register_operand" "=r,r,r"))]
> - "TARGET_VECTOR"
> +(define_insn_and_split "fms<mode>4"
> + [(set (match_operand:VF 0 "register_operand")
> + (minus:VF
> + (mult:VF
> + (match_operand:VF 1 "register_operand")
> + (match_operand:VF 2 "register_operand"))
> + (match_operand:VF 3 "register_operand")))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> "#"
> - "&& reload_completed"
> + "&& 1"
> [(const_int 0)]
> {
> - riscv_vector::emit_vlmax_vsetvl (<VF:MODE>mode, operands[4]);
> - rtx ops[] = {operands[0], operands[1], operands[2],
> operands[3], operands[0]};
> - riscv_vector::emit_vlmax_insn_lra (code_for_pred_mul (MINUS,
> <VF:MODE>mode),
> - riscv_vector::TERNARY_OP_FRM_DYN, ops, operands[4]);
> + rtx ops[] = {operands[0], operands[1], operands[2], operands[3],
> + RVV_VUNDEF(<MODE>mode)};
> + riscv_vector::emit_vlmax_insn (code_for_pred_mul (MINUS,
> <MODE>mode),
> + riscv_vector::TERNARY_OP_FRM_DYN, ops);
> DONE;
> }
> - [(set_attr "type" "vfmuladd")
> - (set_attr "mode" "<VF:MODE>")
> - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
> + [(set_attr "type" "vector")])
> ;;
> -------------------------------------------------------------------------
> ;; ---- [FP] VFNMACC and VFNMADD
> @@ -1330,49 +1218,26 @@
> ;; - vfnmadd
> ;;
> -------------------------------------------------------------------------
> -(define_expand "fnms<mode>4"
> - [(parallel
> - [(set (match_operand:VF 0 "register_operand")
> - (unspec:VF
> - [(fma:VF
> - (neg:VF
> - (match_operand:VF 1 "register_operand"))
> - (match_operand:VF 2 "register_operand")
> - (neg:VF
> - (match_operand:VF 3 "register_operand")))
> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
> - (clobber (match_dup 4))])]
> - "TARGET_VECTOR"
> - {
> - operands[4] = gen_reg_rtx (Pmode);
> - }
> - [(set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
> -
> -(define_insn_and_split "*fnms<VF:mode><P:mode>"
> - [(set (match_operand:VF 0 "register_operand" "=vr, vr, ?&vr")
> - (unspec:VF
> - [(fma:VF
> - (neg:VF
> - (match_operand:VF 1 "register_operand" " %0, vr, vr"))
> - (match_operand:VF 2 "register_operand" " vr, vr, vr")
> - (neg:VF
> - (match_operand:VF 3 "register_operand" " vr, 0, vr")))
> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
> - (clobber (match_operand:P 4 "register_operand" "=r,r,r"))]
> - "TARGET_VECTOR"
> +(define_insn_and_split "fnms<mode>4"
> + [(set (match_operand:VF 0 "register_operand")
> + (minus:VF
> + (neg:VF
> + (mult:VF
> + (match_operand:VF 1 "register_operand")
> + (match_operand:VF 2 "register_operand")))
> + (match_operand:VF 3 "register_operand")))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> "#"
> - "&& reload_completed"
> + "&& 1"
> [(const_int 0)]
> {
> - riscv_vector::emit_vlmax_vsetvl (<VF:MODE>mode, operands[4]);
> - rtx ops[] = {operands[0], operands[1], operands[2],
> operands[3], operands[0]};
> - riscv_vector::emit_vlmax_insn_lra (code_for_pred_mul_neg
> (MINUS, <VF:MODE>mode),
> - riscv_vector::TERNARY_OP_FRM_DYN, ops, operands[4]);
> + rtx ops[] = {operands[0], operands[1], operands[2], operands[3],
> + RVV_VUNDEF(<MODE>mode)};
> + riscv_vector::emit_vlmax_insn (code_for_pred_mul_neg (MINUS,
> <MODE>mode),
> + riscv_vector::TERNARY_OP_FRM_DYN, ops);
> DONE;
> }
> - [(set_attr "type" "vfmuladd")
> - (set_attr "mode" "<VF:MODE>")
> - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
> + [(set_attr "type" "vector")])
> ;;
> =========================================================================
> ;; == SELECT_VL
> diff --git a/gcc/config/riscv/riscv-protos.h
> b/gcc/config/riscv/riscv-protos.h
> index 44fa36c32ab..46fc8f39974 100644
> --- a/gcc/config/riscv/riscv-protos.h
> +++ b/gcc/config/riscv/riscv-protos.h
> @@ -462,7 +462,7 @@ void expand_select_vl (rtx *);
> void expand_load_store (rtx *, bool);
> void expand_gather_scatter (rtx *, bool);
> void expand_cond_len_ternop (unsigned, rtx *);
> -void prepare_ternary_operands (rtx *, bool = false);
> +void prepare_ternary_operands (rtx *);
> void expand_lanes_load_store (rtx *, bool);
> void expand_fold_extract_last (rtx *);
> diff --git a/gcc/config/riscv/riscv-v.cc b/gcc/config/riscv/riscv-v.cc
> index 668594b65ed..f51986aebde 100644
> --- a/gcc/config/riscv/riscv-v.cc
> +++ b/gcc/config/riscv/riscv-v.cc
> @@ -3253,15 +3253,15 @@ expand_reduction (unsigned unspec, rtx *ops,
> rtx init, reduction_type type)
> /* Prepare ops for ternary operations.
> It can be called before or after RA. */
> void
> -prepare_ternary_operands (rtx *ops, bool split_p)
> +prepare_ternary_operands (rtx *ops)
> {
> machine_mode mode = GET_MODE (ops[0]);
> - if (split_p
> - || (!rtx_equal_p (ops[2], ops[5])
> - && !rtx_equal_p (ops[3], ops[5])
> - && !rtx_equal_p (ops[4], ops[5])
> - && riscv_get_v_regno_alignment (mode) == 8))
> + if (!rtx_equal_p (ops[5], RVV_VUNDEF (mode))
> + && (VECTOR_MODE_P (GET_MODE (ops[2]))
> + && !rtx_equal_p (ops[2], ops[5]))
> + && !rtx_equal_p (ops[3], ops[5])
> + && !rtx_equal_p (ops[4], ops[5]))
> {
> /* RA will fail to find vector REG and report ICE, so we
> pre-merge
> the ops for LMUL = 8. */
> @@ -3286,6 +3286,8 @@ prepare_ternary_operands (rtx *ops, bool split_p)
> /* TODO: ??? Maybe we could support splitting FMA (a, 4, b)
> into PLUS (ASHIFT (a, 2), b) according to uarchs. */
> }
> + gcc_assert (rtx_equal_p (ops[5], RVV_VUNDEF (mode))
> + || rtx_equal_p (ops[5], ops[2]) || rtx_equal_p (ops[5],
> ops[4]));
> }
> /* Expand VEC_MASK_LEN_{LOAD_LANES,STORE_LANES}. */
> diff --git a/gcc/config/riscv/vector.md b/gcc/config/riscv/vector.md
> index fb0c8abd995..e174322a459 100644
> --- a/gcc/config/riscv/vector.md
> +++ b/gcc/config/riscv/vector.md
> @@ -5002,12 +5002,40 @@
> (match_operand:VI 2 "register_operand")
> (match_operand:VI 3 "register_operand"))
> (match_operand:VI 4 "register_operand"))
> - (match_operand:VI 5 "register_operand")))]
> + (match_operand:VI 5 "vector_merge_operand")))]
> "TARGET_VECTOR"
> {
> riscv_vector::prepare_ternary_operands (operands);
> })
> +(define_insn "*pred_mul_plus<mode>_undef"
> + [(set (match_operand:VI 0 "register_operand" "=vd,
> vd,?&vd, vr, vr,?&vr")
> + (if_then_else:VI
> + (unspec:<VM>
> + [(match_operand:<VM> 1 "vector_mask_operand" " vm, vm,
> vm,Wc1,Wc1, Wc1")
> + (match_operand 6 "vector_length_operand" " rK, rK, rK,
> rK, rK, rK")
> + (match_operand 7 "const_int_operand" " i, i, i,
> i, i, i")
> + (match_operand 8 "const_int_operand" " i, i, i,
> i, i, i")
> + (match_operand 9 "const_int_operand" " i, i, i,
> i, i, i")
> + (reg:SI VL_REGNUM)
> + (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
> + (plus:VI
> + (mult:VI
> + (match_operand:VI 3 "register_operand" " 0, vr, vr,
> 0, vr, vr")
> + (match_operand:VI 4 "register_operand" " vr, vr, vr,
> vr, vr, vr"))
> + (match_operand:VI 5 "register_operand" " vr, 0, vr,
> vr, 0, vr"))
> + (match_operand:VI 2 "vector_undef_operand")))]
> + "TARGET_VECTOR"
> + "@
> + vmadd.vv\t%0,%4,%5%p1
> + vmacc.vv\t%0,%3,%4%p1
> + vmv.v.v\t%0,%4\;vmacc.vv\t%0,%3,%4%p1
> + vmadd.vv\t%0,%4,%5%p1
> + vmacc.vv\t%0,%3,%4%p1
> + vmv.v.v\t%0,%5\;vmacc.vv\t%0,%3,%4%p1"
> + [(set_attr "type" "vimuladd")
> + (set_attr "mode" "<MODE>")])
> +
> (define_insn "*pred_madd<mode>"
> [(set (match_operand:VI 0 "register_operand"
> "=vd,?&vd, vr,?&vr")
> (if_then_else:VI
> @@ -5070,35 +5098,6 @@
> (set (attr "ma") (symbol_ref "riscv_vector::get_ma(operands[7])"))
> (set (attr "avl_type") (symbol_ref "INTVAL (operands[8])"))])
> -(define_insn_and_rewrite "*pred_mul_plus<mode>"
> - [(set (match_operand:VI 0 "register_operand" "=&vr")
> - (if_then_else:VI
> - (unspec:<VM>
> - [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
> - (match_operand 6 "vector_length_operand" " rK")
> - (match_operand 7 "const_int_operand" " i")
> - (match_operand 8 "const_int_operand" " i")
> - (match_operand 9 "const_int_operand" " i")
> - (reg:SI VL_REGNUM)
> - (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
> - (plus:VI
> - (mult:VI
> - (match_operand:VI 2 "register_operand" " vr")
> - (match_operand:VI 3 "register_operand" " vr"))
> - (match_operand:VI 4 "register_operand" " vr"))
> - (match_operand:VI 5 "register_operand" " vr")))]
> - "TARGET_VECTOR
> - && !rtx_equal_p (operands[2], operands[5])
> - && !rtx_equal_p (operands[3], operands[5])
> - && !rtx_equal_p (operands[4], operands[5])"
> - "#"
> - "&& reload_completed"
> - {
> - riscv_vector::prepare_ternary_operands (operands, true);
> - }
> - [(set_attr "type" "vimuladd")
> - (set_attr "mode" "<MODE>")])
> -
> (define_expand "@pred_mul_plus<mode>_scalar"
> [(set (match_operand:VI_QHS 0 "register_operand")
> (if_then_else:VI_QHS
> @@ -5184,35 +5183,6 @@
> (set (attr "ma") (symbol_ref "riscv_vector::get_ma(operands[7])"))
> (set (attr "avl_type") (symbol_ref "INTVAL (operands[8])"))])
> -(define_insn_and_rewrite "*pred_mul_plus<mode>_scalar"
> - [(set (match_operand:VI 0 "register_operand" "=&vr")
> - (if_then_else:VI
> - (unspec:<VM>
> - [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
> - (match_operand 6 "vector_length_operand" " rK")
> - (match_operand 7 "const_int_operand" " i")
> - (match_operand 8 "const_int_operand" " i")
> - (match_operand 9 "const_int_operand" " i")
> - (reg:SI VL_REGNUM)
> - (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
> - (plus:VI
> - (mult:VI
> - (vec_duplicate:VI
> - (match_operand:<VEL> 2 "register_operand" " r"))
> - (match_operand:VI 3 "register_operand" " vr"))
> - (match_operand:VI 4 "vector_arith_operand" " vr"))
> - (match_operand:VI 5 "register_operand" " vr")))]
> - "TARGET_VECTOR
> - && !rtx_equal_p (operands[3], operands[5])
> - && !rtx_equal_p (operands[4], operands[5])"
> - "#"
> - "&& reload_completed"
> - {
> - riscv_vector::prepare_ternary_operands (operands, true);
> - }
> - [(set_attr "type" "vimuladd")
> - (set_attr "mode" "<MODE>")])
> -
> (define_expand "@pred_mul_plus<mode>_scalar"
> [(set (match_operand:VI_D 0 "register_operand")
> (if_then_else:VI_D
> @@ -5313,36 +5283,6 @@
> (set (attr "ma") (symbol_ref "riscv_vector::get_ma(operands[7])"))
> (set (attr "avl_type") (symbol_ref "INTVAL (operands[8])"))])
> -(define_insn_and_rewrite "*pred_mul_plus<mode>_extended_scalar"
> - [(set (match_operand:VI_D 0 "register_operand" "=&vr")
> - (if_then_else:VI_D
> - (unspec:<VM>
> - [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
> - (match_operand 6 "vector_length_operand" " rK")
> - (match_operand 7 "const_int_operand" " i")
> - (match_operand 8 "const_int_operand" " i")
> - (match_operand 9 "const_int_operand" " i")
> - (reg:SI VL_REGNUM)
> - (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
> - (plus:VI_D
> - (mult:VI_D
> - (vec_duplicate:VI_D
> - (sign_extend:<VEL>
> - (match_operand:<VSUBEL> 2 "register_operand" " r")))
> - (match_operand:VI_D 3 "register_operand" " vr"))
> - (match_operand:VI_D 4 "register_operand" " vr"))
> - (match_operand:VI_D 5 "register_operand" " vr")))]
> - "TARGET_VECTOR
> - && !rtx_equal_p (operands[3], operands[5])
> - && !rtx_equal_p (operands[4], operands[5])"
> - "#"
> - "&& reload_completed"
> - {
> - riscv_vector::prepare_ternary_operands (operands, true);
> - }
> - [(set_attr "type" "vimuladd")
> - (set_attr "mode" "<MODE>")])
> -
> (define_expand "@pred_minus_mul<mode>"
> [(set (match_operand:VI 0 "register_operand")
> (if_then_else:VI
> @@ -5359,12 +5299,40 @@
> (mult:VI
> (match_operand:VI 2 "register_operand")
> (match_operand:VI 3 "register_operand")))
> - (match_operand:VI 5 "register_operand")))]
> + (match_operand:VI 5 "vector_merge_operand")))]
> "TARGET_VECTOR"
> {
> riscv_vector::prepare_ternary_operands (operands);
> })
> +(define_insn "*pred_minus_mul<mode>_undef"
> + [(set (match_operand:VI 0 "register_operand" "=vd,
> vd,?&vd, vr, vr,?&vr")
> + (if_then_else:VI
> + (unspec:<VM>
> + [(match_operand:<VM> 1 "vector_mask_operand" " vm, vm,
> vm,Wc1,Wc1, Wc1")
> + (match_operand 6 "vector_length_operand" " rK, rK, rK,
> rK, rK, rK")
> + (match_operand 7 "const_int_operand" " i, i, i,
> i, i, i")
> + (match_operand 8 "const_int_operand" " i, i, i,
> i, i, i")
> + (match_operand 9 "const_int_operand" " i, i, i,
> i, i, i")
> + (reg:SI VL_REGNUM)
> + (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
> + (minus:VI
> + (match_operand:VI 5 "register_operand" " vr, 0,
> vr, vr, 0, vr")
> + (mult:VI
> + (match_operand:VI 3 "register_operand" " 0, vr, vr,
> 0, vr, vr")
> + (match_operand:VI 4 "register_operand" " vr, vr, vr,
> vr, vr, vr")))
> + (match_operand:VI 2 "vector_undef_operand")))]
> + "TARGET_VECTOR"
> + "@
> + vnmsub.vv\t%0,%4,%5%p1
> + vnmsac.vv\t%0,%3,%4%p1
> + vmv.v.v\t%0,%3\;vnmsub.vv\t%0,%4,%5%p1
> + vnmsub.vv\t%0,%4,%5%p1
> + vnmsac.vv\t%0,%3,%4%p1
> + vmv.v.v\t%0,%3\;vnmsub.vv\t%0,%4,%5%p1"
> + [(set_attr "type" "vimuladd")
> + (set_attr "mode" "<MODE>")])
> +
> (define_insn "*pred_nmsub<mode>"
> [(set (match_operand:VI 0 "register_operand"
> "=vd,?&vd, vr,?&vr")
> (if_then_else:VI
> @@ -5427,35 +5395,6 @@
> (set (attr "ma") (symbol_ref "riscv_vector::get_ma(operands[7])"))
> (set (attr "avl_type") (symbol_ref "INTVAL (operands[8])"))])
> -(define_insn_and_rewrite "*pred_minus_mul<mode>"
> - [(set (match_operand:VI 0 "register_operand" "=&vr")
> - (if_then_else:VI
> - (unspec:<VM>
> - [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
> - (match_operand 6 "vector_length_operand" " rK")
> - (match_operand 7 "const_int_operand" " i")
> - (match_operand 8 "const_int_operand" " i")
> - (match_operand 9 "const_int_operand" " i")
> - (reg:SI VL_REGNUM)
> - (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
> - (minus:VI
> - (match_operand:VI 4 "vector_arith_operand" " vr")
> - (mult:VI
> - (match_operand:VI 2 "register_operand" " vr")
> - (match_operand:VI 3 "register_operand" " vr")))
> - (match_operand:VI 5 "register_operand" " vr")))]
> - "TARGET_VECTOR
> - && !rtx_equal_p (operands[2], operands[5])
> - && !rtx_equal_p (operands[3], operands[5])
> - && !rtx_equal_p (operands[4], operands[5])"
> - "#"
> - "&& reload_completed"
> - {
> - riscv_vector::prepare_ternary_operands (operands, true);
> - }
> - [(set_attr "type" "vimuladd")
> - (set_attr "mode" "<MODE>")])
> -
> (define_expand "@pred_minus_mul<mode>_scalar"
> [(set (match_operand:VI_QHS 0 "register_operand")
> (if_then_else:VI_QHS
> @@ -5541,35 +5480,6 @@
> (set (attr "ma") (symbol_ref "riscv_vector::get_ma(operands[7])"))
> (set (attr "avl_type") (symbol_ref "INTVAL (operands[8])"))])
> -(define_insn_and_rewrite "*pred_minus_mul<mode>_scalar"
> - [(set (match_operand:VI 0 "register_operand" "=&vr")
> - (if_then_else:VI
> - (unspec:<VM>
> - [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
> - (match_operand 6 "vector_length_operand" " rK")
> - (match_operand 7 "const_int_operand" " i")
> - (match_operand 8 "const_int_operand" " i")
> - (match_operand 9 "const_int_operand" " i")
> - (reg:SI VL_REGNUM)
> - (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
> - (minus:VI
> - (match_operand:VI 4 "vector_arith_operand" " vr")
> - (mult:VI
> - (vec_duplicate:VI
> - (match_operand:<VEL> 2 "register_operand" " r"))
> - (match_operand:VI 3 "register_operand" " vr")))
> - (match_operand:VI 5 "register_operand" " vr")))]
> - "TARGET_VECTOR
> - && !rtx_equal_p (operands[3], operands[5])
> - && !rtx_equal_p (operands[4], operands[5])"
> - "#"
> - "&& reload_completed"
> - {
> - riscv_vector::prepare_ternary_operands (operands, true);
> - }
> - [(set_attr "type" "vimuladd")
> - (set_attr "mode" "<MODE>")])
> -
> (define_expand "@pred_minus_mul<mode>_scalar"
> [(set (match_operand:VI_D 0 "register_operand")
> (if_then_else:VI_D
> @@ -5670,36 +5580,6 @@
> (set (attr "ma") (symbol_ref "riscv_vector::get_ma(operands[7])"))
> (set (attr "avl_type") (symbol_ref "INTVAL (operands[8])"))])
> -(define_insn_and_rewrite "*pred_minus_mul<mode>_extended_scalar"
> - [(set (match_operand:VI_D 0 "register_operand" "=&vr")
> - (if_then_else:VI_D
> - (unspec:<VM>
> - [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
> - (match_operand 6 "vector_length_operand" " rK")
> - (match_operand 7 "const_int_operand" " i")
> - (match_operand 8 "const_int_operand" " i")
> - (match_operand 9 "const_int_operand" " i")
> - (reg:SI VL_REGNUM)
> - (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
> - (minus:VI_D
> - (match_operand:VI_D 4 "vector_arith_operand" " vr")
> - (mult:VI_D
> - (vec_duplicate:VI_D
> - (sign_extend:<VEL>
> - (match_operand:<VSUBEL> 2 "register_operand" " r")))
> - (match_operand:VI_D 3 "register_operand" " vr")))
> - (match_operand:VI_D 5 "register_operand" " vr")))]
> - "TARGET_VECTOR
> - && !rtx_equal_p (operands[3], operands[5])
> - && !rtx_equal_p (operands[4], operands[5])"
> - "#"
> - "&& reload_completed"
> - {
> - riscv_vector::prepare_ternary_operands (operands, true);
> - }
> - [(set_attr "type" "vimuladd")
> - (set_attr "mode" "<MODE>")])
> -
> ;;
> -------------------------------------------------------------------------------
> ;; ---- Predicated widen integer ternary operations
> ;;
> -------------------------------------------------------------------------------
> @@ -6276,12 +6156,44 @@
> (match_operand:VF 2 "register_operand")
> (match_operand:VF 3 "register_operand"))
> (match_operand:VF 4 "register_operand"))
> - (match_operand:VF 5 "register_operand")))]
> + (match_operand:VF 5 "vector_merge_operand")))]
> "TARGET_VECTOR"
> {
> riscv_vector::prepare_ternary_operands (operands);
> })
> +(define_insn "*pred_mul_<optab><mode>_undef"
> + [(set (match_operand:VF 0 "register_operand"
> "=vd,vd,?&vd, vr, vr,?&vr")
> + (if_then_else:VF
> + (unspec:<VM>
> + [(match_operand:<VM> 1 "vector_mask_operand" " vm,vm,
> vm,Wc1,Wc1, Wc1")
> + (match_operand 6 "vector_length_operand" " rK,rK, rK, rK,
> rK, rK")
> + (match_operand 7 "const_int_operand" " i, i, i,
> i, i, i")
> + (match_operand 8 "const_int_operand" " i, i, i,
> i, i, i")
> + (match_operand 9 "const_int_operand" " i, i, i,
> i, i, i")
> + (match_operand 10 "const_int_operand" " i, i, i,
> i, i, i")
> + (reg:SI VL_REGNUM)
> + (reg:SI VTYPE_REGNUM)
> + (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE)
> + (plus_minus:VF
> + (mult:VF
> + (match_operand:VF 3 "register_operand" " 0,vr, vr, 0,
> vr, vr")
> + (match_operand:VF 4 "register_operand" " vr,vr, vr, vr,
> vr, vr"))
> + (match_operand:VF 5 "register_operand" " vr, 0, vr,
> vr, 0, vr"))
> + (match_operand:VF 2 "vector_undef_operand")))]
> + "TARGET_VECTOR"
> + "@
> + vf<madd_msub>.vv\t%0,%4,%5%p1
> + vf<macc_msac>.vv\t%0,%3,%4%p1
> + vmv.v.v\t%0,%3\;vf<madd_msub>.vv\t%0,%4,%5%p1
> + vf<madd_msub>.vv\t%0,%4,%5%p1
> + vf<macc_msac>.vv\t%0,%3,%4%p1
> + vmv.v.v\t%0,%3\;vf<madd_msub>.vv\t%0,%4,%5%p1"
> + [(set_attr "type" "vfmuladd")
> + (set_attr "mode" "<MODE>")
> + (set (attr "frm_mode")
> + (symbol_ref "riscv_vector::get_frm_mode (operands[10])"))])
> +
> (define_insn "*pred_<madd_msub><mode>"
> [(set (match_operand:VF 0 "register_operand" "=vd,
> ?&vd, vr, ?&vr")
> (if_then_else:VF
> @@ -6352,39 +6264,6 @@
> (set (attr "frm_mode")
> (symbol_ref "riscv_vector::get_frm_mode (operands[9])"))])
> -(define_insn_and_rewrite "*pred_mul_<optab><mode>"
> - [(set (match_operand:VF 0 "register_operand" "=&vr")
> - (if_then_else:VF
> - (unspec:<VM>
> - [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
> - (match_operand 6 "vector_length_operand" " rK")
> - (match_operand 7 "const_int_operand" " i")
> - (match_operand 8 "const_int_operand" " i")
> - (match_operand 9 "const_int_operand" " i")
> - (match_operand 10 "const_int_operand" " i")
> - (reg:SI VL_REGNUM)
> - (reg:SI VTYPE_REGNUM)
> - (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE)
> - (plus_minus:VF
> - (mult:VF
> - (match_operand:VF 2 "register_operand" " vr")
> - (match_operand:VF 3 "register_operand" " vr"))
> - (match_operand:VF 4 "register_operand" " vr"))
> - (match_operand:VF 5 "register_operand" " vr")))]
> - "TARGET_VECTOR
> - && !rtx_equal_p (operands[2], operands[5])
> - && !rtx_equal_p (operands[3], operands[5])
> - && !rtx_equal_p (operands[4], operands[5])"
> - "#"
> - "&& reload_completed"
> - {
> - riscv_vector::prepare_ternary_operands (operands, true);
> - }
> - [(set_attr "type" "vfmuladd")
> - (set_attr "mode" "<MODE>")
> - (set (attr "frm_mode")
> - (symbol_ref "riscv_vector::get_frm_mode (operands[10])"))])
> -
> (define_expand "@pred_mul_<optab><mode>_scalar"
> [(set (match_operand:VF 0 "register_operand")
> (if_then_else:VF
> @@ -6480,39 +6359,6 @@
> (set (attr "frm_mode")
> (symbol_ref "riscv_vector::get_frm_mode (operands[9])"))])
> -(define_insn_and_rewrite "*pred_mul_<optab><mode>_scalar"
> - [(set (match_operand:VF 0 "register_operand" "=&vr")
> - (if_then_else:VF
> - (unspec:<VM>
> - [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
> - (match_operand 6 "vector_length_operand" " rK")
> - (match_operand 7 "const_int_operand" " i")
> - (match_operand 8 "const_int_operand" " i")
> - (match_operand 9 "const_int_operand" " i")
> - (match_operand 10 "const_int_operand" " i")
> - (reg:SI VL_REGNUM)
> - (reg:SI VTYPE_REGNUM)
> - (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE)
> - (plus_minus:VF
> - (mult:VF
> - (vec_duplicate:VF
> - (match_operand:<VEL> 2 "register_operand" " f"))
> - (match_operand:VF 3 "register_operand" " vr"))
> - (match_operand:VF 4 "vector_arith_operand" " vr"))
> - (match_operand:VF 5 "register_operand" " vr")))]
> - "TARGET_VECTOR
> - && !rtx_equal_p (operands[3], operands[5])
> - && !rtx_equal_p (operands[4], operands[5])"
> - "#"
> - "&& reload_completed"
> - {
> - riscv_vector::prepare_ternary_operands (operands, true);
> - }
> - [(set_attr "type" "vfmuladd")
> - (set_attr "mode" "<MODE>")
> - (set (attr "frm_mode")
> - (symbol_ref "riscv_vector::get_frm_mode (operands[10])"))])
> -
> (define_expand "@pred_mul_neg_<optab><mode>"
> [(set (match_operand:VF 0 "register_operand")
> (if_then_else:VF
> @@ -6532,12 +6378,45 @@
> (match_operand:VF 2 "register_operand")
> (match_operand:VF 3 "register_operand")))
> (match_operand:VF 4 "register_operand"))
> - (match_operand:VF 5 "register_operand")))]
> + (match_operand:VF 5 "vector_merge_operand")))]
> "TARGET_VECTOR"
> {
> riscv_vector::prepare_ternary_operands (operands);
> })
> +(define_insn "*pred_mul_neg_<optab><mode>_undef"
> + [(set (match_operand:VF 0 "register_operand"
> "=vd,vd,?&vd, vr, vr,?&vr")
> + (if_then_else:VF
> + (unspec:<VM>
> + [(match_operand:<VM> 1 "vector_mask_operand" " vm,vm,
> vm,Wc1,Wc1, Wc1")
> + (match_operand 6 "vector_length_operand" " rK,rK, rK, rK,
> rK, rK")
> + (match_operand 7 "const_int_operand" " i, i, i,
> i, i, i")
> + (match_operand 8 "const_int_operand" " i, i, i,
> i, i, i")
> + (match_operand 9 "const_int_operand" " i, i, i,
> i, i, i")
> + (match_operand 10 "const_int_operand" " i, i, i,
> i, i, i")
> + (reg:SI VL_REGNUM)
> + (reg:SI VTYPE_REGNUM)
> + (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE)
> + (plus_minus:VF
> + (neg:VF
> + (mult:VF
> + (match_operand:VF 3 "register_operand" " 0,vr, vr, 0,
> vr, vr")
> + (match_operand:VF 4 "register_operand" " vr,vr, vr, vr,
> vr, vr")))
> + (match_operand:VF 5 "register_operand" " vr, 0, vr,
> vr, 0, vr"))
> + (match_operand:VF 2 "vector_undef_operand")))]
> + "TARGET_VECTOR"
> + "@
> + vf<nmsub_nmadd>.vv\t%0,%4,%5%p1
> + vf<nmsac_nmacc>.vv\t%0,%3,%4%p1
> + vmv.v.v\t%0,%3\;vf<nmsub_nmadd>.vv\t%0,%4,%5%p1
> + vf<nmsub_nmadd>.vv\t%0,%4,%5%p1
> + vf<nmsac_nmacc>.vv\t%0,%3,%4%p1
> + vmv.v.v\t%0,%3\;vf<nmsub_nmadd>.vv\t%0,%4,%5%p1"
> + [(set_attr "type" "vfmuladd")
> + (set_attr "mode" "<MODE>")
> + (set (attr "frm_mode")
> + (symbol_ref "riscv_vector::get_frm_mode (operands[10])"))])
> +
> (define_insn "*pred_<nmsub_nmadd><mode>"
> [(set (match_operand:VF 0 "register_operand" "=vd,
> ?&vd, vr, ?&vr")
> (if_then_else:VF
> @@ -6610,40 +6489,6 @@
> (set (attr "frm_mode")
> (symbol_ref "riscv_vector::get_frm_mode (operands[9])"))])
> -(define_insn_and_rewrite "*pred_mul_neg_<optab><mode>"
> - [(set (match_operand:VF 0 "register_operand" "=&vr")
> - (if_then_else:VF
> - (unspec:<VM>
> - [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
> - (match_operand 6 "vector_length_operand" " rK")
> - (match_operand 7 "const_int_operand" " i")
> - (match_operand 8 "const_int_operand" " i")
> - (match_operand 9 "const_int_operand" " i")
> - (match_operand 10 "const_int_operand" " i")
> - (reg:SI VL_REGNUM)
> - (reg:SI VTYPE_REGNUM)
> - (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE)
> - (plus_minus:VF
> - (neg:VF
> - (mult:VF
> - (match_operand:VF 2 "register_operand" " vr")
> - (match_operand:VF 3 "register_operand" " vr")))
> - (match_operand:VF 4 "vector_arith_operand" " vr"))
> - (match_operand:VF 5 "register_operand" " vr")))]
> - "TARGET_VECTOR
> - && !rtx_equal_p (operands[2], operands[5])
> - && !rtx_equal_p (operands[3], operands[5])
> - && !rtx_equal_p (operands[4], operands[5])"
> - "#"
> - "&& reload_completed"
> - {
> - riscv_vector::prepare_ternary_operands (operands, true);
> - }
> - [(set_attr "type" "vfmuladd")
> - (set_attr "mode" "<MODE>")
> - (set (attr "frm_mode")
> - (symbol_ref "riscv_vector::get_frm_mode (operands[10])"))])
> -
> (define_expand "@pred_mul_neg_<optab><mode>_scalar"
> [(set (match_operand:VF 0 "register_operand")
> (if_then_else:VF
> @@ -6742,40 +6587,6 @@
> (set (attr "frm_mode")
> (symbol_ref "riscv_vector::get_frm_mode (operands[9])"))])
> -(define_insn_and_rewrite "*pred_mul_neg_<optab><mode>_scalar"
> - [(set (match_operand:VF 0 "register_operand" "=&vr")
> - (if_then_else:VF
> - (unspec:<VM>
> - [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
> - (match_operand 6 "vector_length_operand" " rK")
> - (match_operand 7 "const_int_operand" " i")
> - (match_operand 8 "const_int_operand" " i")
> - (match_operand 9 "const_int_operand" " i")
> - (match_operand 10 "const_int_operand" " i")
> - (reg:SI VL_REGNUM)
> - (reg:SI VTYPE_REGNUM)
> - (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE)
> - (plus_minus:VF
> - (neg:VF
> - (mult:VF
> - (vec_duplicate:VF
> - (match_operand:<VEL> 2 "register_operand" " f"))
> - (match_operand:VF 3 "register_operand" " vr")))
> - (match_operand:VF 4 "vector_arith_operand" " vr"))
> - (match_operand:VF 5 "register_operand" " vr")))]
> - "TARGET_VECTOR
> - && !rtx_equal_p (operands[3], operands[5])
> - && !rtx_equal_p (operands[4], operands[5])"
> - "#"
> - "&& reload_completed"
> - {
> - riscv_vector::prepare_ternary_operands (operands, true);
> - }
> - [(set_attr "type" "vfmuladd")
> - (set_attr "mode" "<MODE>")
> - (set (attr "frm_mode")
> - (symbol_ref "riscv_vector::get_frm_mode (operands[10])"))])
> -
> ;;
> -------------------------------------------------------------------------------
> ;; ---- Predicated floating-point unary operations
> ;;
> -------------------------------------------------------------------------------
> --
> 2.36.3
>
--
Best,
Lehua
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] RISC-V: Refactor and cleanup fma patterns
2023-09-19 3:23 ` Lehua Ding
@ 2023-09-19 20:32 ` Patrick O'Neill
2023-09-19 21:53 ` Robin Dapp
0 siblings, 1 reply; 7+ messages in thread
From: Patrick O'Neill @ 2023-09-19 20:32 UTC (permalink / raw)
To: Lehua Ding, juzhe.zhong, gcc-patches
Cc: kito.cheng, Robin Dapp, palmer, jeffreyalaw
[-- Attachment #1: Type: text/plain, Size: 98250 bytes --]
Hi Lehua, Juzhe,
I'm seeing 11 new testcases with segmentation faults in rv64gcv GCC
after this patch (95d2ce05fb32e663bc7553438ccee7f4d4e36a35).
Other rv64/rv32 targets are also affected.
This was first seen on the CI, but I bisected it further to this patch
on my local machine. I tested locally using newlib rv64gcv.
The failures per-target can be seen here:
https://github.com/patrick-rivos/riscv-gnu-toolchain/issues/355
AFAICT all new failures listed there are due to this patch.
Example failure:
Executing on host: /scratch/tc-testing/sept-19-bisect-3/build/build-gcc-newlib-stage2/gcc/xgcc -B/scratch/tc-testing/sept-19-bisect-3/build/build-gcc-newlib-stage2/gcc/ /scratch/tc-testing/sept-19-bisect-3/gcc/gcc/testsuite/gcc.target/riscv/rvv/autovec/gather-scatter/gather_load_run-11.c -march=rv64gcv -mabi=lp64d -mcmodel=medlow -fdiagnostics-plain-output -ftree-vectorize -O3 --param riscv-autovec-preference=fixed-vlmax --param riscv-autovec-lmul=m1 -fno-vect-cost-model -ffast-math -lm -o ./gather_load_run-11.exe (timeout = 600)
spawn -ignore SIGHUP /scratch/tc-testing/sept-19-bisect-3/build/build-gcc-newlib-stage2/gcc/xgcc -B/scratch/tc-testing/sept-19-bisect-3/build/build-gcc-newlib-stage2/gcc/ /scratch/tc-testing/sept-19-bisect-3/gcc/gcc/testsuite/gcc.target/riscv/rvv/autovec/gather-scatter/gather_load_run-11.c -march=rv64gcv -mabi=lp64d -mcmodel=medlow -fdiagnostics-plain-output -ftree-vectorize -O3 --param riscv-autovec-preference=fixed-vlmax --param riscv-autovec-lmul=m1 -fno-vect-cost-model -ffast-math -lm -o ./gather_load_run-11.exe
during RTL pass: split1
/scratch/tc-testing/sept-19-bisect-3/gcc/gcc/testsuite/gcc.target/riscv/rvv/autovec/gather-scatter/gather_load_run-11.c: In function 'main':
/scratch/tc-testing/sept-19-bisect-3/gcc/gcc/testsuite/gcc.target/riscv/rvv/autovec/gather-scatter/gather_load_run-11.c:39:1: internal compiler error: Segmentation fault
0x114a023 crash_signal
../../../gcc/gcc/toplev.cc:314
0xd04c0d mark_label_nuses
../../../gcc/gcc/emit-rtl.cc:3755
0xd04c59 mark_label_nuses
../../../gcc/gcc/emit-rtl.cc:3763
0xd04c59 mark_label_nuses
../../../gcc/gcc/emit-rtl.cc:3763
0xd04c59 mark_label_nuses
../../../gcc/gcc/emit-rtl.cc:3763
0xd092ae try_split(rtx_def*, rtx_insn*, int)
../../../gcc/gcc/emit-rtl.cc:3952
0x10a54a5 split_insn
../../../gcc/gcc/recog.cc:3385
0x10aa0b7 split_all_insns()
../../../gcc/gcc/recog.cc:3489
0x10aa17c execute
../../../gcc/gcc/recog.cc:4413
Please submit a full bug report, with preprocessed source (by using -freport-bug).
Please include the complete backtrace with any bug report.
See<https://gcc.gnu.org/bugs/> for instructions.
compiler exited with status 1
FAIL: gcc.target/riscv/rvv/autovec/gather-scatter/gather_load_run-11.c (internal compiler error: Segmentation fault)
All the failures I've seen appear to have the same backtrace, so all
these issues likely have the same root cause.
Thanks,
Patrick
On 9/18/23 20:23, Lehua Ding wrote:
> Committed, thanks Juzhe.
>
> On 2023/9/19 11:10, juzhe.zhong@rivai.ai wrote:
>> Thanks for the refactoring.
>>
>> This patch is needed in VLS fma support and undefined value enabling
>> support.
>>
>> LGTM.
>>
>> ------------------------------------------------------------------------
>> juzhe.zhong@rivai.ai
>>
>> *From:* Lehua Ding <mailto:lehua.ding@rivai.ai>
>> *Date:* 2023-09-18 19:37
>> *To:* gcc-patches <mailto:gcc-patches@gcc.gnu.org>
>> *CC:* juzhe.zhong <mailto:juzhe.zhong@rivai.ai>; kito.cheng
>> <mailto:kito.cheng@gmail.com>; rdapp.gcc
>> <mailto:rdapp.gcc@gmail.com>; palmer <mailto:palmer@rivosinc.com>;
>> jeffreyalaw <mailto:jeffreyalaw@gmail.com>; lehua.ding
>> <mailto:lehua.ding@rivai.ai>
>> *Subject:* [PATCH] RISC-V: Refactor and cleanup fma patterns
>> At present, FMA autovec's patterns do not fully use the
>> corresponding pattern
>> in vector.md. The previous reason is that the merge operand of
>> pattern in
>> vector.md cannot be VUNDEF. Now allowing it to be VUNDEF, reunify
>> insn used for
>> reload pass into vector.md, and the corresponding vlmax pattern in
>> autovec.md
>> is used for combine. This patch also refactors the corresponding
>> combine
>> pattern inside autovec-opt.md and removes the unused ones.
>> gcc/ChangeLog:
>> * config/riscv/autovec-opt.md (*<optab>_fma<mode>):
>> Removed old combine patterns.
>> (*single_<optab>mult_plus<mode>): Ditto.
>> (*double_<optab>mult_plus<mode>): Ditto.
>> (*sign_zero_extend_fma): Ditto.
>> (*zero_sign_extend_fma): Ditto.
>> (*double_widen_fma<mode>): Ditto.
>> (*single_widen_fma<mode>): Ditto.
>> (*double_widen_fnma<mode>): Ditto.
>> (*single_widen_fnma<mode>): Ditto.
>> (*double_widen_fms<mode>): Ditto.
>> (*single_widen_fms<mode>): Ditto.
>> (*double_widen_fnms<mode>): Ditto.
>> (*single_widen_fnms<mode>): Ditto.
>> (*reduc_plus_scal_<mode>): Adjust name.
>> (*widen_reduc_plus_scal_<mode>): Adjust name.
>> (*dual_widen_fma<mode>): New combine pattern.
>> (*dual_widen_fmasu<mode>): Ditto.
>> (*dual_widen_fmaus<mode>): Ditto.
>> (*dual_fma<mode>): Ditto.
>> (*single_fma<mode>): Ditto.
>> (*dual_fnma<mode>): Ditto.
>> (*single_fnma<mode>): Ditto.
>> (*dual_fms<mode>): Ditto.
>> (*single_fms<mode>): Ditto.
>> (*dual_fnms<mode>): Ditto.
>> (*single_fnms<mode>): Ditto.
>> * config/riscv/autovec.md (fma<mode>4):
>> Reafctor fma pattern.
>> (*fma<VI:mode><P:mode>): Removed.
>> (fnma<mode>4): Reafctor.
>> (*fnma<VI:mode><P:mode>): Removed.
>> (*fma<VF:mode><P:mode>): Removed.
>> (*fnma<VF:mode><P:mode>): Removed.
>> (fms<mode>4): Reafctor.
>> (*fms<VF:mode><P:mode>): Removed.
>> (fnms<mode>4): Reafctor.
>> (*fnms<VF:mode><P:mode>): Removed.
>> * config/riscv/riscv-protos.h (prepare_ternary_operands):
>> Adjust prototype.
>> * config/riscv/riscv-v.cc (prepare_ternary_operands): Refactor.
>> * config/riscv/vector.md (*pred_mul_plus<mode>_undef): New pattern.
>> (*pred_mul_plus<mode>): Removed.
>> (*pred_mul_plus<mode>_scalar): Removed.
>> (*pred_mul_plus<mode>_extended_scalar): Removed.
>> (*pred_minus_mul<mode>_undef): New pattern.
>> (*pred_minus_mul<mode>): Removed.
>> (*pred_minus_mul<mode>_scalar): Removed.
>> (*pred_minus_mul<mode>_extended_scalar): Removed.
>> (*pred_mul_<optab><mode>_undef): New pattern.
>> (*pred_mul_<optab><mode>): Removed.
>> (*pred_mul_<optab><mode>_scalar): Removed.
>> (*pred_mul_neg_<optab><mode>_undef): New pattern.
>> (*pred_mul_neg_<optab><mode>): Removed.
>> (*pred_mul_neg_<optab><mode>_scalar): Removed.
>> ---
>> gcc/config/riscv/autovec-opt.md | 736
>> ++++++++++++++------------------
>> gcc/config/riscv/autovec.md | 301 ++++---------
>> gcc/config/riscv/riscv-protos.h | 2 +-
>> gcc/config/riscv/riscv-v.cc | 14 +-
>> gcc/config/riscv/vector.md | 439 ++++++-------------
>> 5 files changed, 528 insertions(+), 964 deletions(-)
>> diff --git a/gcc/config/riscv/autovec-opt.md
>> b/gcc/config/riscv/autovec-opt.md
>> index df516849527..c94cd0ae087 100644
>> --- a/gcc/config/riscv/autovec-opt.md
>> +++ b/gcc/config/riscv/autovec-opt.md
>> @@ -110,166 +110,6 @@
>> [(set_attr "type" "vmalu")
>> (set_attr "mode" "<MODE>")])
>> -;;
>> =========================================================================
>> -;; == Widening Ternary arithmetic
>> -;;
>> =========================================================================
>> -
>> -;;
>> -------------------------------------------------------------------------
>> -;; ---- [INT] VWMACC
>> -;;
>> -------------------------------------------------------------------------
>> -;; Includes:
>> -;; - vwmacc.vv
>> -;; - vwmaccu.vv
>> -;;
>> -------------------------------------------------------------------------
>> -
>> -;; Combine ext + ext + fma ===> widen fma.
>> -;; Most of circumstantces, LoopVectorizer will generate the
>> following IR:
>> -;; vect__8.64_40 = (vector([4,4]) int) vect__7.63_41;
>> -;; vect__11.68_35 = (vector([4,4]) int) vect__10.67_36;
>> -;; vect__13.70_33 = .FMA (vect__11.68_35, vect__8.64_40,
>> vect__4.60_45);
>> -(define_insn_and_split "*<optab>_fma<mode>"
>> - [(set (match_operand:VWEXTI 0 "register_operand")
>> - (plus:VWEXTI
>> - (mult:VWEXTI
>> - (any_extend:VWEXTI
>> - (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
>> - (any_extend:VWEXTI
>> - (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
>> - (match_operand:VWEXTI 1 "register_operand")))]
>> - "TARGET_VECTOR && can_create_pseudo_p ()"
>> - "#"
>> - "&& 1"
>> - [(const_int 0)]
>> - {
>> - riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_plus
>> (<CODE>, <MODE>mode),
>> - riscv_vector::WIDEN_TERNARY_OP, operands);
>> - DONE;
>> - }
>> - [(set_attr "type" "viwmuladd")
>> - (set_attr "mode" "<V_DOUBLE_TRUNC>")])
>> -
>> -;; This helps to match ext + fma.
>> -(define_insn_and_split "*single_<optab>mult_plus<mode>"
>> - [(set (match_operand:VWEXTI 0 "register_operand")
>> - (plus:VWEXTI
>> - (mult:VWEXTI
>> - (any_extend:VWEXTI
>> - (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
>> - (match_operand:VWEXTI 3 "register_operand"))
>> - (match_operand:VWEXTI 1 "register_operand")))]
>> - "TARGET_VECTOR && can_create_pseudo_p ()"
>> - "#"
>> - "&& 1"
>> - [(const_int 0)]
>> - {
>> - insn_code icode = code_for_pred_vf2 (<CODE>, <MODE>mode);
>> - rtx tmp = gen_reg_rtx (<MODE>mode);
>> - rtx ext_ops[] = {tmp, operands[2]};
>> - riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP,
>> ext_ops);
>> -
>> - rtx dst = expand_ternary_op (<MODE>mode, fma_optab, tmp,
>> operands[3],
>> - operands[1], operands[0], 0);
>> - emit_move_insn (operands[0], dst);
>> - DONE;
>> - }
>> - [(set_attr "type" "viwmuladd")
>> - (set_attr "mode" "<V_DOUBLE_TRUNC>")])
>> -
>> -;; Combine ext + ext + mult + plus ===> widen fma.
>> -;; We have some special cases generated by LoopVectorizer:
>> -;; vect__8.18_46 = (vector([8,8]) signed short) vect__7.17_47;
>> -;; vect__11.22_41 = (vector([8,8]) signed short) vect__10.21_42;
>> -;; vect__12.23_40 = vect__11.22_41 * vect__8.18_46;
>> -;; vect__14.25_38 = vect__13.24_39 + vect__5.14_51;
>> -;; This situation doesn't generate FMA IR.
>> -(define_insn_and_split "*double_<optab>mult_plus<mode>"
>> - [(set (match_operand:VWEXTI 0 "register_operand")
>> - (if_then_else:VWEXTI
>> - (unspec:<VM>
>> - [(match_operand:<VM> 1 "vector_mask_operand")
>> - (match_operand 6 "vector_length_operand")
>> - (match_operand 7 "const_int_operand")
>> - (match_operand 8 "const_int_operand")
>> - (match_operand 9 "const_int_operand")
>> - (reg:SI VL_REGNUM)
>> - (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
>> - (plus:VWEXTI
>> - (if_then_else:VWEXTI
>> - (unspec:<VM>
>> - [(match_dup 1)
>> - (match_dup 6)
>> - (match_dup 7)
>> - (match_dup 8)
>> - (match_dup 9)
>> - (reg:SI VL_REGNUM)
>> - (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
>> - (mult:VWEXTI
>> - (any_extend:VWEXTI
>> - (match_operand:<V_DOUBLE_TRUNC> 4 "register_operand"))
>> - (any_extend:VWEXTI
>> - (match_operand:<V_DOUBLE_TRUNC> 5 "register_operand")))
>> - (match_operand:VWEXTI 2 "vector_undef_operand"))
>> - (match_operand:VWEXTI 3 "register_operand"))
>> - (match_dup 2)))]
>> - "TARGET_VECTOR && can_create_pseudo_p ()"
>> - "#"
>> - "&& 1"
>> - [(const_int 0)]
>> - {
>> - emit_insn (gen_pred_widen_mul_plus (<CODE>, <MODE>mode,
>> operands[0],
>> - operands[1], operands[3], operands[4],
>> - operands[5], operands[6], operands[7],
>> - operands[8], operands[9]));
>> - DONE;
>> - }
>> - [(set_attr "type" "viwmuladd")
>> - (set_attr "mode" "<V_DOUBLE_TRUNC>")])
>> -
>> -;; Combine sign_extend + zero_extend + fma ===> widen fma (su).
>> -(define_insn_and_split "*sign_zero_extend_fma"
>> - [(set (match_operand:VWEXTI 0 "register_operand")
>> - (plus:VWEXTI
>> - (mult:VWEXTI
>> - (sign_extend:VWEXTI
>> - (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
>> - (zero_extend:VWEXTI
>> - (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
>> - (match_operand:VWEXTI 1 "register_operand")))]
>> - "TARGET_VECTOR && can_create_pseudo_p ()"
>> - "#"
>> - "&& 1"
>> - [(const_int 0)]
>> - {
>> - riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_plussu
>> (<MODE>mode),
>> - riscv_vector::WIDEN_TERNARY_OP, operands);
>> - DONE;
>> - }
>> - [(set_attr "type" "viwmuladd")
>> - (set_attr "mode" "<V_DOUBLE_TRUNC>")])
>> -
>> -;; This helps to match zero_extend + sign_extend + fma.
>> -(define_insn_and_split "*zero_sign_extend_fma"
>> - [(set (match_operand:VWEXTI 0 "register_operand")
>> - (plus:VWEXTI
>> - (mult:VWEXTI
>> - (zero_extend:VWEXTI
>> - (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
>> - (sign_extend:VWEXTI
>> - (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
>> - (match_operand:VWEXTI 1 "register_operand")))]
>> - "TARGET_VECTOR && can_create_pseudo_p ()"
>> - "#"
>> - "&& 1"
>> - [(const_int 0)]
>> - {
>> - rtx ops[] = {operands[0], operands[1], operands[3],
>> operands[2]};
>> - riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_plussu
>> (<MODE>mode),
>> - riscv_vector::WIDEN_TERNARY_OP, ops);
>> - DONE;
>> - }
>> - [(set_attr "type" "viwmuladd")
>> - (set_attr "mode" "<V_DOUBLE_TRUNC>")])
>> -
>> ;;
>> -------------------------------------------------------------------------
>> ;; ---- [INT] Binary narrow shifts.
>> ;;
>> -------------------------------------------------------------------------
>> @@ -345,269 +185,9 @@
>> [(set_attr "type" "vimovvx")
>> (set_attr "mode" "<MODE>")])
>> -;;
>> -------------------------------------------------------------------------
>> -;; ---- [FP] VFWMACC
>> -;;
>> -------------------------------------------------------------------------
>> -;; Includes:
>> -;; - vfwmacc.vv
>> -;;
>> -------------------------------------------------------------------------
>> -
>> -;; Combine ext + ext + fma ===> widen fma.
>> -;; Most of circumstantces, LoopVectorizer will generate the
>> following IR:
>> -;; vect__8.176_40 = (vector([2,2]) double) vect__7.175_41;
>> -;; vect__11.180_35 = (vector([2,2]) double) vect__10.179_36;
>> -;; vect__13.182_33 = .FMA (vect__11.180_35, vect__8.176_40,
>> vect__4.172_45);
>> -(define_insn_and_split "*double_widen_fma<mode>"
>> - [(set (match_operand:VWEXTF 0 "register_operand")
>> - (unspec:VWEXTF
>> - [(fma:VWEXTF
>> - (float_extend:VWEXTF
>> - (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
>> - (float_extend:VWEXTF
>> - (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand"))
>> - (match_operand:VWEXTF 1 "register_operand"))
>> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
>> - "TARGET_VECTOR && can_create_pseudo_p ()"
>> - "#"
>> - "&& 1"
>> - [(const_int 0)]
>> - {
>> - riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul (PLUS,
>> <MODE>mode),
>> - riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, operands);
>> - DONE;
>> - }
>> - [(set_attr "type" "vfwmuladd")
>> - (set_attr "mode" "<V_DOUBLE_TRUNC>")
>> - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
>> -
>> -;; This helps to match ext + fma.
>> -(define_insn_and_split "*single_widen_fma<mode>"
>> - [(set (match_operand:VWEXTF 0 "register_operand")
>> - (unspec:VWEXTF
>> - [(fma:VWEXTF
>> - (float_extend:VWEXTF
>> - (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
>> - (match_operand:VWEXTF 3 "register_operand")
>> - (match_operand:VWEXTF 1 "register_operand"))
>> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
>> - "TARGET_VECTOR && can_create_pseudo_p ()"
>> - "#"
>> - "&& 1"
>> - [(const_int 0)]
>> - {
>> - insn_code icode = code_for_pred_extend (<MODE>mode);
>> - rtx tmp = gen_reg_rtx (<MODE>mode);
>> - rtx ext_ops[] = {tmp, operands[2]};
>> - riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP,
>> ext_ops);
>> -
>> - rtx dst = expand_ternary_op (<MODE>mode, fma_optab, tmp,
>> operands[3],
>> - operands[1], operands[0], 0);
>> - emit_move_insn (operands[0], dst);
>> - DONE;
>> - }
>> - [(set_attr "type" "vfwmuladd")
>> - (set_attr "mode" "<V_DOUBLE_TRUNC>")
>> - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
>> -
>> -;;
>> -------------------------------------------------------------------------
>> -;; ---- [FP] VFWNMSAC
>> -;;
>> -------------------------------------------------------------------------
>> -;; Includes:
>> -;; - vfwnmsac.vv
>> -;;
>> -------------------------------------------------------------------------
>> -
>> -;; Combine ext + ext + fnma ===> widen fnma.
>> -;; Most of circumstantces, LoopVectorizer will generate the
>> following IR:
>> -;; vect__8.176_40 = (vector([2,2]) double) vect__7.175_41;
>> -;; vect__11.180_35 = (vector([2,2]) double) vect__10.179_36;
>> -;; vect__13.182_33 = .FNMA (vect__11.180_35, vect__8.176_40,
>> vect__4.172_45);
>> -(define_insn_and_split "*double_widen_fnma<mode>"
>> - [(set (match_operand:VWEXTF 0 "register_operand")
>> - (unspec:VWEXTF
>> - [(fma:VWEXTF
>> - (neg:VWEXTF
>> - (float_extend:VWEXTF
>> - (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand")))
>> - (float_extend:VWEXTF
>> - (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand"))
>> - (match_operand:VWEXTF 1 "register_operand"))
>> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
>> - "TARGET_VECTOR && can_create_pseudo_p ()"
>> - "#"
>> - "&& 1"
>> - [(const_int 0)]
>> - {
>> - riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_neg
>> (PLUS, <MODE>mode),
>> - riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, operands);
>> - DONE;
>> - }
>> - [(set_attr "type" "vfwmuladd")
>> - (set_attr "mode" "<V_DOUBLE_TRUNC>")
>> - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
>> -
>> -;; This helps to match ext + fnma.
>> -(define_insn_and_split "*single_widen_fnma<mode>"
>> - [(set (match_operand:VWEXTF 0 "register_operand")
>> - (unspec:VWEXTF
>> - [(fma:VWEXTF
>> - (neg:VWEXTF
>> - (float_extend:VWEXTF
>> - (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand")))
>> - (match_operand:VWEXTF 3 "register_operand")
>> - (match_operand:VWEXTF 1 "register_operand"))
>> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
>> - "TARGET_VECTOR && can_create_pseudo_p ()"
>> - "#"
>> - "&& 1"
>> - [(const_int 0)]
>> - {
>> - insn_code icode = code_for_pred_extend (<MODE>mode);
>> - rtx tmp = gen_reg_rtx (<MODE>mode);
>> - rtx ext_ops[] = {tmp, operands[2]};
>> - riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP,
>> ext_ops);
>> -
>> - rtx dst = expand_ternary_op (<MODE>mode, fnma_optab, tmp,
>> operands[3],
>> - operands[1], operands[0], 0);
>> - emit_move_insn (operands[0], dst);
>> - DONE;
>> - }
>> - [(set_attr "type" "vfwmuladd")
>> - (set_attr "mode" "<V_DOUBLE_TRUNC>")
>> - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
>> -
>> -;;
>> -------------------------------------------------------------------------
>> -;; ---- [FP] VFWMSAC
>> -;;
>> -------------------------------------------------------------------------
>> -;; Includes:
>> -;; - vfwmsac.vv
>> -;;
>> -------------------------------------------------------------------------
>> -
>> -;; Combine ext + ext + fms ===> widen fms.
>> -;; Most of circumstantces, LoopVectorizer will generate the
>> following IR:
>> -;; vect__8.176_40 = (vector([2,2]) double) vect__7.175_41;
>> -;; vect__11.180_35 = (vector([2,2]) double) vect__10.179_36;
>> -;; vect__13.182_33 = .FMS (vect__11.180_35, vect__8.176_40,
>> vect__4.172_45);
>> -(define_insn_and_split "*double_widen_fms<mode>"
>> - [(set (match_operand:VWEXTF 0 "register_operand")
>> - (unspec:VWEXTF
>> - [(fma:VWEXTF
>> - (float_extend:VWEXTF
>> - (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
>> - (float_extend:VWEXTF
>> - (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand"))
>> - (neg:VWEXTF
>> - (match_operand:VWEXTF 1 "register_operand")))
>> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
>> - "TARGET_VECTOR && can_create_pseudo_p ()"
>> - "#"
>> - "&& 1"
>> - [(const_int 0)]
>> - {
>> - riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul (MINUS,
>> <MODE>mode),
>> - riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, operands);
>> - DONE;
>> - }
>> - [(set_attr "type" "vfwmuladd")
>> - (set_attr "mode" "<V_DOUBLE_TRUNC>")
>> - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
>> -
>> -;; This helps to match ext + fms.
>> -(define_insn_and_split "*single_widen_fms<mode>"
>> - [(set (match_operand:VWEXTF 0 "register_operand")
>> - (unspec:VWEXTF
>> - [(fma:VWEXTF
>> - (float_extend:VWEXTF
>> - (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
>> - (match_operand:VWEXTF 3 "register_operand")
>> - (neg:VWEXTF
>> - (match_operand:VWEXTF 1 "register_operand")))
>> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
>> - "TARGET_VECTOR && can_create_pseudo_p ()"
>> - "#"
>> - "&& 1"
>> - [(const_int 0)]
>> - {
>> - insn_code icode = code_for_pred_extend (<MODE>mode);
>> - rtx tmp = gen_reg_rtx (<MODE>mode);
>> - rtx ext_ops[] = {tmp, operands[2]};
>> - riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP,
>> ext_ops);
>> -
>> - rtx dst = expand_ternary_op (<MODE>mode, fms_optab, tmp,
>> operands[3],
>> - operands[1], operands[0], 0);
>> - emit_move_insn (operands[0], dst);
>> - DONE;
>> - }
>> - [(set_attr "type" "vfwmuladd")
>> - (set_attr "mode" "<V_DOUBLE_TRUNC>")
>> - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
>> -
>> -;;
>> -------------------------------------------------------------------------
>> -;; ---- [FP] VFWNMACC
>> -;;
>> -------------------------------------------------------------------------
>> -;; Includes:
>> -;; - vfwnmacc.vv
>> -;;
>> -------------------------------------------------------------------------
>> -
>> -;; Combine ext + ext + fnms ===> widen fnms.
>> -;; Most of circumstantces, LoopVectorizer will generate the
>> following IR:
>> -;; vect__8.176_40 = (vector([2,2]) double) vect__7.175_41;
>> -;; vect__11.180_35 = (vector([2,2]) double) vect__10.179_36;
>> -;; vect__13.182_33 = .FNMS (vect__11.180_35, vect__8.176_40,
>> vect__4.172_45);
>> -(define_insn_and_split "*double_widen_fnms<mode>"
>> - [(set (match_operand:VWEXTF 0 "register_operand")
>> - (unspec:VWEXTF
>> - [(fma:VWEXTF
>> - (neg:VWEXTF
>> - (float_extend:VWEXTF
>> - (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand")))
>> - (float_extend:VWEXTF
>> - (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand"))
>> - (neg:VWEXTF
>> - (match_operand:VWEXTF 1 "register_operand")))
>> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
>> - "TARGET_VECTOR && can_create_pseudo_p ()"
>> - "#"
>> - "&& 1"
>> - [(const_int 0)]
>> - {
>> - riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_neg
>> (MINUS, <MODE>mode),
>> - riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, operands);
>> - DONE;
>> - }
>> - [(set_attr "type" "vfwmuladd")
>> - (set_attr "mode" "<V_DOUBLE_TRUNC>")
>> - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
>> -
>> -;; This helps to match ext + fnms.
>> -(define_insn_and_split "*single_widen_fnms<mode>"
>> - [(set (match_operand:VWEXTF 0 "register_operand")
>> - (unspec:VWEXTF
>> - [(fma:VWEXTF
>> - (neg:VWEXTF
>> - (float_extend:VWEXTF
>> - (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand")))
>> - (match_operand:VWEXTF 3 "register_operand")
>> - (neg:VWEXTF
>> - (match_operand:VWEXTF 1 "register_operand")))
>> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))]
>> - "TARGET_VECTOR && can_create_pseudo_p ()"
>> - "#"
>> - "&& 1"
>> - [(const_int 0)]
>> - {
>> - insn_code icode = code_for_pred_extend (<MODE>mode);
>> - rtx tmp = gen_reg_rtx (<MODE>mode);
>> - rtx ext_ops[] = {tmp, operands[2]};
>> - riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP,
>> ext_ops);
>> -
>> - rtx dst = expand_ternary_op (<MODE>mode, fnms_optab, tmp,
>> operands[3],
>> - operands[1], operands[0], 0);
>> - emit_move_insn (operands[0], dst);
>> - DONE;
>> - }
>> - [(set_attr "type" "vfwmuladd")
>> - (set_attr "mode" "<V_DOUBLE_TRUNC>")
>> - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
>> +;;
>> =============================================================================
>> +;; All combine patterns for combine pass.
>> +;;
>> =============================================================================
>> ;;
>> =============================================================================
>> ;; Combine op + vmerge to cond_op
>> @@ -1197,7 +777,7 @@
>> [(set_attr "type" "vfwmul")])
>> ;; Combine extend + vredsum to vwredsum[u]
>> -(define_insn_and_split "*reduc_plus_scal_<mode>"
>> +(define_insn_and_split "*widen_reduc_plus_scal_<mode>"
>> [(set (match_operand:<V_DOUBLE_EXTEND_VEL> 0 "register_operand")
>> (unspec:<V_DOUBLE_EXTEND_VEL> [
>> (any_extend:<V_DOUBLE_EXTEND>
>> @@ -1215,7 +795,7 @@
>> [(set_attr "type" "vector")])
>> ;; Combine extend + vfredusum to vfwredusum
>> -(define_insn_and_split "*reduc_plus_scal_<mode>"
>> +(define_insn_and_split "*widen_reduc_plus_scal_<mode>"
>> [(set (match_operand:<V_DOUBLE_EXTEND_VEL> 0 "register_operand")
>> (unspec:<V_DOUBLE_EXTEND_VEL> [
>> (float_extend:<V_DOUBLE_EXTEND>
>> @@ -1278,6 +858,312 @@
>> }
>> [(set_attr "type" "vector")])
>> +;;
>> =============================================================================
>> +;; Combine extend + ternop to widen_ternop
>> +;;
>> =============================================================================
>> +
>> +;; Combine ext + fma(vmacc,vmadd) to widen_fma (vwmacc)
>> +(define_insn_and_split "*dual_widen_fma<mode>"
>> + [(set (match_operand:VWEXTI 0 "register_operand")
>> + (plus:VWEXTI
>> + (mult:VWEXTI
>> + (any_extend:VWEXTI
>> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
>> + (any_extend:VWEXTI
>> + (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
>> + (match_operand:VWEXTI 1 "register_operand")))]
>> + "TARGET_VECTOR && can_create_pseudo_p ()"
>> + "#"
>> + "&& 1"
>> + [(const_int 0)]
>> + {
>> + rtx ops[] = {operands[0], operands[1], operands[2],
>> operands[3]};
>> + riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_plus
>> (<CODE>, <MODE>mode),
>> + riscv_vector::WIDEN_TERNARY_OP, ops);
>> + DONE;
>> + }
>> + [(set_attr "type" "viwmuladd")])
>> +
>> +;; Combine sign_extend + zero_extend + fma to widen_fma (vwmaccsu)
>> +(define_insn_and_split "*dual_widen_fmasu<mode>"
>> + [(set (match_operand:VWEXTI 0 "register_operand")
>> + (plus:VWEXTI
>> + (mult:VWEXTI
>> + (sign_extend:VWEXTI
>> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
>> + (zero_extend:VWEXTI
>> + (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
>> + (match_operand:VWEXTI 1 "register_operand")))]
>> + "TARGET_VECTOR && can_create_pseudo_p ()"
>> + "#"
>> + "&& 1"
>> + [(const_int 0)]
>> + {
>> + rtx ops[] = {operands[0], operands[1], operands[2],
>> operands[3]};
>> + riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_plussu
>> (<MODE>mode),
>> + riscv_vector::WIDEN_TERNARY_OP, ops);
>> + DONE;
>> + }
>> + [(set_attr "type" "viwmuladd")])
>> +
>> +;; Combine zero_extend + sign_extend + fma to widen_fma (vwmaccsu)
>> +(define_insn_and_split "*dual_widen_fmaus<mode>"
>> + [(set (match_operand:VWEXTI 0 "register_operand")
>> + (plus:VWEXTI
>> + (mult:VWEXTI
>> + (zero_extend:VWEXTI
>> + (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand"))
>> + (sign_extend:VWEXTI
>> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand")))
>> + (match_operand:VWEXTI 1 "register_operand")))]
>> + "TARGET_VECTOR && can_create_pseudo_p ()"
>> + "#"
>> + "&& 1"
>> + [(const_int 0)]
>> + {
>> + rtx ops[] = {operands[0], operands[1], operands[2],
>> operands[3]};
>> + riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_plussu
>> (<MODE>mode),
>> + riscv_vector::WIDEN_TERNARY_OP, ops);
>> + DONE;
>> + }
>> + [(set_attr "type" "viwmuladd")])
>> +
>> +;; This combine pattern does not correspond to an single
>> instruction.
>> +;; This is a temporary pattern produced by a combine pass and if
>> there
>> +;; is no further combine into widen pattern, then fall back to
>> extend
>> +;; pattern and non-widen fma pattern.
>> +(define_insn_and_split "*single_widen_fma<mode>"
>> + [(set (match_operand:VWEXTI 0 "register_operand")
>> + (plus:VWEXTI
>> + (mult:VWEXTI
>> + (any_extend:VWEXTI
>> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
>> + (match_operand:VWEXTI 3 "register_operand"))
>> + (match_operand:VWEXTI 1 "register_operand")))]
>> + "TARGET_VECTOR && can_create_pseudo_p ()"
>> + "#"
>> + "&& 1"
>> + [(const_int 0)]
>> + {
>> + insn_code extend_icode = code_for_pred_vf2 (<CODE>,
>> <MODE>mode);
>> + rtx tmp = gen_reg_rtx (<MODE>mode);
>> + rtx extend_ops[] = {tmp, operands[2]};
>> + riscv_vector::emit_vlmax_insn (extend_icode,
>> riscv_vector::UNARY_OP,
>> + extend_ops);
>> +
>> + rtx ops[] = {operands[0], tmp, operands[3], operands[1]};
>> + riscv_vector::emit_vlmax_insn (code_for_pred_mul_plus
>> (<MODE>mode),
>> + riscv_vector::TERNARY_OP, ops);
>> + DONE;
>> + }
>> + [(set_attr "type" "viwmuladd")])
>> +
>> +;; Combine extend + fma to widen_fma (vfwmacc)
>> +(define_insn_and_split "*dual_fma<mode>"
>> + [(set (match_operand:VWEXTF 0 "register_operand")
>> + (plus:VWEXTF
>> + (mult:VWEXTF
>> + (float_extend:VWEXTF
>> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
>> + (float_extend:VWEXTF
>> + (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
>> + (match_operand:VWEXTF 1 "register_operand")))]
>> + "TARGET_VECTOR && can_create_pseudo_p ()"
>> + "#"
>> + "&& 1"
>> + [(const_int 0)]
>> + {
>> + rtx ops[] = {operands[0], operands[1], operands[2],
>> operands[3]};
>> + riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul (PLUS,
>> <MODE>mode),
>> + riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, ops);
>> + DONE;
>> + }
>> + [(set_attr "type" "vfwmuladd")])
>> +
>> +;; This combine pattern does not correspond to an single
>> instruction.
>> +;; This is a temporary pattern produced by a combine pass and if
>> there
>> +;; is no further combine into widen pattern, then fall back to
>> extend
>> +;; pattern and non-widen fma pattern.
>> +(define_insn_and_split "*single_fma<mode>"
>> + [(set (match_operand:VWEXTF 0 "register_operand")
>> + (plus:VWEXTF
>> + (mult:VWEXTF
>> + (float_extend:VWEXTF
>> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
>> + (match_operand:VWEXTF 3 "register_operand"))
>> + (match_operand:VWEXTF 1 "register_operand")))]
>> + "TARGET_VECTOR && can_create_pseudo_p ()"
>> + "#"
>> + "&& 1"
>> + [(const_int 0)]
>> + {
>> + insn_code icode = code_for_pred_extend (<MODE>mode);
>> + rtx tmp = gen_reg_rtx (<MODE>mode);
>> + rtx ext_ops[] = {tmp, operands[2]};
>> + riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP,
>> ext_ops);
>> +
>> + rtx ops[] = {operands[0], tmp, operands[3], operands[1]};
>> + riscv_vector::emit_vlmax_insn (code_for_pred_mul (PLUS,
>> <MODE>mode),
>> + riscv_vector::TERNARY_OP_FRM_DYN, ops);
>> + DONE;
>> + }
>> + [(set_attr "type" "vfwmuladd")])
>> +
>> +;; Combine extend + fnma to widen_fnma (vfwnmsac)
>> +(define_insn_and_split "*dual_fnma<mode>"
>> + [(set (match_operand:VWEXTF 0 "register_operand")
>> + (minus:VWEXTF
>> + (match_operand:VWEXTF 1 "register_operand")
>> + (mult:VWEXTF
>> + (float_extend:VWEXTF
>> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
>> + (float_extend:VWEXTF
>> + (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))))]
>> + "TARGET_VECTOR && can_create_pseudo_p ()"
>> + "#"
>> + "&& 1"
>> + [(const_int 0)]
>> + {
>> + rtx ops[] = {operands[0], operands[1], operands[2],
>> operands[3]};
>> + riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_neg
>> (PLUS, <MODE>mode),
>> + riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, ops);
>> + DONE;
>> + }
>> + [(set_attr "type" "vfwmuladd")])
>> +
>> +;; This combine pattern does not correspond to an single
>> instruction.
>> +;; This is a temporary pattern produced by a combine pass and if
>> there
>> +;; is no further combine into widen pattern, then fall back to
>> extend
>> +;; pattern and non-widen fnma pattern.
>> +(define_insn_and_split "*single_fnma<mode>"
>> + [(set (match_operand:VWEXTF 0 "register_operand")
>> + (minus:VWEXTF
>> + (match_operand:VWEXTF 1 "register_operand")
>> + (mult:VWEXTF
>> + (float_extend:VWEXTF
>> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
>> + (match_operand:VWEXTF 3 "register_operand"))))]
>> + "TARGET_VECTOR && can_create_pseudo_p ()"
>> + "#"
>> + "&& 1"
>> + [(const_int 0)]
>> + {
>> + insn_code icode = code_for_pred_extend (<MODE>mode);
>> + rtx tmp = gen_reg_rtx (<MODE>mode);
>> + rtx ext_ops[] = {tmp, operands[2]};
>> + riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP,
>> ext_ops);
>> +
>> + rtx ops[] = {operands[0], tmp, operands[3], operands[1]};
>> + riscv_vector::emit_vlmax_insn (code_for_pred_mul_neg (PLUS,
>> <MODE>mode),
>> + riscv_vector::TERNARY_OP_FRM_DYN, ops);
>> + DONE;
>> + }
>> + [(set_attr "type" "vfwmuladd")])
>> +
>> +;; Combine extend + fms to widen_fms (vfwmsac)
>> +(define_insn_and_split "*dual_fms<mode>"
>> + [(set (match_operand:VWEXTF 0 "register_operand")
>> + (minus:VWEXTF
>> + (mult:VWEXTF
>> + (float_extend:VWEXTF
>> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
>> + (float_extend:VWEXTF
>> + (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
>> + (match_operand:VWEXTF 1 "register_operand")))]
>> + "TARGET_VECTOR && can_create_pseudo_p ()"
>> + "#"
>> + "&& 1"
>> + [(const_int 0)]
>> + {
>> + rtx ops[] = {operands[0], operands[1], operands[2],
>> operands[3]};
>> + riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul (MINUS,
>> <MODE>mode),
>> + riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, ops);
>> + DONE;
>> + }
>> + [(set_attr "type" "vfwmuladd")])
>> +
>> +;; This combine pattern does not correspond to an single
>> instruction.
>> +;; This is a temporary pattern produced by a combine pass and if
>> there
>> +;; is no further combine into widen pattern, then fall back to
>> extend
>> +;; pattern and non-widen fms pattern.
>> +(define_insn_and_split "*single_fms<mode>"
>> + [(set (match_operand:VWEXTF 0 "register_operand")
>> + (minus:VWEXTF
>> + (mult:VWEXTF
>> + (float_extend:VWEXTF
>> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
>> + (match_operand:VWEXTF 3 "register_operand"))
>> + (match_operand:VWEXTF 1 "register_operand")))]
>> + "TARGET_VECTOR && can_create_pseudo_p ()"
>> + "#"
>> + "&& 1"
>> + [(const_int 0)]
>> + {
>> + insn_code icode = code_for_pred_extend (<MODE>mode);
>> + rtx tmp = gen_reg_rtx (<MODE>mode);
>> + rtx ext_ops[] = {tmp, operands[2]};
>> + riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP,
>> ext_ops);
>> +
>> + rtx ops[] = {operands[0], tmp, operands[3], operands[1]};
>> + riscv_vector::emit_vlmax_insn (code_for_pred_mul (MINUS,
>> <MODE>mode),
>> + riscv_vector::TERNARY_OP_FRM_DYN, ops);
>> + DONE;
>> + }
>> + [(set_attr "type" "vfwmuladd")])
>> +
>> +;; Combine extend + fnms to widen_fnms (vfwnmacc)
>> +(define_insn_and_split "*dual_fnms<mode>"
>> + [(set (match_operand:VWEXTF 0 "register_operand")
>> + (minus:VWEXTF
>> + (mult:VWEXTF
>> + (neg:VWEXTF
>> + (float_extend:VWEXTF
>> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand")))
>> + (float_extend:VWEXTF
>> + (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand")))
>> + (match_operand:VWEXTF 1 "register_operand")))]
>> + "TARGET_VECTOR && can_create_pseudo_p ()"
>> + "#"
>> + "&& 1"
>> + [(const_int 0)]
>> + {
>> + rtx ops[] = {operands[0], operands[1], operands[2],
>> operands[3]};
>> + riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_neg
>> (MINUS, <MODE>mode),
>> + riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, ops);
>> + DONE;
>> + }
>> + [(set_attr "type" "vfwmuladd")])
>> +
>> +;; This combine pattern does not correspond to an single
>> instruction.
>> +;; This is a temporary pattern produced by a combine pass and if
>> there
>> +;; is no further combine into widen pattern, then fall back to
>> extend
>> +;; pattern and non-widen fnms pattern.
>> +(define_insn_and_split "*single_fnms<mode>"
>> + [(set (match_operand:VWEXTF 0 "register_operand")
>> + (minus:VWEXTF
>> + (mult:VWEXTF
>> + (neg:VWEXTF
>> + (match_operand:VWEXTF 3 "register_operand"))
>> + (float_extend:VWEXTF
>> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand")))
>> + (match_operand:VWEXTF 1 "register_operand")))]
>> + "TARGET_VECTOR && can_create_pseudo_p ()"
>> + "#"
>> + "&& 1"
>> + [(const_int 0)]
>> + {
>> + insn_code icode = code_for_pred_extend (<MODE>mode);
>> + rtx tmp = gen_reg_rtx (<MODE>mode);
>> + rtx ext_ops[] = {tmp, operands[2]};
>> + riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP,
>> ext_ops);
>> +
>> + rtx ops[] = {operands[0], tmp, operands[3], operands[1]};
>> + riscv_vector::emit_vlmax_insn (code_for_pred_mul_neg (MINUS,
>> <MODE>mode),
>> + riscv_vector::TERNARY_OP_FRM_DYN, ops);
>> + DONE;
>> + }
>> + [(set_attr "type" "vfwmuladd")])
>> +
>> ;;
>> =============================================================================
>> ;; Misc combine patterns
>> ;;
>> =============================================================================
>> diff --git a/gcc/config/riscv/autovec.md
>> b/gcc/config/riscv/autovec.md
>> index 8537b9d41f6..96aaf869f88 100644
>> --- a/gcc/config/riscv/autovec.md
>> +++ b/gcc/config/riscv/autovec.md
>> @@ -1078,57 +1078,25 @@
>> ;; - vmadd
>> ;;
>> -------------------------------------------------------------------------
>> -;; We can't expand FMA for the following reasons:
>> -;; 1. Before RA, we don't know which multiply-add instruction is
>> the ideal one.
>> -;; The vmacc is the ideal instruction when operands[3] overlaps
>> operands[0].
>> -;; The vmadd is the ideal instruction when operands[1|2]
>> overlaps operands[0].
>> -;; 2. According to vector.md, the multiply-add patterns has 'merge'
>> operand which
>> -;; is the operands[5]. Since operands[5] should overlap
>> operands[0], this operand
>> -;; should be allocated the same regno as operands[1|2|3].
>> -;; 3. The 'merge' operand is always a real merge operand and we
>> don't allow undefined
>> -;; operand.
>> -;; 4. The operation of FMA pattern needs VLMAX vsetlvi which needs
>> a VL operand.
>> -;;
>> -;; In this situation, we design the codegen of FMA as follows:
>> -;; 1. clobber a scratch in the expand pattern of FMA.
>> -;; 2. Let's RA decide which input operand (operands[1|2|3]) overlap
>> operands[0].
>> -;; 3. Generate instructions (vmacc or vmadd) according to the
>> register allocation
>> -;; result after reload_completed.
>> -(define_expand "fma<mode>4"
>> - [(parallel
>> - [(set (match_operand:VI 0 "register_operand")
>> - (plus:VI
>> - (mult:VI
>> - (match_operand:VI 1 "register_operand")
>> - (match_operand:VI 2 "register_operand"))
>> - (match_operand:VI 3 "register_operand")))
>> - (clobber (match_dup 4))])]
>> - "TARGET_VECTOR"
>> - {
>> - operands[4] = gen_reg_rtx (Pmode);
>> - })
>> -
>> -(define_insn_and_split "*fma<VI:mode><P:mode>"
>> - [(set (match_operand:VI 0 "register_operand" "=vr, vr, ?&vr")
>> +(define_insn_and_split "fma<mode>4"
>> + [(set (match_operand:VI 0 "register_operand")
>> (plus:VI
>> (mult:VI
>> - (match_operand:VI 1 "register_operand" " %0, vr, vr")
>> - (match_operand:VI 2 "register_operand" " vr, vr, vr"))
>> - (match_operand:VI 3 "register_operand" " vr, 0, vr")))
>> - (clobber (match_operand:P 4 "register_operand" "=r,r,r"))]
>> - "TARGET_VECTOR"
>> + (match_operand:VI 1 "register_operand")
>> + (match_operand:VI 2 "register_operand"))
>> + (match_operand:VI 3 "register_operand")))]
>> + "TARGET_VECTOR && can_create_pseudo_p ()"
>> "#"
>> - "&& reload_completed"
>> + "&& 1"
>> [(const_int 0)]
>> {
>> - riscv_vector::emit_vlmax_vsetvl (<VI:MODE>mode, operands[4]);
>> - rtx ops[] = {operands[0], operands[1], operands[2],
>> operands[3], operands[0]};
>> - riscv_vector::emit_vlmax_insn_lra (code_for_pred_mul_plus
>> (<VI:MODE>mode),
>> - riscv_vector::TERNARY_OP, ops, operands[4]);
>> + rtx ops[] = {operands[0], operands[1], operands[2],
>> operands[3],
>> + RVV_VUNDEF(<MODE>mode)};
>> + riscv_vector::emit_vlmax_insn (code_for_pred_mul_plus
>> (<MODE>mode),
>> + riscv_vector::TERNARY_OP, ops);
>> DONE;
>> }
>> - [(set_attr "type" "vimuladd")
>> - (set_attr "mode" "<VI:MODE>")])
>> + [(set_attr "type" "vector")])
>> ;;
>> -------------------------------------------------------------------------
>> ;; ---- [INT] VNMSAC and VNMSUB
>> @@ -1138,41 +1106,25 @@
>> ;; - vnmsub
>> ;;
>> -------------------------------------------------------------------------
>> -(define_expand "fnma<mode>4"
>> - [(parallel
>> - [(set (match_operand:VI 0 "register_operand")
>> - (minus:VI
>> - (match_operand:VI 3 "register_operand")
>> - (mult:VI
>> - (match_operand:VI 1 "register_operand")
>> - (match_operand:VI 2 "register_operand"))))
>> - (clobber (match_dup 4))])]
>> - "TARGET_VECTOR"
>> - {
>> - operands[4] = gen_reg_rtx (Pmode);
>> - })
>> -
>> -(define_insn_and_split "*fnma<VI:mode><P:mode>"
>> - [(set (match_operand:VI 0 "register_operand" "=vr, vr, ?&vr")
>> - (minus:VI
>> - (match_operand:VI 3 "register_operand" " vr, 0, vr")
>> - (mult:VI
>> - (match_operand:VI 1 "register_operand" " %0, vr, vr")
>> - (match_operand:VI 2 "register_operand" " vr, vr, vr"))))
>> - (clobber (match_operand:P 4 "register_operand" "=r,r,r"))]
>> - "TARGET_VECTOR"
>> +(define_insn_and_split "fnma<mode>4"
>> + [(set (match_operand:VI 0 "register_operand")
>> + (minus:VI
>> + (match_operand:VI 3 "register_operand")
>> + (mult:VI
>> + (match_operand:VI 1 "register_operand")
>> + (match_operand:VI 2 "register_operand"))))]
>> + "TARGET_VECTOR && can_create_pseudo_p ()"
>> "#"
>> - "&& reload_completed"
>> + "&& 1"
>> [(const_int 0)]
>> {
>> - riscv_vector::emit_vlmax_vsetvl (<VI:MODE>mode, operands[4]);
>> - rtx ops[] = {operands[0], operands[1], operands[2],
>> operands[3], operands[0]};
>> - riscv_vector::emit_vlmax_insn_lra (code_for_pred_minus_mul
>> (<VI:MODE>mode),
>> - riscv_vector::TERNARY_OP,
>> ops, operands[4]);
>> + rtx ops[] = {operands[0], operands[1], operands[2],
>> operands[3],
>> + RVV_VUNDEF(<MODE>mode)};
>> + riscv_vector::emit_vlmax_insn (code_for_pred_minus_mul
>> (<MODE>mode),
>> + riscv_vector::TERNARY_OP, ops);
>> DONE;
>> }
>> - [(set_attr "type" "vimuladd")
>> - (set_attr "mode" "<VI:MODE>")])
>> + [(set_attr "type" "vector")])
>> ;;
>> -------------------------------------------------------------------------
>> ;; ---- [FP] VFMACC and VFMADD
>> @@ -1182,45 +1134,25 @@
>> ;; - vfmadd
>> ;;
>> -------------------------------------------------------------------------
>> -(define_expand "fma<mode>4"
>> - [(parallel
>> - [(set (match_operand:VF 0 "register_operand")
>> - (unspec:VF
>> - [(fma:VF
>> - (match_operand:VF 1 "register_operand")
>> - (match_operand:VF 2 "register_operand")
>> - (match_operand:VF 3 "register_operand"))
>> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
>> - (clobber (match_dup 4))])]
>> - "TARGET_VECTOR"
>> - {
>> - operands[4] = gen_reg_rtx (Pmode);
>> - }
>> - [(set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
>> -
>> -(define_insn_and_split "*fma<VF:mode><P:mode>"
>> - [(set (match_operand:VF 0 "register_operand" "=vr, vr, ?&vr")
>> - (unspec:VF
>> - [(fma:VF
>> - (match_operand:VF 1 "register_operand" " %0, vr, vr")
>> - (match_operand:VF 2 "register_operand" " vr, vr, vr")
>> - (match_operand:VF 3 "register_operand" " vr, 0, vr"))
>> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
>> - (clobber (match_operand:P 4 "register_operand" "=r,r,r"))]
>> - "TARGET_VECTOR"
>> +(define_insn_and_split "fma<mode>4"
>> + [(set (match_operand:VF 0 "register_operand")
>> + (plus:VF
>> + (mult:VF
>> + (match_operand:VF 1 "register_operand")
>> + (match_operand:VF 2 "register_operand"))
>> + (match_operand:VF 3 "register_operand")))]
>> + "TARGET_VECTOR && can_create_pseudo_p ()"
>> "#"
>> - "&& reload_completed"
>> + "&& 1"
>> [(const_int 0)]
>> {
>> - riscv_vector::emit_vlmax_vsetvl (<VF:MODE>mode, operands[4]);
>> - rtx ops[] = {operands[0], operands[1], operands[2],
>> operands[3], operands[0]};
>> - riscv_vector::emit_vlmax_insn_lra (code_for_pred_mul (PLUS,
>> <VF:MODE>mode),
>> - riscv_vector::TERNARY_OP_FRM_DYN, ops, operands[4]);
>> + rtx ops[] = {operands[0], operands[1], operands[2],
>> operands[3],
>> + RVV_VUNDEF(<MODE>mode)};
>> + riscv_vector::emit_vlmax_insn (code_for_pred_mul (PLUS,
>> <MODE>mode),
>> + riscv_vector::TERNARY_OP_FRM_DYN, ops);
>> DONE;
>> }
>> - [(set_attr "type" "vfmuladd")
>> - (set_attr "mode" "<VF:MODE>")
>> - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
>> + [(set_attr "type" "vector")])
>> ;;
>> -------------------------------------------------------------------------
>> ;; ---- [FP] VFNMSAC and VFNMSUB
>> @@ -1230,47 +1162,25 @@
>> ;; - vfnmsub
>> ;;
>> -------------------------------------------------------------------------
>> -(define_expand "fnma<mode>4"
>> - [(parallel
>> - [(set (match_operand:VF 0 "register_operand")
>> - (unspec:VF
>> - [(fma:VF
>> - (neg:VF
>> - (match_operand:VF 1 "register_operand"))
>> - (match_operand:VF 2 "register_operand")
>> - (match_operand:VF 3 "register_operand"))
>> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
>> - (clobber (match_dup 4))])]
>> - "TARGET_VECTOR"
>> - {
>> - operands[4] = gen_reg_rtx (Pmode);
>> - }
>> - [(set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
>> -
>> -(define_insn_and_split "*fnma<VF:mode><P:mode>"
>> - [(set (match_operand:VF 0 "register_operand" "=vr, vr, ?&vr")
>> - (unspec:VF
>> - [(fma:VF
>> - (neg:VF
>> - (match_operand:VF 1 "register_operand" " %0, vr, vr"))
>> - (match_operand:VF 2 "register_operand" " vr, vr, vr")
>> - (match_operand:VF 3 "register_operand" " vr, 0, vr"))
>> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
>> - (clobber (match_operand:P 4 "register_operand" "=r,r,r"))]
>> - "TARGET_VECTOR"
>> +(define_insn_and_split "fnma<mode>4"
>> + [(set (match_operand:VF 0 "register_operand")
>> + (minus:VF
>> + (match_operand:VF 3 "register_operand")
>> + (mult:VF
>> + (match_operand:VF 1 "register_operand")
>> + (match_operand:VF 2 "register_operand"))))]
>> + "TARGET_VECTOR && can_create_pseudo_p ()"
>> "#"
>> - "&& reload_completed"
>> + "&& 1"
>> [(const_int 0)]
>> {
>> - riscv_vector::emit_vlmax_vsetvl (<VF:MODE>mode, operands[4]);
>> - rtx ops[] = {operands[0], operands[1], operands[2],
>> operands[3], operands[0]};
>> - riscv_vector::emit_vlmax_insn_lra (code_for_pred_mul_neg (PLUS,
>> <VF:MODE>mode),
>> - riscv_vector::TERNARY_OP_FRM_DYN, ops, operands[4]);
>> + rtx ops[] = {operands[0], operands[1], operands[2],
>> operands[3],
>> + RVV_VUNDEF(<MODE>mode)};
>> + riscv_vector::emit_vlmax_insn (code_for_pred_mul_neg (PLUS,
>> <MODE>mode),
>> + riscv_vector::TERNARY_OP_FRM_DYN, ops);
>> DONE;
>> }
>> - [(set_attr "type" "vfmuladd")
>> - (set_attr "mode" "<VF:MODE>")
>> - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
>> + [(set_attr "type" "vector")])
>> ;;
>> -------------------------------------------------------------------------
>> ;; ---- [FP] VFMSAC and VFMSUB
>> @@ -1280,47 +1190,25 @@
>> ;; - vfmsub
>> ;;
>> -------------------------------------------------------------------------
>> -(define_expand "fms<mode>4"
>> - [(parallel
>> - [(set (match_operand:VF 0 "register_operand")
>> - (unspec:VF
>> - [(fma:VF
>> - (match_operand:VF 1 "register_operand")
>> - (match_operand:VF 2 "register_operand")
>> - (neg:VF
>> - (match_operand:VF 3 "register_operand")))
>> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
>> - (clobber (match_dup 4))])]
>> - "TARGET_VECTOR"
>> - {
>> - operands[4] = gen_reg_rtx (Pmode);
>> - }
>> - [(set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
>> -
>> -(define_insn_and_split "*fms<VF:mode><P:mode>"
>> - [(set (match_operand:VF 0 "register_operand" "=vr, vr, ?&vr")
>> - (unspec:VF
>> - [(fma:VF
>> - (match_operand:VF 1 "register_operand" " %0, vr, vr")
>> - (match_operand:VF 2 "register_operand" " vr, vr, vr")
>> - (neg:VF
>> - (match_operand:VF 3 "register_operand" " vr, 0, vr")))
>> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
>> - (clobber (match_operand:P 4 "register_operand" "=r,r,r"))]
>> - "TARGET_VECTOR"
>> +(define_insn_and_split "fms<mode>4"
>> + [(set (match_operand:VF 0 "register_operand")
>> + (minus:VF
>> + (mult:VF
>> + (match_operand:VF 1 "register_operand")
>> + (match_operand:VF 2 "register_operand"))
>> + (match_operand:VF 3 "register_operand")))]
>> + "TARGET_VECTOR && can_create_pseudo_p ()"
>> "#"
>> - "&& reload_completed"
>> + "&& 1"
>> [(const_int 0)]
>> {
>> - riscv_vector::emit_vlmax_vsetvl (<VF:MODE>mode, operands[4]);
>> - rtx ops[] = {operands[0], operands[1], operands[2],
>> operands[3], operands[0]};
>> - riscv_vector::emit_vlmax_insn_lra (code_for_pred_mul (MINUS,
>> <VF:MODE>mode),
>> - riscv_vector::TERNARY_OP_FRM_DYN, ops, operands[4]);
>> + rtx ops[] = {operands[0], operands[1], operands[2],
>> operands[3],
>> + RVV_VUNDEF(<MODE>mode)};
>> + riscv_vector::emit_vlmax_insn (code_for_pred_mul (MINUS,
>> <MODE>mode),
>> + riscv_vector::TERNARY_OP_FRM_DYN, ops);
>> DONE;
>> }
>> - [(set_attr "type" "vfmuladd")
>> - (set_attr "mode" "<VF:MODE>")
>> - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
>> + [(set_attr "type" "vector")])
>> ;;
>> -------------------------------------------------------------------------
>> ;; ---- [FP] VFNMACC and VFNMADD
>> @@ -1330,49 +1218,26 @@
>> ;; - vfnmadd
>> ;;
>> -------------------------------------------------------------------------
>> -(define_expand "fnms<mode>4"
>> - [(parallel
>> - [(set (match_operand:VF 0 "register_operand")
>> - (unspec:VF
>> - [(fma:VF
>> - (neg:VF
>> - (match_operand:VF 1 "register_operand"))
>> - (match_operand:VF 2 "register_operand")
>> - (neg:VF
>> - (match_operand:VF 3 "register_operand")))
>> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
>> - (clobber (match_dup 4))])]
>> - "TARGET_VECTOR"
>> - {
>> - operands[4] = gen_reg_rtx (Pmode);
>> - }
>> - [(set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
>> -
>> -(define_insn_and_split "*fnms<VF:mode><P:mode>"
>> - [(set (match_operand:VF 0 "register_operand" "=vr, vr, ?&vr")
>> - (unspec:VF
>> - [(fma:VF
>> - (neg:VF
>> - (match_operand:VF 1 "register_operand" " %0, vr, vr"))
>> - (match_operand:VF 2 "register_operand" " vr, vr, vr")
>> - (neg:VF
>> - (match_operand:VF 3 "register_operand" " vr, 0, vr")))
>> - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))
>> - (clobber (match_operand:P 4 "register_operand" "=r,r,r"))]
>> - "TARGET_VECTOR"
>> +(define_insn_and_split "fnms<mode>4"
>> + [(set (match_operand:VF 0 "register_operand")
>> + (minus:VF
>> + (neg:VF
>> + (mult:VF
>> + (match_operand:VF 1 "register_operand")
>> + (match_operand:VF 2 "register_operand")))
>> + (match_operand:VF 3 "register_operand")))]
>> + "TARGET_VECTOR && can_create_pseudo_p ()"
>> "#"
>> - "&& reload_completed"
>> + "&& 1"
>> [(const_int 0)]
>> {
>> - riscv_vector::emit_vlmax_vsetvl (<VF:MODE>mode, operands[4]);
>> - rtx ops[] = {operands[0], operands[1], operands[2],
>> operands[3], operands[0]};
>> - riscv_vector::emit_vlmax_insn_lra (code_for_pred_mul_neg
>> (MINUS, <VF:MODE>mode),
>> - riscv_vector::TERNARY_OP_FRM_DYN, ops, operands[4]);
>> + rtx ops[] = {operands[0], operands[1], operands[2],
>> operands[3],
>> + RVV_VUNDEF(<MODE>mode)};
>> + riscv_vector::emit_vlmax_insn (code_for_pred_mul_neg (MINUS,
>> <MODE>mode),
>> + riscv_vector::TERNARY_OP_FRM_DYN, ops);
>> DONE;
>> }
>> - [(set_attr "type" "vfmuladd")
>> - (set_attr "mode" "<VF:MODE>")
>> - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))])
>> + [(set_attr "type" "vector")])
>> ;;
>> =========================================================================
>> ;; == SELECT_VL
>> diff --git a/gcc/config/riscv/riscv-protos.h
>> b/gcc/config/riscv/riscv-protos.h
>> index 44fa36c32ab..46fc8f39974 100644
>> --- a/gcc/config/riscv/riscv-protos.h
>> +++ b/gcc/config/riscv/riscv-protos.h
>> @@ -462,7 +462,7 @@ void expand_select_vl (rtx *);
>> void expand_load_store (rtx *, bool);
>> void expand_gather_scatter (rtx *, bool);
>> void expand_cond_len_ternop (unsigned, rtx *);
>> -void prepare_ternary_operands (rtx *, bool = false);
>> +void prepare_ternary_operands (rtx *);
>> void expand_lanes_load_store (rtx *, bool);
>> void expand_fold_extract_last (rtx *);
>> diff --git a/gcc/config/riscv/riscv-v.cc
>> b/gcc/config/riscv/riscv-v.cc
>> index 668594b65ed..f51986aebde 100644
>> --- a/gcc/config/riscv/riscv-v.cc
>> +++ b/gcc/config/riscv/riscv-v.cc
>> @@ -3253,15 +3253,15 @@ expand_reduction (unsigned unspec, rtx *ops,
>> rtx init, reduction_type type)
>> /* Prepare ops for ternary operations.
>> It can be called before or after RA. */
>> void
>> -prepare_ternary_operands (rtx *ops, bool split_p)
>> +prepare_ternary_operands (rtx *ops)
>> {
>> machine_mode mode = GET_MODE (ops[0]);
>> - if (split_p
>> - || (!rtx_equal_p (ops[2], ops[5])
>> - && !rtx_equal_p (ops[3], ops[5])
>> - && !rtx_equal_p (ops[4], ops[5])
>> - && riscv_get_v_regno_alignment (mode) == 8))
>> + if (!rtx_equal_p (ops[5], RVV_VUNDEF (mode))
>> + && (VECTOR_MODE_P (GET_MODE (ops[2]))
>> + && !rtx_equal_p (ops[2], ops[5]))
>> + && !rtx_equal_p (ops[3], ops[5])
>> + && !rtx_equal_p (ops[4], ops[5]))
>> {
>> /* RA will fail to find vector REG and report ICE, so we
>> pre-merge
>> the ops for LMUL = 8. */
>> @@ -3286,6 +3286,8 @@ prepare_ternary_operands (rtx *ops, bool
>> split_p)
>> /* TODO: ??? Maybe we could support splitting FMA (a, 4, b)
>> into PLUS (ASHIFT (a, 2), b) according to uarchs. */
>> }
>> + gcc_assert (rtx_equal_p (ops[5], RVV_VUNDEF (mode))
>> + || rtx_equal_p (ops[5], ops[2]) || rtx_equal_p (ops[5],
>> ops[4]));
>> }
>> /* Expand VEC_MASK_LEN_{LOAD_LANES,STORE_LANES}. */
>> diff --git a/gcc/config/riscv/vector.md b/gcc/config/riscv/vector.md
>> index fb0c8abd995..e174322a459 100644
>> --- a/gcc/config/riscv/vector.md
>> +++ b/gcc/config/riscv/vector.md
>> @@ -5002,12 +5002,40 @@
>> (match_operand:VI 2 "register_operand")
>> (match_operand:VI 3 "register_operand"))
>> (match_operand:VI 4 "register_operand"))
>> - (match_operand:VI 5 "register_operand")))]
>> + (match_operand:VI 5 "vector_merge_operand")))]
>> "TARGET_VECTOR"
>> {
>> riscv_vector::prepare_ternary_operands (operands);
>> })
>> +(define_insn "*pred_mul_plus<mode>_undef"
>> + [(set (match_operand:VI 0 "register_operand" "=vd,
>> vd,?&vd, vr, vr,?&vr")
>> + (if_then_else:VI
>> + (unspec:<VM>
>> + [(match_operand:<VM> 1 "vector_mask_operand" " vm, vm,
>> vm,Wc1,Wc1, Wc1")
>> + (match_operand 6 "vector_length_operand" " rK, rK, rK,
>> rK, rK, rK")
>> + (match_operand 7 "const_int_operand" " i, i, i,
>> i, i, i")
>> + (match_operand 8 "const_int_operand" " i, i, i,
>> i, i, i")
>> + (match_operand 9 "const_int_operand" " i, i, i,
>> i, i, i")
>> + (reg:SI VL_REGNUM)
>> + (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
>> + (plus:VI
>> + (mult:VI
>> + (match_operand:VI 3 "register_operand" " 0, vr, vr,
>> 0, vr, vr")
>> + (match_operand:VI 4 "register_operand" " vr, vr, vr,
>> vr, vr, vr"))
>> + (match_operand:VI 5 "register_operand" " vr, 0, vr,
>> vr, 0, vr"))
>> + (match_operand:VI 2 "vector_undef_operand")))]
>> + "TARGET_VECTOR"
>> + "@
>> + vmadd.vv\t%0,%4,%5%p1
>> + vmacc.vv\t%0,%3,%4%p1
>> + vmv.v.v\t%0,%4\;vmacc.vv\t%0,%3,%4%p1
>> + vmadd.vv\t%0,%4,%5%p1
>> + vmacc.vv\t%0,%3,%4%p1
>> + vmv.v.v\t%0,%5\;vmacc.vv\t%0,%3,%4%p1"
>> + [(set_attr "type" "vimuladd")
>> + (set_attr "mode" "<MODE>")])
>> +
>> (define_insn "*pred_madd<mode>"
>> [(set (match_operand:VI 0 "register_operand" "=vd,?&vd,
>> vr,?&vr")
>> (if_then_else:VI
>> @@ -5070,35 +5098,6 @@
>> (set (attr "ma") (symbol_ref
>> "riscv_vector::get_ma(operands[7])"))
>> (set (attr "avl_type") (symbol_ref "INTVAL (operands[8])"))])
>> -(define_insn_and_rewrite "*pred_mul_plus<mode>"
>> - [(set (match_operand:VI 0 "register_operand" "=&vr")
>> - (if_then_else:VI
>> - (unspec:<VM>
>> - [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
>> - (match_operand 6 "vector_length_operand" " rK")
>> - (match_operand 7 "const_int_operand" " i")
>> - (match_operand 8 "const_int_operand" " i")
>> - (match_operand 9 "const_int_operand" " i")
>> - (reg:SI VL_REGNUM)
>> - (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
>> - (plus:VI
>> - (mult:VI
>> - (match_operand:VI 2 "register_operand" " vr")
>> - (match_operand:VI 3 "register_operand" " vr"))
>> - (match_operand:VI 4 "register_operand" " vr"))
>> - (match_operand:VI 5 "register_operand" " vr")))]
>> - "TARGET_VECTOR
>> - && !rtx_equal_p (operands[2], operands[5])
>> - && !rtx_equal_p (operands[3], operands[5])
>> - && !rtx_equal_p (operands[4], operands[5])"
>> - "#"
>> - "&& reload_completed"
>> - {
>> - riscv_vector::prepare_ternary_operands (operands, true);
>> - }
>> - [(set_attr "type" "vimuladd")
>> - (set_attr "mode" "<MODE>")])
>> -
>> (define_expand "@pred_mul_plus<mode>_scalar"
>> [(set (match_operand:VI_QHS 0 "register_operand")
>> (if_then_else:VI_QHS
>> @@ -5184,35 +5183,6 @@
>> (set (attr "ma") (symbol_ref
>> "riscv_vector::get_ma(operands[7])"))
>> (set (attr "avl_type") (symbol_ref "INTVAL (operands[8])"))])
>> -(define_insn_and_rewrite "*pred_mul_plus<mode>_scalar"
>> - [(set (match_operand:VI 0 "register_operand" "=&vr")
>> - (if_then_else:VI
>> - (unspec:<VM>
>> - [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
>> - (match_operand 6 "vector_length_operand" " rK")
>> - (match_operand 7 "const_int_operand" " i")
>> - (match_operand 8 "const_int_operand" " i")
>> - (match_operand 9 "const_int_operand" " i")
>> - (reg:SI VL_REGNUM)
>> - (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
>> - (plus:VI
>> - (mult:VI
>> - (vec_duplicate:VI
>> - (match_operand:<VEL> 2 "register_operand" " r"))
>> - (match_operand:VI 3 "register_operand" " vr"))
>> - (match_operand:VI 4 "vector_arith_operand" " vr"))
>> - (match_operand:VI 5 "register_operand" " vr")))]
>> - "TARGET_VECTOR
>> - && !rtx_equal_p (operands[3], operands[5])
>> - && !rtx_equal_p (operands[4], operands[5])"
>> - "#"
>> - "&& reload_completed"
>> - {
>> - riscv_vector::prepare_ternary_operands (operands, true);
>> - }
>> - [(set_attr "type" "vimuladd")
>> - (set_attr "mode" "<MODE>")])
>> -
>> (define_expand "@pred_mul_plus<mode>_scalar"
>> [(set (match_operand:VI_D 0 "register_operand")
>> (if_then_else:VI_D
>> @@ -5313,36 +5283,6 @@
>> (set (attr "ma") (symbol_ref
>> "riscv_vector::get_ma(operands[7])"))
>> (set (attr "avl_type") (symbol_ref "INTVAL (operands[8])"))])
>> -(define_insn_and_rewrite "*pred_mul_plus<mode>_extended_scalar"
>> - [(set (match_operand:VI_D 0 "register_operand"
>> "=&vr")
>> - (if_then_else:VI_D
>> - (unspec:<VM>
>> - [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
>> - (match_operand 6 "vector_length_operand" " rK")
>> - (match_operand 7 "const_int_operand" " i")
>> - (match_operand 8 "const_int_operand" " i")
>> - (match_operand 9 "const_int_operand" " i")
>> - (reg:SI VL_REGNUM)
>> - (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
>> - (plus:VI_D
>> - (mult:VI_D
>> - (vec_duplicate:VI_D
>> - (sign_extend:<VEL>
>> - (match_operand:<VSUBEL> 2 "register_operand" " r")))
>> - (match_operand:VI_D 3 "register_operand" " vr"))
>> - (match_operand:VI_D 4 "register_operand" " vr"))
>> - (match_operand:VI_D 5 "register_operand" " vr")))]
>> - "TARGET_VECTOR
>> - && !rtx_equal_p (operands[3], operands[5])
>> - && !rtx_equal_p (operands[4], operands[5])"
>> - "#"
>> - "&& reload_completed"
>> - {
>> - riscv_vector::prepare_ternary_operands (operands, true);
>> - }
>> - [(set_attr "type" "vimuladd")
>> - (set_attr "mode" "<MODE>")])
>> -
>> (define_expand "@pred_minus_mul<mode>"
>> [(set (match_operand:VI 0 "register_operand")
>> (if_then_else:VI
>> @@ -5359,12 +5299,40 @@
>> (mult:VI
>> (match_operand:VI 2 "register_operand")
>> (match_operand:VI 3 "register_operand")))
>> - (match_operand:VI 5 "register_operand")))]
>> + (match_operand:VI 5 "vector_merge_operand")))]
>> "TARGET_VECTOR"
>> {
>> riscv_vector::prepare_ternary_operands (operands);
>> })
>> +(define_insn "*pred_minus_mul<mode>_undef"
>> + [(set (match_operand:VI 0 "register_operand" "=vd,
>> vd,?&vd, vr, vr,?&vr")
>> + (if_then_else:VI
>> + (unspec:<VM>
>> + [(match_operand:<VM> 1 "vector_mask_operand" " vm, vm,
>> vm,Wc1,Wc1, Wc1")
>> + (match_operand 6 "vector_length_operand" " rK, rK, rK,
>> rK, rK, rK")
>> + (match_operand 7 "const_int_operand" " i, i, i,
>> i, i, i")
>> + (match_operand 8 "const_int_operand" " i, i, i,
>> i, i, i")
>> + (match_operand 9 "const_int_operand" " i, i, i,
>> i, i, i")
>> + (reg:SI VL_REGNUM)
>> + (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
>> + (minus:VI
>> + (match_operand:VI 5 "register_operand" " vr,
>> 0, vr, vr, 0, vr")
>> + (mult:VI
>> + (match_operand:VI 3 "register_operand" " 0, vr, vr,
>> 0, vr, vr")
>> + (match_operand:VI 4 "register_operand" " vr, vr, vr,
>> vr, vr, vr")))
>> + (match_operand:VI 2 "vector_undef_operand")))]
>> + "TARGET_VECTOR"
>> + "@
>> + vnmsub.vv\t%0,%4,%5%p1
>> + vnmsac.vv\t%0,%3,%4%p1
>> + vmv.v.v\t%0,%3\;vnmsub.vv\t%0,%4,%5%p1
>> + vnmsub.vv\t%0,%4,%5%p1
>> + vnmsac.vv\t%0,%3,%4%p1
>> + vmv.v.v\t%0,%3\;vnmsub.vv\t%0,%4,%5%p1"
>> + [(set_attr "type" "vimuladd")
>> + (set_attr "mode" "<MODE>")])
>> +
>> (define_insn "*pred_nmsub<mode>"
>> [(set (match_operand:VI 0 "register_operand" "=vd,?&vd,
>> vr,?&vr")
>> (if_then_else:VI
>> @@ -5427,35 +5395,6 @@
>> (set (attr "ma") (symbol_ref
>> "riscv_vector::get_ma(operands[7])"))
>> (set (attr "avl_type") (symbol_ref "INTVAL (operands[8])"))])
>> -(define_insn_and_rewrite "*pred_minus_mul<mode>"
>> - [(set (match_operand:VI 0 "register_operand" "=&vr")
>> - (if_then_else:VI
>> - (unspec:<VM>
>> - [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
>> - (match_operand 6 "vector_length_operand" " rK")
>> - (match_operand 7 "const_int_operand" " i")
>> - (match_operand 8 "const_int_operand" " i")
>> - (match_operand 9 "const_int_operand" " i")
>> - (reg:SI VL_REGNUM)
>> - (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
>> - (minus:VI
>> - (match_operand:VI 4 "vector_arith_operand" " vr")
>> - (mult:VI
>> - (match_operand:VI 2 "register_operand" " vr")
>> - (match_operand:VI 3 "register_operand" " vr")))
>> - (match_operand:VI 5 "register_operand" " vr")))]
>> - "TARGET_VECTOR
>> - && !rtx_equal_p (operands[2], operands[5])
>> - && !rtx_equal_p (operands[3], operands[5])
>> - && !rtx_equal_p (operands[4], operands[5])"
>> - "#"
>> - "&& reload_completed"
>> - {
>> - riscv_vector::prepare_ternary_operands (operands, true);
>> - }
>> - [(set_attr "type" "vimuladd")
>> - (set_attr "mode" "<MODE>")])
>> -
>> (define_expand "@pred_minus_mul<mode>_scalar"
>> [(set (match_operand:VI_QHS 0 "register_operand")
>> (if_then_else:VI_QHS
>> @@ -5541,35 +5480,6 @@
>> (set (attr "ma") (symbol_ref
>> "riscv_vector::get_ma(operands[7])"))
>> (set (attr "avl_type") (symbol_ref "INTVAL (operands[8])"))])
>> -(define_insn_and_rewrite "*pred_minus_mul<mode>_scalar"
>> - [(set (match_operand:VI 0 "register_operand" "=&vr")
>> - (if_then_else:VI
>> - (unspec:<VM>
>> - [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
>> - (match_operand 6 "vector_length_operand" " rK")
>> - (match_operand 7 "const_int_operand" " i")
>> - (match_operand 8 "const_int_operand" " i")
>> - (match_operand 9 "const_int_operand" " i")
>> - (reg:SI VL_REGNUM)
>> - (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
>> - (minus:VI
>> - (match_operand:VI 4 "vector_arith_operand" " vr")
>> - (mult:VI
>> - (vec_duplicate:VI
>> - (match_operand:<VEL> 2 "register_operand" " r"))
>> - (match_operand:VI 3 "register_operand" " vr")))
>> - (match_operand:VI 5 "register_operand" " vr")))]
>> - "TARGET_VECTOR
>> - && !rtx_equal_p (operands[3], operands[5])
>> - && !rtx_equal_p (operands[4], operands[5])"
>> - "#"
>> - "&& reload_completed"
>> - {
>> - riscv_vector::prepare_ternary_operands (operands, true);
>> - }
>> - [(set_attr "type" "vimuladd")
>> - (set_attr "mode" "<MODE>")])
>> -
>> (define_expand "@pred_minus_mul<mode>_scalar"
>> [(set (match_operand:VI_D 0 "register_operand")
>> (if_then_else:VI_D
>> @@ -5670,36 +5580,6 @@
>> (set (attr "ma") (symbol_ref
>> "riscv_vector::get_ma(operands[7])"))
>> (set (attr "avl_type") (symbol_ref "INTVAL (operands[8])"))])
>> -(define_insn_and_rewrite "*pred_minus_mul<mode>_extended_scalar"
>> - [(set (match_operand:VI_D 0 "register_operand"
>> "=&vr")
>> - (if_then_else:VI_D
>> - (unspec:<VM>
>> - [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
>> - (match_operand 6 "vector_length_operand" " rK")
>> - (match_operand 7 "const_int_operand" " i")
>> - (match_operand 8 "const_int_operand" " i")
>> - (match_operand 9 "const_int_operand" " i")
>> - (reg:SI VL_REGNUM)
>> - (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
>> - (minus:VI_D
>> - (match_operand:VI_D 4 "vector_arith_operand" " vr")
>> - (mult:VI_D
>> - (vec_duplicate:VI_D
>> - (sign_extend:<VEL>
>> - (match_operand:<VSUBEL> 2 "register_operand" " r")))
>> - (match_operand:VI_D 3 "register_operand" " vr")))
>> - (match_operand:VI_D 5 "register_operand" " vr")))]
>> - "TARGET_VECTOR
>> - && !rtx_equal_p (operands[3], operands[5])
>> - && !rtx_equal_p (operands[4], operands[5])"
>> - "#"
>> - "&& reload_completed"
>> - {
>> - riscv_vector::prepare_ternary_operands (operands, true);
>> - }
>> - [(set_attr "type" "vimuladd")
>> - (set_attr "mode" "<MODE>")])
>> -
>> ;;
>> -------------------------------------------------------------------------------
>> ;; ---- Predicated widen integer ternary operations
>> ;;
>> -------------------------------------------------------------------------------
>> @@ -6276,12 +6156,44 @@
>> (match_operand:VF 2 "register_operand")
>> (match_operand:VF 3 "register_operand"))
>> (match_operand:VF 4 "register_operand"))
>> - (match_operand:VF 5 "register_operand")))]
>> + (match_operand:VF 5 "vector_merge_operand")))]
>> "TARGET_VECTOR"
>> {
>> riscv_vector::prepare_ternary_operands (operands);
>> })
>> +(define_insn "*pred_mul_<optab><mode>_undef"
>> + [(set (match_operand:VF 0 "register_operand" "=vd,vd,?&vd, vr,
>> vr,?&vr")
>> + (if_then_else:VF
>> + (unspec:<VM>
>> + [(match_operand:<VM> 1 "vector_mask_operand" " vm,vm,
>> vm,Wc1,Wc1, Wc1")
>> + (match_operand 6 "vector_length_operand" " rK,rK, rK, rK,
>> rK, rK")
>> + (match_operand 7 "const_int_operand" " i, i, i,
>> i, i, i")
>> + (match_operand 8 "const_int_operand" " i, i, i,
>> i, i, i")
>> + (match_operand 9 "const_int_operand" " i, i, i,
>> i, i, i")
>> + (match_operand 10 "const_int_operand" " i, i, i,
>> i, i, i")
>> + (reg:SI VL_REGNUM)
>> + (reg:SI VTYPE_REGNUM)
>> + (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE)
>> + (plus_minus:VF
>> + (mult:VF
>> + (match_operand:VF 3 "register_operand" " 0,vr, vr, 0,
>> vr, vr")
>> + (match_operand:VF 4 "register_operand" " vr,vr, vr, vr,
>> vr, vr"))
>> + (match_operand:VF 5 "register_operand" " vr, 0, vr,
>> vr, 0, vr"))
>> + (match_operand:VF 2 "vector_undef_operand")))]
>> + "TARGET_VECTOR"
>> + "@
>> + vf<madd_msub>.vv\t%0,%4,%5%p1
>> + vf<macc_msac>.vv\t%0,%3,%4%p1
>> + vmv.v.v\t%0,%3\;vf<madd_msub>.vv\t%0,%4,%5%p1
>> + vf<madd_msub>.vv\t%0,%4,%5%p1
>> + vf<macc_msac>.vv\t%0,%3,%4%p1
>> + vmv.v.v\t%0,%3\;vf<madd_msub>.vv\t%0,%4,%5%p1"
>> + [(set_attr "type" "vfmuladd")
>> + (set_attr "mode" "<MODE>")
>> + (set (attr "frm_mode")
>> + (symbol_ref "riscv_vector::get_frm_mode (operands[10])"))])
>> +
>> (define_insn "*pred_<madd_msub><mode>"
>> [(set (match_operand:VF 0 "register_operand" "=vd,
>> ?&vd, vr, ?&vr")
>> (if_then_else:VF
>> @@ -6352,39 +6264,6 @@
>> (set (attr "frm_mode")
>> (symbol_ref "riscv_vector::get_frm_mode (operands[9])"))])
>> -(define_insn_and_rewrite "*pred_mul_<optab><mode>"
>> - [(set (match_operand:VF 0 "register_operand" "=&vr")
>> - (if_then_else:VF
>> - (unspec:<VM>
>> - [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
>> - (match_operand 6 "vector_length_operand" " rK")
>> - (match_operand 7 "const_int_operand" " i")
>> - (match_operand 8 "const_int_operand" " i")
>> - (match_operand 9 "const_int_operand" " i")
>> - (match_operand 10 "const_int_operand" " i")
>> - (reg:SI VL_REGNUM)
>> - (reg:SI VTYPE_REGNUM)
>> - (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE)
>> - (plus_minus:VF
>> - (mult:VF
>> - (match_operand:VF 2 "register_operand" " vr")
>> - (match_operand:VF 3 "register_operand" " vr"))
>> - (match_operand:VF 4 "register_operand" " vr"))
>> - (match_operand:VF 5 "register_operand" " vr")))]
>> - "TARGET_VECTOR
>> - && !rtx_equal_p (operands[2], operands[5])
>> - && !rtx_equal_p (operands[3], operands[5])
>> - && !rtx_equal_p (operands[4], operands[5])"
>> - "#"
>> - "&& reload_completed"
>> - {
>> - riscv_vector::prepare_ternary_operands (operands, true);
>> - }
>> - [(set_attr "type" "vfmuladd")
>> - (set_attr "mode" "<MODE>")
>> - (set (attr "frm_mode")
>> - (symbol_ref "riscv_vector::get_frm_mode (operands[10])"))])
>> -
>> (define_expand "@pred_mul_<optab><mode>_scalar"
>> [(set (match_operand:VF 0 "register_operand")
>> (if_then_else:VF
>> @@ -6480,39 +6359,6 @@
>> (set (attr "frm_mode")
>> (symbol_ref "riscv_vector::get_frm_mode (operands[9])"))])
>> -(define_insn_and_rewrite "*pred_mul_<optab><mode>_scalar"
>> - [(set (match_operand:VF 0 "register_operand" "=&vr")
>> - (if_then_else:VF
>> - (unspec:<VM>
>> - [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
>> - (match_operand 6 "vector_length_operand" " rK")
>> - (match_operand 7 "const_int_operand" " i")
>> - (match_operand 8 "const_int_operand" " i")
>> - (match_operand 9 "const_int_operand" " i")
>> - (match_operand 10 "const_int_operand" " i")
>> - (reg:SI VL_REGNUM)
>> - (reg:SI VTYPE_REGNUM)
>> - (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE)
>> - (plus_minus:VF
>> - (mult:VF
>> - (vec_duplicate:VF
>> - (match_operand:<VEL> 2 "register_operand" " f"))
>> - (match_operand:VF 3 "register_operand" " vr"))
>> - (match_operand:VF 4 "vector_arith_operand" " vr"))
>> - (match_operand:VF 5 "register_operand" " vr")))]
>> - "TARGET_VECTOR
>> - && !rtx_equal_p (operands[3], operands[5])
>> - && !rtx_equal_p (operands[4], operands[5])"
>> - "#"
>> - "&& reload_completed"
>> - {
>> - riscv_vector::prepare_ternary_operands (operands, true);
>> - }
>> - [(set_attr "type" "vfmuladd")
>> - (set_attr "mode" "<MODE>")
>> - (set (attr "frm_mode")
>> - (symbol_ref "riscv_vector::get_frm_mode (operands[10])"))])
>> -
>> (define_expand "@pred_mul_neg_<optab><mode>"
>> [(set (match_operand:VF 0 "register_operand")
>> (if_then_else:VF
>> @@ -6532,12 +6378,45 @@
>> (match_operand:VF 2 "register_operand")
>> (match_operand:VF 3 "register_operand")))
>> (match_operand:VF 4 "register_operand"))
>> - (match_operand:VF 5 "register_operand")))]
>> + (match_operand:VF 5 "vector_merge_operand")))]
>> "TARGET_VECTOR"
>> {
>> riscv_vector::prepare_ternary_operands (operands);
>> })
>> +(define_insn "*pred_mul_neg_<optab><mode>_undef"
>> + [(set (match_operand:VF 0 "register_operand" "=vd,vd,?&vd, vr,
>> vr,?&vr")
>> + (if_then_else:VF
>> + (unspec:<VM>
>> + [(match_operand:<VM> 1 "vector_mask_operand" " vm,vm,
>> vm,Wc1,Wc1, Wc1")
>> + (match_operand 6 "vector_length_operand" " rK,rK, rK, rK,
>> rK, rK")
>> + (match_operand 7 "const_int_operand" " i, i, i,
>> i, i, i")
>> + (match_operand 8 "const_int_operand" " i, i, i,
>> i, i, i")
>> + (match_operand 9 "const_int_operand" " i, i, i,
>> i, i, i")
>> + (match_operand 10 "const_int_operand" " i, i, i,
>> i, i, i")
>> + (reg:SI VL_REGNUM)
>> + (reg:SI VTYPE_REGNUM)
>> + (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE)
>> + (plus_minus:VF
>> + (neg:VF
>> + (mult:VF
>> + (match_operand:VF 3 "register_operand" " 0,vr, vr, 0,
>> vr, vr")
>> + (match_operand:VF 4 "register_operand" " vr,vr, vr, vr,
>> vr, vr")))
>> + (match_operand:VF 5 "register_operand" " vr, 0, vr,
>> vr, 0, vr"))
>> + (match_operand:VF 2 "vector_undef_operand")))]
>> + "TARGET_VECTOR"
>> + "@
>> + vf<nmsub_nmadd>.vv\t%0,%4,%5%p1
>> + vf<nmsac_nmacc>.vv\t%0,%3,%4%p1
>> + vmv.v.v\t%0,%3\;vf<nmsub_nmadd>.vv\t%0,%4,%5%p1
>> + vf<nmsub_nmadd>.vv\t%0,%4,%5%p1
>> + vf<nmsac_nmacc>.vv\t%0,%3,%4%p1
>> + vmv.v.v\t%0,%3\;vf<nmsub_nmadd>.vv\t%0,%4,%5%p1"
>> + [(set_attr "type" "vfmuladd")
>> + (set_attr "mode" "<MODE>")
>> + (set (attr "frm_mode")
>> + (symbol_ref "riscv_vector::get_frm_mode (operands[10])"))])
>> +
>> (define_insn "*pred_<nmsub_nmadd><mode>"
>> [(set (match_operand:VF 0 "register_operand" "=vd,
>> ?&vd, vr, ?&vr")
>> (if_then_else:VF
>> @@ -6610,40 +6489,6 @@
>> (set (attr "frm_mode")
>> (symbol_ref "riscv_vector::get_frm_mode (operands[9])"))])
>> -(define_insn_and_rewrite "*pred_mul_neg_<optab><mode>"
>> - [(set (match_operand:VF 0 "register_operand" "=&vr")
>> - (if_then_else:VF
>> - (unspec:<VM>
>> - [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
>> - (match_operand 6 "vector_length_operand" " rK")
>> - (match_operand 7 "const_int_operand" " i")
>> - (match_operand 8 "const_int_operand" " i")
>> - (match_operand 9 "const_int_operand" " i")
>> - (match_operand 10 "const_int_operand" " i")
>> - (reg:SI VL_REGNUM)
>> - (reg:SI VTYPE_REGNUM)
>> - (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE)
>> - (plus_minus:VF
>> - (neg:VF
>> - (mult:VF
>> - (match_operand:VF 2 "register_operand" " vr")
>> - (match_operand:VF 3 "register_operand" " vr")))
>> - (match_operand:VF 4 "vector_arith_operand" " vr"))
>> - (match_operand:VF 5 "register_operand" " vr")))]
>> - "TARGET_VECTOR
>> - && !rtx_equal_p (operands[2], operands[5])
>> - && !rtx_equal_p (operands[3], operands[5])
>> - && !rtx_equal_p (operands[4], operands[5])"
>> - "#"
>> - "&& reload_completed"
>> - {
>> - riscv_vector::prepare_ternary_operands (operands, true);
>> - }
>> - [(set_attr "type" "vfmuladd")
>> - (set_attr "mode" "<MODE>")
>> - (set (attr "frm_mode")
>> - (symbol_ref "riscv_vector::get_frm_mode (operands[10])"))])
>> -
>> (define_expand "@pred_mul_neg_<optab><mode>_scalar"
>> [(set (match_operand:VF 0 "register_operand")
>> (if_then_else:VF
>> @@ -6742,40 +6587,6 @@
>> (set (attr "frm_mode")
>> (symbol_ref "riscv_vector::get_frm_mode (operands[9])"))])
>> -(define_insn_and_rewrite "*pred_mul_neg_<optab><mode>_scalar"
>> - [(set (match_operand:VF 0 "register_operand"
>> "=&vr")
>> - (if_then_else:VF
>> - (unspec:<VM>
>> - [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1")
>> - (match_operand 6 "vector_length_operand" " rK")
>> - (match_operand 7 "const_int_operand" " i")
>> - (match_operand 8 "const_int_operand" " i")
>> - (match_operand 9 "const_int_operand" " i")
>> - (match_operand 10 "const_int_operand" " i")
>> - (reg:SI VL_REGNUM)
>> - (reg:SI VTYPE_REGNUM)
>> - (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE)
>> - (plus_minus:VF
>> - (neg:VF
>> - (mult:VF
>> - (vec_duplicate:VF
>> - (match_operand:<VEL> 2 "register_operand" " f"))
>> - (match_operand:VF 3 "register_operand" " vr")))
>> - (match_operand:VF 4 "vector_arith_operand" " vr"))
>> - (match_operand:VF 5 "register_operand" " vr")))]
>> - "TARGET_VECTOR
>> - && !rtx_equal_p (operands[3], operands[5])
>> - && !rtx_equal_p (operands[4], operands[5])"
>> - "#"
>> - "&& reload_completed"
>> - {
>> - riscv_vector::prepare_ternary_operands (operands, true);
>> - }
>> - [(set_attr "type" "vfmuladd")
>> - (set_attr "mode" "<MODE>")
>> - (set (attr "frm_mode")
>> - (symbol_ref "riscv_vector::get_frm_mode (operands[10])"))])
>> -
>> ;;
>> -------------------------------------------------------------------------------
>> ;; ---- Predicated floating-point unary operations
>> ;;
>> -------------------------------------------------------------------------------
>> --
>> 2.36.3
>>
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] RISC-V: Refactor and cleanup fma patterns
2023-09-19 20:32 ` Patrick O'Neill
@ 2023-09-19 21:53 ` Robin Dapp
2023-09-20 1:00 ` Lehua Ding
0 siblings, 1 reply; 7+ messages in thread
From: Robin Dapp @ 2023-09-19 21:53 UTC (permalink / raw)
To: Patrick O'Neill, Lehua Ding, juzhe.zhong, gcc-patches
Cc: rdapp.gcc, kito.cheng, palmer, jeffreyalaw
Hi Patrick,
thanks for reporting. Before seeing your message here I already opened a PR
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111488
Regards
Robin
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] RISC-V: Refactor and cleanup fma patterns
2023-09-19 21:53 ` Robin Dapp
@ 2023-09-20 1:00 ` Lehua Ding
2023-09-20 5:44 ` Lehua Ding
0 siblings, 1 reply; 7+ messages in thread
From: Lehua Ding @ 2023-09-20 1:00 UTC (permalink / raw)
To: Robin Dapp, Patrick O'Neill
Cc: kito.cheng, palmer, jeffreyalaw, juzhe.zhong, gcc-patches
Hi Patrick and Robin,
Thanks for reporting, I'll fix this as soon as I can.
On 2023/9/20 05:53, Robin Dapp wrote:
> Hi Patrick,
>
> thanks for reporting. Before seeing your message here I already opened a PR
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111488
>
> Regards
> Robin
>
>
--
Best,
Lehua
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] RISC-V: Refactor and cleanup fma patterns
2023-09-20 1:00 ` Lehua Ding
@ 2023-09-20 5:44 ` Lehua Ding
0 siblings, 0 replies; 7+ messages in thread
From: Lehua Ding @ 2023-09-20 5:44 UTC (permalink / raw)
To: Robin Dapp, Patrick O'Neill
Cc: kito.cheng, palmer, jeffreyalaw, juzhe.zhong, gcc-patches
Hi Patrick and Robin,
I fixed this ICE by this patch
(https://gcc.gnu.org/pipermail/gcc-patches/2023-September/630969.html).
I didn't repeat this problem at first because I was using -O0 compiled
GCC. It reappeared after it was changed to -O2. Sorry for introducing
this problem, and thanks again for reporting this bug.
On 2023/9/20 9:00, Lehua Ding wrote:
> Hi Patrick and Robin,
>
> Thanks for reporting, I'll fix this as soon as I can.
>
> On 2023/9/20 05:53, Robin Dapp wrote:
>> Hi Patrick,
>>
>> thanks for reporting. Before seeing your message here I already
>> opened a PR
>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111488
>>
>> Regards
>> Robin
>>
>>
>
--
Best,
Lehua
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2023-09-20 5:44 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-09-18 11:37 [PATCH] RISC-V: Refactor and cleanup fma patterns Lehua Ding
2023-09-19 3:10 ` juzhe.zhong
2023-09-19 3:23 ` Lehua Ding
2023-09-19 20:32 ` Patrick O'Neill
2023-09-19 21:53 ` Robin Dapp
2023-09-20 1:00 ` Lehua Ding
2023-09-20 5:44 ` Lehua Ding
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).