* [PATCH] Middle-end: Change order of LEN_MASK_LOAD/LEN_MASK_STORE arguments
@ 2023-07-03 4:07 juzhe.zhong
2023-07-03 7:45 ` Richard Sandiford
0 siblings, 1 reply; 2+ messages in thread
From: juzhe.zhong @ 2023-07-03 4:07 UTC (permalink / raw)
To: gcc-patches; +Cc: richard.sandiford, rguenther, Ju-Zhe Zhong
From: Ju-Zhe Zhong <juzhe.zhong@rivai.ai>
Hi, Richard and Richi.
According to Richard's review comments:
https://gcc.gnu.org/pipermail/gcc-patches/2023-July/623405.html
current len, bias and mask order is not reasonable.
Change {len,mask,bias} into {len,bias,mask}.
This patch is a simple fix patch, ok for trunk ?
gcc/ChangeLog:
* config/riscv/autovec.md: Change order of LEN_MASK_LOAD/LEN_MASK_STORE arguments.
* config/riscv/riscv-v.cc (expand_load_store): Ditto.
* doc/md.texi: Ditto.
* gimple-fold.cc (gimple_fold_partial_load_store_mem_ref): Ditto.
* internal-fn.cc (len_maskload_direct): Ditto.
(len_maskstore_direct): Ditto.
(add_len_bias_and_mask_args): Ditto.
(expand_partial_load_optab_fn): Ditto.
(expand_partial_store_optab_fn): Ditto.
(internal_fn_mask_index): Ditto.
(internal_fn_len_index): Ditto.
(internal_fn_bias_index): Ditto.
(internal_fn_stored_value_index): Ditto.
(internal_len_load_store_bias): Ditto.
* tree-ssa-dse.cc (initialize_ao_ref_for_dse): Ditto.
* tree-vect-stmts.cc (vectorizable_store): Ditto.
(vectorizable_load): Ditto.
---
gcc/config/riscv/autovec.md | 8 +-
gcc/config/riscv/riscv-v.cc | 2 +-
gcc/doc/md.texi | 16 ++--
gcc/gimple-fold.cc | 7 +-
gcc/internal-fn.cc | 176 +++++++++++++-----------------------
gcc/tree-ssa-dse.cc | 13 +--
gcc/tree-vect-stmts.cc | 6 +-
7 files changed, 90 insertions(+), 138 deletions(-)
diff --git a/gcc/config/riscv/autovec.md b/gcc/config/riscv/autovec.md
index 1488f2be1be..4ab0e9f99eb 100644
--- a/gcc/config/riscv/autovec.md
+++ b/gcc/config/riscv/autovec.md
@@ -26,8 +26,8 @@
[(match_operand:V 0 "register_operand")
(match_operand:V 1 "memory_operand")
(match_operand 2 "autovec_length_operand")
- (match_operand:<VM> 3 "vector_mask_operand")
- (match_operand 4 "const_0_operand")]
+ (match_operand 3 "const_0_operand")
+ (match_operand:<VM> 4 "vector_mask_operand")]
"TARGET_VECTOR"
{
riscv_vector::expand_load_store (operands, true);
@@ -38,8 +38,8 @@
[(match_operand:V 0 "memory_operand")
(match_operand:V 1 "register_operand")
(match_operand 2 "autovec_length_operand")
- (match_operand:<VM> 3 "vector_mask_operand")
- (match_operand 4 "const_0_operand")]
+ (match_operand 3 "const_0_operand")
+ (match_operand:<VM> 4 "vector_mask_operand")]
"TARGET_VECTOR"
{
riscv_vector::expand_load_store (operands, false);
diff --git a/gcc/config/riscv/riscv-v.cc b/gcc/config/riscv/riscv-v.cc
index adb8d7d36a5..8d5bed7ebe4 100644
--- a/gcc/config/riscv/riscv-v.cc
+++ b/gcc/config/riscv/riscv-v.cc
@@ -2777,7 +2777,7 @@ expand_load_store (rtx *ops, bool is_load)
{
poly_int64 value;
rtx len = ops[2];
- rtx mask = ops[3];
+ rtx mask = ops[4];
machine_mode mode = GET_MODE (ops[0]);
if (poly_int_rtx_p (len, &value) && known_eq (value, GET_MODE_NUNITS (mode)))
diff --git a/gcc/doc/md.texi b/gcc/doc/md.texi
index b44d1ba3af9..f14dd32b2dc 100644
--- a/gcc/doc/md.texi
+++ b/gcc/doc/md.texi
@@ -5319,15 +5319,15 @@ This pattern is not allowed to @code{FAIL}.
@cindex @code{len_maskload@var{m}@var{n}} instruction pattern
@item @samp{len_maskload@var{m}@var{n}}
Perform a masked load from the memory location pointed to by operand 1
-into register operand 0. (operand 2 + operand 4) elements are loaded from
+into register operand 0. (operand 2 + operand 3) elements are loaded from
memory and other elements in operand 0 are set to undefined values.
This is a combination of len_load and maskload.
Operands 0 and 1 have mode @var{m}, which must be a vector mode. Operand 2
has whichever integer mode the target prefers. A mask is specified in
-operand 3 which must be of type @var{n}. The mask has lower precedence than
+operand 4 which must be of type @var{n}. The mask has lower precedence than
the length and is itself subject to length masking,
-i.e. only mask indices < (operand 2 + operand 4) are used.
-Operand 4 conceptually has mode @code{QI}.
+i.e. only mask indices < (operand 2 + operand 3) are used.
+Operand 3 conceptually has mode @code{QI}.
Operand 2 can be a variable or a constant amount. Operand 4 specifies a
constant bias: it is either a constant 0 or a constant -1. The predicate on
@@ -5346,14 +5346,14 @@ This pattern is not allowed to @code{FAIL}.
@cindex @code{len_maskstore@var{m}@var{n}} instruction pattern
@item @samp{len_maskstore@var{m}@var{n}}
Perform a masked store from vector register operand 1 into memory operand 0.
-(operand 2 + operand 4) elements are stored to memory
+(operand 2 + operand 3) elements are stored to memory
and leave the other elements of operand 0 unchanged.
This is a combination of len_store and maskstore.
Operands 0 and 1 have mode @var{m}, which must be a vector mode. Operand 2 has whichever
-integer mode the target prefers. A mask is specified in operand 3 which must be
+integer mode the target prefers. A mask is specified in operand 4 which must be
of type @var{n}. The mask has lower precedence than the length and is itself subject to
-length masking, i.e. only mask indices < (operand 2 + operand 4) are used.
-Operand 4 conceptually has mode @code{QI}.
+length masking, i.e. only mask indices < (operand 2 + operand 3) are used.
+Operand 3 conceptually has mode @code{QI}.
Operand 2 can be a variable or a constant amount. Operand 3 specifies a
constant bias: it is either a constant 0 or a constant -1. The predicate on
diff --git a/gcc/gimple-fold.cc b/gcc/gimple-fold.cc
index 8434274f69d..d2d758f6435 100644
--- a/gcc/gimple-fold.cc
+++ b/gcc/gimple-fold.cc
@@ -5391,11 +5391,11 @@ gimple_fold_partial_load_store_mem_ref (gcall *call, tree vectype, bool mask_p)
}
else
{
- tree basic_len = gimple_call_arg (call, 2);
+ internal_fn ifn = gimple_call_internal_fn (call);
+ tree basic_len = gimple_call_arg (call, internal_fn_len_index (ifn));
if (!poly_int_tree_p (basic_len))
return NULL_TREE;
- unsigned int nargs = gimple_call_num_args (call);
- tree bias = gimple_call_arg (call, nargs - 1);
+ tree bias = gimple_call_arg (call, internal_fn_bias_index (ifn));
gcc_assert (TREE_CODE (bias) == INTEGER_CST);
/* For LEN_LOAD/LEN_STORE/LEN_MASK_LOAD/LEN_MASK_STORE,
we don't fold when (bias + len) != VF. */
@@ -5405,7 +5405,6 @@ gimple_fold_partial_load_store_mem_ref (gcall *call, tree vectype, bool mask_p)
/* For LEN_MASK_{LOAD,STORE}, we should also check whether
the mask is all ones mask. */
- internal_fn ifn = gimple_call_internal_fn (call);
if (ifn == IFN_LEN_MASK_LOAD || ifn == IFN_LEN_MASK_STORE)
{
tree mask = gimple_call_arg (call, internal_fn_mask_index (ifn));
diff --git a/gcc/internal-fn.cc b/gcc/internal-fn.cc
index 43ac5b37cc9..e394c05c895 100644
--- a/gcc/internal-fn.cc
+++ b/gcc/internal-fn.cc
@@ -165,7 +165,7 @@ init_internal_fns ()
#define mask_load_lanes_direct { -1, -1, false }
#define gather_load_direct { 3, 1, false }
#define len_load_direct { -1, -1, false }
-#define len_maskload_direct { -1, 3, false }
+#define len_maskload_direct { -1, 4, false }
#define mask_store_direct { 3, 2, false }
#define store_lanes_direct { 0, 0, false }
#define mask_store_lanes_direct { 0, 0, false }
@@ -173,7 +173,7 @@ init_internal_fns ()
#define vec_cond_direct { 2, 0, false }
#define scatter_store_direct { 3, 1, false }
#define len_store_direct { 3, 3, false }
-#define len_maskstore_direct { 4, 3, false }
+#define len_maskstore_direct { 4, 5, false }
#define vec_set_direct { 3, 3, false }
#define unary_direct { 0, 0, true }
#define unary_convert_direct { -1, 0, true }
@@ -293,6 +293,40 @@ get_multi_vector_move (tree array_type, convert_optab optab)
return convert_optab_handler (optab, imode, vmode);
}
+/* Add len, bias and mask arguments according to the STMT. */
+
+static unsigned int
+add_len_bias_and_mask_args (expand_operand *ops, unsigned int opno, gcall *stmt)
+{
+ internal_fn ifn = gimple_call_internal_fn (stmt);
+ int len_index = internal_fn_len_index (ifn);
+ int bias_index = internal_fn_bias_index (ifn);
+ int mask_index = internal_fn_mask_index (ifn);
+ /* The order of arguments are always {len,bias,mask}. */
+ if (len_index >= 0)
+ {
+ tree len = gimple_call_arg (stmt, len_index);
+ rtx len_rtx = expand_normal (len);
+ create_convert_operand_from (&ops[opno++], len_rtx,
+ TYPE_MODE (TREE_TYPE (len)),
+ TYPE_UNSIGNED (TREE_TYPE (len)));
+ }
+ if (bias_index >= 0)
+ {
+ tree biast = gimple_call_arg (stmt, bias_index);
+ rtx bias = expand_normal (biast);
+ create_input_operand (&ops[opno++], bias, QImode);
+ }
+ if (mask_index >= 0)
+ {
+ tree mask = gimple_call_arg (stmt, mask_index);
+ rtx mask_rtx = expand_normal (mask);
+ create_input_operand (&ops[opno++], mask_rtx,
+ TYPE_MODE (TREE_TYPE (mask)));
+ }
+ return opno;
+}
+
/* Expand LOAD_LANES call STMT using optab OPTAB. */
static void
@@ -2879,14 +2913,15 @@ expand_call_mem_ref (tree type, gcall *stmt, int index)
* OPTAB. */
static void
-expand_partial_load_optab_fn (internal_fn, gcall *stmt, convert_optab optab)
+expand_partial_load_optab_fn (internal_fn ifn, gcall *stmt, convert_optab optab)
{
+ int i = 0;
class expand_operand ops[5];
- tree type, lhs, rhs, maskt, biast;
- rtx mem, target, mask, bias;
+ tree type, lhs, rhs, maskt;
+ rtx mem, target;
insn_code icode;
- maskt = gimple_call_arg (stmt, 2);
+ maskt = gimple_call_arg (stmt, internal_fn_mask_index (ifn));
lhs = gimple_call_lhs (stmt);
if (lhs == NULL_TREE)
return;
@@ -2903,38 +2938,11 @@ expand_partial_load_optab_fn (internal_fn, gcall *stmt, convert_optab optab)
mem = expand_expr (rhs, NULL_RTX, VOIDmode, EXPAND_WRITE);
gcc_assert (MEM_P (mem));
- mask = expand_normal (maskt);
target = expand_expr (lhs, NULL_RTX, VOIDmode, EXPAND_WRITE);
- create_output_operand (&ops[0], target, TYPE_MODE (type));
- create_fixed_operand (&ops[1], mem);
- if (optab == len_load_optab)
- {
- create_convert_operand_from (&ops[2], mask, TYPE_MODE (TREE_TYPE (maskt)),
- TYPE_UNSIGNED (TREE_TYPE (maskt)));
- biast = gimple_call_arg (stmt, 3);
- bias = expand_normal (biast);
- create_input_operand (&ops[3], bias, QImode);
- expand_insn (icode, 4, ops);
- }
- else if (optab == len_maskload_optab)
- {
- create_convert_operand_from (&ops[2], mask, TYPE_MODE (TREE_TYPE (maskt)),
- TYPE_UNSIGNED (TREE_TYPE (maskt)));
- maskt = gimple_call_arg (stmt, 3);
- mask = expand_normal (maskt);
- create_input_operand (&ops[3], mask, TYPE_MODE (TREE_TYPE (maskt)));
- icode = convert_optab_handler (optab, TYPE_MODE (type),
- TYPE_MODE (TREE_TYPE (maskt)));
- biast = gimple_call_arg (stmt, 4);
- bias = expand_normal (biast);
- create_input_operand (&ops[4], bias, QImode);
- expand_insn (icode, 5, ops);
- }
- else
- {
- create_input_operand (&ops[2], mask, TYPE_MODE (TREE_TYPE (maskt)));
- expand_insn (icode, 3, ops);
- }
+ create_output_operand (&ops[i++], target, TYPE_MODE (type));
+ create_fixed_operand (&ops[i++], mem);
+ i = add_len_bias_and_mask_args (ops, i, stmt);
+ expand_insn (icode, i, ops);
if (!rtx_equal_p (target, ops[0].value))
emit_move_insn (target, ops[0].value);
@@ -2951,12 +2959,13 @@ expand_partial_load_optab_fn (internal_fn, gcall *stmt, convert_optab optab)
static void
expand_partial_store_optab_fn (internal_fn ifn, gcall *stmt, convert_optab optab)
{
+ int i = 0;
class expand_operand ops[5];
- tree type, lhs, rhs, maskt, biast;
- rtx mem, reg, mask, bias;
+ tree type, lhs, rhs, maskt;
+ rtx mem, reg;
insn_code icode;
- maskt = gimple_call_arg (stmt, 2);
+ maskt = gimple_call_arg (stmt, internal_fn_mask_index (ifn));
rhs = gimple_call_arg (stmt, internal_fn_stored_value_index (ifn));
type = TREE_TYPE (rhs);
lhs = expand_call_mem_ref (type, stmt, 0);
@@ -2971,37 +2980,11 @@ expand_partial_store_optab_fn (internal_fn ifn, gcall *stmt, convert_optab optab
mem = expand_expr (lhs, NULL_RTX, VOIDmode, EXPAND_WRITE);
gcc_assert (MEM_P (mem));
- mask = expand_normal (maskt);
reg = expand_normal (rhs);
- create_fixed_operand (&ops[0], mem);
- create_input_operand (&ops[1], reg, TYPE_MODE (type));
- if (optab == len_store_optab)
- {
- create_convert_operand_from (&ops[2], mask, TYPE_MODE (TREE_TYPE (maskt)),
- TYPE_UNSIGNED (TREE_TYPE (maskt)));
- biast = gimple_call_arg (stmt, 4);
- bias = expand_normal (biast);
- create_input_operand (&ops[3], bias, QImode);
- expand_insn (icode, 4, ops);
- }
- else if (optab == len_maskstore_optab)
- {
- create_convert_operand_from (&ops[2], mask, TYPE_MODE (TREE_TYPE (maskt)),
- TYPE_UNSIGNED (TREE_TYPE (maskt)));
- maskt = gimple_call_arg (stmt, 3);
- mask = expand_normal (maskt);
- create_input_operand (&ops[3], mask, TYPE_MODE (TREE_TYPE (maskt)));
- biast = gimple_call_arg (stmt, 5);
- bias = expand_normal (biast);
- create_input_operand (&ops[4], bias, QImode);
- icode = convert_optab_handler (optab, TYPE_MODE (type), GET_MODE (mask));
- expand_insn (icode, 5, ops);
- }
- else
- {
- create_input_operand (&ops[2], mask, TYPE_MODE (TREE_TYPE (maskt)));
- expand_insn (icode, 3, ops);
- }
+ create_fixed_operand (&ops[i++], mem);
+ create_input_operand (&ops[i++], reg, TYPE_MODE (type));
+ i = add_len_bias_and_mask_args (ops, i, stmt);
+ expand_insn (icode, i, ops);
}
#define expand_mask_store_optab_fn expand_partial_store_optab_fn
@@ -3519,40 +3502,6 @@ expand_LAUNDER (internal_fn, gcall *call)
expand_assignment (lhs, gimple_call_arg (call, 0), false);
}
-/* Add len, bias and mask arguments according to the STMT. */
-
-static unsigned int
-add_len_bias_and_mask_args (expand_operand *ops, unsigned int opno, gcall *stmt)
-{
- internal_fn ifn = gimple_call_internal_fn (stmt);
- int len_index = internal_fn_len_index (ifn);
- int bias_index = internal_fn_bias_index (ifn);
- int mask_index = internal_fn_mask_index (ifn);
- /* The order of arguments are always {len,bias,mask}. */
- if (len_index >= 0)
- {
- tree len = gimple_call_arg (stmt, len_index);
- rtx len_rtx = expand_normal (len);
- create_convert_operand_from (&ops[opno++], len_rtx,
- TYPE_MODE (TREE_TYPE (len)),
- TYPE_UNSIGNED (TREE_TYPE (len)));
- }
- if (bias_index >= 0)
- {
- tree biast = gimple_call_arg (stmt, bias_index);
- rtx bias = expand_normal (biast);
- create_input_operand (&ops[opno++], bias, QImode);
- }
- if (mask_index >= 0)
- {
- tree mask = gimple_call_arg (stmt, mask_index);
- rtx mask_rtx = expand_normal (mask);
- create_input_operand (&ops[opno++], mask_rtx,
- TYPE_MODE (TREE_TYPE (mask)));
- }
- return opno;
-}
-
/* Expand {MASK_,}SCATTER_STORE{S,U} call CALL using optab OPTAB. */
static void
@@ -4532,8 +4481,9 @@ internal_fn_mask_index (internal_fn fn)
return 6;
case IFN_LEN_MASK_LOAD:
+ return 4;
case IFN_LEN_MASK_STORE:
- return 3;
+ return 5;
default:
return (conditional_internal_fn_code (fn) != ERROR_MARK
@@ -4550,8 +4500,10 @@ internal_fn_len_index (internal_fn fn)
switch (fn)
{
case IFN_LEN_LOAD:
- case IFN_LEN_STORE:
case IFN_LEN_MASK_LOAD:
+ return 2;
+
+ case IFN_LEN_STORE:
case IFN_LEN_MASK_STORE:
return 3;
@@ -4573,8 +4525,10 @@ internal_fn_bias_index (internal_fn fn)
switch (fn)
{
case IFN_LEN_LOAD:
- case IFN_LEN_STORE:
case IFN_LEN_MASK_LOAD:
+ return 3;
+
+ case IFN_LEN_STORE:
case IFN_LEN_MASK_STORE:
return 4;
@@ -4604,7 +4558,7 @@ internal_fn_stored_value_index (internal_fn fn)
return 3;
case IFN_LEN_MASK_STORE:
- return 4;
+ return 2;
default:
return -1;
@@ -4670,7 +4624,6 @@ internal_len_load_store_bias (internal_fn ifn, machine_mode mode)
{
optab optab = direct_internal_fn_optab (ifn);
insn_code icode = direct_optab_handler (optab, mode);
- int bias_opno = 3;
if (icode == CODE_FOR_nothing)
{
@@ -4688,15 +4641,14 @@ internal_len_load_store_bias (internal_fn ifn, machine_mode mode)
optab = direct_internal_fn_optab (IFN_LEN_MASK_STORE);
}
icode = convert_optab_handler (optab, mode, mask_mode);
- bias_opno = 4;
}
if (icode != CODE_FOR_nothing)
{
/* For now we only support biases of 0 or -1. Try both of them. */
- if (insn_operand_matches (icode, bias_opno, GEN_INT (0)))
+ if (insn_operand_matches (icode, 3, GEN_INT (0)))
return 0;
- if (insn_operand_matches (icode, bias_opno, GEN_INT (-1)))
+ if (insn_operand_matches (icode, 3, GEN_INT (-1)))
return -1;
}
diff --git a/gcc/tree-ssa-dse.cc b/gcc/tree-ssa-dse.cc
index f8338037a61..aa58f7b900d 100644
--- a/gcc/tree-ssa-dse.cc
+++ b/gcc/tree-ssa-dse.cc
@@ -159,14 +159,15 @@ initialize_ao_ref_for_dse (gimple *stmt, ao_ref *write, bool may_def_ok = false)
{
case IFN_LEN_STORE:
case IFN_MASK_STORE:
- case IFN_LEN_MASK_STORE:
+ case IFN_LEN_MASK_STORE:
{
- int stored_value_index
- = internal_fn_stored_value_index (gimple_call_internal_fn (stmt));
- if (gimple_call_internal_fn (stmt) == IFN_LEN_STORE)
+ internal_fn ifn = gimple_call_internal_fn (stmt);
+ int stored_value_index = internal_fn_stored_value_index (ifn);
+ if (ifn == IFN_LEN_STORE)
{
- tree len = gimple_call_arg (stmt, 2);
- tree bias = gimple_call_arg (stmt, 4);
+ tree len = gimple_call_arg (stmt, internal_fn_len_index (ifn));
+ tree bias
+ = gimple_call_arg (stmt, internal_fn_bias_index (ifn));
if (tree_fits_uhwi_p (len))
{
ao_ref_init_from_ptr_and_size (write,
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index 68faa8ead39..0c61e123e5e 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -9122,8 +9122,8 @@ vectorizable_store (vec_info *vinfo,
if (partial_ifn == IFN_LEN_MASK_STORE)
call = gimple_build_call_internal (IFN_LEN_MASK_STORE, 6,
dataref_ptr, ptr,
- final_len, final_mask,
- vec_oprnd, bias);
+ vec_oprnd, final_len,
+ bias, final_mask);
else
call
= gimple_build_call_internal (IFN_LEN_STORE, 5,
@@ -10523,7 +10523,7 @@ vectorizable_load (vec_info *vinfo,
call = gimple_build_call_internal (IFN_LEN_MASK_LOAD,
5, dataref_ptr,
ptr, final_len,
- final_mask, bias);
+ bias, final_mask);
else
call = gimple_build_call_internal (IFN_LEN_LOAD, 4,
dataref_ptr, ptr,
--
2.36.1
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: [PATCH] Middle-end: Change order of LEN_MASK_LOAD/LEN_MASK_STORE arguments
2023-07-03 4:07 [PATCH] Middle-end: Change order of LEN_MASK_LOAD/LEN_MASK_STORE arguments juzhe.zhong
@ 2023-07-03 7:45 ` Richard Sandiford
0 siblings, 0 replies; 2+ messages in thread
From: Richard Sandiford @ 2023-07-03 7:45 UTC (permalink / raw)
To: juzhe.zhong; +Cc: gcc-patches, rguenther
juzhe.zhong@rivai.ai writes:
> From: Ju-Zhe Zhong <juzhe.zhong@rivai.ai>
>
> Hi, Richard and Richi.
>
> According to Richard's review comments:
> https://gcc.gnu.org/pipermail/gcc-patches/2023-July/623405.html
>
> current len, bias and mask order is not reasonable.
>
> Change {len,mask,bias} into {len,bias,mask}.
>
> This patch is a simple fix patch, ok for trunk ?
Sorry to be awkward, but can you put the patches in the opposite order,
so that this one comes first, and then the gather/scatter patch comes
second?
I don't think we want internal_fn_bias_index. The point of reordering
the arguments was to put the bias immediately after the length in all
cases. In other words, the length and the bias form a pair: the bias
only exists if the length does, and the length only exists if the bias
does.
In other respects the patch LGTM. Thanks for doing this.
Richard
>
> gcc/ChangeLog:
>
> * config/riscv/autovec.md: Change order of LEN_MASK_LOAD/LEN_MASK_STORE arguments.
> * config/riscv/riscv-v.cc (expand_load_store): Ditto.
> * doc/md.texi: Ditto.
> * gimple-fold.cc (gimple_fold_partial_load_store_mem_ref): Ditto.
> * internal-fn.cc (len_maskload_direct): Ditto.
> (len_maskstore_direct): Ditto.
> (add_len_bias_and_mask_args): Ditto.
> (expand_partial_load_optab_fn): Ditto.
> (expand_partial_store_optab_fn): Ditto.
> (internal_fn_mask_index): Ditto.
> (internal_fn_len_index): Ditto.
> (internal_fn_bias_index): Ditto.
> (internal_fn_stored_value_index): Ditto.
> (internal_len_load_store_bias): Ditto.
> * tree-ssa-dse.cc (initialize_ao_ref_for_dse): Ditto.
> * tree-vect-stmts.cc (vectorizable_store): Ditto.
> (vectorizable_load): Ditto.
>
> ---
> gcc/config/riscv/autovec.md | 8 +-
> gcc/config/riscv/riscv-v.cc | 2 +-
> gcc/doc/md.texi | 16 ++--
> gcc/gimple-fold.cc | 7 +-
> gcc/internal-fn.cc | 176 +++++++++++++-----------------------
> gcc/tree-ssa-dse.cc | 13 +--
> gcc/tree-vect-stmts.cc | 6 +-
> 7 files changed, 90 insertions(+), 138 deletions(-)
>
> diff --git a/gcc/config/riscv/autovec.md b/gcc/config/riscv/autovec.md
> index 1488f2be1be..4ab0e9f99eb 100644
> --- a/gcc/config/riscv/autovec.md
> +++ b/gcc/config/riscv/autovec.md
> @@ -26,8 +26,8 @@
> [(match_operand:V 0 "register_operand")
> (match_operand:V 1 "memory_operand")
> (match_operand 2 "autovec_length_operand")
> - (match_operand:<VM> 3 "vector_mask_operand")
> - (match_operand 4 "const_0_operand")]
> + (match_operand 3 "const_0_operand")
> + (match_operand:<VM> 4 "vector_mask_operand")]
> "TARGET_VECTOR"
> {
> riscv_vector::expand_load_store (operands, true);
> @@ -38,8 +38,8 @@
> [(match_operand:V 0 "memory_operand")
> (match_operand:V 1 "register_operand")
> (match_operand 2 "autovec_length_operand")
> - (match_operand:<VM> 3 "vector_mask_operand")
> - (match_operand 4 "const_0_operand")]
> + (match_operand 3 "const_0_operand")
> + (match_operand:<VM> 4 "vector_mask_operand")]
> "TARGET_VECTOR"
> {
> riscv_vector::expand_load_store (operands, false);
> diff --git a/gcc/config/riscv/riscv-v.cc b/gcc/config/riscv/riscv-v.cc
> index adb8d7d36a5..8d5bed7ebe4 100644
> --- a/gcc/config/riscv/riscv-v.cc
> +++ b/gcc/config/riscv/riscv-v.cc
> @@ -2777,7 +2777,7 @@ expand_load_store (rtx *ops, bool is_load)
> {
> poly_int64 value;
> rtx len = ops[2];
> - rtx mask = ops[3];
> + rtx mask = ops[4];
> machine_mode mode = GET_MODE (ops[0]);
>
> if (poly_int_rtx_p (len, &value) && known_eq (value, GET_MODE_NUNITS (mode)))
> diff --git a/gcc/doc/md.texi b/gcc/doc/md.texi
> index b44d1ba3af9..f14dd32b2dc 100644
> --- a/gcc/doc/md.texi
> +++ b/gcc/doc/md.texi
> @@ -5319,15 +5319,15 @@ This pattern is not allowed to @code{FAIL}.
> @cindex @code{len_maskload@var{m}@var{n}} instruction pattern
> @item @samp{len_maskload@var{m}@var{n}}
> Perform a masked load from the memory location pointed to by operand 1
> -into register operand 0. (operand 2 + operand 4) elements are loaded from
> +into register operand 0. (operand 2 + operand 3) elements are loaded from
> memory and other elements in operand 0 are set to undefined values.
> This is a combination of len_load and maskload.
> Operands 0 and 1 have mode @var{m}, which must be a vector mode. Operand 2
> has whichever integer mode the target prefers. A mask is specified in
> -operand 3 which must be of type @var{n}. The mask has lower precedence than
> +operand 4 which must be of type @var{n}. The mask has lower precedence than
> the length and is itself subject to length masking,
> -i.e. only mask indices < (operand 2 + operand 4) are used.
> -Operand 4 conceptually has mode @code{QI}.
> +i.e. only mask indices < (operand 2 + operand 3) are used.
> +Operand 3 conceptually has mode @code{QI}.
>
> Operand 2 can be a variable or a constant amount. Operand 4 specifies a
> constant bias: it is either a constant 0 or a constant -1. The predicate on
> @@ -5346,14 +5346,14 @@ This pattern is not allowed to @code{FAIL}.
> @cindex @code{len_maskstore@var{m}@var{n}} instruction pattern
> @item @samp{len_maskstore@var{m}@var{n}}
> Perform a masked store from vector register operand 1 into memory operand 0.
> -(operand 2 + operand 4) elements are stored to memory
> +(operand 2 + operand 3) elements are stored to memory
> and leave the other elements of operand 0 unchanged.
> This is a combination of len_store and maskstore.
> Operands 0 and 1 have mode @var{m}, which must be a vector mode. Operand 2 has whichever
> -integer mode the target prefers. A mask is specified in operand 3 which must be
> +integer mode the target prefers. A mask is specified in operand 4 which must be
> of type @var{n}. The mask has lower precedence than the length and is itself subject to
> -length masking, i.e. only mask indices < (operand 2 + operand 4) are used.
> -Operand 4 conceptually has mode @code{QI}.
> +length masking, i.e. only mask indices < (operand 2 + operand 3) are used.
> +Operand 3 conceptually has mode @code{QI}.
>
> Operand 2 can be a variable or a constant amount. Operand 3 specifies a
> constant bias: it is either a constant 0 or a constant -1. The predicate on
> diff --git a/gcc/gimple-fold.cc b/gcc/gimple-fold.cc
> index 8434274f69d..d2d758f6435 100644
> --- a/gcc/gimple-fold.cc
> +++ b/gcc/gimple-fold.cc
> @@ -5391,11 +5391,11 @@ gimple_fold_partial_load_store_mem_ref (gcall *call, tree vectype, bool mask_p)
> }
> else
> {
> - tree basic_len = gimple_call_arg (call, 2);
> + internal_fn ifn = gimple_call_internal_fn (call);
> + tree basic_len = gimple_call_arg (call, internal_fn_len_index (ifn));
> if (!poly_int_tree_p (basic_len))
> return NULL_TREE;
> - unsigned int nargs = gimple_call_num_args (call);
> - tree bias = gimple_call_arg (call, nargs - 1);
> + tree bias = gimple_call_arg (call, internal_fn_bias_index (ifn));
> gcc_assert (TREE_CODE (bias) == INTEGER_CST);
> /* For LEN_LOAD/LEN_STORE/LEN_MASK_LOAD/LEN_MASK_STORE,
> we don't fold when (bias + len) != VF. */
> @@ -5405,7 +5405,6 @@ gimple_fold_partial_load_store_mem_ref (gcall *call, tree vectype, bool mask_p)
>
> /* For LEN_MASK_{LOAD,STORE}, we should also check whether
> the mask is all ones mask. */
> - internal_fn ifn = gimple_call_internal_fn (call);
> if (ifn == IFN_LEN_MASK_LOAD || ifn == IFN_LEN_MASK_STORE)
> {
> tree mask = gimple_call_arg (call, internal_fn_mask_index (ifn));
> diff --git a/gcc/internal-fn.cc b/gcc/internal-fn.cc
> index 43ac5b37cc9..e394c05c895 100644
> --- a/gcc/internal-fn.cc
> +++ b/gcc/internal-fn.cc
> @@ -165,7 +165,7 @@ init_internal_fns ()
> #define mask_load_lanes_direct { -1, -1, false }
> #define gather_load_direct { 3, 1, false }
> #define len_load_direct { -1, -1, false }
> -#define len_maskload_direct { -1, 3, false }
> +#define len_maskload_direct { -1, 4, false }
> #define mask_store_direct { 3, 2, false }
> #define store_lanes_direct { 0, 0, false }
> #define mask_store_lanes_direct { 0, 0, false }
> @@ -173,7 +173,7 @@ init_internal_fns ()
> #define vec_cond_direct { 2, 0, false }
> #define scatter_store_direct { 3, 1, false }
> #define len_store_direct { 3, 3, false }
> -#define len_maskstore_direct { 4, 3, false }
> +#define len_maskstore_direct { 4, 5, false }
> #define vec_set_direct { 3, 3, false }
> #define unary_direct { 0, 0, true }
> #define unary_convert_direct { -1, 0, true }
> @@ -293,6 +293,40 @@ get_multi_vector_move (tree array_type, convert_optab optab)
> return convert_optab_handler (optab, imode, vmode);
> }
>
> +/* Add len, bias and mask arguments according to the STMT. */
> +
> +static unsigned int
> +add_len_bias_and_mask_args (expand_operand *ops, unsigned int opno, gcall *stmt)
> +{
> + internal_fn ifn = gimple_call_internal_fn (stmt);
> + int len_index = internal_fn_len_index (ifn);
> + int bias_index = internal_fn_bias_index (ifn);
> + int mask_index = internal_fn_mask_index (ifn);
> + /* The order of arguments are always {len,bias,mask}. */
> + if (len_index >= 0)
> + {
> + tree len = gimple_call_arg (stmt, len_index);
> + rtx len_rtx = expand_normal (len);
> + create_convert_operand_from (&ops[opno++], len_rtx,
> + TYPE_MODE (TREE_TYPE (len)),
> + TYPE_UNSIGNED (TREE_TYPE (len)));
> + }
> + if (bias_index >= 0)
> + {
> + tree biast = gimple_call_arg (stmt, bias_index);
> + rtx bias = expand_normal (biast);
> + create_input_operand (&ops[opno++], bias, QImode);
> + }
> + if (mask_index >= 0)
> + {
> + tree mask = gimple_call_arg (stmt, mask_index);
> + rtx mask_rtx = expand_normal (mask);
> + create_input_operand (&ops[opno++], mask_rtx,
> + TYPE_MODE (TREE_TYPE (mask)));
> + }
> + return opno;
> +}
> +
> /* Expand LOAD_LANES call STMT using optab OPTAB. */
>
> static void
> @@ -2879,14 +2913,15 @@ expand_call_mem_ref (tree type, gcall *stmt, int index)
> * OPTAB. */
>
> static void
> -expand_partial_load_optab_fn (internal_fn, gcall *stmt, convert_optab optab)
> +expand_partial_load_optab_fn (internal_fn ifn, gcall *stmt, convert_optab optab)
> {
> + int i = 0;
> class expand_operand ops[5];
> - tree type, lhs, rhs, maskt, biast;
> - rtx mem, target, mask, bias;
> + tree type, lhs, rhs, maskt;
> + rtx mem, target;
> insn_code icode;
>
> - maskt = gimple_call_arg (stmt, 2);
> + maskt = gimple_call_arg (stmt, internal_fn_mask_index (ifn));
> lhs = gimple_call_lhs (stmt);
> if (lhs == NULL_TREE)
> return;
> @@ -2903,38 +2938,11 @@ expand_partial_load_optab_fn (internal_fn, gcall *stmt, convert_optab optab)
>
> mem = expand_expr (rhs, NULL_RTX, VOIDmode, EXPAND_WRITE);
> gcc_assert (MEM_P (mem));
> - mask = expand_normal (maskt);
> target = expand_expr (lhs, NULL_RTX, VOIDmode, EXPAND_WRITE);
> - create_output_operand (&ops[0], target, TYPE_MODE (type));
> - create_fixed_operand (&ops[1], mem);
> - if (optab == len_load_optab)
> - {
> - create_convert_operand_from (&ops[2], mask, TYPE_MODE (TREE_TYPE (maskt)),
> - TYPE_UNSIGNED (TREE_TYPE (maskt)));
> - biast = gimple_call_arg (stmt, 3);
> - bias = expand_normal (biast);
> - create_input_operand (&ops[3], bias, QImode);
> - expand_insn (icode, 4, ops);
> - }
> - else if (optab == len_maskload_optab)
> - {
> - create_convert_operand_from (&ops[2], mask, TYPE_MODE (TREE_TYPE (maskt)),
> - TYPE_UNSIGNED (TREE_TYPE (maskt)));
> - maskt = gimple_call_arg (stmt, 3);
> - mask = expand_normal (maskt);
> - create_input_operand (&ops[3], mask, TYPE_MODE (TREE_TYPE (maskt)));
> - icode = convert_optab_handler (optab, TYPE_MODE (type),
> - TYPE_MODE (TREE_TYPE (maskt)));
> - biast = gimple_call_arg (stmt, 4);
> - bias = expand_normal (biast);
> - create_input_operand (&ops[4], bias, QImode);
> - expand_insn (icode, 5, ops);
> - }
> - else
> - {
> - create_input_operand (&ops[2], mask, TYPE_MODE (TREE_TYPE (maskt)));
> - expand_insn (icode, 3, ops);
> - }
> + create_output_operand (&ops[i++], target, TYPE_MODE (type));
> + create_fixed_operand (&ops[i++], mem);
> + i = add_len_bias_and_mask_args (ops, i, stmt);
> + expand_insn (icode, i, ops);
>
> if (!rtx_equal_p (target, ops[0].value))
> emit_move_insn (target, ops[0].value);
> @@ -2951,12 +2959,13 @@ expand_partial_load_optab_fn (internal_fn, gcall *stmt, convert_optab optab)
> static void
> expand_partial_store_optab_fn (internal_fn ifn, gcall *stmt, convert_optab optab)
> {
> + int i = 0;
> class expand_operand ops[5];
> - tree type, lhs, rhs, maskt, biast;
> - rtx mem, reg, mask, bias;
> + tree type, lhs, rhs, maskt;
> + rtx mem, reg;
> insn_code icode;
>
> - maskt = gimple_call_arg (stmt, 2);
> + maskt = gimple_call_arg (stmt, internal_fn_mask_index (ifn));
> rhs = gimple_call_arg (stmt, internal_fn_stored_value_index (ifn));
> type = TREE_TYPE (rhs);
> lhs = expand_call_mem_ref (type, stmt, 0);
> @@ -2971,37 +2980,11 @@ expand_partial_store_optab_fn (internal_fn ifn, gcall *stmt, convert_optab optab
>
> mem = expand_expr (lhs, NULL_RTX, VOIDmode, EXPAND_WRITE);
> gcc_assert (MEM_P (mem));
> - mask = expand_normal (maskt);
> reg = expand_normal (rhs);
> - create_fixed_operand (&ops[0], mem);
> - create_input_operand (&ops[1], reg, TYPE_MODE (type));
> - if (optab == len_store_optab)
> - {
> - create_convert_operand_from (&ops[2], mask, TYPE_MODE (TREE_TYPE (maskt)),
> - TYPE_UNSIGNED (TREE_TYPE (maskt)));
> - biast = gimple_call_arg (stmt, 4);
> - bias = expand_normal (biast);
> - create_input_operand (&ops[3], bias, QImode);
> - expand_insn (icode, 4, ops);
> - }
> - else if (optab == len_maskstore_optab)
> - {
> - create_convert_operand_from (&ops[2], mask, TYPE_MODE (TREE_TYPE (maskt)),
> - TYPE_UNSIGNED (TREE_TYPE (maskt)));
> - maskt = gimple_call_arg (stmt, 3);
> - mask = expand_normal (maskt);
> - create_input_operand (&ops[3], mask, TYPE_MODE (TREE_TYPE (maskt)));
> - biast = gimple_call_arg (stmt, 5);
> - bias = expand_normal (biast);
> - create_input_operand (&ops[4], bias, QImode);
> - icode = convert_optab_handler (optab, TYPE_MODE (type), GET_MODE (mask));
> - expand_insn (icode, 5, ops);
> - }
> - else
> - {
> - create_input_operand (&ops[2], mask, TYPE_MODE (TREE_TYPE (maskt)));
> - expand_insn (icode, 3, ops);
> - }
> + create_fixed_operand (&ops[i++], mem);
> + create_input_operand (&ops[i++], reg, TYPE_MODE (type));
> + i = add_len_bias_and_mask_args (ops, i, stmt);
> + expand_insn (icode, i, ops);
> }
>
> #define expand_mask_store_optab_fn expand_partial_store_optab_fn
> @@ -3519,40 +3502,6 @@ expand_LAUNDER (internal_fn, gcall *call)
> expand_assignment (lhs, gimple_call_arg (call, 0), false);
> }
>
> -/* Add len, bias and mask arguments according to the STMT. */
> -
> -static unsigned int
> -add_len_bias_and_mask_args (expand_operand *ops, unsigned int opno, gcall *stmt)
> -{
> - internal_fn ifn = gimple_call_internal_fn (stmt);
> - int len_index = internal_fn_len_index (ifn);
> - int bias_index = internal_fn_bias_index (ifn);
> - int mask_index = internal_fn_mask_index (ifn);
> - /* The order of arguments are always {len,bias,mask}. */
> - if (len_index >= 0)
> - {
> - tree len = gimple_call_arg (stmt, len_index);
> - rtx len_rtx = expand_normal (len);
> - create_convert_operand_from (&ops[opno++], len_rtx,
> - TYPE_MODE (TREE_TYPE (len)),
> - TYPE_UNSIGNED (TREE_TYPE (len)));
> - }
> - if (bias_index >= 0)
> - {
> - tree biast = gimple_call_arg (stmt, bias_index);
> - rtx bias = expand_normal (biast);
> - create_input_operand (&ops[opno++], bias, QImode);
> - }
> - if (mask_index >= 0)
> - {
> - tree mask = gimple_call_arg (stmt, mask_index);
> - rtx mask_rtx = expand_normal (mask);
> - create_input_operand (&ops[opno++], mask_rtx,
> - TYPE_MODE (TREE_TYPE (mask)));
> - }
> - return opno;
> -}
> -
> /* Expand {MASK_,}SCATTER_STORE{S,U} call CALL using optab OPTAB. */
>
> static void
> @@ -4532,8 +4481,9 @@ internal_fn_mask_index (internal_fn fn)
> return 6;
>
> case IFN_LEN_MASK_LOAD:
> + return 4;
> case IFN_LEN_MASK_STORE:
> - return 3;
> + return 5;
>
> default:
> return (conditional_internal_fn_code (fn) != ERROR_MARK
> @@ -4550,8 +4500,10 @@ internal_fn_len_index (internal_fn fn)
> switch (fn)
> {
> case IFN_LEN_LOAD:
> - case IFN_LEN_STORE:
> case IFN_LEN_MASK_LOAD:
> + return 2;
> +
> + case IFN_LEN_STORE:
> case IFN_LEN_MASK_STORE:
> return 3;
>
> @@ -4573,8 +4525,10 @@ internal_fn_bias_index (internal_fn fn)
> switch (fn)
> {
> case IFN_LEN_LOAD:
> - case IFN_LEN_STORE:
> case IFN_LEN_MASK_LOAD:
> + return 3;
> +
> + case IFN_LEN_STORE:
> case IFN_LEN_MASK_STORE:
> return 4;
>
> @@ -4604,7 +4558,7 @@ internal_fn_stored_value_index (internal_fn fn)
> return 3;
>
> case IFN_LEN_MASK_STORE:
> - return 4;
> + return 2;
>
> default:
> return -1;
> @@ -4670,7 +4624,6 @@ internal_len_load_store_bias (internal_fn ifn, machine_mode mode)
> {
> optab optab = direct_internal_fn_optab (ifn);
> insn_code icode = direct_optab_handler (optab, mode);
> - int bias_opno = 3;
>
> if (icode == CODE_FOR_nothing)
> {
> @@ -4688,15 +4641,14 @@ internal_len_load_store_bias (internal_fn ifn, machine_mode mode)
> optab = direct_internal_fn_optab (IFN_LEN_MASK_STORE);
> }
> icode = convert_optab_handler (optab, mode, mask_mode);
> - bias_opno = 4;
> }
>
> if (icode != CODE_FOR_nothing)
> {
> /* For now we only support biases of 0 or -1. Try both of them. */
> - if (insn_operand_matches (icode, bias_opno, GEN_INT (0)))
> + if (insn_operand_matches (icode, 3, GEN_INT (0)))
> return 0;
> - if (insn_operand_matches (icode, bias_opno, GEN_INT (-1)))
> + if (insn_operand_matches (icode, 3, GEN_INT (-1)))
> return -1;
> }
>
> diff --git a/gcc/tree-ssa-dse.cc b/gcc/tree-ssa-dse.cc
> index f8338037a61..aa58f7b900d 100644
> --- a/gcc/tree-ssa-dse.cc
> +++ b/gcc/tree-ssa-dse.cc
> @@ -159,14 +159,15 @@ initialize_ao_ref_for_dse (gimple *stmt, ao_ref *write, bool may_def_ok = false)
> {
> case IFN_LEN_STORE:
> case IFN_MASK_STORE:
> - case IFN_LEN_MASK_STORE:
> + case IFN_LEN_MASK_STORE:
> {
> - int stored_value_index
> - = internal_fn_stored_value_index (gimple_call_internal_fn (stmt));
> - if (gimple_call_internal_fn (stmt) == IFN_LEN_STORE)
> + internal_fn ifn = gimple_call_internal_fn (stmt);
> + int stored_value_index = internal_fn_stored_value_index (ifn);
> + if (ifn == IFN_LEN_STORE)
> {
> - tree len = gimple_call_arg (stmt, 2);
> - tree bias = gimple_call_arg (stmt, 4);
> + tree len = gimple_call_arg (stmt, internal_fn_len_index (ifn));
> + tree bias
> + = gimple_call_arg (stmt, internal_fn_bias_index (ifn));
> if (tree_fits_uhwi_p (len))
> {
> ao_ref_init_from_ptr_and_size (write,
> diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
> index 68faa8ead39..0c61e123e5e 100644
> --- a/gcc/tree-vect-stmts.cc
> +++ b/gcc/tree-vect-stmts.cc
> @@ -9122,8 +9122,8 @@ vectorizable_store (vec_info *vinfo,
> if (partial_ifn == IFN_LEN_MASK_STORE)
> call = gimple_build_call_internal (IFN_LEN_MASK_STORE, 6,
> dataref_ptr, ptr,
> - final_len, final_mask,
> - vec_oprnd, bias);
> + vec_oprnd, final_len,
> + bias, final_mask);
> else
> call
> = gimple_build_call_internal (IFN_LEN_STORE, 5,
> @@ -10523,7 +10523,7 @@ vectorizable_load (vec_info *vinfo,
> call = gimple_build_call_internal (IFN_LEN_MASK_LOAD,
> 5, dataref_ptr,
> ptr, final_len,
> - final_mask, bias);
> + bias, final_mask);
> else
> call = gimple_build_call_internal (IFN_LEN_LOAD, 4,
> dataref_ptr, ptr,
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2023-07-03 7:46 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-07-03 4:07 [PATCH] Middle-end: Change order of LEN_MASK_LOAD/LEN_MASK_STORE arguments juzhe.zhong
2023-07-03 7:45 ` Richard Sandiford
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).