* [PATCH V3] VECT: Add decrement IV iteration loop control by variable amount support
@ 2023-04-26 3:59 juzhe.zhong
2023-04-26 8:13 ` Richard Biener
0 siblings, 1 reply; 2+ messages in thread
From: juzhe.zhong @ 2023-04-26 3:59 UTC (permalink / raw)
To: gcc-patches; +Cc: richard.sandiford, rguenther, rep.dot.nop, Juzhe-Zhong
From: Juzhe-Zhong <juzhe.zhong@rivai.ai>
This patch address all comments from Richard && Bernhard.
Fix all codes following their comments:
1. Remove ifn-specific handling of WHILE_LEN, so there is no change
in internal-fn.cc in this patch.
2. Add comments for changing interface of "create_iv" function.
3. Reorder the CODE of interface "create_iv", make CODE argument between
base and step.
4. Fix the misleading and incorrect comments of "vect_set_loop_controls_by_while_len".
5. remove redundant "enum" of "create_iv" interface and function.
Feel free to comment if I meet something.
Thanks.
gcc/ChangeLog:
* cfgloopmanip.cc (create_empty_loop_on_edge): Adapt create_iv interface change.
* doc/md.texi: Add WHILE_LEN pattern description.
* gimple-loop-interchange.cc (tree_loop_interchange::map_inductions_to_loop): Adapt create_iv interface change.
* internal-fn.def (WHILE_LEN): New ifn.
* optabs.def (OPTAB_D): New optab pattern.
* tree-ssa-loop-ivcanon.cc (create_canonical_iv): Adapt create_iv interface change.
* tree-ssa-loop-ivopts.cc (create_new_iv): Ditto.
* tree-ssa-loop-manip.cc (create_iv): Ditto.
(tree_transform_and_unroll_loop): Ditto.
(canonicalize_loop_ivs): Ditto.
* tree-ssa-loop-manip.h (create_iv): Add code in the interface.
* tree-vect-data-refs.cc (vect_create_data_ref_ptr): New function.
* tree-vect-loop-manip.cc (vect_set_loop_controls_directly): Add decrementing IV loop control support.
(vect_set_loop_controls_by_while_len): New function.
(vect_set_loop_condition_partial_vectors): Add decrementing IV loop control support.
(vect_set_loop_condition_normal): Adapt create_iv interface change.
* tree-vect-loop.cc (vect_create_epilog_for_reduction): Ditto.
(vect_get_loop_len): Adapt SLP auto-vectorization for WHILE_LEN.
* tree-vect-stmts.cc (get_while_len_data_ref_ptr): New function.
(vectorizable_store): Add WHILE_LEN data reference pointer IV.
(vectorizable_load): Ditto.
* tree-vectorizer.h (vect_get_loop_len): Add new arguments to interface.
---
gcc/cfgloopmanip.cc | 2 +-
gcc/doc/md.texi | 34 ++++
gcc/gimple-loop-interchange.cc | 2 +-
gcc/internal-fn.def | 1 +
gcc/optabs.def | 1 +
gcc/tree-ssa-loop-ivcanon.cc | 2 +-
gcc/tree-ssa-loop-ivopts.cc | 2 +-
gcc/tree-ssa-loop-manip.cc | 18 ++-
gcc/tree-ssa-loop-manip.h | 4 +-
gcc/tree-vect-data-refs.cc | 8 +-
gcc/tree-vect-loop-manip.cc | 280 +++++++++++++++++++++++++++++++--
gcc/tree-vect-loop.cc | 29 +++-
gcc/tree-vect-stmts.cc | 95 ++++++++++-
gcc/tree-vectorizer.h | 4 +-
14 files changed, 445 insertions(+), 37 deletions(-)
diff --git a/gcc/cfgloopmanip.cc b/gcc/cfgloopmanip.cc
index 0e3ad8ed742..6e09dcbb0b1 100644
--- a/gcc/cfgloopmanip.cc
+++ b/gcc/cfgloopmanip.cc
@@ -826,7 +826,7 @@ create_empty_loop_on_edge (edge entry_edge,
}
gsi = gsi_last_bb (loop_header);
- create_iv (initial_value, stride, iv, loop, &gsi, false,
+ create_iv (initial_value, PLUS_EXPR, stride, iv, loop, &gsi, false,
iv_before, iv_after);
/* Insert loop exit condition. */
diff --git a/gcc/doc/md.texi b/gcc/doc/md.texi
index 07bf8bdebff..307d3c407d8 100644
--- a/gcc/doc/md.texi
+++ b/gcc/doc/md.texi
@@ -4965,6 +4965,40 @@ for (i = 1; i < operand3; i++)
operand0[i] = operand0[i - 1] && (operand1 + i < operand2);
@end smallexample
+@cindex @code{while_len@var{m}@var{n}} instruction pattern
+@item @code{while_len@var{m}@var{n}}
+Set operand 0 to the number of active elements in vector will be updated value.
+operand 1 is the total elements need to be updated value.
+operand 2 is the vectorization factor.
+The value of operand 0 is target dependent and flexible in each iteration.
+The operation of this pattern can be:
+
+@smallexample
+Case 1:
+operand0 = MIN (operand1, operand2);
+operand2 can be const_poly_int or poly_int related to vector mode size.
+Some target like RISC-V has a standalone instruction to get MIN (n, MODE SIZE) so
+that we can reduce a use of general purpose register.
+
+In this case, only the last iteration of the loop is partial iteration.
+@end smallexample
+
+@smallexample
+Case 2:
+if (operand1 <= operand2)
+ operand0 = operand1;
+else if (operand1 < 2 * operand2)
+ operand0 = IN_RANGE (ceil (operand1 / 2), operand2);
+else
+ operand0 = operand2;
+
+This case will evenly distribute work over the last 2 iterations of a stripmine loop.
+@end smallexample
+
+The output of this pattern is not only used as IV of loop control counter, but also
+is used as the IV of address calculation with multiply/shift operation. This allow
+us dynamic adjust the number of elements is processed in each iteration of the loop.
+
@cindex @code{check_raw_ptrs@var{m}} instruction pattern
@item @samp{check_raw_ptrs@var{m}}
Check whether, given two pointers @var{a} and @var{b} and a length @var{len},
diff --git a/gcc/gimple-loop-interchange.cc b/gcc/gimple-loop-interchange.cc
index 1b77bfd46b2..e5590374e59 100644
--- a/gcc/gimple-loop-interchange.cc
+++ b/gcc/gimple-loop-interchange.cc
@@ -1185,7 +1185,7 @@ tree_loop_interchange::map_inductions_to_loop (loop_cand &src, loop_cand &tgt)
tree var_before, var_after;
tree base = unshare_expr (iv->init_expr);
tree step = unshare_expr (iv->step);
- create_iv (base, step, SSA_NAME_VAR (iv->var),
+ create_iv (base, PLUS_EXPR, step, SSA_NAME_VAR (iv->var),
tgt.m_loop, &incr_pos, false, &var_before, &var_after);
bitmap_set_bit (m_dce_seeds, SSA_NAME_VERSION (var_before));
bitmap_set_bit (m_dce_seeds, SSA_NAME_VERSION (var_after));
diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def
index 7fe742c2ae7..9f7269f8f60 100644
--- a/gcc/internal-fn.def
+++ b/gcc/internal-fn.def
@@ -153,6 +153,7 @@ DEF_INTERNAL_OPTAB_FN (VEC_SET, 0, vec_set, vec_set)
DEF_INTERNAL_OPTAB_FN (LEN_STORE, 0, len_store, len_store)
DEF_INTERNAL_OPTAB_FN (WHILE_ULT, ECF_CONST | ECF_NOTHROW, while_ult, while)
+DEF_INTERNAL_OPTAB_FN (WHILE_LEN, ECF_CONST | ECF_NOTHROW, while_len, binary)
DEF_INTERNAL_OPTAB_FN (CHECK_RAW_PTRS, ECF_CONST | ECF_NOTHROW,
check_raw_ptrs, check_ptrs)
DEF_INTERNAL_OPTAB_FN (CHECK_WAR_PTRS, ECF_CONST | ECF_NOTHROW,
diff --git a/gcc/optabs.def b/gcc/optabs.def
index 695f5911b30..f5938bd2c24 100644
--- a/gcc/optabs.def
+++ b/gcc/optabs.def
@@ -476,3 +476,4 @@ OPTAB_DC (vec_series_optab, "vec_series$a", VEC_SERIES)
OPTAB_D (vec_shl_insert_optab, "vec_shl_insert_$a")
OPTAB_D (len_load_optab, "len_load_$a")
OPTAB_D (len_store_optab, "len_store_$a")
+OPTAB_D (while_len_optab, "while_len$a")
diff --git a/gcc/tree-ssa-loop-ivcanon.cc b/gcc/tree-ssa-loop-ivcanon.cc
index 9f72d534b7c..6941d0a4d1d 100644
--- a/gcc/tree-ssa-loop-ivcanon.cc
+++ b/gcc/tree-ssa-loop-ivcanon.cc
@@ -113,7 +113,7 @@ create_canonical_iv (class loop *loop, edge exit, tree niter,
niter,
build_int_cst (type, 1));
incr_at = gsi_last_bb (in->src);
- create_iv (niter,
+ create_iv (niter, PLUS_EXPR,
build_int_cst (type, -1),
NULL_TREE, loop,
&incr_at, false, var_before, &var);
diff --git a/gcc/tree-ssa-loop-ivopts.cc b/gcc/tree-ssa-loop-ivopts.cc
index 0dd47910f97..a7377f91f9e 100644
--- a/gcc/tree-ssa-loop-ivopts.cc
+++ b/gcc/tree-ssa-loop-ivopts.cc
@@ -7267,7 +7267,7 @@ create_new_iv (struct ivopts_data *data, struct iv_cand *cand)
base = unshare_expr (cand->iv->base);
- create_iv (base, unshare_expr (cand->iv->step),
+ create_iv (base, PLUS_EXPR, unshare_expr (cand->iv->step),
cand->var_before, data->current_loop,
&incr_pos, after, &cand->var_before, &cand->var_after);
}
diff --git a/gcc/tree-ssa-loop-manip.cc b/gcc/tree-ssa-loop-manip.cc
index a52277abdbf..b0d254b0c89 100644
--- a/gcc/tree-ssa-loop-manip.cc
+++ b/gcc/tree-ssa-loop-manip.cc
@@ -47,7 +47,9 @@ along with GCC; see the file COPYING3. If not see
so that we can free them all at once. */
static bitmap_obstack loop_renamer_obstack;
-/* Creates an induction variable with value BASE + STEP * iteration in LOOP.
+/* Creates an induction variable with value BASE (+/-) STEP * iteration in LOOP.
+ If CODE is PLUS_EXPR, the induction variable is BASE + STEP * iteration.
+ If CODE is MINUS_EXPR, the induction variable is BASE - STEP * iteration.
It is expected that neither BASE nor STEP are shared with other expressions
(unless the sharing rules allow this). Use VAR as a base var_decl for it
(if NULL, a new temporary will be created). The increment will occur at
@@ -57,8 +59,8 @@ static bitmap_obstack loop_renamer_obstack;
VAR_AFTER (unless they are NULL). */
void
-create_iv (tree base, tree step, tree var, class loop *loop,
- gimple_stmt_iterator *incr_pos, bool after,
+create_iv (tree base, tree_code code, tree step, tree var,
+ class loop *loop, gimple_stmt_iterator *incr_pos, bool after,
tree *var_before, tree *var_after)
{
gassign *stmt;
@@ -66,7 +68,9 @@ create_iv (tree base, tree step, tree var, class loop *loop,
tree initial, step1;
gimple_seq stmts;
tree vb, va;
- enum tree_code incr_op = PLUS_EXPR;
+ /* The code can only be PLUS_EXPR or MINUS_EXPR. */
+ gcc_assert (code == PLUS_EXPR || code == MINUS_EXPR);
+ tree_code incr_op = code;
edge pe = loop_preheader_edge (loop);
if (var != NULL_TREE)
@@ -1368,7 +1372,7 @@ tree_transform_and_unroll_loop (class loop *loop, unsigned factor,
tree ctr_before, ctr_after;
gimple_stmt_iterator bsi = gsi_last_nondebug_bb (new_exit->src);
exit_if = as_a <gcond *> (gsi_stmt (bsi));
- create_iv (exit_base, exit_step, NULL_TREE, loop,
+ create_iv (exit_base, PLUS_EXPR, exit_step, NULL_TREE, loop,
&bsi, false, &ctr_before, &ctr_after);
gimple_cond_set_code (exit_if, exit_cmp);
gimple_cond_set_lhs (exit_if, ctr_after);
@@ -1583,8 +1587,8 @@ canonicalize_loop_ivs (class loop *loop, tree *nit, bool bump_in_latch)
gsi = gsi_last_bb (loop->latch);
else
gsi = gsi_last_nondebug_bb (loop->header);
- create_iv (build_int_cst_type (type, 0), build_int_cst (type, 1), NULL_TREE,
- loop, &gsi, bump_in_latch, &var_before, NULL);
+ create_iv (build_int_cst_type (type, 0), PLUS_EXPR, build_int_cst (type, 1),
+ NULL_TREE, loop, &gsi, bump_in_latch, &var_before, NULL);
rewrite_all_phi_nodes_with_iv (loop, var_before);
diff --git a/gcc/tree-ssa-loop-manip.h b/gcc/tree-ssa-loop-manip.h
index d49273a3987..bda09f51d56 100644
--- a/gcc/tree-ssa-loop-manip.h
+++ b/gcc/tree-ssa-loop-manip.h
@@ -22,8 +22,8 @@ along with GCC; see the file COPYING3. If not see
typedef void (*transform_callback)(class loop *, void *);
-extern void create_iv (tree, tree, tree, class loop *, gimple_stmt_iterator *,
- bool, tree *, tree *);
+extern void create_iv (tree, tree_code, tree, tree, class loop *,
+ gimple_stmt_iterator *, bool, tree *, tree *);
extern void rewrite_into_loop_closed_ssa (bitmap, unsigned);
extern void verify_loop_closed_ssa (bool, class loop * = NULL);
diff --git a/gcc/tree-vect-data-refs.cc b/gcc/tree-vect-data-refs.cc
index c03ffb3aaf1..355ede5c0d9 100644
--- a/gcc/tree-vect-data-refs.cc
+++ b/gcc/tree-vect-data-refs.cc
@@ -5101,7 +5101,7 @@ vect_create_data_ref_ptr (vec_info *vinfo, stmt_vec_info stmt_info,
standard_iv_increment_position (loop, &incr_gsi, &insert_after);
- create_iv (aggr_ptr_init,
+ create_iv (aggr_ptr_init, PLUS_EXPR,
fold_convert (aggr_ptr_type, iv_step),
aggr_ptr, loop, &incr_gsi, insert_after,
&indx_before_incr, &indx_after_incr);
@@ -5131,9 +5131,9 @@ vect_create_data_ref_ptr (vec_info *vinfo, stmt_vec_info stmt_info,
{
standard_iv_increment_position (containing_loop, &incr_gsi,
&insert_after);
- create_iv (aptr, fold_convert (aggr_ptr_type, DR_STEP (dr)), aggr_ptr,
- containing_loop, &incr_gsi, insert_after, &indx_before_incr,
- &indx_after_incr);
+ create_iv (aptr, PLUS_EXPR, fold_convert (aggr_ptr_type, DR_STEP (dr)),
+ aggr_ptr, containing_loop, &incr_gsi, insert_after,
+ &indx_before_incr, &indx_after_incr);
incr = gsi_stmt (incr_gsi);
/* Copy the points-to information if it exists. */
diff --git a/gcc/tree-vect-loop-manip.cc b/gcc/tree-vect-loop-manip.cc
index f60fa50e8f4..6973314adb5 100644
--- a/gcc/tree-vect-loop-manip.cc
+++ b/gcc/tree-vect-loop-manip.cc
@@ -468,8 +468,9 @@ vect_set_loop_controls_directly (class loop *loop, loop_vec_info loop_vinfo,
gimple_stmt_iterator incr_gsi;
bool insert_after;
standard_iv_increment_position (loop, &incr_gsi, &insert_after);
- create_iv (build_int_cst (iv_type, 0), nitems_step, NULL_TREE, loop,
- &incr_gsi, insert_after, &index_before_incr, &index_after_incr);
+ create_iv (build_int_cst (iv_type, 0), PLUS_EXPR, nitems_step, NULL_TREE,
+ loop, &incr_gsi, insert_after, &index_before_incr,
+ &index_after_incr);
tree zero_index = build_int_cst (compare_type, 0);
tree test_index, test_limit, first_limit;
@@ -682,6 +683,260 @@ vect_set_loop_controls_directly (class loop *loop, loop_vec_info loop_vinfo,
return next_ctrl;
}
+/* Helper for vect_set_loop_condition_partial_vectors. Generate definitions
+ for all the rgroup controls in RGC and return a control that is nonzero
+ when the loop needs to iterate. Add any new preheader statements to
+ PREHEADER_SEQ. Use LOOP_COND_GSI to insert code before the exit gcond.
+
+ RGC belongs to loop LOOP. The loop originally iterated NITERS
+ times and has been vectorized according to LOOP_VINFO.
+
+ Unlike vect_set_loop_controls_directly which is iterating from 0-based IV
+ to TEST_LIMIT - bias.
+
+ In vect_set_loop_controls_by_while_len, we are iterating from start at
+ IV = TEST_LIMIT - bias and keep subtract IV by the length calculated by
+ IFN_WHILE_LEN pattern.
+
+ 1. Single rgroup, the Gimple IR should be:
+
+ # vectp_B.6_8 = PHI <vectp_B.6_13(6), &B(5)>
+ # vectp_B.8_16 = PHI <vectp_B.8_17(6), &B(5)>
+ # vectp_A.11_19 = PHI <vectp_A.11_20(6), &A(5)>
+ # vectp_A.13_22 = PHI <vectp_A.13_23(6), &A(5)>
+ # ivtmp_26 = PHI <ivtmp_27(6), _25(5)>
+ _28 = .WHILE_LEN (ivtmp_26, POLY_INT_CST [4, 4]);
+ ivtmp_15 = _28 * 4;
+ vect__1.10_18 = .LEN_LOAD (vectp_B.8_16, 128B, _28, 0);
+ _1 = B[i_10];
+ .LEN_STORE (vectp_A.13_22, 128B, _28, vect__1.10_18, 0);
+ i_7 = i_10 + 1;
+ vectp_B.8_17 = vectp_B.8_16 + ivtmp_15;
+ vectp_A.13_23 = vectp_A.13_22 + ivtmp_15;
+ ivtmp_27 = ivtmp_26 - _28;
+ if (ivtmp_27 != 0)
+ goto <bb 6>; [83.33%]
+ else
+ goto <bb 7>; [16.67%]
+
+ Note: We use the outcome of .WHILE_LEN to adjust both loop control IV and
+ data reference pointer IV.
+
+ 1). The result of .WHILE_LEN:
+ _28 = .WHILE_LEN (ivtmp_26, POLY_INT_CST [4, 4]);
+ The _28 is not necessary to be VF in any iteration, instead, we allow
+ _28 to be any value as long as it it <= VF. Such flexible WHILE_LEN
+ pattern allows target have various flexible optimizations in vector
+ loop iterations. Target like RISC-V has special application vector
+ length calculation instruction which will distribute even workload
+ in the last 2 iterations.
+
+ Other example is that we can allow even generate _28 <= VF / 2 so
+ that some machine can run vector codes in low power mode.
+
+ 2). Loop control IV:
+ ivtmp_27 = ivtmp_26 - _28;
+ if (ivtmp_27 != 0)
+ goto <bb 6>; [83.33%]
+ else
+ goto <bb 7>; [16.67%]
+
+ This is the saturating-subtraction towards zero, the outcome of
+ .WHILE_LEN wil make ivtmp_27 never underflow zero.
+
+ 3). Data reference pointer IV:
+ ivtmp_15 = _28 * 4;
+ vectp_B.8_17 = vectp_B.8_16 + ivtmp_15;
+ vectp_A.13_23 = vectp_A.13_22 + ivtmp_15;
+
+ The pointer IV is adjusted accurately according to the .WHILE_LEN.
+
+ 2. Multiple rgroup, the Gimple IR should be:
+
+ # i_23 = PHI <i_20(6), 0(11)>
+ # vectp_f.8_51 = PHI <vectp_f.8_52(6), f_15(D)(11)>
+ # vectp_d.10_59 = PHI <vectp_d.10_60(6), d_18(D)(11)>
+ # ivtmp_70 = PHI <ivtmp_71(6), _69(11)>
+ # ivtmp_73 = PHI <ivtmp_74(6), _67(11)>
+ _72 = MIN_EXPR <ivtmp_70, 16>;
+ _75 = MIN_EXPR <ivtmp_73, 16>;
+ _1 = i_23 * 2;
+ _2 = (long unsigned int) _1;
+ _3 = _2 * 2;
+ _4 = f_15(D) + _3;
+ _5 = _2 + 1;
+ _6 = _5 * 2;
+ _7 = f_15(D) + _6;
+ .LEN_STORE (vectp_f.8_51, 128B, _75, { 1, 2, 1, 2, 1, 2, 1, 2 }, 0);
+ vectp_f.8_56 = vectp_f.8_51 + 16;
+ .LEN_STORE (vectp_f.8_56, 128B, _72, { 1, 2, 1, 2, 1, 2, 1, 2 }, 0);
+ _8 = (long unsigned int) i_23;
+ _9 = _8 * 4;
+ _10 = d_18(D) + _9;
+ _61 = _75 / 2;
+ .LEN_STORE (vectp_d.10_59, 128B, _61, { 3, 3, 3, 3 }, 0);
+ vectp_d.10_63 = vectp_d.10_59 + 16;
+ _64 = _72 / 2;
+ .LEN_STORE (vectp_d.10_63, 128B, _64, { 3, 3, 3, 3 }, 0);
+ i_20 = i_23 + 1;
+ vectp_f.8_52 = vectp_f.8_56 + 16;
+ vectp_d.10_60 = vectp_d.10_63 + 16;
+ ivtmp_74 = ivtmp_73 - _75;
+ ivtmp_71 = ivtmp_70 - _72;
+ if (ivtmp_74 != 0)
+ goto <bb 6>; [83.33%]
+ else
+ goto <bb 13>; [16.67%]
+
+ Note: We DO NOT use .WHILE_LEN in SLP auto-vectorization for multiple
+ rgroups. Instead, we use MIN_EXPR to guarantee we always use VF as the
+ iteration amount for mutiple rgroups.
+
+ The analysis of the flow of multiple rgroups:
+ _72 = MIN_EXPR <ivtmp_70, 16>;
+ _75 = MIN_EXPR <ivtmp_73, 16>;
+ ...
+ .LEN_STORE (vectp_f.8_51, 128B, _75, { 1, 2, 1, 2, 1, 2, 1, 2 }, 0);
+ vectp_f.8_56 = vectp_f.8_51 + 16;
+ .LEN_STORE (vectp_f.8_56, 128B, _72, { 1, 2, 1, 2, 1, 2, 1, 2 }, 0);
+ ...
+ _61 = _75 / 2;
+ .LEN_STORE (vectp_d.10_59, 128B, _61, { 3, 3, 3, 3 }, 0);
+ vectp_d.10_63 = vectp_d.10_59 + 16;
+ _64 = _72 / 2;
+ .LEN_STORE (vectp_d.10_63, 128B, _64, { 3, 3, 3, 3 }, 0);
+
+ We use _72 = MIN_EXPR <ivtmp_70, 16>; to generate the number of the elements
+ to be processed in each iteration.
+
+ The related STOREs:
+ _72 = MIN_EXPR <ivtmp_70, 16>;
+ .LEN_STORE (vectp_f.8_56, 128B, _72, { 1, 2, 1, 2, 1, 2, 1, 2 }, 0);
+ _64 = _72 / 2;
+ .LEN_STORE (vectp_d.10_63, 128B, _64, { 3, 3, 3, 3 }, 0);
+ Since these 2 STOREs store 2 vectors that the second vector is half elements
+ of the first vector. So the length of second STORE will be _64 = _72 / 2;
+ It's similar to the VIEW_CONVERT of handling masks in SLP.
+*/
+
+static tree
+vect_set_loop_controls_by_while_len (class loop *loop, loop_vec_info loop_vinfo,
+ gimple_seq *preheader_seq,
+ gimple_seq *header_seq,
+ rgroup_controls *rgc, tree niters)
+{
+ tree compare_type = LOOP_VINFO_RGROUP_COMPARE_TYPE (loop_vinfo);
+ tree iv_type = LOOP_VINFO_RGROUP_IV_TYPE (loop_vinfo);
+ /* We are not allowing masked approach in WHILE_LEN. */
+ gcc_assert (!LOOP_VINFO_FULLY_MASKED_P (loop_vinfo));
+
+ tree ctrl_type = rgc->type;
+ unsigned int nitems_per_iter = rgc->max_nscalars_per_iter * rgc->factor;
+ poly_uint64 nitems_per_ctrl = TYPE_VECTOR_SUBPARTS (ctrl_type) * rgc->factor;
+ poly_uint64 vf = LOOP_VINFO_VECT_FACTOR (loop_vinfo);
+
+ /* Calculate the maximum number of item values that the rgroup
+ handles in total, the number that it handles for each iteration
+ of the vector loop. */
+ tree nitems_total = niters;
+ if (nitems_per_iter != 1)
+ {
+ /* We checked before setting LOOP_VINFO_USING_PARTIAL_VECTORS_P that
+ these multiplications don't overflow. */
+ tree compare_factor = build_int_cst (compare_type, nitems_per_iter);
+ nitems_total = gimple_build (preheader_seq, MULT_EXPR, compare_type,
+ nitems_total, compare_factor);
+ }
+
+ /* Convert the comparison value to the IV type (either a no-op or
+ a promotion). */
+ nitems_total = gimple_convert (preheader_seq, iv_type, nitems_total);
+
+ /* Create an induction variable that counts the number of items
+ processed. */
+ tree index_before_incr, index_after_incr;
+ gimple_stmt_iterator incr_gsi;
+ bool insert_after;
+ standard_iv_increment_position (loop, &incr_gsi, &insert_after);
+
+ /* Test the decremented IV, which will never underflow 0 since we have
+ IFN_WHILE_LEN to gurantee that. */
+ tree test_limit = nitems_total;
+
+ /* Provide a definition of each control in the group. */
+ tree ctrl;
+ unsigned int i;
+ FOR_EACH_VEC_ELT_REVERSE (rgc->controls, i, ctrl)
+ {
+ /* Previous controls will cover BIAS items. This control covers the
+ next batch. */
+ poly_uint64 bias = nitems_per_ctrl * i;
+ tree bias_tree = build_int_cst (iv_type, bias);
+
+ /* Rather than have a new IV that starts at TEST_LIMIT and goes down to
+ BIAS, prefer to use the same TEST_LIMIT - BIAS based IV for each
+ control and adjust the bound down by BIAS. */
+ tree this_test_limit = test_limit;
+ if (i != 0)
+ {
+ this_test_limit = gimple_build (preheader_seq, MAX_EXPR, iv_type,
+ this_test_limit, bias_tree);
+ this_test_limit = gimple_build (preheader_seq, MINUS_EXPR, iv_type,
+ this_test_limit, bias_tree);
+ }
+
+ /* Create decrement IV. */
+ create_iv (this_test_limit, MINUS_EXPR, ctrl, NULL_TREE, loop, &incr_gsi,
+ insert_after, &index_before_incr, &index_after_incr);
+
+ poly_uint64 final_vf = vf * nitems_per_iter;
+ tree vf_step = build_int_cst (iv_type, final_vf);
+ tree res_len;
+ if (nitems_per_iter != 1)
+ {
+ /* For SLP, we can't allow non-VF number of elements to be processed
+ in non-final iteration. We force the number of elements to be
+ processed in each non-final iteration is VF elements. If we allow
+ non-VF elements processing in non-final iteration will make SLP too
+ complicated and produce inferior codegen.
+
+ For example:
+
+ If non-final iteration process VF elements.
+
+ ...
+ .LEN_STORE (vectp_f.8_51, 128B, _71, { 1, 2, 1, 2 }, 0);
+ .LEN_STORE (vectp_f.8_56, 128B, _72, { 1, 2, 1, 2 }, 0);
+ ...
+
+ If non-final iteration process non-VF elements.
+
+ ...
+ .LEN_STORE (vectp_f.8_51, 128B, _71, { 1, 2, 1, 2 }, 0);
+ if (_71 % 2 == 0)
+ .LEN_STORE (vectp_f.8_56, 128B, _72, { 1, 2, 1, 2 }, 0);
+ else
+ .LEN_STORE (vectp_f.8_56, 128B, _72, { 2, 1, 2, 1 }, 0);
+ ...
+
+ This is the simple case of 2-elements interleaved vector SLP. We
+ consider other interleave vector, the situation will become more
+ complicated. */
+ res_len = gimple_build (header_seq, MIN_EXPR, iv_type,
+ index_before_incr, vf_step);
+ }
+ else
+ {
+ res_len = gimple_build (header_seq, IFN_WHILE_LEN, iv_type,
+ index_before_incr, vf_step);
+ }
+ gassign *assign = gimple_build_assign (ctrl, res_len);
+ gimple_seq_add_stmt (header_seq, assign);
+ }
+
+ return index_after_incr;
+}
+
/* Set up the iteration condition and rgroup controls for LOOP, given
that LOOP_VINFO_USING_PARTIAL_VECTORS_P is true for the vectorized
loop. LOOP_VINFO describes the vectorization of LOOP. NITERS is
@@ -703,6 +958,7 @@ vect_set_loop_condition_partial_vectors (class loop *loop,
bool use_masks_p = LOOP_VINFO_FULLY_MASKED_P (loop_vinfo);
tree compare_type = LOOP_VINFO_RGROUP_COMPARE_TYPE (loop_vinfo);
+ tree iv_type = LOOP_VINFO_RGROUP_IV_TYPE (loop_vinfo);
unsigned int compare_precision = TYPE_PRECISION (compare_type);
tree orig_niters = niters;
@@ -757,12 +1013,18 @@ vect_set_loop_condition_partial_vectors (class loop *loop,
bool might_wrap_p = vect_rgroup_iv_might_wrap_p (loop_vinfo, rgc);
/* Set up all controls for this group. */
- test_ctrl = vect_set_loop_controls_directly (loop, loop_vinfo,
- &preheader_seq,
- &header_seq,
- loop_cond_gsi, rgc,
- niters, niters_skip,
- might_wrap_p);
+ if (direct_internal_fn_supported_p (IFN_WHILE_LEN, iv_type,
+ OPTIMIZE_FOR_SPEED))
+ test_ctrl
+ = vect_set_loop_controls_by_while_len (loop, loop_vinfo,
+ &preheader_seq, &header_seq,
+ rgc, niters);
+ else
+ test_ctrl
+ = vect_set_loop_controls_directly (loop, loop_vinfo, &preheader_seq,
+ &header_seq, loop_cond_gsi, rgc,
+ niters, niters_skip,
+ might_wrap_p);
}
/* Emit all accumulated statements. */
@@ -893,7 +1155,7 @@ vect_set_loop_condition_normal (class loop *loop, tree niters, tree step,
}
standard_iv_increment_position (loop, &incr_gsi, &insert_after);
- create_iv (init, step, NULL_TREE, loop,
+ create_iv (init, PLUS_EXPR, step, NULL_TREE, loop,
&incr_gsi, insert_after, &indx_before_incr, &indx_after_incr);
indx_after_incr = force_gimple_operand_gsi (&loop_cond_gsi, indx_after_incr,
true, NULL_TREE, true,
diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
index 6ea0f21fd13..d6e90ed93cd 100644
--- a/gcc/tree-vect-loop.cc
+++ b/gcc/tree-vect-loop.cc
@@ -5567,7 +5567,7 @@ vect_create_epilog_for_reduction (loop_vec_info loop_vinfo,
gimple_stmt_iterator incr_gsi;
bool insert_after;
standard_iv_increment_position (loop, &incr_gsi, &insert_after);
- create_iv (series_vect, vec_step, NULL_TREE, loop, &incr_gsi,
+ create_iv (series_vect, PLUS_EXPR, vec_step, NULL_TREE, loop, &incr_gsi,
insert_after, &indx_before_incr, &indx_after_incr);
/* Next create a new phi node vector (NEW_PHI_TREE) which starts
@@ -10364,12 +10364,14 @@ vect_record_loop_len (loop_vec_info loop_vinfo, vec_loop_lens *lens,
rgroup that operates on NVECTORS vectors, where 0 <= INDEX < NVECTORS. */
tree
-vect_get_loop_len (loop_vec_info loop_vinfo, vec_loop_lens *lens,
- unsigned int nvectors, unsigned int index)
+vect_get_loop_len (gimple_stmt_iterator *gsi, loop_vec_info loop_vinfo,
+ vec_loop_lens *lens, unsigned int nvectors, tree vectype,
+ unsigned int index)
{
rgroup_controls *rgl = &(*lens)[nvectors - 1];
bool use_bias_adjusted_len =
LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo) != 0;
+ tree iv_type = LOOP_VINFO_RGROUP_IV_TYPE (loop_vinfo);
/* Populate the rgroup's len array, if this is the first time we've
used it. */
@@ -10400,6 +10402,27 @@ vect_get_loop_len (loop_vec_info loop_vinfo, vec_loop_lens *lens,
if (use_bias_adjusted_len)
return rgl->bias_adjusted_ctrl;
+ else if (direct_internal_fn_supported_p (IFN_WHILE_LEN, iv_type,
+ OPTIMIZE_FOR_SPEED))
+ {
+ tree loop_len = rgl->controls[index];
+ poly_int64 nunits1 = TYPE_VECTOR_SUBPARTS (rgl->type);
+ poly_int64 nunits2 = TYPE_VECTOR_SUBPARTS (vectype);
+ if (maybe_ne (nunits1, nunits2))
+ {
+ /* A loop len for data type X can be reused for data type Y
+ if X has N times more elements than Y and if Y's elements
+ are N times bigger than X's. */
+ gcc_assert (multiple_p (nunits1, nunits2));
+ unsigned int factor = exact_div (nunits1, nunits2).to_constant ();
+ gimple_seq seq = NULL;
+ loop_len = gimple_build (&seq, RDIV_EXPR, iv_type, loop_len,
+ build_int_cst (iv_type, factor));
+ if (seq)
+ gsi_insert_seq_before (gsi, seq, GSI_SAME_STMT);
+ }
+ return loop_len;
+ }
else
return rgl->controls[index];
}
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index 6b7dbfd4a23..98bef8f4fc9 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -3144,6 +3144,70 @@ vect_get_data_ptr_increment (vec_info *vinfo,
return iv_step;
}
+/* Prepare the pointer IVs which needs to be updated by a variable amount.
+ Such variable amount is the outcome of .WHILE_LEN. In this case, we can
+ allow each iteration process the flexible number of elements as long as
+ the number <= vf elments.
+
+ Return data reference according to WHILE_LEN.
+ If new statements are needed, insert them before GSI. */
+
+static tree
+get_while_len_data_ref_ptr (vec_info *vinfo, stmt_vec_info stmt_info,
+ tree aggr_type, class loop *at_loop, tree offset,
+ tree *dummy, gimple_stmt_iterator *gsi,
+ bool simd_lane_access_p, vec_loop_lens *loop_lens,
+ dr_vec_info *dr_info,
+ vect_memory_access_type memory_access_type)
+{
+ if (!loop_lens || loop_lens->length () != 1)
+ return NULL_TREE;
+ loop_vec_info loop_vinfo = dyn_cast<loop_vec_info> (vinfo);
+ tree iv_type = LOOP_VINFO_RGROUP_IV_TYPE (loop_vinfo);
+ tree step = vect_dr_behavior (vinfo, dr_info)->step;
+ if (!direct_internal_fn_supported_p (IFN_WHILE_LEN, iv_type,
+ OPTIMIZE_FOR_SPEED))
+ return NULL_TREE;
+
+ if (memory_access_type == VMAT_INVARIANT)
+ return NULL_TREE;
+
+ /* TODO: We don't support gather/scatter or load_lanes/store_lanes for pointer
+ IVs are updated by variable amount but we will support them in the future.
+ */
+ gcc_assert (memory_access_type != VMAT_GATHER_SCATTER
+ && memory_access_type != VMAT_LOAD_STORE_LANES);
+
+ /* When we support WHILE_LEN pattern, we dynamic adjust
+ the memory address by .WHILE_LEN result.
+
+ The result of .WHILE_LEN is the number of elements to
+ be processed of each iteration. So the memory address
+ adjustment operation should be:
+
+ bytesize = GET_MODE_SIZE (element_mode (aggr_type));
+ addr = addr + .WHILE_LEN (ARG..) * bytesize;
+ */
+ gimple *ptr_incr;
+ tree loop_len
+ = vect_get_loop_len (gsi, loop_vinfo, loop_lens, 1, aggr_type, 0);
+ tree len_type = TREE_TYPE (loop_len);
+ poly_uint64 bytesize = GET_MODE_SIZE (element_mode (aggr_type));
+ /* Since the outcome of .WHILE_LEN is element size, we should adjust
+ it into bytesize so that it can be used in address pointer variable
+ amount IVs adjustment. */
+ tree tmp = fold_build2 (MULT_EXPR, len_type, loop_len,
+ build_int_cst (len_type, bytesize));
+ if (tree_int_cst_sgn (step) == -1)
+ tmp = fold_build1 (NEGATE_EXPR, len_type, tmp);
+ tree bump = make_temp_ssa_name (len_type, NULL, "ivtmp");
+ gassign *assign = gimple_build_assign (bump, tmp);
+ gsi_insert_before (gsi, assign, GSI_SAME_STMT);
+ return vect_create_data_ref_ptr (vinfo, stmt_info, aggr_type, at_loop, offset,
+ dummy, gsi, &ptr_incr, simd_lane_access_p,
+ bump);
+}
+
/* Check and perform vectorization of BUILT_IN_BSWAP{16,32,64,128}. */
static bool
@@ -8195,7 +8259,7 @@ vectorizable_store (vec_info *vinfo,
stride_base = cse_and_gimplify_to_preheader (loop_vinfo, stride_base);
ivstep = cse_and_gimplify_to_preheader (loop_vinfo, ivstep);
- create_iv (stride_base, ivstep, NULL,
+ create_iv (stride_base, PLUS_EXPR, ivstep, NULL,
loop, &incr_gsi, insert_after,
&offvar, NULL);
incr = gsi_stmt (incr_gsi);
@@ -8465,6 +8529,15 @@ vectorizable_store (vec_info *vinfo,
simd_lane_access_p ? loop : NULL,
offset, &dummy, gsi, &ptr_incr,
simd_lane_access_p, bump);
+
+ tree while_len_data_ref_ptr
+ = get_while_len_data_ref_ptr (vinfo, stmt_info, aggr_type,
+ simd_lane_access_p ? loop : NULL,
+ offset, &dummy, gsi,
+ simd_lane_access_p, loop_lens,
+ dr_info, memory_access_type);
+ if (while_len_data_ref_ptr)
+ dataref_ptr = while_len_data_ref_ptr;
}
else
{
@@ -8652,8 +8725,9 @@ vectorizable_store (vec_info *vinfo,
else if (loop_lens)
{
tree final_len
- = vect_get_loop_len (loop_vinfo, loop_lens,
- vec_num * ncopies, vec_num * j + i);
+ = vect_get_loop_len (gsi, loop_vinfo, loop_lens,
+ vec_num * ncopies, vectype,
+ vec_num * j + i);
tree ptr = build_int_cst (ref_type, align * BITS_PER_UNIT);
machine_mode vmode = TYPE_MODE (vectype);
opt_machine_mode new_ovmode
@@ -9314,7 +9388,7 @@ vectorizable_load (vec_info *vinfo,
stride_base = cse_and_gimplify_to_preheader (loop_vinfo, stride_base);
ivstep = cse_and_gimplify_to_preheader (loop_vinfo, ivstep);
- create_iv (stride_base, ivstep, NULL,
+ create_iv (stride_base, PLUS_EXPR, ivstep, NULL,
loop, &incr_gsi, insert_after,
&offvar, NULL);
@@ -9798,6 +9872,15 @@ vectorizable_load (vec_info *vinfo,
at_loop,
offset, &dummy, gsi, &ptr_incr,
simd_lane_access_p, bump);
+
+ tree while_len_data_ref_ptr
+ = get_while_len_data_ref_ptr (vinfo, stmt_info, aggr_type, at_loop,
+ offset, &dummy, gsi,
+ simd_lane_access_p, loop_lens,
+ dr_info, memory_access_type);
+ if (while_len_data_ref_ptr)
+ dataref_ptr = while_len_data_ref_ptr;
+
if (mask)
vec_mask = vec_masks[0];
}
@@ -10008,8 +10091,8 @@ vectorizable_load (vec_info *vinfo,
else if (loop_lens && memory_access_type != VMAT_INVARIANT)
{
tree final_len
- = vect_get_loop_len (loop_vinfo, loop_lens,
- vec_num * ncopies,
+ = vect_get_loop_len (gsi, loop_vinfo, loop_lens,
+ vec_num * ncopies, vectype,
vec_num * j + i);
tree ptr = build_int_cst (ref_type,
align * BITS_PER_UNIT);
diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
index 9cf2fb23fe3..811c17e1d3d 100644
--- a/gcc/tree-vectorizer.h
+++ b/gcc/tree-vectorizer.h
@@ -2293,8 +2293,8 @@ extern tree vect_get_loop_mask (gimple_stmt_iterator *, vec_loop_masks *,
unsigned int, tree, unsigned int);
extern void vect_record_loop_len (loop_vec_info, vec_loop_lens *, unsigned int,
tree, unsigned int);
-extern tree vect_get_loop_len (loop_vec_info, vec_loop_lens *, unsigned int,
- unsigned int);
+extern tree vect_get_loop_len (gimple_stmt_iterator *, loop_vec_info, vec_loop_lens *,
+ unsigned int, tree, unsigned int);
extern gimple_seq vect_gen_len (tree, tree, tree, tree);
extern stmt_vec_info info_for_reduction (vec_info *, stmt_vec_info);
extern bool reduction_fn_for_scalar_code (code_helper, internal_fn *);
--
2.36.1
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: [PATCH V3] VECT: Add decrement IV iteration loop control by variable amount support
2023-04-26 3:59 [PATCH V3] VECT: Add decrement IV iteration loop control by variable amount support juzhe.zhong
@ 2023-04-26 8:13 ` Richard Biener
0 siblings, 0 replies; 2+ messages in thread
From: Richard Biener @ 2023-04-26 8:13 UTC (permalink / raw)
To: Juzhe-Zhong; +Cc: gcc-patches, richard.sandiford, rep.dot.nop
On Wed, 26 Apr 2023, juzhe.zhong@rivai.ai wrote:
> From: Juzhe-Zhong <juzhe.zhong@rivai.ai>
>
> This patch address all comments from Richard && Bernhard.
> Fix all codes following their comments:
>
> 1. Remove ifn-specific handling of WHILE_LEN, so there is no change
> in internal-fn.cc in this patch.
>
> 2. Add comments for changing interface of "create_iv" function.
>
> 3. Reorder the CODE of interface "create_iv", make CODE argument between
> base and step.
>
> 4. Fix the misleading and incorrect comments of "vect_set_loop_controls_by_while_len".
>
> 5. remove redundant "enum" of "create_iv" interface and function.
>
> Feel free to comment if I meet something.
>
> Thanks.
>
> gcc/ChangeLog:
>
> * cfgloopmanip.cc (create_empty_loop_on_edge): Adapt create_iv interface change.
> * doc/md.texi: Add WHILE_LEN pattern description.
> * gimple-loop-interchange.cc (tree_loop_interchange::map_inductions_to_loop): Adapt create_iv interface change.
> * internal-fn.def (WHILE_LEN): New ifn.
> * optabs.def (OPTAB_D): New optab pattern.
> * tree-ssa-loop-ivcanon.cc (create_canonical_iv): Adapt create_iv interface change.
> * tree-ssa-loop-ivopts.cc (create_new_iv): Ditto.
> * tree-ssa-loop-manip.cc (create_iv): Ditto.
> (tree_transform_and_unroll_loop): Ditto.
> (canonicalize_loop_ivs): Ditto.
> * tree-ssa-loop-manip.h (create_iv): Add code in the interface.
> * tree-vect-data-refs.cc (vect_create_data_ref_ptr): New function.
> * tree-vect-loop-manip.cc (vect_set_loop_controls_directly): Add decrementing IV loop control support.
> (vect_set_loop_controls_by_while_len): New function.
> (vect_set_loop_condition_partial_vectors): Add decrementing IV loop control support.
> (vect_set_loop_condition_normal): Adapt create_iv interface change.
> * tree-vect-loop.cc (vect_create_epilog_for_reduction): Ditto.
> (vect_get_loop_len): Adapt SLP auto-vectorization for WHILE_LEN.
> * tree-vect-stmts.cc (get_while_len_data_ref_ptr): New function.
> (vectorizable_store): Add WHILE_LEN data reference pointer IV.
> (vectorizable_load): Ditto.
> * tree-vectorizer.h (vect_get_loop_len): Add new arguments to interface.
>
>
> ---
> gcc/cfgloopmanip.cc | 2 +-
> gcc/doc/md.texi | 34 ++++
> gcc/gimple-loop-interchange.cc | 2 +-
> gcc/internal-fn.def | 1 +
> gcc/optabs.def | 1 +
> gcc/tree-ssa-loop-ivcanon.cc | 2 +-
> gcc/tree-ssa-loop-ivopts.cc | 2 +-
> gcc/tree-ssa-loop-manip.cc | 18 ++-
> gcc/tree-ssa-loop-manip.h | 4 +-
> gcc/tree-vect-data-refs.cc | 8 +-
> gcc/tree-vect-loop-manip.cc | 280 +++++++++++++++++++++++++++++++--
> gcc/tree-vect-loop.cc | 29 +++-
> gcc/tree-vect-stmts.cc | 95 ++++++++++-
> gcc/tree-vectorizer.h | 4 +-
> 14 files changed, 445 insertions(+), 37 deletions(-)
>
> diff --git a/gcc/cfgloopmanip.cc b/gcc/cfgloopmanip.cc
> index 0e3ad8ed742..6e09dcbb0b1 100644
> --- a/gcc/cfgloopmanip.cc
> +++ b/gcc/cfgloopmanip.cc
> @@ -826,7 +826,7 @@ create_empty_loop_on_edge (edge entry_edge,
> }
>
> gsi = gsi_last_bb (loop_header);
> - create_iv (initial_value, stride, iv, loop, &gsi, false,
> + create_iv (initial_value, PLUS_EXPR, stride, iv, loop, &gsi, false,
> iv_before, iv_after);
>
> /* Insert loop exit condition. */
> diff --git a/gcc/doc/md.texi b/gcc/doc/md.texi
> index 07bf8bdebff..307d3c407d8 100644
> --- a/gcc/doc/md.texi
> +++ b/gcc/doc/md.texi
> @@ -4965,6 +4965,40 @@ for (i = 1; i < operand3; i++)
> operand0[i] = operand0[i - 1] && (operand1 + i < operand2);
> @end smallexample
>
> +@cindex @code{while_len@var{m}@var{n}} instruction pattern
> +@item @code{while_len@var{m}@var{n}}
> +Set operand 0 to the number of active elements in vector will be updated value.
> +operand 1 is the total elements need to be updated value.
> +operand 2 is the vectorization factor.
> +The value of operand 0 is target dependent and flexible in each iteration.
I can't really parse the above - there's information missing on what
the modes M and N are and how they relate. I suppose you take the
while_ult description and mangle it to the while_len semantics?
> +The operation of this pattern can be:
> +
> +@smallexample
> +Case 1:
> +operand0 = MIN (operand1, operand2);
> +operand2 can be const_poly_int or poly_int related to vector mode size.
> +Some target like RISC-V has a standalone instruction to get MIN (n, MODE SIZE) so
> +that we can reduce a use of general purpose register.
> +
> +In this case, only the last iteration of the loop is partial iteration.
> +@end smallexample
> +
> +@smallexample
> +Case 2:
> +if (operand1 <= operand2)
> + operand0 = operand1;
> +else if (operand1 < 2 * operand2)
> + operand0 = IN_RANGE (ceil (operand1 / 2), operand2);
> +else
> + operand0 = operand2;
> +
> +This case will evenly distribute work over the last 2 iterations of a stripmine loop.
> +@end smallexample
> +
> +The output of this pattern is not only used as IV of loop control counter, but also
> +is used as the IV of address calculation with multiply/shift operation. This allow
> +us dynamic adjust the number of elements is processed in each iteration of the loop.
> +
> @cindex @code{check_raw_ptrs@var{m}} instruction pattern
> @item @samp{check_raw_ptrs@var{m}}
> Check whether, given two pointers @var{a} and @var{b} and a length @var{len},
> diff --git a/gcc/gimple-loop-interchange.cc b/gcc/gimple-loop-interchange.cc
> index 1b77bfd46b2..e5590374e59 100644
> --- a/gcc/gimple-loop-interchange.cc
> +++ b/gcc/gimple-loop-interchange.cc
> @@ -1185,7 +1185,7 @@ tree_loop_interchange::map_inductions_to_loop (loop_cand &src, loop_cand &tgt)
> tree var_before, var_after;
> tree base = unshare_expr (iv->init_expr);
> tree step = unshare_expr (iv->step);
> - create_iv (base, step, SSA_NAME_VAR (iv->var),
> + create_iv (base, PLUS_EXPR, step, SSA_NAME_VAR (iv->var),
> tgt.m_loop, &incr_pos, false, &var_before, &var_after);
> bitmap_set_bit (m_dce_seeds, SSA_NAME_VERSION (var_before));
> bitmap_set_bit (m_dce_seeds, SSA_NAME_VERSION (var_after));
> diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def
> index 7fe742c2ae7..9f7269f8f60 100644
> --- a/gcc/internal-fn.def
> +++ b/gcc/internal-fn.def
> @@ -153,6 +153,7 @@ DEF_INTERNAL_OPTAB_FN (VEC_SET, 0, vec_set, vec_set)
> DEF_INTERNAL_OPTAB_FN (LEN_STORE, 0, len_store, len_store)
>
> DEF_INTERNAL_OPTAB_FN (WHILE_ULT, ECF_CONST | ECF_NOTHROW, while_ult, while)
> +DEF_INTERNAL_OPTAB_FN (WHILE_LEN, ECF_CONST | ECF_NOTHROW, while_len, binary)
> DEF_INTERNAL_OPTAB_FN (CHECK_RAW_PTRS, ECF_CONST | ECF_NOTHROW,
> check_raw_ptrs, check_ptrs)
> DEF_INTERNAL_OPTAB_FN (CHECK_WAR_PTRS, ECF_CONST | ECF_NOTHROW,
> diff --git a/gcc/optabs.def b/gcc/optabs.def
> index 695f5911b30..f5938bd2c24 100644
> --- a/gcc/optabs.def
> +++ b/gcc/optabs.def
> @@ -476,3 +476,4 @@ OPTAB_DC (vec_series_optab, "vec_series$a", VEC_SERIES)
> OPTAB_D (vec_shl_insert_optab, "vec_shl_insert_$a")
> OPTAB_D (len_load_optab, "len_load_$a")
> OPTAB_D (len_store_optab, "len_store_$a")
> +OPTAB_D (while_len_optab, "while_len$a")
But your description mentions two modes?
> diff --git a/gcc/tree-ssa-loop-ivcanon.cc b/gcc/tree-ssa-loop-ivcanon.cc
> index 9f72d534b7c..6941d0a4d1d 100644
> --- a/gcc/tree-ssa-loop-ivcanon.cc
> +++ b/gcc/tree-ssa-loop-ivcanon.cc
> @@ -113,7 +113,7 @@ create_canonical_iv (class loop *loop, edge exit, tree niter,
> niter,
> build_int_cst (type, 1));
> incr_at = gsi_last_bb (in->src);
> - create_iv (niter,
> + create_iv (niter, PLUS_EXPR,
> build_int_cst (type, -1),
> NULL_TREE, loop,
> &incr_at, false, var_before, &var);
> diff --git a/gcc/tree-ssa-loop-ivopts.cc b/gcc/tree-ssa-loop-ivopts.cc
> index 0dd47910f97..a7377f91f9e 100644
> --- a/gcc/tree-ssa-loop-ivopts.cc
> +++ b/gcc/tree-ssa-loop-ivopts.cc
> @@ -7267,7 +7267,7 @@ create_new_iv (struct ivopts_data *data, struct iv_cand *cand)
>
> base = unshare_expr (cand->iv->base);
>
> - create_iv (base, unshare_expr (cand->iv->step),
> + create_iv (base, PLUS_EXPR, unshare_expr (cand->iv->step),
> cand->var_before, data->current_loop,
> &incr_pos, after, &cand->var_before, &cand->var_after);
> }
> diff --git a/gcc/tree-ssa-loop-manip.cc b/gcc/tree-ssa-loop-manip.cc
> index a52277abdbf..b0d254b0c89 100644
> --- a/gcc/tree-ssa-loop-manip.cc
> +++ b/gcc/tree-ssa-loop-manip.cc
> @@ -47,7 +47,9 @@ along with GCC; see the file COPYING3. If not see
> so that we can free them all at once. */
> static bitmap_obstack loop_renamer_obstack;
>
> -/* Creates an induction variable with value BASE + STEP * iteration in LOOP.
> +/* Creates an induction variable with value BASE (+/-) STEP * iteration in LOOP.
> + If CODE is PLUS_EXPR, the induction variable is BASE + STEP * iteration.
> + If CODE is MINUS_EXPR, the induction variable is BASE - STEP * iteration.
> It is expected that neither BASE nor STEP are shared with other expressions
> (unless the sharing rules allow this). Use VAR as a base var_decl for it
> (if NULL, a new temporary will be created). The increment will occur at
> @@ -57,8 +59,8 @@ static bitmap_obstack loop_renamer_obstack;
> VAR_AFTER (unless they are NULL). */
>
> void
> -create_iv (tree base, tree step, tree var, class loop *loop,
> - gimple_stmt_iterator *incr_pos, bool after,
> +create_iv (tree base, tree_code code, tree step, tree var,
> + class loop *loop, gimple_stmt_iterator *incr_pos, bool after,
> tree *var_before, tree *var_after)
> {
> gassign *stmt;
> @@ -66,7 +68,9 @@ create_iv (tree base, tree step, tree var, class loop *loop,
> tree initial, step1;
> gimple_seq stmts;
> tree vb, va;
> - enum tree_code incr_op = PLUS_EXPR;
> + /* The code can only be PLUS_EXPR or MINUS_EXPR. */
> + gcc_assert (code == PLUS_EXPR || code == MINUS_EXPR);
Ah, you already have an assert.
> + tree_code incr_op = code;
If you name the parameter incr_op you can get rid of this assignment.
> edge pe = loop_preheader_edge (loop);
>
> if (var != NULL_TREE)
> @@ -1368,7 +1372,7 @@ tree_transform_and_unroll_loop (class loop *loop, unsigned factor,
> tree ctr_before, ctr_after;
> gimple_stmt_iterator bsi = gsi_last_nondebug_bb (new_exit->src);
> exit_if = as_a <gcond *> (gsi_stmt (bsi));
> - create_iv (exit_base, exit_step, NULL_TREE, loop,
> + create_iv (exit_base, PLUS_EXPR, exit_step, NULL_TREE, loop,
> &bsi, false, &ctr_before, &ctr_after);
> gimple_cond_set_code (exit_if, exit_cmp);
> gimple_cond_set_lhs (exit_if, ctr_after);
> @@ -1583,8 +1587,8 @@ canonicalize_loop_ivs (class loop *loop, tree *nit, bool bump_in_latch)
> gsi = gsi_last_bb (loop->latch);
> else
> gsi = gsi_last_nondebug_bb (loop->header);
> - create_iv (build_int_cst_type (type, 0), build_int_cst (type, 1), NULL_TREE,
> - loop, &gsi, bump_in_latch, &var_before, NULL);
> + create_iv (build_int_cst_type (type, 0), PLUS_EXPR, build_int_cst (type, 1),
> + NULL_TREE, loop, &gsi, bump_in_latch, &var_before, NULL);
>
> rewrite_all_phi_nodes_with_iv (loop, var_before);
>
> diff --git a/gcc/tree-ssa-loop-manip.h b/gcc/tree-ssa-loop-manip.h
> index d49273a3987..bda09f51d56 100644
> --- a/gcc/tree-ssa-loop-manip.h
> +++ b/gcc/tree-ssa-loop-manip.h
> @@ -22,8 +22,8 @@ along with GCC; see the file COPYING3. If not see
>
> typedef void (*transform_callback)(class loop *, void *);
>
> -extern void create_iv (tree, tree, tree, class loop *, gimple_stmt_iterator *,
> - bool, tree *, tree *);
> +extern void create_iv (tree, tree_code, tree, tree, class loop *,
> + gimple_stmt_iterator *, bool, tree *, tree *);
> extern void rewrite_into_loop_closed_ssa (bitmap, unsigned);
> extern void verify_loop_closed_ssa (bool, class loop * = NULL);
>
> diff --git a/gcc/tree-vect-data-refs.cc b/gcc/tree-vect-data-refs.cc
> index c03ffb3aaf1..355ede5c0d9 100644
> --- a/gcc/tree-vect-data-refs.cc
> +++ b/gcc/tree-vect-data-refs.cc
> @@ -5101,7 +5101,7 @@ vect_create_data_ref_ptr (vec_info *vinfo, stmt_vec_info stmt_info,
>
> standard_iv_increment_position (loop, &incr_gsi, &insert_after);
>
> - create_iv (aggr_ptr_init,
> + create_iv (aggr_ptr_init, PLUS_EXPR,
> fold_convert (aggr_ptr_type, iv_step),
> aggr_ptr, loop, &incr_gsi, insert_after,
> &indx_before_incr, &indx_after_incr);
> @@ -5131,9 +5131,9 @@ vect_create_data_ref_ptr (vec_info *vinfo, stmt_vec_info stmt_info,
> {
> standard_iv_increment_position (containing_loop, &incr_gsi,
> &insert_after);
> - create_iv (aptr, fold_convert (aggr_ptr_type, DR_STEP (dr)), aggr_ptr,
> - containing_loop, &incr_gsi, insert_after, &indx_before_incr,
> - &indx_after_incr);
> + create_iv (aptr, PLUS_EXPR, fold_convert (aggr_ptr_type, DR_STEP (dr)),
> + aggr_ptr, containing_loop, &incr_gsi, insert_after,
> + &indx_before_incr, &indx_after_incr);
> incr = gsi_stmt (incr_gsi);
>
> /* Copy the points-to information if it exists. */
> diff --git a/gcc/tree-vect-loop-manip.cc b/gcc/tree-vect-loop-manip.cc
> index f60fa50e8f4..6973314adb5 100644
> --- a/gcc/tree-vect-loop-manip.cc
> +++ b/gcc/tree-vect-loop-manip.cc
> @@ -468,8 +468,9 @@ vect_set_loop_controls_directly (class loop *loop, loop_vec_info loop_vinfo,
> gimple_stmt_iterator incr_gsi;
> bool insert_after;
> standard_iv_increment_position (loop, &incr_gsi, &insert_after);
> - create_iv (build_int_cst (iv_type, 0), nitems_step, NULL_TREE, loop,
> - &incr_gsi, insert_after, &index_before_incr, &index_after_incr);
> + create_iv (build_int_cst (iv_type, 0), PLUS_EXPR, nitems_step, NULL_TREE,
> + loop, &incr_gsi, insert_after, &index_before_incr,
> + &index_after_incr);
>
> tree zero_index = build_int_cst (compare_type, 0);
> tree test_index, test_limit, first_limit;
> @@ -682,6 +683,260 @@ vect_set_loop_controls_directly (class loop *loop, loop_vec_info loop_vinfo,
> return next_ctrl;
> }
>
> +/* Helper for vect_set_loop_condition_partial_vectors. Generate definitions
> + for all the rgroup controls in RGC and return a control that is nonzero
> + when the loop needs to iterate. Add any new preheader statements to
> + PREHEADER_SEQ. Use LOOP_COND_GSI to insert code before the exit gcond.
> +
> + RGC belongs to loop LOOP. The loop originally iterated NITERS
> + times and has been vectorized according to LOOP_VINFO.
> +
> + Unlike vect_set_loop_controls_directly which is iterating from 0-based IV
> + to TEST_LIMIT - bias.
> +
> + In vect_set_loop_controls_by_while_len, we are iterating from start at
> + IV = TEST_LIMIT - bias and keep subtract IV by the length calculated by
> + IFN_WHILE_LEN pattern.
> +
> + 1. Single rgroup, the Gimple IR should be:
> +
> + # vectp_B.6_8 = PHI <vectp_B.6_13(6), &B(5)>
> + # vectp_B.8_16 = PHI <vectp_B.8_17(6), &B(5)>
> + # vectp_A.11_19 = PHI <vectp_A.11_20(6), &A(5)>
> + # vectp_A.13_22 = PHI <vectp_A.13_23(6), &A(5)>
> + # ivtmp_26 = PHI <ivtmp_27(6), _25(5)>
> + _28 = .WHILE_LEN (ivtmp_26, POLY_INT_CST [4, 4]);
> + ivtmp_15 = _28 * 4;
> + vect__1.10_18 = .LEN_LOAD (vectp_B.8_16, 128B, _28, 0);
> + _1 = B[i_10];
> + .LEN_STORE (vectp_A.13_22, 128B, _28, vect__1.10_18, 0);
> + i_7 = i_10 + 1;
> + vectp_B.8_17 = vectp_B.8_16 + ivtmp_15;
> + vectp_A.13_23 = vectp_A.13_22 + ivtmp_15;
> + ivtmp_27 = ivtmp_26 - _28;
> + if (ivtmp_27 != 0)
> + goto <bb 6>; [83.33%]
> + else
> + goto <bb 7>; [16.67%]
> +
> + Note: We use the outcome of .WHILE_LEN to adjust both loop control IV and
> + data reference pointer IV.
> +
> + 1). The result of .WHILE_LEN:
> + _28 = .WHILE_LEN (ivtmp_26, POLY_INT_CST [4, 4]);
> + The _28 is not necessary to be VF in any iteration, instead, we allow
> + _28 to be any value as long as it it <= VF. Such flexible WHILE_LEN
"it is", but I don't really understand what you are trying to say? Is it
that loop control cannot be implemented as
if (_28 != POLY_INT_CST [4, 4])
next iteration will be the last;
but the result can only be used to decrement the remaining scalar
iterations? That is, .WHILE_LEN (a, b) doesn't actually subtract
'b' but may choose to subtract any value in [1, b]?
How can this work when the vectorizer "knows" that 'b' covers two
vectors and thus it emits two vector stmts?
> + pattern allows target have various flexible optimizations in vector
> + loop iterations. Target like RISC-V has special application vector
> + length calculation instruction which will distribute even workload
> + in the last 2 iterations.
> +
> + Other example is that we can allow even generate _28 <= VF / 2 so
> + that some machine can run vector codes in low power mode.
> +
> + 2). Loop control IV:
> + ivtmp_27 = ivtmp_26 - _28;
> + if (ivtmp_27 != 0)
> + goto <bb 6>; [83.33%]
> + else
> + goto <bb 7>; [16.67%]
> +
> + This is the saturating-subtraction towards zero, the outcome of
> + .WHILE_LEN wil make ivtmp_27 never underflow zero.
So it's not actually saturating. The saturating operation is
done by .WHILE_LEN?
> + 3). Data reference pointer IV:
> + ivtmp_15 = _28 * 4;
> + vectp_B.8_17 = vectp_B.8_16 + ivtmp_15;
> + vectp_A.13_23 = vectp_A.13_22 + ivtmp_15;
> +
> + The pointer IV is adjusted accurately according to the .WHILE_LEN.
Note that this will be bad for IVOPTs, it would be _way_ better if
the increment were (poly-)constant. For powerpc this would mean
complex addressing modes cannot be used (not sure if they have those).
> + 2. Multiple rgroup, the Gimple IR should be:
> +
> + # i_23 = PHI <i_20(6), 0(11)>
> + # vectp_f.8_51 = PHI <vectp_f.8_52(6), f_15(D)(11)>
> + # vectp_d.10_59 = PHI <vectp_d.10_60(6), d_18(D)(11)>
> + # ivtmp_70 = PHI <ivtmp_71(6), _69(11)>
> + # ivtmp_73 = PHI <ivtmp_74(6), _67(11)>
> + _72 = MIN_EXPR <ivtmp_70, 16>;
> + _75 = MIN_EXPR <ivtmp_73, 16>;
> + _1 = i_23 * 2;
> + _2 = (long unsigned int) _1;
> + _3 = _2 * 2;
> + _4 = f_15(D) + _3;
> + _5 = _2 + 1;
> + _6 = _5 * 2;
> + _7 = f_15(D) + _6;
> + .LEN_STORE (vectp_f.8_51, 128B, _75, { 1, 2, 1, 2, 1, 2, 1, 2 }, 0);
> + vectp_f.8_56 = vectp_f.8_51 + 16;
> + .LEN_STORE (vectp_f.8_56, 128B, _72, { 1, 2, 1, 2, 1, 2, 1, 2 }, 0);
> + _8 = (long unsigned int) i_23;
> + _9 = _8 * 4;
> + _10 = d_18(D) + _9;
> + _61 = _75 / 2;
> + .LEN_STORE (vectp_d.10_59, 128B, _61, { 3, 3, 3, 3 }, 0);
> + vectp_d.10_63 = vectp_d.10_59 + 16;
> + _64 = _72 / 2;
> + .LEN_STORE (vectp_d.10_63, 128B, _64, { 3, 3, 3, 3 }, 0);
> + i_20 = i_23 + 1;
> + vectp_f.8_52 = vectp_f.8_56 + 16;
> + vectp_d.10_60 = vectp_d.10_63 + 16;
> + ivtmp_74 = ivtmp_73 - _75;
> + ivtmp_71 = ivtmp_70 - _72;
> + if (ivtmp_74 != 0)
> + goto <bb 6>; [83.33%]
> + else
> + goto <bb 13>; [16.67%]
> +
> + Note: We DO NOT use .WHILE_LEN in SLP auto-vectorization for multiple
> + rgroups. Instead, we use MIN_EXPR to guarantee we always use VF as the
> + iteration amount for mutiple rgroups.
I see. I wonder if it makes sense to leave .WHILE_LEN aside for a start,
the above scheme should also work for single rgroups, no?
> + The analysis of the flow of multiple rgroups:
> + _72 = MIN_EXPR <ivtmp_70, 16>;
> + _75 = MIN_EXPR <ivtmp_73, 16>;
> + ...
> + .LEN_STORE (vectp_f.8_51, 128B, _75, { 1, 2, 1, 2, 1, 2, 1, 2 }, 0);
> + vectp_f.8_56 = vectp_f.8_51 + 16;
> + .LEN_STORE (vectp_f.8_56, 128B, _72, { 1, 2, 1, 2, 1, 2, 1, 2 }, 0);
> + ...
> + _61 = _75 / 2;
> + .LEN_STORE (vectp_d.10_59, 128B, _61, { 3, 3, 3, 3 }, 0);
> + vectp_d.10_63 = vectp_d.10_59 + 16;
> + _64 = _72 / 2;
> + .LEN_STORE (vectp_d.10_63, 128B, _64, { 3, 3, 3, 3 }, 0);
> +
> + We use _72 = MIN_EXPR <ivtmp_70, 16>; to generate the number of the elements
> + to be processed in each iteration.
> +
> + The related STOREs:
> + _72 = MIN_EXPR <ivtmp_70, 16>;
> + .LEN_STORE (vectp_f.8_56, 128B, _72, { 1, 2, 1, 2, 1, 2, 1, 2 }, 0);
> + _64 = _72 / 2;
> + .LEN_STORE (vectp_d.10_63, 128B, _64, { 3, 3, 3, 3 }, 0);
> + Since these 2 STOREs store 2 vectors that the second vector is half elements
> + of the first vector. So the length of second STORE will be _64 = _72 / 2;
> + It's similar to the VIEW_CONVERT of handling masks in SLP.
> +*/
> +
> +static tree
> +vect_set_loop_controls_by_while_len (class loop *loop, loop_vec_info loop_vinfo,
> + gimple_seq *preheader_seq,
> + gimple_seq *header_seq,
> + rgroup_controls *rgc, tree niters)
> +{
> + tree compare_type = LOOP_VINFO_RGROUP_COMPARE_TYPE (loop_vinfo);
> + tree iv_type = LOOP_VINFO_RGROUP_IV_TYPE (loop_vinfo);
> + /* We are not allowing masked approach in WHILE_LEN. */
> + gcc_assert (!LOOP_VINFO_FULLY_MASKED_P (loop_vinfo));
> +
> + tree ctrl_type = rgc->type;
> + unsigned int nitems_per_iter = rgc->max_nscalars_per_iter * rgc->factor;
> + poly_uint64 nitems_per_ctrl = TYPE_VECTOR_SUBPARTS (ctrl_type) * rgc->factor;
> + poly_uint64 vf = LOOP_VINFO_VECT_FACTOR (loop_vinfo);
> +
> + /* Calculate the maximum number of item values that the rgroup
> + handles in total, the number that it handles for each iteration
> + of the vector loop. */
> + tree nitems_total = niters;
> + if (nitems_per_iter != 1)
> + {
> + /* We checked before setting LOOP_VINFO_USING_PARTIAL_VECTORS_P that
> + these multiplications don't overflow. */
> + tree compare_factor = build_int_cst (compare_type, nitems_per_iter);
> + nitems_total = gimple_build (preheader_seq, MULT_EXPR, compare_type,
> + nitems_total, compare_factor);
> + }
> +
> + /* Convert the comparison value to the IV type (either a no-op or
> + a promotion). */
> + nitems_total = gimple_convert (preheader_seq, iv_type, nitems_total);
> +
> + /* Create an induction variable that counts the number of items
> + processed. */
> + tree index_before_incr, index_after_incr;
> + gimple_stmt_iterator incr_gsi;
> + bool insert_after;
> + standard_iv_increment_position (loop, &incr_gsi, &insert_after);
> +
> + /* Test the decremented IV, which will never underflow 0 since we have
> + IFN_WHILE_LEN to gurantee that. */
> + tree test_limit = nitems_total;
> +
> + /* Provide a definition of each control in the group. */
> + tree ctrl;
> + unsigned int i;
> + FOR_EACH_VEC_ELT_REVERSE (rgc->controls, i, ctrl)
> + {
> + /* Previous controls will cover BIAS items. This control covers the
> + next batch. */
> + poly_uint64 bias = nitems_per_ctrl * i;
> + tree bias_tree = build_int_cst (iv_type, bias);
> +
> + /* Rather than have a new IV that starts at TEST_LIMIT and goes down to
> + BIAS, prefer to use the same TEST_LIMIT - BIAS based IV for each
> + control and adjust the bound down by BIAS. */
> + tree this_test_limit = test_limit;
> + if (i != 0)
> + {
> + this_test_limit = gimple_build (preheader_seq, MAX_EXPR, iv_type,
> + this_test_limit, bias_tree);
> + this_test_limit = gimple_build (preheader_seq, MINUS_EXPR, iv_type,
> + this_test_limit, bias_tree);
> + }
> +
> + /* Create decrement IV. */
> + create_iv (this_test_limit, MINUS_EXPR, ctrl, NULL_TREE, loop, &incr_gsi,
> + insert_after, &index_before_incr, &index_after_incr);
> +
> + poly_uint64 final_vf = vf * nitems_per_iter;
> + tree vf_step = build_int_cst (iv_type, final_vf);
> + tree res_len;
> + if (nitems_per_iter != 1)
> + {
> + /* For SLP, we can't allow non-VF number of elements to be processed
> + in non-final iteration. We force the number of elements to be
> + processed in each non-final iteration is VF elements. If we allow
> + non-VF elements processing in non-final iteration will make SLP too
> + complicated and produce inferior codegen.
> +
> + For example:
> +
> + If non-final iteration process VF elements.
> +
> + ...
> + .LEN_STORE (vectp_f.8_51, 128B, _71, { 1, 2, 1, 2 }, 0);
> + .LEN_STORE (vectp_f.8_56, 128B, _72, { 1, 2, 1, 2 }, 0);
> + ...
> +
> + If non-final iteration process non-VF elements.
> +
> + ...
> + .LEN_STORE (vectp_f.8_51, 128B, _71, { 1, 2, 1, 2 }, 0);
> + if (_71 % 2 == 0)
> + .LEN_STORE (vectp_f.8_56, 128B, _72, { 1, 2, 1, 2 }, 0);
> + else
> + .LEN_STORE (vectp_f.8_56, 128B, _72, { 2, 1, 2, 1 }, 0);
> + ...
> +
> + This is the simple case of 2-elements interleaved vector SLP. We
> + consider other interleave vector, the situation will become more
> + complicated. */
> + res_len = gimple_build (header_seq, MIN_EXPR, iv_type,
> + index_before_incr, vf_step);
> + }
> + else
> + {
> + res_len = gimple_build (header_seq, IFN_WHILE_LEN, iv_type,
> + index_before_incr, vf_step);
> + }
> + gassign *assign = gimple_build_assign (ctrl, res_len);
> + gimple_seq_add_stmt (header_seq, assign);
> + }
> +
> + return index_after_incr;
> +}
> +
> /* Set up the iteration condition and rgroup controls for LOOP, given
> that LOOP_VINFO_USING_PARTIAL_VECTORS_P is true for the vectorized
> loop. LOOP_VINFO describes the vectorization of LOOP. NITERS is
> @@ -703,6 +958,7 @@ vect_set_loop_condition_partial_vectors (class loop *loop,
>
> bool use_masks_p = LOOP_VINFO_FULLY_MASKED_P (loop_vinfo);
> tree compare_type = LOOP_VINFO_RGROUP_COMPARE_TYPE (loop_vinfo);
> + tree iv_type = LOOP_VINFO_RGROUP_IV_TYPE (loop_vinfo);
> unsigned int compare_precision = TYPE_PRECISION (compare_type);
> tree orig_niters = niters;
>
> @@ -757,12 +1013,18 @@ vect_set_loop_condition_partial_vectors (class loop *loop,
> bool might_wrap_p = vect_rgroup_iv_might_wrap_p (loop_vinfo, rgc);
>
> /* Set up all controls for this group. */
> - test_ctrl = vect_set_loop_controls_directly (loop, loop_vinfo,
> - &preheader_seq,
> - &header_seq,
> - loop_cond_gsi, rgc,
> - niters, niters_skip,
> - might_wrap_p);
> + if (direct_internal_fn_supported_p (IFN_WHILE_LEN, iv_type,
> + OPTIMIZE_FOR_SPEED))
> + test_ctrl
> + = vect_set_loop_controls_by_while_len (loop, loop_vinfo,
> + &preheader_seq, &header_seq,
> + rgc, niters);
> + else
> + test_ctrl
> + = vect_set_loop_controls_directly (loop, loop_vinfo, &preheader_seq,
> + &header_seq, loop_cond_gsi, rgc,
> + niters, niters_skip,
> + might_wrap_p);
> }
>
> /* Emit all accumulated statements. */
> @@ -893,7 +1155,7 @@ vect_set_loop_condition_normal (class loop *loop, tree niters, tree step,
> }
>
> standard_iv_increment_position (loop, &incr_gsi, &insert_after);
> - create_iv (init, step, NULL_TREE, loop,
> + create_iv (init, PLUS_EXPR, step, NULL_TREE, loop,
> &incr_gsi, insert_after, &indx_before_incr, &indx_after_incr);
> indx_after_incr = force_gimple_operand_gsi (&loop_cond_gsi, indx_after_incr,
> true, NULL_TREE, true,
> diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
> index 6ea0f21fd13..d6e90ed93cd 100644
> --- a/gcc/tree-vect-loop.cc
> +++ b/gcc/tree-vect-loop.cc
> @@ -5567,7 +5567,7 @@ vect_create_epilog_for_reduction (loop_vec_info loop_vinfo,
> gimple_stmt_iterator incr_gsi;
> bool insert_after;
> standard_iv_increment_position (loop, &incr_gsi, &insert_after);
> - create_iv (series_vect, vec_step, NULL_TREE, loop, &incr_gsi,
> + create_iv (series_vect, PLUS_EXPR, vec_step, NULL_TREE, loop, &incr_gsi,
> insert_after, &indx_before_incr, &indx_after_incr);
>
> /* Next create a new phi node vector (NEW_PHI_TREE) which starts
> @@ -10364,12 +10364,14 @@ vect_record_loop_len (loop_vec_info loop_vinfo, vec_loop_lens *lens,
> rgroup that operates on NVECTORS vectors, where 0 <= INDEX < NVECTORS. */
>
> tree
> -vect_get_loop_len (loop_vec_info loop_vinfo, vec_loop_lens *lens,
> - unsigned int nvectors, unsigned int index)
> +vect_get_loop_len (gimple_stmt_iterator *gsi, loop_vec_info loop_vinfo,
please keep loop_vinfo as first parameter and update the comment for the
new.
> + vec_loop_lens *lens, unsigned int nvectors, tree vectype,
> + unsigned int index)
> {
> rgroup_controls *rgl = &(*lens)[nvectors - 1];
> bool use_bias_adjusted_len =
> LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo) != 0;
> + tree iv_type = LOOP_VINFO_RGROUP_IV_TYPE (loop_vinfo);
>
> /* Populate the rgroup's len array, if this is the first time we've
> used it. */
> @@ -10400,6 +10402,27 @@ vect_get_loop_len (loop_vec_info loop_vinfo, vec_loop_lens *lens,
>
> if (use_bias_adjusted_len)
> return rgl->bias_adjusted_ctrl;
> + else if (direct_internal_fn_supported_p (IFN_WHILE_LEN, iv_type,
> + OPTIMIZE_FOR_SPEED))
It's odd to see this test here - why's the .WHILE_LEN sufficiently
different from "other" loop_len cases?
> + {
> + tree loop_len = rgl->controls[index];
> + poly_int64 nunits1 = TYPE_VECTOR_SUBPARTS (rgl->type);
> + poly_int64 nunits2 = TYPE_VECTOR_SUBPARTS (vectype);
> + if (maybe_ne (nunits1, nunits2))
> + {
> + /* A loop len for data type X can be reused for data type Y
> + if X has N times more elements than Y and if Y's elements
> + are N times bigger than X's. */
> + gcc_assert (multiple_p (nunits1, nunits2));
> + unsigned int factor = exact_div (nunits1, nunits2).to_constant ();
> + gimple_seq seq = NULL;
> + loop_len = gimple_build (&seq, RDIV_EXPR, iv_type, loop_len,
> + build_int_cst (iv_type, factor));
> + if (seq)
> + gsi_insert_seq_before (gsi, seq, GSI_SAME_STMT);
> + }
> + return loop_len;
You are going to re-emit this division for each load/store, shouldn't
this be integrated in the loop populating controls[] at the first call?
> + }
> else
> return rgl->controls[index];
> }
> diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
> index 6b7dbfd4a23..98bef8f4fc9 100644
> --- a/gcc/tree-vect-stmts.cc
> +++ b/gcc/tree-vect-stmts.cc
> @@ -3144,6 +3144,70 @@ vect_get_data_ptr_increment (vec_info *vinfo,
> return iv_step;
> }
>
> +/* Prepare the pointer IVs which needs to be updated by a variable amount.
> + Such variable amount is the outcome of .WHILE_LEN. In this case, we can
> + allow each iteration process the flexible number of elements as long as
> + the number <= vf elments.
> +
> + Return data reference according to WHILE_LEN.
> + If new statements are needed, insert them before GSI. */
> +
> +static tree
> +get_while_len_data_ref_ptr (vec_info *vinfo, stmt_vec_info stmt_info,
> + tree aggr_type, class loop *at_loop, tree offset,
> + tree *dummy, gimple_stmt_iterator *gsi,
> + bool simd_lane_access_p, vec_loop_lens *loop_lens,
> + dr_vec_info *dr_info,
> + vect_memory_access_type memory_access_type)
> +{
> + if (!loop_lens || loop_lens->length () != 1)
> + return NULL_TREE;
> + loop_vec_info loop_vinfo = dyn_cast<loop_vec_info> (vinfo);
> + tree iv_type = LOOP_VINFO_RGROUP_IV_TYPE (loop_vinfo);
> + tree step = vect_dr_behavior (vinfo, dr_info)->step;
> + if (!direct_internal_fn_supported_p (IFN_WHILE_LEN, iv_type,
> + OPTIMIZE_FOR_SPEED))
> + return NULL_TREE;
> +
> + if (memory_access_type == VMAT_INVARIANT)
> + return NULL_TREE;
> +
> + /* TODO: We don't support gather/scatter or load_lanes/store_lanes for pointer
> + IVs are updated by variable amount but we will support them in the future.
> + */
> + gcc_assert (memory_access_type != VMAT_GATHER_SCATTER
> + && memory_access_type != VMAT_LOAD_STORE_LANES);
All this should be gated at the caller site. Quoting one:
@@ -9798,6 +9872,15 @@ vectorizable_load (vec_info *vinfo,
at_loop,
offset, &dummy, gsi, &ptr_incr,
simd_lane_access_p, bump);
+
+ tree while_len_data_ref_ptr
+ = get_while_len_data_ref_ptr (vinfo, stmt_info, aggr_type,
at_loop,
+ offset, &dummy, gsi,
+ simd_lane_access_p, loop_lens,
+ dr_info, memory_access_type);
+ if (while_len_data_ref_ptr)
+ dataref_ptr = while_len_data_ref_ptr;
you call this after other methods have computed a dataref_ptr, that's
very ugly. Instead you should compute the "update method" during
load/store _analysis_ and then conciously decide on using this
function (which then better should succeed). That will also be required
to properly apply costing during analysis.
I also wonder how you can get away without touching the j == 1 case.
I suppose we'll ever use the .WHILE_LEN path then.
> + /* When we support WHILE_LEN pattern, we dynamic adjust
> + the memory address by .WHILE_LEN result.
> +
> + The result of .WHILE_LEN is the number of elements to
> + be processed of each iteration. So the memory address
> + adjustment operation should be:
> +
> + bytesize = GET_MODE_SIZE (element_mode (aggr_type));
> + addr = addr + .WHILE_LEN (ARG..) * bytesize;
> + */
> + gimple *ptr_incr;
> + tree loop_len
> + = vect_get_loop_len (gsi, loop_vinfo, loop_lens, 1, aggr_type, 0);
> + tree len_type = TREE_TYPE (loop_len);
> + poly_uint64 bytesize = GET_MODE_SIZE (element_mode (aggr_type));
> + /* Since the outcome of .WHILE_LEN is element size, we should adjust
> + it into bytesize so that it can be used in address pointer variable
> + amount IVs adjustment. */
> + tree tmp = fold_build2 (MULT_EXPR, len_type, loop_len,
> + build_int_cst (len_type, bytesize));
> + if (tree_int_cst_sgn (step) == -1)
> + tmp = fold_build1 (NEGATE_EXPR, len_type, tmp);
> + tree bump = make_temp_ssa_name (len_type, NULL, "ivtmp");
> + gassign *assign = gimple_build_assign (bump, tmp);
> + gsi_insert_before (gsi, assign, GSI_SAME_STMT);
> + return vect_create_data_ref_ptr (vinfo, stmt_info, aggr_type, at_loop, offset,
> + dummy, gsi, &ptr_incr, simd_lane_access_p,
> + bump);
> +}
> +
> /* Check and perform vectorization of BUILT_IN_BSWAP{16,32,64,128}. */
>
> static bool
> @@ -8195,7 +8259,7 @@ vectorizable_store (vec_info *vinfo,
>
> stride_base = cse_and_gimplify_to_preheader (loop_vinfo, stride_base);
> ivstep = cse_and_gimplify_to_preheader (loop_vinfo, ivstep);
> - create_iv (stride_base, ivstep, NULL,
> + create_iv (stride_base, PLUS_EXPR, ivstep, NULL,
> loop, &incr_gsi, insert_after,
> &offvar, NULL);
> incr = gsi_stmt (incr_gsi);
> @@ -8465,6 +8529,15 @@ vectorizable_store (vec_info *vinfo,
> simd_lane_access_p ? loop : NULL,
> offset, &dummy, gsi, &ptr_incr,
> simd_lane_access_p, bump);
> +
> + tree while_len_data_ref_ptr
> + = get_while_len_data_ref_ptr (vinfo, stmt_info, aggr_type,
> + simd_lane_access_p ? loop : NULL,
> + offset, &dummy, gsi,
> + simd_lane_access_p, loop_lens,
> + dr_info, memory_access_type);
> + if (while_len_data_ref_ptr)
> + dataref_ptr = while_len_data_ref_ptr;
> }
> else
> {
> @@ -8652,8 +8725,9 @@ vectorizable_store (vec_info *vinfo,
> else if (loop_lens)
> {
> tree final_len
> - = vect_get_loop_len (loop_vinfo, loop_lens,
> - vec_num * ncopies, vec_num * j + i);
> + = vect_get_loop_len (gsi, loop_vinfo, loop_lens,
> + vec_num * ncopies, vectype,
> + vec_num * j + i);
> tree ptr = build_int_cst (ref_type, align * BITS_PER_UNIT);
> machine_mode vmode = TYPE_MODE (vectype);
> opt_machine_mode new_ovmode
> @@ -9314,7 +9388,7 @@ vectorizable_load (vec_info *vinfo,
>
> stride_base = cse_and_gimplify_to_preheader (loop_vinfo, stride_base);
> ivstep = cse_and_gimplify_to_preheader (loop_vinfo, ivstep);
> - create_iv (stride_base, ivstep, NULL,
> + create_iv (stride_base, PLUS_EXPR, ivstep, NULL,
> loop, &incr_gsi, insert_after,
> &offvar, NULL);
>
> @@ -9798,6 +9872,15 @@ vectorizable_load (vec_info *vinfo,
> at_loop,
> offset, &dummy, gsi, &ptr_incr,
> simd_lane_access_p, bump);
> +
> + tree while_len_data_ref_ptr
> + = get_while_len_data_ref_ptr (vinfo, stmt_info, aggr_type, at_loop,
> + offset, &dummy, gsi,
> + simd_lane_access_p, loop_lens,
> + dr_info, memory_access_type);
> + if (while_len_data_ref_ptr)
> + dataref_ptr = while_len_data_ref_ptr;
> +
> if (mask)
> vec_mask = vec_masks[0];
> }
> @@ -10008,8 +10091,8 @@ vectorizable_load (vec_info *vinfo,
> else if (loop_lens && memory_access_type != VMAT_INVARIANT)
> {
> tree final_len
> - = vect_get_loop_len (loop_vinfo, loop_lens,
> - vec_num * ncopies,
> + = vect_get_loop_len (gsi, loop_vinfo, loop_lens,
> + vec_num * ncopies, vectype,
> vec_num * j + i);
> tree ptr = build_int_cst (ref_type,
> align * BITS_PER_UNIT);
> diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
> index 9cf2fb23fe3..811c17e1d3d 100644
> --- a/gcc/tree-vectorizer.h
> +++ b/gcc/tree-vectorizer.h
> @@ -2293,8 +2293,8 @@ extern tree vect_get_loop_mask (gimple_stmt_iterator *, vec_loop_masks *,
> unsigned int, tree, unsigned int);
> extern void vect_record_loop_len (loop_vec_info, vec_loop_lens *, unsigned int,
> tree, unsigned int);
> -extern tree vect_get_loop_len (loop_vec_info, vec_loop_lens *, unsigned int,
> - unsigned int);
> +extern tree vect_get_loop_len (gimple_stmt_iterator *, loop_vec_info, vec_loop_lens *,
> + unsigned int, tree, unsigned int);
> extern gimple_seq vect_gen_len (tree, tree, tree, tree);
> extern stmt_vec_info info_for_reduction (vec_info *, stmt_vec_info);
> extern bool reduction_fn_for_scalar_code (code_helper, internal_fn *);
Otherwise looks reasonable.
As said, it _looks_ like you can progress without .WHILE_LEN and using
.WHILE_LEN is a pure optimization?
Thanks,
Richard.
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2023-04-26 8:13 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-04-26 3:59 [PATCH V3] VECT: Add decrement IV iteration loop control by variable amount support juzhe.zhong
2023-04-26 8:13 ` Richard Biener
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).