From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtpbgeu1.qq.com (smtpbgeu1.qq.com [52.59.177.22]) by sourceware.org (Postfix) with ESMTPS id A87F93858D33 for ; Wed, 26 Apr 2023 03:59:40 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org A87F93858D33 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivai.ai Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivai.ai X-QQ-mid: bizesmtp67t1682481574tk4fp1uj Received: from server1.localdomain ( [58.60.1.22]) by bizesmtp.qq.com (ESMTP) with id ; Wed, 26 Apr 2023 11:59:33 +0800 (CST) X-QQ-SSF: 01400000000000F0P000000A0000000 X-QQ-FEAT: U4uNK7LTfdrtTPcqQ+aC96J5XRO8jWwbjK7MDDcBKGKO91c0E3CWTKMl53TqO aJOGgPSO4ggvQYJSYXhpNDk7YU/AH/o0xc5F87/pdtuq2M7qvBSHrRukddoeWQC3sY8shjN F9MwIPIiTLxVPsQGtQmJmECeXTXQTk1j9UjvGenOiWOeaBNp3exutFZpXsuP6IeDgtjiL0z 0griw/8iwJASljgQNhXvGtxSYH/XX+zOqPjOcCVxc46h7OyI8VFKyOFsWvTY879aOAwd9+r rujnLvkVIQWrDQUippDmNL4Ko4uYM8KaRlqIeTm3ezob6Rq+E33DgZri2nz9QdFdVby2byw foqjYv9b/k7TgWmIUsbDTtLxqL60Eyvq2mXiWzeKmsY14DBzzDkef6kMReYY19TQh36ska1 X-QQ-GoodBg: 2 X-BIZMAIL-ID: 12690462481909400483 From: juzhe.zhong@rivai.ai To: gcc-patches@gcc.gnu.org Cc: richard.sandiford@arm.com, rguenther@suse.de, rep.dot.nop@gmail.com, Juzhe-Zhong Subject: [PATCH V3] VECT: Add decrement IV iteration loop control by variable amount support Date: Wed, 26 Apr 2023 11:59:29 +0800 Message-Id: <20230426035929.330213-1-juzhe.zhong@rivai.ai> X-Mailer: git-send-email 2.36.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-QQ-SENDSIZE: 520 Feedback-ID: bizesmtp:rivai.ai:qybglogicsvr:qybglogicsvr7 X-Spam-Status: No, score=-10.7 required=5.0 tests=BAYES_00,GIT_PATCH_0,KAM_DMARC_STATUS,RCVD_IN_BARRACUDACENTRAL,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_PASS,TXREP,T_SCC_BODY_TEXT_LINE,T_SPF_HELO_TEMPERROR autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: From: Juzhe-Zhong This patch address all comments from Richard && Bernhard. Fix all codes following their comments: 1. Remove ifn-specific handling of WHILE_LEN, so there is no change in internal-fn.cc in this patch. 2. Add comments for changing interface of "create_iv" function. 3. Reorder the CODE of interface "create_iv", make CODE argument between base and step. 4. Fix the misleading and incorrect comments of "vect_set_loop_controls_by_while_len". 5. remove redundant "enum" of "create_iv" interface and function. Feel free to comment if I meet something. Thanks. gcc/ChangeLog: * cfgloopmanip.cc (create_empty_loop_on_edge): Adapt create_iv interface change. * doc/md.texi: Add WHILE_LEN pattern description. * gimple-loop-interchange.cc (tree_loop_interchange::map_inductions_to_loop): Adapt create_iv interface change. * internal-fn.def (WHILE_LEN): New ifn. * optabs.def (OPTAB_D): New optab pattern. * tree-ssa-loop-ivcanon.cc (create_canonical_iv): Adapt create_iv interface change. * tree-ssa-loop-ivopts.cc (create_new_iv): Ditto. * tree-ssa-loop-manip.cc (create_iv): Ditto. (tree_transform_and_unroll_loop): Ditto. (canonicalize_loop_ivs): Ditto. * tree-ssa-loop-manip.h (create_iv): Add code in the interface. * tree-vect-data-refs.cc (vect_create_data_ref_ptr): New function. * tree-vect-loop-manip.cc (vect_set_loop_controls_directly): Add decrementing IV loop control support. (vect_set_loop_controls_by_while_len): New function. (vect_set_loop_condition_partial_vectors): Add decrementing IV loop control support. (vect_set_loop_condition_normal): Adapt create_iv interface change. * tree-vect-loop.cc (vect_create_epilog_for_reduction): Ditto. (vect_get_loop_len): Adapt SLP auto-vectorization for WHILE_LEN. * tree-vect-stmts.cc (get_while_len_data_ref_ptr): New function. (vectorizable_store): Add WHILE_LEN data reference pointer IV. (vectorizable_load): Ditto. * tree-vectorizer.h (vect_get_loop_len): Add new arguments to interface. --- gcc/cfgloopmanip.cc | 2 +- gcc/doc/md.texi | 34 ++++ gcc/gimple-loop-interchange.cc | 2 +- gcc/internal-fn.def | 1 + gcc/optabs.def | 1 + gcc/tree-ssa-loop-ivcanon.cc | 2 +- gcc/tree-ssa-loop-ivopts.cc | 2 +- gcc/tree-ssa-loop-manip.cc | 18 ++- gcc/tree-ssa-loop-manip.h | 4 +- gcc/tree-vect-data-refs.cc | 8 +- gcc/tree-vect-loop-manip.cc | 280 +++++++++++++++++++++++++++++++-- gcc/tree-vect-loop.cc | 29 +++- gcc/tree-vect-stmts.cc | 95 ++++++++++- gcc/tree-vectorizer.h | 4 +- 14 files changed, 445 insertions(+), 37 deletions(-) diff --git a/gcc/cfgloopmanip.cc b/gcc/cfgloopmanip.cc index 0e3ad8ed742..6e09dcbb0b1 100644 --- a/gcc/cfgloopmanip.cc +++ b/gcc/cfgloopmanip.cc @@ -826,7 +826,7 @@ create_empty_loop_on_edge (edge entry_edge, } gsi = gsi_last_bb (loop_header); - create_iv (initial_value, stride, iv, loop, &gsi, false, + create_iv (initial_value, PLUS_EXPR, stride, iv, loop, &gsi, false, iv_before, iv_after); /* Insert loop exit condition. */ diff --git a/gcc/doc/md.texi b/gcc/doc/md.texi index 07bf8bdebff..307d3c407d8 100644 --- a/gcc/doc/md.texi +++ b/gcc/doc/md.texi @@ -4965,6 +4965,40 @@ for (i = 1; i < operand3; i++) operand0[i] = operand0[i - 1] && (operand1 + i < operand2); @end smallexample +@cindex @code{while_len@var{m}@var{n}} instruction pattern +@item @code{while_len@var{m}@var{n}} +Set operand 0 to the number of active elements in vector will be updated value. +operand 1 is the total elements need to be updated value. +operand 2 is the vectorization factor. +The value of operand 0 is target dependent and flexible in each iteration. +The operation of this pattern can be: + +@smallexample +Case 1: +operand0 = MIN (operand1, operand2); +operand2 can be const_poly_int or poly_int related to vector mode size. +Some target like RISC-V has a standalone instruction to get MIN (n, MODE SIZE) so +that we can reduce a use of general purpose register. + +In this case, only the last iteration of the loop is partial iteration. +@end smallexample + +@smallexample +Case 2: +if (operand1 <= operand2) + operand0 = operand1; +else if (operand1 < 2 * operand2) + operand0 = IN_RANGE (ceil (operand1 / 2), operand2); +else + operand0 = operand2; + +This case will evenly distribute work over the last 2 iterations of a stripmine loop. +@end smallexample + +The output of this pattern is not only used as IV of loop control counter, but also +is used as the IV of address calculation with multiply/shift operation. This allow +us dynamic adjust the number of elements is processed in each iteration of the loop. + @cindex @code{check_raw_ptrs@var{m}} instruction pattern @item @samp{check_raw_ptrs@var{m}} Check whether, given two pointers @var{a} and @var{b} and a length @var{len}, diff --git a/gcc/gimple-loop-interchange.cc b/gcc/gimple-loop-interchange.cc index 1b77bfd46b2..e5590374e59 100644 --- a/gcc/gimple-loop-interchange.cc +++ b/gcc/gimple-loop-interchange.cc @@ -1185,7 +1185,7 @@ tree_loop_interchange::map_inductions_to_loop (loop_cand &src, loop_cand &tgt) tree var_before, var_after; tree base = unshare_expr (iv->init_expr); tree step = unshare_expr (iv->step); - create_iv (base, step, SSA_NAME_VAR (iv->var), + create_iv (base, PLUS_EXPR, step, SSA_NAME_VAR (iv->var), tgt.m_loop, &incr_pos, false, &var_before, &var_after); bitmap_set_bit (m_dce_seeds, SSA_NAME_VERSION (var_before)); bitmap_set_bit (m_dce_seeds, SSA_NAME_VERSION (var_after)); diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def index 7fe742c2ae7..9f7269f8f60 100644 --- a/gcc/internal-fn.def +++ b/gcc/internal-fn.def @@ -153,6 +153,7 @@ DEF_INTERNAL_OPTAB_FN (VEC_SET, 0, vec_set, vec_set) DEF_INTERNAL_OPTAB_FN (LEN_STORE, 0, len_store, len_store) DEF_INTERNAL_OPTAB_FN (WHILE_ULT, ECF_CONST | ECF_NOTHROW, while_ult, while) +DEF_INTERNAL_OPTAB_FN (WHILE_LEN, ECF_CONST | ECF_NOTHROW, while_len, binary) DEF_INTERNAL_OPTAB_FN (CHECK_RAW_PTRS, ECF_CONST | ECF_NOTHROW, check_raw_ptrs, check_ptrs) DEF_INTERNAL_OPTAB_FN (CHECK_WAR_PTRS, ECF_CONST | ECF_NOTHROW, diff --git a/gcc/optabs.def b/gcc/optabs.def index 695f5911b30..f5938bd2c24 100644 --- a/gcc/optabs.def +++ b/gcc/optabs.def @@ -476,3 +476,4 @@ OPTAB_DC (vec_series_optab, "vec_series$a", VEC_SERIES) OPTAB_D (vec_shl_insert_optab, "vec_shl_insert_$a") OPTAB_D (len_load_optab, "len_load_$a") OPTAB_D (len_store_optab, "len_store_$a") +OPTAB_D (while_len_optab, "while_len$a") diff --git a/gcc/tree-ssa-loop-ivcanon.cc b/gcc/tree-ssa-loop-ivcanon.cc index 9f72d534b7c..6941d0a4d1d 100644 --- a/gcc/tree-ssa-loop-ivcanon.cc +++ b/gcc/tree-ssa-loop-ivcanon.cc @@ -113,7 +113,7 @@ create_canonical_iv (class loop *loop, edge exit, tree niter, niter, build_int_cst (type, 1)); incr_at = gsi_last_bb (in->src); - create_iv (niter, + create_iv (niter, PLUS_EXPR, build_int_cst (type, -1), NULL_TREE, loop, &incr_at, false, var_before, &var); diff --git a/gcc/tree-ssa-loop-ivopts.cc b/gcc/tree-ssa-loop-ivopts.cc index 0dd47910f97..a7377f91f9e 100644 --- a/gcc/tree-ssa-loop-ivopts.cc +++ b/gcc/tree-ssa-loop-ivopts.cc @@ -7267,7 +7267,7 @@ create_new_iv (struct ivopts_data *data, struct iv_cand *cand) base = unshare_expr (cand->iv->base); - create_iv (base, unshare_expr (cand->iv->step), + create_iv (base, PLUS_EXPR, unshare_expr (cand->iv->step), cand->var_before, data->current_loop, &incr_pos, after, &cand->var_before, &cand->var_after); } diff --git a/gcc/tree-ssa-loop-manip.cc b/gcc/tree-ssa-loop-manip.cc index a52277abdbf..b0d254b0c89 100644 --- a/gcc/tree-ssa-loop-manip.cc +++ b/gcc/tree-ssa-loop-manip.cc @@ -47,7 +47,9 @@ along with GCC; see the file COPYING3. If not see so that we can free them all at once. */ static bitmap_obstack loop_renamer_obstack; -/* Creates an induction variable with value BASE + STEP * iteration in LOOP. +/* Creates an induction variable with value BASE (+/-) STEP * iteration in LOOP. + If CODE is PLUS_EXPR, the induction variable is BASE + STEP * iteration. + If CODE is MINUS_EXPR, the induction variable is BASE - STEP * iteration. It is expected that neither BASE nor STEP are shared with other expressions (unless the sharing rules allow this). Use VAR as a base var_decl for it (if NULL, a new temporary will be created). The increment will occur at @@ -57,8 +59,8 @@ static bitmap_obstack loop_renamer_obstack; VAR_AFTER (unless they are NULL). */ void -create_iv (tree base, tree step, tree var, class loop *loop, - gimple_stmt_iterator *incr_pos, bool after, +create_iv (tree base, tree_code code, tree step, tree var, + class loop *loop, gimple_stmt_iterator *incr_pos, bool after, tree *var_before, tree *var_after) { gassign *stmt; @@ -66,7 +68,9 @@ create_iv (tree base, tree step, tree var, class loop *loop, tree initial, step1; gimple_seq stmts; tree vb, va; - enum tree_code incr_op = PLUS_EXPR; + /* The code can only be PLUS_EXPR or MINUS_EXPR. */ + gcc_assert (code == PLUS_EXPR || code == MINUS_EXPR); + tree_code incr_op = code; edge pe = loop_preheader_edge (loop); if (var != NULL_TREE) @@ -1368,7 +1372,7 @@ tree_transform_and_unroll_loop (class loop *loop, unsigned factor, tree ctr_before, ctr_after; gimple_stmt_iterator bsi = gsi_last_nondebug_bb (new_exit->src); exit_if = as_a (gsi_stmt (bsi)); - create_iv (exit_base, exit_step, NULL_TREE, loop, + create_iv (exit_base, PLUS_EXPR, exit_step, NULL_TREE, loop, &bsi, false, &ctr_before, &ctr_after); gimple_cond_set_code (exit_if, exit_cmp); gimple_cond_set_lhs (exit_if, ctr_after); @@ -1583,8 +1587,8 @@ canonicalize_loop_ivs (class loop *loop, tree *nit, bool bump_in_latch) gsi = gsi_last_bb (loop->latch); else gsi = gsi_last_nondebug_bb (loop->header); - create_iv (build_int_cst_type (type, 0), build_int_cst (type, 1), NULL_TREE, - loop, &gsi, bump_in_latch, &var_before, NULL); + create_iv (build_int_cst_type (type, 0), PLUS_EXPR, build_int_cst (type, 1), + NULL_TREE, loop, &gsi, bump_in_latch, &var_before, NULL); rewrite_all_phi_nodes_with_iv (loop, var_before); diff --git a/gcc/tree-ssa-loop-manip.h b/gcc/tree-ssa-loop-manip.h index d49273a3987..bda09f51d56 100644 --- a/gcc/tree-ssa-loop-manip.h +++ b/gcc/tree-ssa-loop-manip.h @@ -22,8 +22,8 @@ along with GCC; see the file COPYING3. If not see typedef void (*transform_callback)(class loop *, void *); -extern void create_iv (tree, tree, tree, class loop *, gimple_stmt_iterator *, - bool, tree *, tree *); +extern void create_iv (tree, tree_code, tree, tree, class loop *, + gimple_stmt_iterator *, bool, tree *, tree *); extern void rewrite_into_loop_closed_ssa (bitmap, unsigned); extern void verify_loop_closed_ssa (bool, class loop * = NULL); diff --git a/gcc/tree-vect-data-refs.cc b/gcc/tree-vect-data-refs.cc index c03ffb3aaf1..355ede5c0d9 100644 --- a/gcc/tree-vect-data-refs.cc +++ b/gcc/tree-vect-data-refs.cc @@ -5101,7 +5101,7 @@ vect_create_data_ref_ptr (vec_info *vinfo, stmt_vec_info stmt_info, standard_iv_increment_position (loop, &incr_gsi, &insert_after); - create_iv (aggr_ptr_init, + create_iv (aggr_ptr_init, PLUS_EXPR, fold_convert (aggr_ptr_type, iv_step), aggr_ptr, loop, &incr_gsi, insert_after, &indx_before_incr, &indx_after_incr); @@ -5131,9 +5131,9 @@ vect_create_data_ref_ptr (vec_info *vinfo, stmt_vec_info stmt_info, { standard_iv_increment_position (containing_loop, &incr_gsi, &insert_after); - create_iv (aptr, fold_convert (aggr_ptr_type, DR_STEP (dr)), aggr_ptr, - containing_loop, &incr_gsi, insert_after, &indx_before_incr, - &indx_after_incr); + create_iv (aptr, PLUS_EXPR, fold_convert (aggr_ptr_type, DR_STEP (dr)), + aggr_ptr, containing_loop, &incr_gsi, insert_after, + &indx_before_incr, &indx_after_incr); incr = gsi_stmt (incr_gsi); /* Copy the points-to information if it exists. */ diff --git a/gcc/tree-vect-loop-manip.cc b/gcc/tree-vect-loop-manip.cc index f60fa50e8f4..6973314adb5 100644 --- a/gcc/tree-vect-loop-manip.cc +++ b/gcc/tree-vect-loop-manip.cc @@ -468,8 +468,9 @@ vect_set_loop_controls_directly (class loop *loop, loop_vec_info loop_vinfo, gimple_stmt_iterator incr_gsi; bool insert_after; standard_iv_increment_position (loop, &incr_gsi, &insert_after); - create_iv (build_int_cst (iv_type, 0), nitems_step, NULL_TREE, loop, - &incr_gsi, insert_after, &index_before_incr, &index_after_incr); + create_iv (build_int_cst (iv_type, 0), PLUS_EXPR, nitems_step, NULL_TREE, + loop, &incr_gsi, insert_after, &index_before_incr, + &index_after_incr); tree zero_index = build_int_cst (compare_type, 0); tree test_index, test_limit, first_limit; @@ -682,6 +683,260 @@ vect_set_loop_controls_directly (class loop *loop, loop_vec_info loop_vinfo, return next_ctrl; } +/* Helper for vect_set_loop_condition_partial_vectors. Generate definitions + for all the rgroup controls in RGC and return a control that is nonzero + when the loop needs to iterate. Add any new preheader statements to + PREHEADER_SEQ. Use LOOP_COND_GSI to insert code before the exit gcond. + + RGC belongs to loop LOOP. The loop originally iterated NITERS + times and has been vectorized according to LOOP_VINFO. + + Unlike vect_set_loop_controls_directly which is iterating from 0-based IV + to TEST_LIMIT - bias. + + In vect_set_loop_controls_by_while_len, we are iterating from start at + IV = TEST_LIMIT - bias and keep subtract IV by the length calculated by + IFN_WHILE_LEN pattern. + + 1. Single rgroup, the Gimple IR should be: + + # vectp_B.6_8 = PHI + # vectp_B.8_16 = PHI + # vectp_A.11_19 = PHI + # vectp_A.13_22 = PHI + # ivtmp_26 = PHI + _28 = .WHILE_LEN (ivtmp_26, POLY_INT_CST [4, 4]); + ivtmp_15 = _28 * 4; + vect__1.10_18 = .LEN_LOAD (vectp_B.8_16, 128B, _28, 0); + _1 = B[i_10]; + .LEN_STORE (vectp_A.13_22, 128B, _28, vect__1.10_18, 0); + i_7 = i_10 + 1; + vectp_B.8_17 = vectp_B.8_16 + ivtmp_15; + vectp_A.13_23 = vectp_A.13_22 + ivtmp_15; + ivtmp_27 = ivtmp_26 - _28; + if (ivtmp_27 != 0) + goto ; [83.33%] + else + goto ; [16.67%] + + Note: We use the outcome of .WHILE_LEN to adjust both loop control IV and + data reference pointer IV. + + 1). The result of .WHILE_LEN: + _28 = .WHILE_LEN (ivtmp_26, POLY_INT_CST [4, 4]); + The _28 is not necessary to be VF in any iteration, instead, we allow + _28 to be any value as long as it it <= VF. Such flexible WHILE_LEN + pattern allows target have various flexible optimizations in vector + loop iterations. Target like RISC-V has special application vector + length calculation instruction which will distribute even workload + in the last 2 iterations. + + Other example is that we can allow even generate _28 <= VF / 2 so + that some machine can run vector codes in low power mode. + + 2). Loop control IV: + ivtmp_27 = ivtmp_26 - _28; + if (ivtmp_27 != 0) + goto ; [83.33%] + else + goto ; [16.67%] + + This is the saturating-subtraction towards zero, the outcome of + .WHILE_LEN wil make ivtmp_27 never underflow zero. + + 3). Data reference pointer IV: + ivtmp_15 = _28 * 4; + vectp_B.8_17 = vectp_B.8_16 + ivtmp_15; + vectp_A.13_23 = vectp_A.13_22 + ivtmp_15; + + The pointer IV is adjusted accurately according to the .WHILE_LEN. + + 2. Multiple rgroup, the Gimple IR should be: + + # i_23 = PHI + # vectp_f.8_51 = PHI + # vectp_d.10_59 = PHI + # ivtmp_70 = PHI + # ivtmp_73 = PHI + _72 = MIN_EXPR ; + _75 = MIN_EXPR ; + _1 = i_23 * 2; + _2 = (long unsigned int) _1; + _3 = _2 * 2; + _4 = f_15(D) + _3; + _5 = _2 + 1; + _6 = _5 * 2; + _7 = f_15(D) + _6; + .LEN_STORE (vectp_f.8_51, 128B, _75, { 1, 2, 1, 2, 1, 2, 1, 2 }, 0); + vectp_f.8_56 = vectp_f.8_51 + 16; + .LEN_STORE (vectp_f.8_56, 128B, _72, { 1, 2, 1, 2, 1, 2, 1, 2 }, 0); + _8 = (long unsigned int) i_23; + _9 = _8 * 4; + _10 = d_18(D) + _9; + _61 = _75 / 2; + .LEN_STORE (vectp_d.10_59, 128B, _61, { 3, 3, 3, 3 }, 0); + vectp_d.10_63 = vectp_d.10_59 + 16; + _64 = _72 / 2; + .LEN_STORE (vectp_d.10_63, 128B, _64, { 3, 3, 3, 3 }, 0); + i_20 = i_23 + 1; + vectp_f.8_52 = vectp_f.8_56 + 16; + vectp_d.10_60 = vectp_d.10_63 + 16; + ivtmp_74 = ivtmp_73 - _75; + ivtmp_71 = ivtmp_70 - _72; + if (ivtmp_74 != 0) + goto ; [83.33%] + else + goto ; [16.67%] + + Note: We DO NOT use .WHILE_LEN in SLP auto-vectorization for multiple + rgroups. Instead, we use MIN_EXPR to guarantee we always use VF as the + iteration amount for mutiple rgroups. + + The analysis of the flow of multiple rgroups: + _72 = MIN_EXPR ; + _75 = MIN_EXPR ; + ... + .LEN_STORE (vectp_f.8_51, 128B, _75, { 1, 2, 1, 2, 1, 2, 1, 2 }, 0); + vectp_f.8_56 = vectp_f.8_51 + 16; + .LEN_STORE (vectp_f.8_56, 128B, _72, { 1, 2, 1, 2, 1, 2, 1, 2 }, 0); + ... + _61 = _75 / 2; + .LEN_STORE (vectp_d.10_59, 128B, _61, { 3, 3, 3, 3 }, 0); + vectp_d.10_63 = vectp_d.10_59 + 16; + _64 = _72 / 2; + .LEN_STORE (vectp_d.10_63, 128B, _64, { 3, 3, 3, 3 }, 0); + + We use _72 = MIN_EXPR ; to generate the number of the elements + to be processed in each iteration. + + The related STOREs: + _72 = MIN_EXPR ; + .LEN_STORE (vectp_f.8_56, 128B, _72, { 1, 2, 1, 2, 1, 2, 1, 2 }, 0); + _64 = _72 / 2; + .LEN_STORE (vectp_d.10_63, 128B, _64, { 3, 3, 3, 3 }, 0); + Since these 2 STOREs store 2 vectors that the second vector is half elements + of the first vector. So the length of second STORE will be _64 = _72 / 2; + It's similar to the VIEW_CONVERT of handling masks in SLP. +*/ + +static tree +vect_set_loop_controls_by_while_len (class loop *loop, loop_vec_info loop_vinfo, + gimple_seq *preheader_seq, + gimple_seq *header_seq, + rgroup_controls *rgc, tree niters) +{ + tree compare_type = LOOP_VINFO_RGROUP_COMPARE_TYPE (loop_vinfo); + tree iv_type = LOOP_VINFO_RGROUP_IV_TYPE (loop_vinfo); + /* We are not allowing masked approach in WHILE_LEN. */ + gcc_assert (!LOOP_VINFO_FULLY_MASKED_P (loop_vinfo)); + + tree ctrl_type = rgc->type; + unsigned int nitems_per_iter = rgc->max_nscalars_per_iter * rgc->factor; + poly_uint64 nitems_per_ctrl = TYPE_VECTOR_SUBPARTS (ctrl_type) * rgc->factor; + poly_uint64 vf = LOOP_VINFO_VECT_FACTOR (loop_vinfo); + + /* Calculate the maximum number of item values that the rgroup + handles in total, the number that it handles for each iteration + of the vector loop. */ + tree nitems_total = niters; + if (nitems_per_iter != 1) + { + /* We checked before setting LOOP_VINFO_USING_PARTIAL_VECTORS_P that + these multiplications don't overflow. */ + tree compare_factor = build_int_cst (compare_type, nitems_per_iter); + nitems_total = gimple_build (preheader_seq, MULT_EXPR, compare_type, + nitems_total, compare_factor); + } + + /* Convert the comparison value to the IV type (either a no-op or + a promotion). */ + nitems_total = gimple_convert (preheader_seq, iv_type, nitems_total); + + /* Create an induction variable that counts the number of items + processed. */ + tree index_before_incr, index_after_incr; + gimple_stmt_iterator incr_gsi; + bool insert_after; + standard_iv_increment_position (loop, &incr_gsi, &insert_after); + + /* Test the decremented IV, which will never underflow 0 since we have + IFN_WHILE_LEN to gurantee that. */ + tree test_limit = nitems_total; + + /* Provide a definition of each control in the group. */ + tree ctrl; + unsigned int i; + FOR_EACH_VEC_ELT_REVERSE (rgc->controls, i, ctrl) + { + /* Previous controls will cover BIAS items. This control covers the + next batch. */ + poly_uint64 bias = nitems_per_ctrl * i; + tree bias_tree = build_int_cst (iv_type, bias); + + /* Rather than have a new IV that starts at TEST_LIMIT and goes down to + BIAS, prefer to use the same TEST_LIMIT - BIAS based IV for each + control and adjust the bound down by BIAS. */ + tree this_test_limit = test_limit; + if (i != 0) + { + this_test_limit = gimple_build (preheader_seq, MAX_EXPR, iv_type, + this_test_limit, bias_tree); + this_test_limit = gimple_build (preheader_seq, MINUS_EXPR, iv_type, + this_test_limit, bias_tree); + } + + /* Create decrement IV. */ + create_iv (this_test_limit, MINUS_EXPR, ctrl, NULL_TREE, loop, &incr_gsi, + insert_after, &index_before_incr, &index_after_incr); + + poly_uint64 final_vf = vf * nitems_per_iter; + tree vf_step = build_int_cst (iv_type, final_vf); + tree res_len; + if (nitems_per_iter != 1) + { + /* For SLP, we can't allow non-VF number of elements to be processed + in non-final iteration. We force the number of elements to be + processed in each non-final iteration is VF elements. If we allow + non-VF elements processing in non-final iteration will make SLP too + complicated and produce inferior codegen. + + For example: + + If non-final iteration process VF elements. + + ... + .LEN_STORE (vectp_f.8_51, 128B, _71, { 1, 2, 1, 2 }, 0); + .LEN_STORE (vectp_f.8_56, 128B, _72, { 1, 2, 1, 2 }, 0); + ... + + If non-final iteration process non-VF elements. + + ... + .LEN_STORE (vectp_f.8_51, 128B, _71, { 1, 2, 1, 2 }, 0); + if (_71 % 2 == 0) + .LEN_STORE (vectp_f.8_56, 128B, _72, { 1, 2, 1, 2 }, 0); + else + .LEN_STORE (vectp_f.8_56, 128B, _72, { 2, 1, 2, 1 }, 0); + ... + + This is the simple case of 2-elements interleaved vector SLP. We + consider other interleave vector, the situation will become more + complicated. */ + res_len = gimple_build (header_seq, MIN_EXPR, iv_type, + index_before_incr, vf_step); + } + else + { + res_len = gimple_build (header_seq, IFN_WHILE_LEN, iv_type, + index_before_incr, vf_step); + } + gassign *assign = gimple_build_assign (ctrl, res_len); + gimple_seq_add_stmt (header_seq, assign); + } + + return index_after_incr; +} + /* Set up the iteration condition and rgroup controls for LOOP, given that LOOP_VINFO_USING_PARTIAL_VECTORS_P is true for the vectorized loop. LOOP_VINFO describes the vectorization of LOOP. NITERS is @@ -703,6 +958,7 @@ vect_set_loop_condition_partial_vectors (class loop *loop, bool use_masks_p = LOOP_VINFO_FULLY_MASKED_P (loop_vinfo); tree compare_type = LOOP_VINFO_RGROUP_COMPARE_TYPE (loop_vinfo); + tree iv_type = LOOP_VINFO_RGROUP_IV_TYPE (loop_vinfo); unsigned int compare_precision = TYPE_PRECISION (compare_type); tree orig_niters = niters; @@ -757,12 +1013,18 @@ vect_set_loop_condition_partial_vectors (class loop *loop, bool might_wrap_p = vect_rgroup_iv_might_wrap_p (loop_vinfo, rgc); /* Set up all controls for this group. */ - test_ctrl = vect_set_loop_controls_directly (loop, loop_vinfo, - &preheader_seq, - &header_seq, - loop_cond_gsi, rgc, - niters, niters_skip, - might_wrap_p); + if (direct_internal_fn_supported_p (IFN_WHILE_LEN, iv_type, + OPTIMIZE_FOR_SPEED)) + test_ctrl + = vect_set_loop_controls_by_while_len (loop, loop_vinfo, + &preheader_seq, &header_seq, + rgc, niters); + else + test_ctrl + = vect_set_loop_controls_directly (loop, loop_vinfo, &preheader_seq, + &header_seq, loop_cond_gsi, rgc, + niters, niters_skip, + might_wrap_p); } /* Emit all accumulated statements. */ @@ -893,7 +1155,7 @@ vect_set_loop_condition_normal (class loop *loop, tree niters, tree step, } standard_iv_increment_position (loop, &incr_gsi, &insert_after); - create_iv (init, step, NULL_TREE, loop, + create_iv (init, PLUS_EXPR, step, NULL_TREE, loop, &incr_gsi, insert_after, &indx_before_incr, &indx_after_incr); indx_after_incr = force_gimple_operand_gsi (&loop_cond_gsi, indx_after_incr, true, NULL_TREE, true, diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc index 6ea0f21fd13..d6e90ed93cd 100644 --- a/gcc/tree-vect-loop.cc +++ b/gcc/tree-vect-loop.cc @@ -5567,7 +5567,7 @@ vect_create_epilog_for_reduction (loop_vec_info loop_vinfo, gimple_stmt_iterator incr_gsi; bool insert_after; standard_iv_increment_position (loop, &incr_gsi, &insert_after); - create_iv (series_vect, vec_step, NULL_TREE, loop, &incr_gsi, + create_iv (series_vect, PLUS_EXPR, vec_step, NULL_TREE, loop, &incr_gsi, insert_after, &indx_before_incr, &indx_after_incr); /* Next create a new phi node vector (NEW_PHI_TREE) which starts @@ -10364,12 +10364,14 @@ vect_record_loop_len (loop_vec_info loop_vinfo, vec_loop_lens *lens, rgroup that operates on NVECTORS vectors, where 0 <= INDEX < NVECTORS. */ tree -vect_get_loop_len (loop_vec_info loop_vinfo, vec_loop_lens *lens, - unsigned int nvectors, unsigned int index) +vect_get_loop_len (gimple_stmt_iterator *gsi, loop_vec_info loop_vinfo, + vec_loop_lens *lens, unsigned int nvectors, tree vectype, + unsigned int index) { rgroup_controls *rgl = &(*lens)[nvectors - 1]; bool use_bias_adjusted_len = LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo) != 0; + tree iv_type = LOOP_VINFO_RGROUP_IV_TYPE (loop_vinfo); /* Populate the rgroup's len array, if this is the first time we've used it. */ @@ -10400,6 +10402,27 @@ vect_get_loop_len (loop_vec_info loop_vinfo, vec_loop_lens *lens, if (use_bias_adjusted_len) return rgl->bias_adjusted_ctrl; + else if (direct_internal_fn_supported_p (IFN_WHILE_LEN, iv_type, + OPTIMIZE_FOR_SPEED)) + { + tree loop_len = rgl->controls[index]; + poly_int64 nunits1 = TYPE_VECTOR_SUBPARTS (rgl->type); + poly_int64 nunits2 = TYPE_VECTOR_SUBPARTS (vectype); + if (maybe_ne (nunits1, nunits2)) + { + /* A loop len for data type X can be reused for data type Y + if X has N times more elements than Y and if Y's elements + are N times bigger than X's. */ + gcc_assert (multiple_p (nunits1, nunits2)); + unsigned int factor = exact_div (nunits1, nunits2).to_constant (); + gimple_seq seq = NULL; + loop_len = gimple_build (&seq, RDIV_EXPR, iv_type, loop_len, + build_int_cst (iv_type, factor)); + if (seq) + gsi_insert_seq_before (gsi, seq, GSI_SAME_STMT); + } + return loop_len; + } else return rgl->controls[index]; } diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc index 6b7dbfd4a23..98bef8f4fc9 100644 --- a/gcc/tree-vect-stmts.cc +++ b/gcc/tree-vect-stmts.cc @@ -3144,6 +3144,70 @@ vect_get_data_ptr_increment (vec_info *vinfo, return iv_step; } +/* Prepare the pointer IVs which needs to be updated by a variable amount. + Such variable amount is the outcome of .WHILE_LEN. In this case, we can + allow each iteration process the flexible number of elements as long as + the number <= vf elments. + + Return data reference according to WHILE_LEN. + If new statements are needed, insert them before GSI. */ + +static tree +get_while_len_data_ref_ptr (vec_info *vinfo, stmt_vec_info stmt_info, + tree aggr_type, class loop *at_loop, tree offset, + tree *dummy, gimple_stmt_iterator *gsi, + bool simd_lane_access_p, vec_loop_lens *loop_lens, + dr_vec_info *dr_info, + vect_memory_access_type memory_access_type) +{ + if (!loop_lens || loop_lens->length () != 1) + return NULL_TREE; + loop_vec_info loop_vinfo = dyn_cast (vinfo); + tree iv_type = LOOP_VINFO_RGROUP_IV_TYPE (loop_vinfo); + tree step = vect_dr_behavior (vinfo, dr_info)->step; + if (!direct_internal_fn_supported_p (IFN_WHILE_LEN, iv_type, + OPTIMIZE_FOR_SPEED)) + return NULL_TREE; + + if (memory_access_type == VMAT_INVARIANT) + return NULL_TREE; + + /* TODO: We don't support gather/scatter or load_lanes/store_lanes for pointer + IVs are updated by variable amount but we will support them in the future. + */ + gcc_assert (memory_access_type != VMAT_GATHER_SCATTER + && memory_access_type != VMAT_LOAD_STORE_LANES); + + /* When we support WHILE_LEN pattern, we dynamic adjust + the memory address by .WHILE_LEN result. + + The result of .WHILE_LEN is the number of elements to + be processed of each iteration. So the memory address + adjustment operation should be: + + bytesize = GET_MODE_SIZE (element_mode (aggr_type)); + addr = addr + .WHILE_LEN (ARG..) * bytesize; + */ + gimple *ptr_incr; + tree loop_len + = vect_get_loop_len (gsi, loop_vinfo, loop_lens, 1, aggr_type, 0); + tree len_type = TREE_TYPE (loop_len); + poly_uint64 bytesize = GET_MODE_SIZE (element_mode (aggr_type)); + /* Since the outcome of .WHILE_LEN is element size, we should adjust + it into bytesize so that it can be used in address pointer variable + amount IVs adjustment. */ + tree tmp = fold_build2 (MULT_EXPR, len_type, loop_len, + build_int_cst (len_type, bytesize)); + if (tree_int_cst_sgn (step) == -1) + tmp = fold_build1 (NEGATE_EXPR, len_type, tmp); + tree bump = make_temp_ssa_name (len_type, NULL, "ivtmp"); + gassign *assign = gimple_build_assign (bump, tmp); + gsi_insert_before (gsi, assign, GSI_SAME_STMT); + return vect_create_data_ref_ptr (vinfo, stmt_info, aggr_type, at_loop, offset, + dummy, gsi, &ptr_incr, simd_lane_access_p, + bump); +} + /* Check and perform vectorization of BUILT_IN_BSWAP{16,32,64,128}. */ static bool @@ -8195,7 +8259,7 @@ vectorizable_store (vec_info *vinfo, stride_base = cse_and_gimplify_to_preheader (loop_vinfo, stride_base); ivstep = cse_and_gimplify_to_preheader (loop_vinfo, ivstep); - create_iv (stride_base, ivstep, NULL, + create_iv (stride_base, PLUS_EXPR, ivstep, NULL, loop, &incr_gsi, insert_after, &offvar, NULL); incr = gsi_stmt (incr_gsi); @@ -8465,6 +8529,15 @@ vectorizable_store (vec_info *vinfo, simd_lane_access_p ? loop : NULL, offset, &dummy, gsi, &ptr_incr, simd_lane_access_p, bump); + + tree while_len_data_ref_ptr + = get_while_len_data_ref_ptr (vinfo, stmt_info, aggr_type, + simd_lane_access_p ? loop : NULL, + offset, &dummy, gsi, + simd_lane_access_p, loop_lens, + dr_info, memory_access_type); + if (while_len_data_ref_ptr) + dataref_ptr = while_len_data_ref_ptr; } else { @@ -8652,8 +8725,9 @@ vectorizable_store (vec_info *vinfo, else if (loop_lens) { tree final_len - = vect_get_loop_len (loop_vinfo, loop_lens, - vec_num * ncopies, vec_num * j + i); + = vect_get_loop_len (gsi, loop_vinfo, loop_lens, + vec_num * ncopies, vectype, + vec_num * j + i); tree ptr = build_int_cst (ref_type, align * BITS_PER_UNIT); machine_mode vmode = TYPE_MODE (vectype); opt_machine_mode new_ovmode @@ -9314,7 +9388,7 @@ vectorizable_load (vec_info *vinfo, stride_base = cse_and_gimplify_to_preheader (loop_vinfo, stride_base); ivstep = cse_and_gimplify_to_preheader (loop_vinfo, ivstep); - create_iv (stride_base, ivstep, NULL, + create_iv (stride_base, PLUS_EXPR, ivstep, NULL, loop, &incr_gsi, insert_after, &offvar, NULL); @@ -9798,6 +9872,15 @@ vectorizable_load (vec_info *vinfo, at_loop, offset, &dummy, gsi, &ptr_incr, simd_lane_access_p, bump); + + tree while_len_data_ref_ptr + = get_while_len_data_ref_ptr (vinfo, stmt_info, aggr_type, at_loop, + offset, &dummy, gsi, + simd_lane_access_p, loop_lens, + dr_info, memory_access_type); + if (while_len_data_ref_ptr) + dataref_ptr = while_len_data_ref_ptr; + if (mask) vec_mask = vec_masks[0]; } @@ -10008,8 +10091,8 @@ vectorizable_load (vec_info *vinfo, else if (loop_lens && memory_access_type != VMAT_INVARIANT) { tree final_len - = vect_get_loop_len (loop_vinfo, loop_lens, - vec_num * ncopies, + = vect_get_loop_len (gsi, loop_vinfo, loop_lens, + vec_num * ncopies, vectype, vec_num * j + i); tree ptr = build_int_cst (ref_type, align * BITS_PER_UNIT); diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h index 9cf2fb23fe3..811c17e1d3d 100644 --- a/gcc/tree-vectorizer.h +++ b/gcc/tree-vectorizer.h @@ -2293,8 +2293,8 @@ extern tree vect_get_loop_mask (gimple_stmt_iterator *, vec_loop_masks *, unsigned int, tree, unsigned int); extern void vect_record_loop_len (loop_vec_info, vec_loop_lens *, unsigned int, tree, unsigned int); -extern tree vect_get_loop_len (loop_vec_info, vec_loop_lens *, unsigned int, - unsigned int); +extern tree vect_get_loop_len (gimple_stmt_iterator *, loop_vec_info, vec_loop_lens *, + unsigned int, tree, unsigned int); extern gimple_seq vect_gen_len (tree, tree, tree, tree); extern stmt_vec_info info_for_reduction (vec_info *, stmt_vec_info); extern bool reduction_fn_for_scalar_code (code_helper, internal_fn *); -- 2.36.1