Re: [PATCH V6] VECT: Add decrement IV support in Loop Vectorizer

[Richard Sandiford <richard.sandiford@arm.com>]

"juzhe.zhong" <juzhe.zhong@rivai.ai> writes:
> Hi, Richard.  For "can iterate more than once", is it correct use the condition
> "LOOP_LENS ().length >1".     

No, that says whether any LOAD_LENs or STORE_LENs operate on multiple
vectors, rather than just single vectors.

I meant: whether the vector loop body might be executed more than once
(i.e. whether the branch-back condition can be true).

This is true for a scalar loop that goes from 0 to some unbounded
variable n.  It's false for a scalar loop that goes from 0 to 6,
if the vectors are known to have at least 8 elements.

Thanks,
Richard

> ---- Replied Message ----
>
> From      Richard Sandiford<richard.sandiford@arm.com>
>
> Date      05/12/2023 19:39
>
> To        juzhe.zhong<juzhe.zhong@rivai.ai>
>
> Cc        gcc-patches@gcc.gnu.org<gcc-patches@gcc.gnu.org>,
>           kito.cheng@gmail.com<kito.cheng@gmail.com>,
>           palmer@dabbelt.com<palmer@dabbelt.com>,
>           richard.guenther@gmail.com<richard.guenther@gmail.com>
>
> Subject   Re: [PATCH V6] VECT: Add decrement IV support in Loop Vectorizer
>
> "juzhe.zhong" <juzhe.zhong@rivai.ai> writes:
>> Thanks Richard.
>>  I will do that as you suggested. I have a question for the first patch. How
> to
>> enable decrement IV？ Should I add a target hook or something to let target
>> decide whether enable decrement IV？
>
> At the moment, the only other targets that use IFN_LOAD_LEN and
> IFN_STORE_LEN are PowerPC and s390.  Both targets default to
> --param vect-partial-vector-usage=1 (i.e. use partial vectors
> for epilogues only).
>
> So I think the condition should be that the loop:
>
>  (a) uses length "controls"; and
>  (b) can iterate more than once
>
> No target checks should be needed.
>
> Thanks,
> Richard
>
>> ---- Replied Message ----
>>
>> From      Richard Sandiford<richard.sandiford@arm.com>
>>
>> Date      05/12/2023 19:08
>>
>> To        juzhe.zhong@rivai.ai<juzhe.zhong@rivai.ai>
>>
>> Cc        gcc-patches@gcc.gnu.org<gcc-patches@gcc.gnu.org>,
>>           kito.cheng@gmail.com<kito.cheng@gmail.com>,
>>           palmer@dabbelt.com<palmer@dabbelt.com>,
>>           richard.guenther@gmail.com<richard.guenther@gmail.com>
>>
>> Subject   Re: [PATCH V6] VECT: Add decrement IV support in Loop Vectorizer
>>
>> juzhe.zhong@rivai.ai writes:
>>> From: Ju-Zhe Zhong <juzhe.zhong@rivai.ai>
>>>
>>> 1. Fix document description according Jeff && Richard.
>>> 2. Add LOOP_VINFO_USING_SELECT_VL_P for single rgroup.
>>> 3. Add LOOP_VINFO_USING_SLP_ADJUSTED_LEN_P for SLP multiple rgroup.
>>>
>>> Fix bugs for V5 after testing:
>>> https://gcc.gnu.org/pipermail/gcc-patches/2023-May/618209.html
>>>
>>> gcc/ChangeLog:
>>>
>>>         * doc/md.texi: Add seletc_vl pattern.
>>>         * internal-fn.def (SELECT_VL): New ifn.
>>>         * optabs.def (OPTAB_D): New optab.
>>>         * tree-vect-loop-manip.cc (vect_adjust_loop_lens): New function.
>>>         (vect_set_loop_controls_by_select_vl): Ditto.
>>>         (vect_set_loop_condition_partial_vectors): Add loop control for
>> decrement IV.
>>>         * tree-vect-loop.cc (vect_get_loop_len): Adjust loop len for SLP.
>>>         * tree-vect-stmts.cc (get_select_vl_data_ref_ptr): New function.
>>>         (vectorizable_store): Support data reference IV added by outcome of
>> SELECT_VL.
>>>         (vectorizable_load): Ditto.
>>>         * tree-vectorizer.h (LOOP_VINFO_USING_SELECT_VL_P): New macro.
>>>         (LOOP_VINFO_USING_SLP_ADJUSTED_LEN_P): Ditto.
>>>         (vect_get_loop_len): Adjust loop len for SLP.
>>>
>>> ---
>>>  gcc/doc/md.texi             |  36 ++++
>>>  gcc/internal-fn.def         |   1 +
>>>  gcc/optabs.def              |   1 +
>>>  gcc/tree-vect-loop-manip.cc | 380 +++++++++++++++++++++++++++++++++++-
>>>  gcc/tree-vect-loop.cc       |  31 ++-
>>>  gcc/tree-vect-stmts.cc      |  79 +++++++-
>>>  gcc/tree-vectorizer.h       |  12 +-
>>>  7 files changed, 526 insertions(+), 14 deletions(-)
>>>
>>> diff --git a/gcc/doc/md.texi b/gcc/doc/md.texi
>>> index 8ebce31ba78..a94ffc4456d 100644
>>> --- a/gcc/doc/md.texi
>>> +++ b/gcc/doc/md.texi
>>> @@ -4974,6 +4974,42 @@ for (i = 1; i < operand3; i++)
>>>    operand0[i] = operand0[i - 1] && (operand1 + i < operand2);
>>>  @end smallexample
>>>  
>>> +@cindex @code{select_vl@var{m}} instruction pattern
>>> +@item @code{select_vl@var{m}}
>>> +Set operand 0 to the number of active elements in a vector to be updated
>>> +in a loop iteration based on the total number of elements to be updated,
>>> +the vectorization factor and vector properties of the target.
>>> +operand 1 is the total elements in the vector to be updated.
>>> +operand 2 is the vectorization factor.
>>> +The value of operand 0 is target dependent and flexible in each iteration.
>>> +The operation of this pattern can be:
>>> +
>>> +@smallexample
>>> +Case 1:
>>> +operand0 = MIN (operand1, operand2);
>>> +operand2 can be const_poly_int or poly_int related to vector mode size.
>>> +Some target like RISC-V has a standalone instruction to get MIN (n, MODE
>> SIZE) so
>>> +that we can reduce a use of general purpose register.
>>> +
>>> +In this case, only the last iteration of the loop is partial iteration.
>>> +@end smallexample
>>> +
>>> +@smallexample
>>> +Case 2:
>>> +if (operand1 <= operand2)
>>> +  operand0 = operand1;
>>> +else if (operand1 < 2 * operand2)
>>> +  operand0 = ceil (operand1 / 2);
>>> +else
>>> +  operand0 = operand2;
>>> +
>>> +This case will evenly distribute work over the last 2 iterations of a
>> stripmine loop.
>>> +@end smallexample
>>> +
>>> +The output of this pattern is not only used as IV of loop control counter,
>> but also
>>> +is used as the IV of address calculation with multiply/shift operation.
> This
>> allows
>>> +dynamic adjustment of the number of elements processed each loop iteration.
>>> +
>>
>> I don't think we need to restrict the definition to the two RVV cases.
>> How about:
>>
>> -----------------------------------------------------------------------
>> Set operand 0 to the number of scalar iterations that should be handled
>> by one iteration of a vector loop.  Operand 1 is the total number of
>> scalar iterations that the loop needs to process and operand 2 is a
>> maximum bound on the result (also known as the maximum ``vectorization
>> factor'').
>>
>> The maximum value of operand 0 is given by:
>> @smallexample
>> operand0 = MIN (operand1, operand2)
>> @end smallexample
>> However, targets might choose a lower value than this, based on
>> target-specific criteria.  Each iteration of the vector loop might
>> therefore process a different number of scalar iterations, which in turn
>> means that induction variables will have a variable step.  Because of
>> this, it is generally not useful to define this instruction if it will
>> always calculate the maximum value.
>>
>> This optab is only useful on targets that implement @samp{len_load_@var{m}}
>> and/or @samp{len_store_@var{m}}.
>> -----------------------------------------------------------------------
>>
>>>  @cindex @code{check_raw_ptrs@var{m}} instruction pattern
>>>  @item @samp{check_raw_ptrs@var{m}}
>>>  Check whether, given two pointers @var{a} and @var{b} and a length @var
>> {len},
>>> diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def
>>> index 7fe742c2ae7..6f6fa7d37f9 100644
>>> --- a/gcc/internal-fn.def
>>> +++ b/gcc/internal-fn.def
>>> @@ -153,6 +153,7 @@ DEF_INTERNAL_OPTAB_FN (VEC_SET, 0, vec_set, vec_set)
>>>  DEF_INTERNAL_OPTAB_FN (LEN_STORE, 0, len_store, len_store)
>>>  
>>>  DEF_INTERNAL_OPTAB_FN (WHILE_ULT, ECF_CONST | ECF_NOTHROW, while_ult,
> while)
>>> +DEF_INTERNAL_OPTAB_FN (SELECT_VL, ECF_CONST | ECF_NOTHROW, select_vl,
>> binary)
>>>  DEF_INTERNAL_OPTAB_FN (CHECK_RAW_PTRS, ECF_CONST | ECF_NOTHROW,
>>>                 check_raw_ptrs, check_ptrs)
>>>  DEF_INTERNAL_OPTAB_FN (CHECK_WAR_PTRS, ECF_CONST | ECF_NOTHROW,
>>> diff --git a/gcc/optabs.def b/gcc/optabs.def
>>> index 695f5911b30..b637471b76e 100644
>>> --- a/gcc/optabs.def
>>> +++ b/gcc/optabs.def
>>> @@ -476,3 +476,4 @@ OPTAB_DC (vec_series_optab, "vec_series$a", VEC_SERIES)
>>>  OPTAB_D (vec_shl_insert_optab, "vec_shl_insert_$a")
>>>  OPTAB_D (len_load_optab, "len_load_$a")
>>>  OPTAB_D (len_store_optab, "len_store_$a")
>>> +OPTAB_D (select_vl_optab, "select_vl$a")
>>> diff --git a/gcc/tree-vect-loop-manip.cc b/gcc/tree-vect-loop-manip.cc
>>> index ff6159e08d5..81334f4f171 100644
>>> --- a/gcc/tree-vect-loop-manip.cc
>>> +++ b/gcc/tree-vect-loop-manip.cc
>>> @@ -385,6 +385,353 @@ vect_maybe_permute_loop_masks (gimple_seq *seq,
>> rgroup_controls *dest_rgm,
>>>    return false;
>>>  }
>>>  
>>> +/* Try to use adjust loop lens for non-SLP multiple-rgroups.
>>> +
>>> +     _36 = MIN_EXPR <ivtmp_34, POLY_INT_CST [8, 8]>;
>>> +
>>> +     First length (MIN (X, VF/N)):
>>> +       loop_len_15 = MIN_EXPR <_36, POLY_INT_CST [2, 2]>;
>>> +
>>> +     Second length (X - MIN (X, 1 * VF/N)):
>>> +       loop_len_16 = _36 - loop_len_15;
>>> +
>>> +     Third length (X - MIN (X, 2 * VF/N)):
>>> +       _38 = MIN_EXPR <_36, POLY_INT_CST [4, 4]>;
>>> +       loop_len_17 = _36 - _38;
>>> +
>>> +     Forth length (X - MIN (X, 3 * VF/N)):
>>> +       _39 = MIN_EXPR <_36, POLY_INT_CST [6, 6]>;
>>> +       loop_len_18 = _36 - _39;  */
>>> +
>>> +static void
>>> +vect_adjust_loop_lens (tree iv_type, gimple_seq *seq, rgroup_controls
>> *dest_rgm,
>>> +               rgroup_controls *src_rgm)
>>> +{
>>> +  tree ctrl_type = dest_rgm->type;
>>> +  poly_uint64 nitems_per_ctrl
>>> +    = TYPE_VECTOR_SUBPARTS (ctrl_type) * dest_rgm->factor;
>>> +
>>> +  for (unsigned int i = 0; i < dest_rgm->controls.length (); ++i)
>>> +    {
>>> +      tree src = src_rgm->controls[i / dest_rgm->controls.length ()];
>>> +      tree dest = dest_rgm->controls[i];
>>> +      gassign *stmt;
>>> +      if (i == 0)
>>> +    {
>>> +      /* MIN (X, VF*I/N) capped to the range [0, VF/N].  */
>>> +      tree factor = build_int_cst (iv_type, nitems_per_ctrl);
>>> +      stmt = gimple_build_assign (dest, MIN_EXPR, src, factor);
>>> +      gimple_seq_add_stmt (seq, stmt);
>>> +    }
>>> +      else
>>> +    {
>>> +      /* (X - MIN (X, VF*I/N)) capped to the range [0, VF/N].  */
>>> +      tree factor = build_int_cst (iv_type, nitems_per_ctrl * i);
>>> +      tree temp = make_ssa_name (iv_type);
>>> +      stmt = gimple_build_assign (temp, MIN_EXPR, src, factor);
>>> +      gimple_seq_add_stmt (seq, stmt);
>>> +      stmt = gimple_build_assign (dest, MINUS_EXPR, src, temp);
>>> +      gimple_seq_add_stmt (seq, stmt);
>>> +    }
>>> +    }
>>> +}
>>> +
>>> +/* Helper for vect_set_loop_condition_partial_vectors.  Generate
> definitions
>>> +   for all the rgroup controls in RGC and return a control that is nonzero
>>> +   when the loop needs to iterate.  Add any new preheader statements to
>>> +   PREHEADER_SEQ.  Use LOOP_COND_GSI to insert code before the exit gcond.
>>> +
>>> +   RGC belongs to loop LOOP.  The loop originally iterated NITERS
>>> +   times and has been vectorized according to LOOP_VINFO.
>>> +
>>> +   Unlike vect_set_loop_controls_directly which is iterating from 0-based
> IV
>>> +   to TEST_LIMIT - bias.
>>> +
>>> +   In vect_set_loop_controls_by_select_vl, we are iterating from start at
>>> +   IV = TEST_LIMIT - bias and keep subtract IV by the length calculated by
>>> +   IFN_SELECT_VL pattern.
>>> +
>>> +   1. Single rgroup, the Gimple IR should be:
>>> +
>>> +    # vectp_B.6_8 = PHI <vectp_B.6_13(6), &B(5)>
>>> +    # vectp_B.8_16 = PHI <vectp_B.8_17(6), &B(5)>
>>> +    # vectp_A.11_19 = PHI <vectp_A.11_20(6), &A(5)>
>>> +    # vectp_A.13_22 = PHI <vectp_A.13_23(6), &A(5)>
>>> +    # ivtmp_26 = PHI <ivtmp_27(6), _25(5)>
>>> +    _28 = .SELECT_VL (ivtmp_26, POLY_INT_CST [4, 4]);
>>> +    ivtmp_15 = _28 * 4;
>>> +    vect__1.10_18 = .LEN_LOAD (vectp_B.8_16, 128B, _28, 0);
>>> +    _1 = B[i_10];
>>> +    .LEN_STORE (vectp_A.13_22, 128B, _28, vect__1.10_18, 0);
>>> +    i_7 = i_10 + 1;
>>> +    vectp_B.8_17 = vectp_B.8_16 + ivtmp_15;
>>> +    vectp_A.13_23 = vectp_A.13_22 + ivtmp_15;
>>> +    ivtmp_27 = ivtmp_26 - _28;
>>> +    if (ivtmp_27 != 0)
>>> +      goto <bb 6>; [83.33%]
>>> +    else
>>> +      goto <bb 7>; [16.67%]
>>> +
>>> +   Note: We use the outcome of .SELECT_VL to adjust both loop control IV
> and
>>> +   data reference pointer IV.
>>> +
>>> +   1). The result of .SELECT_VL:
>>> +       _28 = .SELECT_VL (ivtmp_26, POLY_INT_CST [4, 4]);
>>> +       The _28 is not necessary to be VF in any iteration, instead, we
> allow
>>> +       _28 to be any value as long as _28 <= VF. Such flexible SELECT_VL
>>> +       pattern allows target have various flexible optimizations in vector
>>> +       loop iterations. Target like RISC-V has special application vector
>>> +       length calculation instruction which will distribute even workload
>>> +       in the last 2 iterations.
>>> +
>>> +       Other example is that we can allow even generate _28 <= VF / 2 so
>>> +       that some machine can run vector codes in low power mode.
>>> +
>>> +   2). Loop control IV:
>>> +       ivtmp_27 = ivtmp_26 - _28;
>>> +       if (ivtmp_27 != 0)
>>> +     goto <bb 6>; [83.33%]
>>> +       else
>>> +     goto <bb 7>; [16.67%]
>>> +
>>> +       This is the saturating-subtraction towards zero, the outcome of
>>> +       .SELECT_VL wil make ivtmp_27 never underflow zero.
>>> +
>>> +   3). Data reference pointer IV:
>>> +       ivtmp_15 = _28 * 4;
>>> +       vectp_B.8_17 = vectp_B.8_16 + ivtmp_15;
>>> +       vectp_A.13_23 = vectp_A.13_22 + ivtmp_15;
>>> +
>>> +       The pointer IV is adjusted accurately according to the .SELECT_VL.
>>> +
>>> +   2. Multiple rgroup, the Gimple IR should be:
>>> +
>>> +    # i_23 = PHI <i_20(6), 0(11)>
>>> +    # vectp_f.8_51 = PHI <vectp_f.8_52(6), f_15(D)(11)>
>>> +    # vectp_d.10_59 = PHI <vectp_d.10_60(6), d_18(D)(11)>
>>> +    # ivtmp_70 = PHI <ivtmp_71(6), _69(11)>
>>> +    # ivtmp_73 = PHI <ivtmp_74(6), _67(11)>
>>> +    _72 = MIN_EXPR <ivtmp_70, 16>;
>>> +    _75 = MIN_EXPR <ivtmp_73, 16>;
>>> +    _1 = i_23 * 2;
>>> +    _2 = (long unsigned int) _1;
>>> +    _3 = _2 * 2;
>>> +    _4 = f_15(D) + _3;
>>> +    _5 = _2 + 1;
>>> +    _6 = _5 * 2;
>>> +    _7 = f_15(D) + _6;
>>> +    .LEN_STORE (vectp_f.8_51, 128B, _75, { 1, 2, 1, 2, 1, 2, 1, 2 }, 0);
>>> +    vectp_f.8_56 = vectp_f.8_51 + 16;
>>> +    .LEN_STORE (vectp_f.8_56, 128B, _72, { 1, 2, 1, 2, 1, 2, 1, 2 }, 0);
>>> +    _8 = (long unsigned int) i_23;
>>> +    _9 = _8 * 4;
>>> +    _10 = d_18(D) + _9;
>>> +    _61 = _75 / 2;
>>> +    .LEN_STORE (vectp_d.10_59, 128B, _61, { 3, 3, 3, 3 }, 0);
>>> +    vectp_d.10_63 = vectp_d.10_59 + 16;
>>> +    _64 = _72 / 2;
>>> +    .LEN_STORE (vectp_d.10_63, 128B, _64, { 3, 3, 3, 3 }, 0);
>>> +    i_20 = i_23 + 1;
>>> +    vectp_f.8_52 = vectp_f.8_56 + 16;
>>> +    vectp_d.10_60 = vectp_d.10_63 + 16;
>>> +    ivtmp_74 = ivtmp_73 - _75;
>>> +    ivtmp_71 = ivtmp_70 - _72;
>>> +    if (ivtmp_74 != 0)
>>> +      goto <bb 6>; [83.33%]
>>> +    else
>>> +      goto <bb 13>; [16.67%]
>>
>> In the gimple examples, I think it would help to quote only the relevant
>> parts and use ellipsis to hide things that don't directly matter.
>> E.g. in the above samples, the old scalar code isn't relevant, whereas
>> it's difficult to follow the example without knowing how _69 and _67
>> relate to each other.  It would also help to say which scalar loop
>> is being vectorised here.
>>
>>> +
>>> +   Note: We DO NOT use .SELECT_VL in SLP auto-vectorization for multiple
>>> +   rgroups. Instead, we use MIN_EXPR to guarantee we always use VF as the
>>> +   iteration amount for mutiple rgroups.
>>> +
>>> +   The analysis of the flow of multiple rgroups:
>>> +    _72 = MIN_EXPR <ivtmp_70, 16>;
>>> +    _75 = MIN_EXPR <ivtmp_73, 16>;
>>> +    ...
>>> +    .LEN_STORE (vectp_f.8_51, 128B, _75, { 1, 2, 1, 2, 1, 2, 1, 2 }, 0);
>>> +    vectp_f.8_56 = vectp_f.8_51 + 16;
>>> +    .LEN_STORE (vectp_f.8_56, 128B, _72, { 1, 2, 1, 2, 1, 2, 1, 2 }, 0);
>>> +    ...
>>> +    _61 = _75 / 2;
>>> +    .LEN_STORE (vectp_d.10_59, 128B, _61, { 3, 3, 3, 3 }, 0);
>>> +    vectp_d.10_63 = vectp_d.10_59 + 16;
>>> +    _64 = _72 / 2;
>>> +    .LEN_STORE (vectp_d.10_63, 128B, _64, { 3, 3, 3, 3 }, 0);
>>> +
>>> +  We use _72 = MIN_EXPR <ivtmp_70, 16>; to generate the number of the
>> elements
>>> +  to be processed in each iteration.
>>> +
>>> +  The related STOREs:
>>> +    _72 = MIN_EXPR <ivtmp_70, 16>;
>>> +    .LEN_STORE (vectp_f.8_56, 128B, _72, { 1, 2, 1, 2, 1, 2, 1, 2 }, 0);
>>> +    _64 = _72 / 2;
>>> +    .LEN_STORE (vectp_d.10_63, 128B, _64, { 3, 3, 3, 3 }, 0);
>>> +  Since these 2 STOREs store 2 vectors that the second vector is half
>> elements
>>> +  of the first vector. So the length of second STORE will be _64 = _72 / 2;
>>> +  It's similar to the VIEW_CONVERT of handling masks in SLP.
>>
>>
>>
>>> +
>>> +  3. Multiple rgroups for non-SLP auto-vectorization.
>>> +
>>> +     # ivtmp_26 = PHI <ivtmp_27(4), _25(3)>
>>> +     # ivtmp.35_10 = PHI <ivtmp.35_11(4), ivtmp.35_1(3)>
>>> +     # ivtmp.36_2 = PHI <ivtmp.36_8(4), ivtmp.36_23(3)>
>>> +     _28 = MIN_EXPR <ivtmp_26, POLY_INT_CST [8, 8]>;
>>> +     loop_len_15 = MIN_EXPR <_28, POLY_INT_CST [4, 4]>;
>>> +     loop_len_16   = _28 - loop_len_15;
>>> +     _29 = (void *) ivtmp.35_10;
>>> +     _7 = &MEM <vector([4,4]) int> [(int *)_29];
>>> +     vect__1.25_17 = .LEN_LOAD (_7, 128B, loop_len_15, 0);
>>> +     _33 = _29 + POLY_INT_CST [16, 16];
>>> +     _34 = &MEM <vector([4,4]) int> [(int *)_33];
>>> +     vect__1.26_19 = .LEN_LOAD (_34, 128B, loop_len_16, 0);
>>> +     vect__2.27_20 = VEC_PACK_TRUNC_EXPR <vect__1.25_17, vect__1.26_19>;
>>> +     _30 = (void *) ivtmp.36_2;
>>> +     _31 = &MEM <vector([8,8]) short int> [(short int *)_30];
>>> +     .LEN_STORE (_31, 128B, _28, vect__2.27_20, 0);
>>> +     ivtmp_27 = ivtmp_26 - _28;
>>> +     ivtmp.35_11 = ivtmp.35_10 + POLY_INT_CST [32, 32];
>>> +     ivtmp.36_8 = ivtmp.36_2 + POLY_INT_CST [16, 16];
>>> +     if (ivtmp_27 != 0)
>>> +       goto <bb 4>; [83.33%]
>>> +     else
>>> +       goto <bb 5>; [16.67%]
>>> +
>>> +     The total length: _28 = MIN_EXPR <ivtmp_26, POLY_INT_CST [8, 8]>;
>>> +
>>> +     The length of first half vector:
>>> +       loop_len_15 = MIN_EXPR <_28, POLY_INT_CST [4, 4]>;
>>> +
>>> +     The length of second half vector:
>>> +       loop_len_15 = MIN_EXPR <_28, POLY_INT_CST [4, 4]>;
>>> +       loop_len_16 = _28 - loop_len_15;
>>> +
>>> +     1). _28 always <= POLY_INT_CST [8, 8].
>>> +     2). When _28 <= POLY_INT_CST [4, 4], second half vector is not
>> processed.
>>> +     3). When _28 > POLY_INT_CST [4, 4], second half vector is processed.
>>> +*/
>>> +
>>> +static tree
>>> +vect_set_loop_controls_by_select_vl (class loop *loop, loop_vec_info
>> loop_vinfo,
>>> +                     gimple_seq *preheader_seq,
>>> +                     gimple_seq *header_seq,
>>> +                     rgroup_controls *rgc, tree niters)
>>> +{
>>> +  tree compare_type = LOOP_VINFO_RGROUP_COMPARE_TYPE (loop_vinfo);
>>> +  tree iv_type = LOOP_VINFO_RGROUP_IV_TYPE (loop_vinfo);
>>> +  /* We are not allowing masked approach in SELECT_VL.  */
>>> +  gcc_assert (!LOOP_VINFO_FULLY_MASKED_P (loop_vinfo));
>>> +
>>> +  tree ctrl_type = rgc->type;
>>> +  unsigned int nitems_per_iter = rgc->max_nscalars_per_iter * rgc->factor;
>>> +  poly_uint64 nitems_per_ctrl = TYPE_VECTOR_SUBPARTS (ctrl_type) * rgc->
>> factor;
>>> +  poly_uint64 vf = LOOP_VINFO_VECT_FACTOR (loop_vinfo);
>>> +
>>> +  /* Calculate the maximum number of item values that the rgroup
>>> +     handles in total, the number that it handles for each iteration
>>> +     of the vector loop.  */
>>> +  tree nitems_total = niters;
>>> +  if (nitems_per_iter != 1)
>>> +    {
>>> +      /* We checked before setting LOOP_VINFO_USING_PARTIAL_VECTORS_P that
>>> +     these multiplications don't overflow.  */
>>> +      tree compare_factor = build_int_cst (compare_type, nitems_per_iter);
>>> +      nitems_total = gimple_build (preheader_seq, MULT_EXPR, compare_type,
>>> +                   nitems_total, compare_factor);
>>> +    }
>>> +
>>> +  /* Convert the comparison value to the IV type (either a no-op or
>>> +     a promotion).  */
>>> +  nitems_total = gimple_convert (preheader_seq, iv_type, nitems_total);
>>> +
>>> +  /* Create an induction variable that counts the number of items
>>> +     processed.  */
>>> +  tree index_before_incr, index_after_incr;
>>> +  gimple_stmt_iterator incr_gsi;
>>> +  bool insert_after;
>>> +  standard_iv_increment_position (loop, &incr_gsi, &insert_after);
>>> +
>>> +  /* Test the decremented IV, which will never underflow 0 since we have
>>> +     IFN_SELECT_VL to gurantee that.  */
>>> +  tree test_limit = nitems_total;
>>> +
>>> +  /* Provide a definition of each control in the group.  */
>>> +  tree ctrl;
>>> +  unsigned int i;
>>> +  FOR_EACH_VEC_ELT_REVERSE (rgc->controls, i, ctrl)
>>> +    {
>>> +      /* Previous controls will cover BIAS items.  This control covers the
>>> +     next batch.  */
>>> +      poly_uint64 bias = nitems_per_ctrl * i;
>>> +      tree bias_tree = build_int_cst (iv_type, bias);
>>> +
>>> +      /* Rather than have a new IV that starts at TEST_LIMIT and goes down
>> to
>>> +     BIAS, prefer to use the same TEST_LIMIT - BIAS based IV for each
>>> +     control and adjust the bound down by BIAS.  */
>>> +      tree this_test_limit = test_limit;
>>> +      if (i != 0)
>>> +    {
>>> +      this_test_limit = gimple_build (preheader_seq, MAX_EXPR, iv_type,
>>> +                      this_test_limit, bias_tree);
>>> +      this_test_limit = gimple_build (preheader_seq, MINUS_EXPR, iv_type,
>>> +                      this_test_limit, bias_tree);
>>> +    }
>>> +
>>> +      /* Create decrement IV.  */
>>> +      create_iv (this_test_limit, MINUS_EXPR, ctrl, NULL_TREE, loop, &
>> incr_gsi,
>>> +         insert_after, &index_before_incr, &index_after_incr);
>>> +
>>> +      poly_uint64 final_vf = vf * nitems_per_iter;
>>> +      tree vf_step = build_int_cst (iv_type, final_vf);
>>> +      tree res_len;
>>> +      if (LOOP_VINFO_LENS (loop_vinfo).length () == 1)
>>> +    {
>>> +      res_len = gimple_build (header_seq, IFN_SELECT_VL, iv_type,
>>> +                  index_before_incr, vf_step);
>>> +      LOOP_VINFO_USING_SELECT_VL_P (loop_vinfo) = true;
>>
>> The middle of this loop seems too "deep down" to be setting this.
>> I think it would make sense to do it after:
>>
>>  /* If we still have the option of using partial vectors,
>>     check whether we can generate the necessary loop controls.  */
>>  if (LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo)
>>      && !vect_verify_full_masking (loop_vinfo)
>>      && !vect_verify_loop_lens (loop_vinfo))
>>    LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) = false;
>>
>> in vect_analyze_loop_2.
>>
>> I think it'd help for review purposes to split this patch into two
>> (independently tested) pieces:
>>
>> (1) Your cases 2 and 3, where AIUI the main change is to use
>>    a decrementing loop control IV that counts scalars.  This can
>>    be done for any loop that:
>>
>>    (a) uses length "controls"; and
>>    (b) can iterate more than once
>>
>>    Initially this patch would handle case 1 in the same way.
>>
>>    Conceptually, I think it would make sense for this case to use:
>>
>>    - a signed control IV
>>    - with a constant VF step
>>    - and a loop-back test for > 0
>>
>>    in cases where we can prove that that doesn't overflow.  But I
>>    accept that using:
>>
>>    - an unsigned control IV
>>    - with a variable step
>>    - and a loop-back test for != 0
>>
>>    is more general.  So it's OK to handle just that case.  The
>>    optimisation to use signed control IVs could be left to future work.
>>
>> (2) Add SELECT_VL, where AIUI the main change (relative to (1))
>>    is to use a variable step for other IVs too.
>>
>> This is just for review purposes, and to help to separate concepts.
>> SELECT_VL is still an important part of the end result.
>>
>> Thanks,
>> Richard
>>
>>> +    }
>>> +      else
>>> +    {
>>> +      /* For SLP, we can't allow non-VF number of elements to be processed
>>> +         in non-final iteration. We force the number of elements to be
>>> +         processed in each non-final iteration is VF elements. If we allow
>>> +         non-VF elements processing in non-final iteration will make SLP
> too
>>> +         complicated and produce inferior codegen.
>>> +
>>> +           For example:
>>> +
>>> +        If non-final iteration process VF elements.
>>> +
>>> +          ...
>>> +          .LEN_STORE (vectp_f.8_51, 128B, _71, { 1, 2, 1, 2 }, 0);
>>> +          .LEN_STORE (vectp_f.8_56, 128B, _72, { 1, 2, 1, 2 }, 0);
>>> +          ...
>>> +
>>> +        If non-final iteration process non-VF elements.
>>> +
>>> +          ...
>>> +          .LEN_STORE (vectp_f.8_51, 128B, _71, { 1, 2, 1, 2 }, 0);
>>> +          if (_71 % 2 == 0)
>>> +           .LEN_STORE (vectp_f.8_56, 128B, _72, { 1, 2, 1, 2 }, 0);
>>> +          else
>>> +           .LEN_STORE (vectp_f.8_56, 128B, _72, { 2, 1, 2, 1 }, 0);
>>> +          ...
>>> +
>>> +         This is the simple case of 2-elements interleaved vector SLP. We
>>> +         consider other interleave vector, the situation will become more
>>> +         complicated.  */
>>> +      res_len = gimple_build (header_seq, MIN_EXPR, iv_type,
>>> +                  index_before_incr, vf_step);
>>> +      if (rgc->max_nscalars_per_iter != 1)
>>> +        LOOP_VINFO_USING_SLP_ADJUSTED_LEN_P (loop_vinfo) = true;
>>> +    }
>>> +      gassign *assign = gimple_build_assign (ctrl, res_len);
>>> +      gimple_seq_add_stmt (header_seq, assign);
>>> +    }
>>> +
>>> +  return index_after_incr;
>>> +}
>>> +
>>>  /* Helper for vect_set_loop_condition_partial_vectors.  Generate
> definitions
>>>     for all the rgroup controls in RGC and return a control that is nonzero
>>>     when the loop needs to iterate.  Add any new preheader statements to
>>> @@ -704,6 +1051,10 @@ vect_set_loop_condition_partial_vectors (class loop
>> *loop,
>>>  
>>>    bool use_masks_p = LOOP_VINFO_FULLY_MASKED_P (loop_vinfo);
>>>    tree compare_type = LOOP_VINFO_RGROUP_COMPARE_TYPE (loop_vinfo);
>>> +  tree iv_type = LOOP_VINFO_RGROUP_IV_TYPE (loop_vinfo);
>>> +  bool use_vl_p = !use_masks_p
>>> +          && direct_internal_fn_supported_p (IFN_SELECT_VL, iv_type,
>>> +                             OPTIMIZE_FOR_SPEED);
>>>    unsigned int compare_precision = TYPE_PRECISION (compare_type);
>>>    tree orig_niters = niters;
>>>  
>>> @@ -753,17 +1104,34 @@ vect_set_loop_condition_partial_vectors (class loop
>> *loop,
>>>            continue;
>>>        }
>>>  
>>> +    if (use_vl_p && rgc->max_nscalars_per_iter == 1
>>> +        && rgc != &LOOP_VINFO_LENS (loop_vinfo)[0])
>>> +      {
>>> +        rgroup_controls *sub_rgc
>>> +          = &(*controls)[nmasks / rgc->controls.length () - 1];
>>> +        if (!sub_rgc->controls.is_empty ())
>>> +          {
>>> +        vect_adjust_loop_lens (iv_type, &header_seq, rgc, sub_rgc);
>>> +        continue;
>>> +          }
>>> +      }
>>> +
>>>      /* See whether zero-based IV would ever generate all-false masks
>>>         or zero length before wrapping around.  */
>>>      bool might_wrap_p = vect_rgroup_iv_might_wrap_p (loop_vinfo, rgc);
>>>  
>>>      /* Set up all controls for this group.  */
>>> -    test_ctrl = vect_set_loop_controls_directly (loop, loop_vinfo,
>>> -                             &preheader_seq,
>>> -                             &header_seq,
>>> -                             loop_cond_gsi, rgc,
>>> -                             niters, niters_skip,
>>> -                             might_wrap_p);
>>> +    if (use_vl_p)
>>> +      test_ctrl
>>> +        = vect_set_loop_controls_by_select_vl (loop, loop_vinfo,
>>> +                           &preheader_seq, &header_seq,
>>> +                           rgc, niters);
>>> +    else
>>> +      test_ctrl
>>> +        = vect_set_loop_controls_directly (loop, loop_vinfo, &
> preheader_seq,
>>> +                           &header_seq, loop_cond_gsi, rgc,
>>> +                           niters, niters_skip,
>>> +                           might_wrap_p);
>>>        }
>>>  
>>>    /* Emit all accumulated statements.  */
>>> diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
>>> index ed0166fedab..fe6af4286bf 100644
>>> --- a/gcc/tree-vect-loop.cc
>>> +++ b/gcc/tree-vect-loop.cc
>>> @@ -973,6 +973,8 @@ _loop_vec_info::_loop_vec_info (class loop *loop_in,
>> vec_info_shared *shared)
>>>      vectorizable (false),
>>>      can_use_partial_vectors_p (param_vect_partial_vector_usage != 0),
>>>      using_partial_vectors_p (false),
>>> +    using_select_vl_p (false),
>>> +    using_slp_adjusted_len_p (false),
>>>      epil_using_partial_vectors_p (false),
>>>      partial_load_store_bias (0),
>>>      peeling_for_gaps (false),
>>> @@ -10361,15 +10363,18 @@ vect_record_loop_len (loop_vec_info loop_vinfo,
>> vec_loop_lens *lens,
>>>  }
>>>  
>>>  /* Given a complete set of length LENS, extract length number INDEX for an
>>> -   rgroup that operates on NVECTORS vectors, where 0 <= INDEX < NVECTORS.
>  *
>> /
>>> +   rgroup that operates on NVECTORS vectors, where 0 <= INDEX < NVECTORS.
>>> +   Insert any set-up statements before GSI.  */
>>>  
>>>  tree
>>> -vect_get_loop_len (loop_vec_info loop_vinfo, vec_loop_lens *lens,
>>> -           unsigned int nvectors, unsigned int index)
>>> +vect_get_loop_len (loop_vec_info loop_vinfo, gimple_stmt_iterator *gsi,
>>> +           vec_loop_lens *lens, unsigned int nvectors, tree vectype,
>>> +           unsigned int index)
>>>  {
>>>    rgroup_controls *rgl = &(*lens)[nvectors - 1];
>>>    bool use_bias_adjusted_len =
>>>      LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo) != 0;
>>> +  tree iv_type = LOOP_VINFO_RGROUP_IV_TYPE (loop_vinfo);
>>>  
>>>    /* Populate the rgroup's len array, if this is the first time we've
>>>       used it.  */
>>> @@ -10400,6 +10405,26 @@ vect_get_loop_len (loop_vec_info loop_vinfo,
>> vec_loop_lens *lens,
>>>  
>>>    if (use_bias_adjusted_len)
>>>      return rgl->bias_adjusted_ctrl;
>>> +  else if (LOOP_VINFO_USING_SLP_ADJUSTED_LEN_P (loop_vinfo))
>>> +    {
>>> +      tree loop_len = rgl->controls[index];
>>> +      poly_int64 nunits1 = TYPE_VECTOR_SUBPARTS (rgl->type);
>>> +      poly_int64 nunits2 = TYPE_VECTOR_SUBPARTS (vectype);
>>> +      if (maybe_ne (nunits1, nunits2))
>>> +    {
>>> +      /* A loop len for data type X can be reused for data type Y
>>> +         if X has N times more elements than Y and if Y's elements
>>> +         are N times bigger than X's.  */
>>> +      gcc_assert (multiple_p (nunits1, nunits2));
>>> +      unsigned int factor = exact_div (nunits1, nunits2).to_constant ();
>>> +      gimple_seq seq = NULL;
>>> +      loop_len = gimple_build (&seq, RDIV_EXPR, iv_type, loop_len,
>>> +                   build_int_cst (iv_type, factor));
>>> +      if (seq)
>>> +        gsi_insert_seq_before (gsi, seq, GSI_SAME_STMT);
>>> +    }
>>> +      return loop_len;
>>> +    }
>>>    else
>>>      return rgl->controls[index];
>>>  }
>>> diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
>>> index 7313191b0db..15b22132bd6 100644
>>> --- a/gcc/tree-vect-stmts.cc
>>> +++ b/gcc/tree-vect-stmts.cc
>>> @@ -3147,6 +3147,61 @@ vect_get_data_ptr_increment (vec_info *vinfo,
>>>    return iv_step;
>>>  }
>>>  
>>> +/* Prepare the pointer IVs which needs to be updated by a variable amount.
>>> +   Such variable amount is the outcome of .SELECT_VL. In this case, we can
>>> +   allow each iteration process the flexible number of elements as long as
>>> +   the number <= vf elments.
>>> +
>>> +   Return data reference according to SELECT_VL.
>>> +   If new statements are needed, insert them before GSI.  */
>>> +
>>> +static tree
>>> +get_select_vl_data_ref_ptr (vec_info *vinfo, stmt_vec_info stmt_info,
>>> +                tree aggr_type, class loop *at_loop, tree offset,
>>> +                tree *dummy, gimple_stmt_iterator *gsi,
>>> +                bool simd_lane_access_p, vec_loop_lens *loop_lens,
>>> +                dr_vec_info *dr_info,
>>> +                vect_memory_access_type memory_access_type)
>>> +{
>>> +  loop_vec_info loop_vinfo = dyn_cast<loop_vec_info> (vinfo);
>>> +  tree step = vect_dr_behavior (vinfo, dr_info)->step;
>>> +
>>> +  /* TODO: We don't support gather/scatter or load_lanes/store_lanes for
>> pointer
>>> +     IVs are updated by variable amount but we will support them in the
>> future.
>>> +   */
>>> +  gcc_assert (memory_access_type != VMAT_GATHER_SCATTER
>>> +          && memory_access_type != VMAT_LOAD_STORE_LANES);
>>> +
>>> +  /* When we support SELECT_VL pattern, we dynamic adjust
>>> +     the memory address by .SELECT_VL result.
>>> +
>>> +     The result of .SELECT_VL is the number of elements to
>>> +     be processed of each iteration. So the memory address
>>> +     adjustment operation should be:
>>> +
>>> +     bytesize = GET_MODE_SIZE (element_mode (aggr_type));
>>> +     addr = addr + .SELECT_VL (ARG..) * bytesize;
>>> +  */
>>> +  gimple *ptr_incr;
>>> +  tree loop_len
>>> +    = vect_get_loop_len (loop_vinfo, gsi, loop_lens, 1, aggr_type, 0);
>>> +  tree len_type = TREE_TYPE (loop_len);
>>> +  poly_uint64 bytesize = GET_MODE_SIZE (element_mode (aggr_type));
>>> +  /* Since the outcome of .SELECT_VL is element size, we should adjust
>>> +     it into bytesize so that it can be used in address pointer variable
>>> +     amount IVs adjustment.  */
>>> +  tree tmp = fold_build2 (MULT_EXPR, len_type, loop_len,
>>> +              build_int_cst (len_type, bytesize));
>>> +  if (tree_int_cst_sgn (step) == -1)
>>> +    tmp = fold_build1 (NEGATE_EXPR, len_type, tmp);
>>> +  tree bump = make_temp_ssa_name (len_type, NULL, "ivtmp");
>>> +  gassign *assign = gimple_build_assign (bump, tmp);
>>> +  gsi_insert_before (gsi, assign, GSI_SAME_STMT);
>>> +  return vect_create_data_ref_ptr (vinfo, stmt_info, aggr_type, at_loop,
>> offset,
>>> +                   dummy, gsi, &ptr_incr, simd_lane_access_p,
>>> +                   bump);
>>> +}
>>> +
>>>  /* Check and perform vectorization of BUILT_IN_BSWAP{16,32,64,128}.  */
>>>  
>>>  static bool
>>> @@ -8547,6 +8602,14 @@ vectorizable_store (vec_info *vinfo,
>>>          vect_get_gather_scatter_ops (loop_vinfo, loop, stmt_info,
>>>                       slp_node, &gs_info, &dataref_ptr,
>>>                       &vec_offsets);
>>> +      else if (loop_vinfo && LOOP_VINFO_USING_SELECT_VL_P (loop_vinfo)
>>> +           && memory_access_type != VMAT_INVARIANT)
>>> +        dataref_ptr
>>> +          = get_select_vl_data_ref_ptr (vinfo, stmt_info, aggr_type,
>>> +                        simd_lane_access_p ? loop : NULL,
>>> +                        offset, &dummy, gsi,
>>> +                        simd_lane_access_p, loop_lens,
>>> +                        dr_info, memory_access_type);
>>>        else
>>>          dataref_ptr
>>>            = vect_create_data_ref_ptr (vinfo, first_stmt_info, aggr_type,
>>> @@ -8795,8 +8858,9 @@ vectorizable_store (vec_info *vinfo,
>>>            else if (loop_lens)
>>>          {
>>>            tree final_len
>>> -            = vect_get_loop_len (loop_vinfo, loop_lens,
>>> -                     vec_num * ncopies, vec_num * j + i);
>>> +            = vect_get_loop_len (loop_vinfo, gsi, loop_lens,
>>> +                     vec_num * ncopies, vectype,
>>> +                     vec_num * j + i);
>>>            tree ptr = build_int_cst (ref_type, align * BITS_PER_UNIT);
>>>            machine_mode vmode = TYPE_MODE (vectype);
>>>            opt_machine_mode new_ovmode
>>> @@ -9935,6 +9999,13 @@ vectorizable_load (vec_info *vinfo,
>>>                         slp_node, &gs_info, &dataref_ptr,
>>>                         &vec_offsets);
>>>          }
>>> +      else if (loop_vinfo && LOOP_VINFO_USING_SELECT_VL_P (loop_vinfo)
>>> +           && memory_access_type != VMAT_INVARIANT)
>>> +        dataref_ptr
>>> +          = get_select_vl_data_ref_ptr (vinfo, stmt_info, aggr_type,
>>> +                        at_loop, offset, &dummy, gsi,
>>> +                        simd_lane_access_p, loop_lens,
>>> +                        dr_info, memory_access_type);
>>>        else
>>>          dataref_ptr
>>>            = vect_create_data_ref_ptr (vinfo, first_stmt_info, aggr_type,
>>> @@ -10151,8 +10222,8 @@ vectorizable_load (vec_info *vinfo,
>>>              else if (loop_lens && memory_access_type != VMAT_INVARIANT)
>>>                {
>>>              tree final_len
>>> -              = vect_get_loop_len (loop_vinfo, loop_lens,
>>> -                           vec_num * ncopies,
>>> +              = vect_get_loop_len (loop_vinfo, gsi, loop_lens,
>>> +                           vec_num * ncopies, vectype,
>>>                             vec_num * j + i);
>>>              tree ptr = build_int_cst (ref_type,
>>>                            align * BITS_PER_UNIT);
>>> diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
>>> index 9cf2fb23fe3..3d21e23513d 100644
>>> --- a/gcc/tree-vectorizer.h
>>> +++ b/gcc/tree-vectorizer.h
>>> @@ -818,6 +818,13 @@ public:
>>>       the vector loop can handle fewer than VF scalars.  */
>>>    bool using_partial_vectors_p;
>>>  
>>> +  /* True if we've decided to use SELECT_VL to get the number of active
>>> +     elements in a vector loop to be updated.  */
>>> +  bool using_select_vl_p;
>>> +
>>> +  /* True if use adjusted loop length for SLP.  */
>>> +  bool using_slp_adjusted_len_p;
>>> +
>>>    /* True if we've decided to use partially-populated vectors for the
>>>       epilogue of loop.  */
>>>    bool epil_using_partial_vectors_p;
>>> @@ -890,6 +897,8 @@ public:
>>>  #define LOOP_VINFO_VECTORIZABLE_P(L)       (L)->vectorizable
>>>  #define LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P(L) (L)->
>> can_use_partial_vectors_p
>>>  #define LOOP_VINFO_USING_PARTIAL_VECTORS_P(L) (L)->using_partial_vectors_p
>>> +#define LOOP_VINFO_USING_SELECT_VL_P(L) (L)->using_select_vl_p
>>> +#define LOOP_VINFO_USING_SLP_ADJUSTED_LEN_P(L) (L)->
> using_slp_adjusted_len_p
>>>  #define LOOP_VINFO_EPIL_USING_PARTIAL_VECTORS_P(L)
>>                             \
>>>    (L)->epil_using_partial_vectors_p
>>>  #define LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS(L) (L)->partial_load_store_bias
>>> @@ -2293,7 +2302,8 @@ extern tree vect_get_loop_mask (gimple_stmt_iterator
> *,
>> vec_loop_masks *,
>>>                  unsigned int, tree, unsigned int);
>>>  extern void vect_record_loop_len (loop_vec_info, vec_loop_lens *, unsigned
>> int,
>>>                    tree, unsigned int);
>>> -extern tree vect_get_loop_len (loop_vec_info, vec_loop_lens *, unsigned
> int,
>>> +extern tree vect_get_loop_len (loop_vec_info, gimple_stmt_iterator *,
>>> +                   vec_loop_lens *, unsigned int, tree,
>>>                     unsigned int);
>>>  extern gimple_seq vect_gen_len (tree, tree, tree, tree);
>>>  extern stmt_vec_info info_for_reduction (vec_info *, stmt_vec_info);
>> 8
> C