* Re: Add a loop versioning pass
@ 2018-10-30 10:55 Richard Biener
2018-11-09 10:46 ` Kyrill Tkachov
2018-11-28 16:48 ` Richard Sandiford
0 siblings, 2 replies; 17+ messages in thread
From: Richard Biener @ 2018-10-30 10:55 UTC (permalink / raw)
To: gcc-patches, richard.sandiford
(sorry for breaking threading -- I composed a review mail offline but
gmail has no way of nicely sending that neither has it a way to bounce
messages...)
> This patch adds a pass that versions loops with variable index strides
> for the case in which the stride is 1. E.g.:
>
> for (int i = 0; i < n; ++i)
> x[i * stride] = ...;
>
> becomes:
>
> if (stepx == 1)
> for (int i = 0; i < n; ++i)
> x[i] = ...;
> else
> for (int i = 0; i < n; ++i)
> x[i * stride] = ...;
>
> This is useful for both vector code and scalar code, and in some cases
> can enable further optimisations like loop interchange or pattern
> recognition.
>
> The pass gives a 7.6% improvement on Cortex-A72 for 554.roms_r at -O3
> and a 2.4% improvement for 465.tonto. I haven't found any SPEC tests
> that regress.
>
> Sizewise, there's a 10% increase in .text for both 554.roms_r and
> 465.tonto. That's obviously a lot, but in tonto's case it's because
> the whole program is written using assumed-shape arrays and pointers,
> so a large number of functions really do benefit from versioning.
> roms likewise makes heavy use of assumed-shape arrays, and that
> improvement in performance IMO justifies the code growth.
Ouch. I know that at least with LTO IPA-CP can do "quite" some
propagation of constant strides. Not sure if we're aggressive
enough in actually doing the cloning for all cases we figure out
strides though. But my question is how we can avoid doing the
versioning for loops in the copy that did not have the IPA-CPed
stride of one? Ideally we'd be able to mark individual references
as {definitely,likely,unlikely,not}-unit-stride?
> The next biggest .text increase is 4.5% for 548.exchange2_r. I did see
> a small (0.4%) speed improvement there, but although both 3-iteration runs
> produced stable results, that might still be noise. There was a slightly
> larger (non-noise) improvement for a 256-bit SVE model.
>
> 481.wrf and 521.wrf_r .text grew by 2.8% and 2.5% respectively, but
> without any noticeable improvement in performance. No other test grew
> by more than 2%.
>
> Although the main SPEC beneficiaries are all Fortran tests, the
> benchmarks we use for SVE also include some C and C++ tests that
> benefit.
Did you see any slowdown, for example because versioning was forced
to be on an innermost loop? I'm thinking of the testcase in
PR87561 where we do have strided accesses in the innermost loop.
Since you cite performance numbers how did you measure them?
I assume -Ofast -march=native but did you check with -flto?
> Using -frepack-arrays gives the same benefits in many Fortran cases.
> The problem is that using that option inappropriately can force a full
> array copy for arguments that the function only reads once, and so it
> isn't really something we can turn on by default. The new pass is
> supposed to give most of the benefits of -frepack-arrays without
> the risk of unnecessary repacking.
>
> The patch therefore enables the pass by default at -O3.
I think that's reasonable.
One possible enhancement would be to add a value-profile for the
strides so we can guide this optimization better.
The pass falls foul of C++ class make small methods of everything.
That makes following the code very hard. Please inline single-used
methods in callers wherever possible to make the code read
more like GCC code (using GCC API).
The pass contains an awful lot of heuristics :/ Like last year
with the interchange pass I would suggest to rip most of it out
and first lay infrastructure with the cases you can positively
identify without applying heuristics or "hacks" like stripping
semantically required casts. That makes it also clear which
testcases test which code-path. That said, all the analyze
multiplications/plusses/factors stuff was extremely hard to review
and I have no overall picture why this is all so complicated or
necessary.
> Tested on aarch64-linux-gnu and x86_64-linux-gnu. OK to install?
>
> Richard
>
>
> 2018-10-24 Richard Sandiford <richard.sandiford@arm.com>
>
> gcc/
> * doc/invoke.texi (-fversion-loops-for-strides): Document
> (loop-versioning-group-size, loop-versioning-max-inner-insns)
> (loop-versioning-max-outer-insns): Document new --params.
> * Makefile.in (OBJS): Add gimple-loop-versioning.o.
> * common.opt (fversion-loops-for-strides): New option.
> * opts.c (default_options_table): Enable fversion-loops-for-strides
> at -O3.
> * params.def (PARAM_LOOP_VERSIONING_GROUP_SIZE)
> (PARAM_LOOP_VERSIONING_MAX_INNER_INSNS)
> (PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS): New parameters.
> * passes.def: Add pass_loop_versioning.
> * timevar.def (TV_LOOP_VERSIONING): New time variable.
> * tree-ssa-propagate.h
> (substitute_and_fold_engine::substitute_and_fold): Add an optional
> block parameter.
> * tree-ssa-propagate.c
> (substitute_and_fold_engine::substitute_and_fold): Likewise.
> When passed, only walk blocks dominated by that block.
> * tree-vrp.h (range_includes_p): Declare.
> (range_includes_zero_p): Turn into an inline wrapper around
> range_includes_p.
> * tree-vrp.c (range_includes_p): New function, generalizing...
> (range_includes_zero_p): ...this.
> * tree-pass.h (make_pass_loop_versioning): Declare.
> * gimple-loop-versioning.cc: New file.
>
> gcc/testsuite/
> * gcc.dg/loop-versioning-1.c: New test.
> * gcc.dg/loop-versioning-10.c: Likewise.
> * gcc.dg/loop-versioning-11.c: Likewise.
> * gcc.dg/loop-versioning-2.c: Likewise.
> * gcc.dg/loop-versioning-3.c: Likewise.
> * gcc.dg/loop-versioning-4.c: Likewise.
> * gcc.dg/loop-versioning-5.c: Likewise.
> * gcc.dg/loop-versioning-6.c: Likewise.
> * gcc.dg/loop-versioning-7.c: Likewise.
> * gcc.dg/loop-versioning-8.c: Likewise.
> * gcc.dg/loop-versioning-9.c: Likewise.
> * gfortran.dg/loop_versioning_1.f90: Likewise.
> * gfortran.dg/loop_versioning_2.f90: Likewise.
> * gfortran.dg/loop_versioning_3.f90: Likewise.
> * gfortran.dg/loop_versioning_4.f90: Likewise.
> * gfortran.dg/loop_versioning_5.f90: Likewise.
> * gfortran.dg/loop_versioning_6.f90: Likewise.
> * gfortran.dg/loop_versioning_7.f90: Likewise.
> * gfortran.dg/loop_versioning_8.f90: Likewise.
>
> Index: gcc/doc/invoke.texi
> ===================================================================
> --- gcc/doc/invoke.texi 2018-10-24 14:02:14.000000000 +0100
> +++ gcc/doc/invoke.texi 2018-10-24 14:02:15.184152693 +0100
> @@ -7934,7 +7934,8 @@ by @option{-O2} and also turns on the fo
> -fvect-cost-model @gol
> -ftree-partial-pre @gol
> -fpeel-loops @gol
> --fipa-cp-clone}
> +-fipa-cp-clone @gol
> +-fversion-loops-for-strides}
>
> @item -O0
> @opindex O0
> @@ -10358,6 +10359,29 @@ for one side of the iteration space and
> Move branches with loop invariant conditions out of the loop, with duplicates
> of the loop on both branches (modified according to result of the condition).
>
> +@item -fversion-loops-for-strides
> +@opindex fversion-loops-for-strides
> +If a loop iterates over an array with a variable stride, create another
> +version of the loop that assumes the stride is always 1. For example:
> +
> +@smallexample
> +for (int i = 0; i < n; ++i)
> + x[i * stride] = @dots{};
> +@end smallexample
> +
> +becomes:
> +
> +@smallexample
> +if (stride == 1)
> + for (int i = 0; i < n; ++i)
> + x[i] = @dots{};
> +else
> + for (int i = 0; i < n; ++i)
> + x[i * stride] = @dots{};
> +@end smallexample
> +
> +This is particularly useful for assumed-shape arrays in Fortran.
"where it allows better vectorization assuming contiguous accesses."
> +
> @item -ffunction-sections
> @itemx -fdata-sections
> @opindex ffunction-sections
> @@ -11567,6 +11591,20 @@ Hardware autoprefetcher scheduler model
> Number of lookahead cycles the model looks into; at '
> ' only enable instruction sorting heuristic.
>
> +@item loop-versioning-group-size
> +Make the loop versioning pass optimize @samp{a[i * index * @var{N}]}
> +in the same way as it would optimize @samp{a[i * index]} when @var{N}
> +is less than or equal to this value.
> +
> +@item loop-versioning-max-inner-insns
> +The maximum number of instructions that an inner loop can have
> +before the loop versioning pass considers it too big to copy.
> +
> +@item loop-versioning-max-outer-insns
> +The maximum number of instructions that an outer loop can have
> +before the loop versioning pass considers it too big to copy,
> +discounting any instructions in inner loops that directly benefit
> +from versioning.
>
> @end table
> @end table
> Index: gcc/Makefile.in
> ===================================================================
> --- gcc/Makefile.in 2018-10-24 14:02:14.000000000 +0100
> +++ gcc/Makefile.in 2018-10-24 14:02:15.180152727 +0100
> @@ -1312,6 +1312,7 @@ OBJS = \
> gimple-laddress.o \
> gimple-loop-interchange.o \
> gimple-loop-jam.o \
> + gimple-loop-versioning.o \
> gimple-low.o \
> gimple-pretty-print.o \
> gimple-ssa-backprop.o \
> Index: gcc/common.opt
> ===================================================================
> --- gcc/common.opt 2018-10-24 14:02:14.000000000 +0100
> +++ gcc/common.opt 2018-10-24 14:02:15.180152727 +0100
> @@ -2712,6 +2712,10 @@ fsplit-loops
> Common Report Var(flag_split_loops) Optimization
> Perform loop splitting.
>
> +fversion-loops-for-strides
> +Common Report Var(flag_version_loops_for_strides) Optimization
> +Version loops based on whether indices have a stride of 1.
stride of one.
> +
> funwind-tables
> Common Report Var(flag_unwind_tables) Optimization
> Just generate unwind tables for exception handling.
> Index: gcc/opts.c
> ===================================================================
> --- gcc/opts.c 2018-10-24 14:02:14.000000000 +0100
> +++ gcc/opts.c 2018-10-24 14:02:15.184152693 +0100
> @@ -544,6 +544,7 @@ static const struct default_options defa
> { OPT_LEVELS_3_PLUS, OPT_fipa_cp_clone, NULL, 1 },
> { OPT_LEVELS_3_PLUS, OPT_ftree_partial_pre, NULL, 1 },
> { OPT_LEVELS_3_PLUS, OPT_fpeel_loops, NULL, 1 },
> + { OPT_LEVELS_3_PLUS, OPT_fversion_loops_for_strides, NULL, 1 },
>
> /* -Ofast adds optimizations to -O3. */
> { OPT_LEVELS_FAST, OPT_ffast_math, NULL, 1 },
> Index: gcc/params.def
> ===================================================================
> --- gcc/params.def 2018-10-24 14:02:14.000000000 +0100
> +++ gcc/params.def 2018-10-24 14:02:15.184152693 +0100
> @@ -1360,6 +1360,25 @@ DEFPARAM(PARAM_AVOID_FMA_MAX_BITS,
> "Maximum number of bits for which we avoid creating FMAs.",
> 0, 0, 512)
>
> +DEFPARAM(PARAM_LOOP_VERSIONING_GROUP_SIZE,
> + "loop-versioning-group-size",
> + "The maximum constant N for which accesses of the form x[N * step]"
> + " are worth versioning for the case in which step is 1",
> + 4, 1, 0)
> +
> +DEFPARAM(PARAM_LOOP_VERSIONING_MAX_INNER_INSNS,
> + "loop-versioning-max-inner-insns",
> + "The maximum number of instructions in an inner loop that is being"
> + " considered for versioning",
> + 200, 0, 0)
> +
> +DEFPARAM(PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS,
> + "loop-versioning-max-outer-insns",
> + "The maximum number of instructions in an outer loop that is being"
> + " considered for versioning, on top of the instructions in inner"
> + " loops",
> + 100, 0, 0)
> +
> /*
>
> Local variables:
> Index: gcc/passes.def
> ===================================================================
> --- gcc/passes.def 2018-10-24 14:02:14.000000000 +0100
> +++ gcc/passes.def 2018-10-24 14:02:15.184152693 +0100
> @@ -260,6 +260,7 @@ along with GCC; see the file COPYING3.
> NEXT_PASS (pass_tree_loop);
> PUSH_INSERT_PASSES_WITHIN (pass_tree_loop)
> NEXT_PASS (pass_tree_loop_init);
> + NEXT_PASS (pass_loop_versioning);
I think neither of the following visible passes benefit from the
versioning but they might need to duplicate work (and code).
pass_loop_jam benefits because it needs to do dependence analysis.
Can you move the pass after loop splitting please?
> NEXT_PASS (pass_tree_unswitch);
> NEXT_PASS (pass_scev_cprop);
> NEXT_PASS (pass_loop_split);
> Index: gcc/timevar.def
> ===================================================================
> --- gcc/timevar.def 2018-10-24 14:02:14.000000000 +0100
> +++ gcc/timevar.def 2018-10-24 14:02:15.188152659 +0100
> @@ -234,6 +234,7 @@ DEFTIMEVAR (TV_DSE1 , "
> DEFTIMEVAR (TV_DSE2 , "dead store elim2")
> DEFTIMEVAR (TV_LOOP , "loop analysis")
> DEFTIMEVAR (TV_LOOP_INIT , "loop init")
> +DEFTIMEVAR (TV_LOOP_VERSIONING , "loop versioning")
> DEFTIMEVAR (TV_LOOP_MOVE_INVARIANTS , "loop invariant motion")
> DEFTIMEVAR (TV_LOOP_UNROLL , "loop unrolling")
> DEFTIMEVAR (TV_LOOP_DOLOOP , "loop doloop")
> Index: gcc/tree-ssa-propagate.h
> ===================================================================
> --- gcc/tree-ssa-propagate.h 2018-10-24 14:02:14.000000000 +0100
> +++ gcc/tree-ssa-propagate.h 2018-10-24 14:02:15.188152659 +0100
> @@ -104,7 +104,7 @@ extern void propagate_tree_value_into_st
> virtual bool fold_stmt (gimple_stmt_iterator *) { return false; }
> virtual tree get_value (tree) { return NULL_TREE; }
>
> - bool substitute_and_fold (void);
> + bool substitute_and_fold (basic_block = NULL);
I'm not sure I approve of your use of the SSA propagator but
at least dominance constrains the versioned loop body appropriately...
> bool replace_uses_in (gimple *);
> bool replace_phi_args_in (gphi *);
> };
> Index: gcc/tree-ssa-propagate.c
> ===================================================================
> --- gcc/tree-ssa-propagate.c 2018-10-24 14:02:14.000000000 +0100
> +++ gcc/tree-ssa-propagate.c 2018-10-24 14:02:15.188152659 +0100
> @@ -1152,6 +1152,10 @@ substitute_and_fold_dom_walker::before_d
>
>
> /* Perform final substitution and folding of propagated values.
> + Process the whole function if BLOCK is null, otherwise only
> + process the blocks that BLOCK dominates. In the latter case,
> + it is the caller's responsibility to ensure that dominator
> + information is available and up-to-date.
>
> PROP_VALUE[I] contains the single value that should be substituted
> at every use of SSA name N_I. If PROP_VALUE is NULL, no values are
> @@ -1168,16 +1172,24 @@ substitute_and_fold_dom_walker::before_d
> Return TRUE when something changed. */
>
> bool
> -substitute_and_fold_engine::substitute_and_fold (void)
> +substitute_and_fold_engine::substitute_and_fold (basic_block block)
> {
> if (dump_file && (dump_flags & TDF_DETAILS))
> fprintf (dump_file, "\nSubstituting values and folding statements\n\n");
>
> memset (&prop_stats, 0, sizeof (prop_stats));
>
> - calculate_dominance_info (CDI_DOMINATORS);
> + /* Don't call calculate_dominance_info when iterating over a subgraph.
> + Callers that are using the interface this way are likely to want to
> + iterate over several disjoint subgraphs, and it would be expensive
> + in enable-checking builds to revalidate the whole dominance tree
> + each time. */
> + if (block)
> + gcc_assert (dom_info_state (CDI_DOMINATORS));
> + else
> + calculate_dominance_info (CDI_DOMINATORS);
> substitute_and_fold_dom_walker walker (CDI_DOMINATORS, this);
> - walker.walk (ENTRY_BLOCK_PTR_FOR_FN (cfun));
> + walker.walk (block ? block : ENTRY_BLOCK_PTR_FOR_FN (cfun));
>
> /* We cannot remove stmts during the BB walk, especially not release
> SSA names there as that destroys the lattice of our callers.
> Index: gcc/tree-vrp.h
> ===================================================================
> --- gcc/tree-vrp.h 2018-10-24 14:02:14.000000000 +0100
> +++ gcc/tree-vrp.h 2018-10-24 14:02:15.188152659 +0100
> @@ -86,7 +86,7 @@ extern void register_edge_assert_for (tr
> tree, tree, vec<assert_info> &);
> extern bool stmt_interesting_for_vrp (gimple *);
> extern void set_value_range_to_varying (value_range *);
> -extern bool range_includes_zero_p (const value_range *);
> +extern bool range_includes_p (const value_range *, HOST_WIDE_INT);
> extern bool infer_value_range (gimple *, tree, tree_code *, tree *);
>
> extern void set_value_range_to_nonnull (value_range *, tree);
> @@ -122,4 +122,13 @@ extern int value_inside_range (tree, tre
> extern tree get_single_symbol (tree, bool *, tree *);
> extern void maybe_set_nonzero_bits (edge, tree);
> extern value_range_type determine_value_range (tree, wide_int *, wide_int *);
> +
> +/* Return TRUE if *VR includes the value zero. */
> +
> +inline bool
> +range_includes_zero_p (const value_range *vr)
> +{
> + return range_includes_p (vr, 0);
> +}
> +
> #endif /* GCC_TREE_VRP_H */
> Index: gcc/tree-vrp.c
> ===================================================================
> --- gcc/tree-vrp.c 2018-10-24 14:02:14.000000000 +0100
> +++ gcc/tree-vrp.c 2018-10-24 14:02:15.188152659 +0100
> @@ -844,10 +844,10 @@ value_inside_range (tree val, tree min,
> }
>
>
> -/* Return TRUE if *VR includes the value zero. */
> +/* Return TRUE if *VR includes the value X. */
>
> bool
> -range_includes_zero_p (const value_range *vr)
> +range_includes_p (const value_range *vr, HOST_WIDE_INT x)
> {
> if (vr->type == VR_VARYING)
> return true;
> @@ -856,13 +856,13 @@ range_includes_zero_p (const value_range
> if (vr->type == VR_UNDEFINED)
> return true;
>
> - tree zero = build_int_cst (TREE_TYPE (vr->min), 0);
> + tree x_cst = build_int_cst (TREE_TYPE (vr->min), x);
> if (vr->type == VR_ANTI_RANGE)
> {
> - int res = value_inside_range (zero, vr->min, vr->max);
> + int res = value_inside_range (x_cst, vr->min, vr->max);
> return res == 0 || res == -2;
> }
> - return value_inside_range (zero, vr->min, vr->max) != 0;
> + return value_inside_range (x_cst, vr->min, vr->max) != 0;
> }
>
> /* If *VR has a value rante that is a single constant value return that,
> Index: gcc/tree-pass.h
> ===================================================================
> --- gcc/tree-pass.h 2018-10-24 14:02:14.000000000 +0100
> +++ gcc/tree-pass.h 2018-10-24 14:02:15.188152659 +0100
> @@ -362,6 +362,7 @@ extern gimple_opt_pass *make_pass_fix_lo
> extern gimple_opt_pass *make_pass_tree_loop (gcc::context *ctxt);
> extern gimple_opt_pass *make_pass_tree_no_loop (gcc::context *ctxt);
> extern gimple_opt_pass *make_pass_tree_loop_init (gcc::context *ctxt);
> +extern gimple_opt_pass *make_pass_loop_versioning (gcc::context *ctxt);
> extern gimple_opt_pass *make_pass_lim (gcc::context *ctxt);
> extern gimple_opt_pass *make_pass_linterchange (gcc::context *ctxt);
> extern gimple_opt_pass *make_pass_tree_unswitch (gcc::context *ctxt);
> Index: gcc/gimple-loop-versioning.cc
> ===================================================================
> --- /dev/null 2018-09-14 11:16:31.122530289 +0100
> +++ gcc/gimple-loop-versioning.cc 2018-10-24 14:02:15.184152693 +0100
> @@ -0,0 +1,1417 @@
> +/* Loop versioning pass.
> + Copyright (C) 2018 Free Software Foundation, Inc.
> +
> +This file is part of GCC.
> +
> +GCC is free software; you can redistribute it and/or modify it
> +under the terms of the GNU General Public License as published by the
> +Free Software Foundation; either version 3, or (at your option) any
> +later version.
> +
> +GCC is distributed in the hope that it will be useful, but WITHOUT
> +ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
> +FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
> +for more details.
> +
> +You should have received a copy of the GNU General Public License
> +along with GCC; see the file COPYING3. If not see
> +<http://www.gnu.org/licenses/>. */
> +
> +#include "config.h"
> +#include "system.h"
> +#include "coretypes.h"
> +#include "backend.h"
> +#include "tree.h"
> +#include "gimple.h"
> +#include "gimple-iterator.h"
> +#include "tree-pass.h"
> +#include "gimplify-me.h"
> +#include "cfgloop.h"
> +#include "tree-ssa-loop.h"
> +#include "ssa.h"
> +#include "tree-scalar-evolution.h"
> +#include "tree-chrec.h"
> +#include "tree-ssa-loop-ivopts.h"
> +#include "fold-const.h"
> +#include "tree-ssa-propagate.h"
> +#include "tree-inline.h"
> +#include "domwalk.h"
> +#include "alloc-pool.h"
> +#include "vr-values.h"
> +#include "gimple-ssa-evrp-analyze.h"
> +#include "gimple-pretty-print.h"
> +#include "params.h"
> +
> +/* This pass looks for loops that could be simplified if certain loop
> + invariant conditions were true. It is effectively a form of loop
> + splitting in which the pass produces the split conditions itself,
> + instead of using ones that are already present in the IL.
> +
> + Versioning for when strides are 1
> + ---------------------------------
> +
> + At the moment the only thing the pass looks for are memory references
> + like:
> +
> + for (auto i : ...)
> + ...x[i * stride]...
> +
> + It considers changing such loops to:
> +
> + if (stride == 1)
> + for (auto i : ...) [A]
> + ...x[i]...
> + else
> + for (auto i : ...) [B]
> + ...x[i * stride]...
> +
> + This can have several benefits:
> +
> + (1) [A] is often easier or cheaper to vectorize than [B].
> +
> + (2) The scalar code in [A] is simpler than the scalar code in [B]
> + (if the loops cannot be vectorized or need an epilogue loop).
> +
> + (3) We might recognize [A] as a pattern, such as a memcpy or memset.
> +
> + (4) [A] has simpler address evolutions, which can help other passes
> + like loop interchange.
> +
> + The optimization is particularly useful for assumed-shape arrays in
> + Fortran, where the stride of the innermost dimension depends on the
> + array descriptor but is often equal to 1 in practice. For example:
> +
> + subroutine f1(x)
> + real :: x(:)
> + x(:) = 100
> + end subroutine f1
> +
> + generates the equivalent of:
> +
> + raw_stride = *x.dim[0].stride;
> + stride = raw_stride != 0 ? raw_stride : 1;
> + x_base = *x.data;
> + ...
> + tmp1 = stride * S;
> + tmp2 = tmp1 - stride;
> + *x_base[tmp2] = 1.0e+2;
> +
> + but in the common case that stride == 1, the last three statements
> + simplify to:
> +
> + tmp3 = S + -1;
> + *x_base[tmp3] = 1.0e+2;
> +
> + The optimization is in principle very simple. The difficult parts are:
> +
> + (a) deciding which parts of a general address calculation correspond
> + to the inner dimension of an array, since this usually isn't explicit
> + in the IL, and for C often isn't even explicit in the source code
> +
> + (b) estimating when the transformation is worthwhile
> +
> + Structure
> + ---------
> +
> + The pass has four phases:
> +
> + (1) Walk through the statements looking for and recording potential
> + versioning opportunities. Stop if there are none.
> +
> + (2) Use context-sensitive range information to see whether any versioning
> + conditions are impossible in practice. Remove them if so, and stop
> + if no opportunities remain.
> +
> + (We do this only after (1) to keep compile time down when no
> + versioning opportunities exist.)
> +
> + (3) Apply the cost model. Decide which versioning opportunities are
> + worthwhile and at which nesting level they should be applied.
> +
> + (4) Attempt to version all the loops selected by (3), so that:
> +
> + for (...)
> + ...
> +
> + becomes:
> +
> + if (!cond)
> + for (...) // Original loop
> + ...
> + else
> + for (...) // New loop
> + ...
> +
> + Use the version condition COND to simplify the new loop. */
> +class loop_versioning {
> +public:
> + loop_versioning (function *);
> + ~loop_versioning ();
> + unsigned int run ();
> +
> +private:
> + /* Information about the versioning we'd like to apply to a loop. */
> + struct loop_info {
> + bool worth_versioning_p () const;
> +
> + /* True if we've decided not to version this loop. The remaining
> + fields are meaningless if so. */
> + bool rejected_p;
> +
> + /* True if at least one subloop of this loop benefits from versioning. */
> + bool subloops_benefit_p;
> +
> + /* An estimate of the total number of instructions in the loop,
> + excluding those in subloops that benefit from versioning. */
> + unsigned int num_insns;
> +
> + /* The outermost loop that can handle all the version checks
> + described below. */
> + struct loop *outermost;
> +
> + /* We'd like to version the loop for the case in which these
> + SSA_NAMEs are all equal to 1 at runtime. */
> + vec<tree> unity_names;
> +
> + /* The set of SSA_NAMEs in UNITY_NAMES, keyed off the SSA_NAME_VERSION. */
> + bitmap_head unity_name_ids;
I wonder why you need both, a vector and a bitmap since you can
cheaply iterate over the bitmap and get the SSA name via ssa_name (version)?
> + };
> +
> + /* Used to walk the dominator tree to find loop versioning conditions
> + that are always false. */
> + class lv_dom_walker : public dom_walker
> + {
> + public:
> + lv_dom_walker (loop_versioning &);
> +
> + edge before_dom_children (basic_block) FINAL OVERRIDE;
> + void after_dom_children (basic_block) FINAL OVERRIDE;
> +
> + private:
> + /* The parent pass. */
> + loop_versioning &m_lv;
> +
> + /* Used to build context-dependent range information. */
> + evrp_range_analyzer m_range_analyzer;
> + };
> +
> + /* Used to simplify statements based on conditions that are established
> + by the version checks. */
> + class name_prop : public substitute_and_fold_engine
> + {
> + public:
> + name_prop (loop_info &li) : m_li (li) {}
> + tree get_value (tree) FINAL OVERRIDE;
> +
> + private:
> + /* Information about the versioning we've performed on the loop. */
> + loop_info &m_li;
> + };
> +
> + loop_info &get_loop_info (struct loop *loop) { return m_loops[loop->num]; }
> +
> + unsigned int max_insns_for_loop (struct loop *);
> + bool expensive_stmt_p (gimple *);
> +
> + void version_for_unity (struct loop *, tree);
> + bool acceptable_scale_p (tree, poly_uint64);
> + tree get_step_if_innermost (tree, poly_uint64);
> + tree extract_step (tree, poly_uint64, tree *);
> + void analyze_evolution (struct loop *, tree, poly_uint64);
> + bool analyze_product (struct loop *, gassign *, poly_uint64);
> + bool analyze_sum_of_products (struct loop *, tree, poly_uint64);
> + void analyze_pointer (struct loop *, tree, tree);
> + void analyze_expr (struct loop *, tree);
> + void analyze_stmt (gimple *);
> + void analyze_block (basic_block);
> + bool analyze_blocks ();
> +
> + void prune_loop_conditions (struct loop *, vr_values *);
> + bool prune_conditions ();
> +
> + void merge_loop_info (struct loop *, struct loop *);
> + void add_loop_to_queue (struct loop *);
> + bool decide_whether_loop_is_versionable (struct loop *);
> + bool make_versioning_decisions ();
> +
> + bool version_loop (struct loop *);
> + bool implement_versioning_decisions ();
> +
> + /* The function we're optimizing. */
> + function *m_fn;
> +
> + /* The obstack to use for all pass-specific bitmaps. */
> + bitmap_obstack m_obstack;
> +
> + /* The number of loops in the function. */
> + unsigned int m_nloops;
> +
> + /* The total number of loop version conditions we've found. */
> + unsigned int m_num_conditions;
> +
> + /* Information about each loop. */
> + auto_vec<loop_info> m_loops;
> +
> + /* The list of loops that we've decided to version. */
> + auto_vec<struct loop *> m_loops_to_version;
> +};
> +
> +/* If EXPR is an SSA name and not a default definition, return the
> + defining statement, otherwise return null. */
> +
> +static gimple *
> +maybe_get_stmt (tree expr)
> +{
> + if (TREE_CODE (expr) == SSA_NAME && !SSA_NAME_IS_DEFAULT_DEF (expr))
> + return SSA_NAME_DEF_STMT (expr);
> + return NULL;
> +}
> +
> +/* Like maybe_get_stmt, but also return null if the defining
> + statement isn't an assignment. */
> +
> +static gassign *
> +maybe_get_assign (tree expr)
> +{
> + return safe_dyn_cast <gassign *> (maybe_get_stmt (expr));
> +}
> +
> +/* If EXPR is an SSA name, look through any casts to see whether the
> + unconverted value is defined in LOOP by a gassign. Return the
> + gassign if so, otherwise return null. */
> +
> +gassign *
> +maybe_get_assign_strip_casts (struct loop *loop, tree expr)
> +{
> + const unsigned int MAX_NITERS = 4;
> +
> + tree type = TREE_TYPE (expr);
> + for (unsigned int niters = 0; niters < MAX_NITERS; ++niters)
> + {
> + gassign *assign = maybe_get_assign (expr);
> + if (!assign || gimple_bb (assign)->loop_father != loop)
> + return NULL;
> + expr = gimple_assign_rhs1 (assign);
> + if (CONVERT_EXPR_CODE_P (gimple_assign_rhs_code (assign))
> + && INTEGRAL_TYPE_P (TREE_TYPE (expr)) == INTEGRAL_TYPE_P (type)
> + && POINTER_TYPE_P (TREE_TYPE (expr)) == POINTER_TYPE_P (type))
> + ;
> + else
> + return assign;
> + }
> + return NULL;
> +}
> +
> +/* Strip all conversions of integers from EXPR, regardless of whether
> + the conversions are nops. This is useful in the context of this pass
> + because we're not trying to fold or simulate the expression; we just
> + want to see how it's structured. */
> +
> +static tree
> +strip_casts (tree expr)
> +{
> + while (CONVERT_EXPR_P (expr)
> + && INTEGRAL_TYPE_P (TREE_TYPE (TREE_OPERAND (expr, 0))))
> + expr = TREE_OPERAND (expr, 0);
> + return expr;
> +}
> +
> +/* Return true if we want to version the loop, i.e. if we have a
> + specific reason for doing so and no specific reason not to. */
> +
> +bool
> +loop_versioning::loop_info::worth_versioning_p () const
> +{
> + return !rejected_p && (!unity_names.is_empty () || subloops_benefit_p);
> +}
> +
> +loop_versioning::lv_dom_walker::lv_dom_walker (loop_versioning &lv)
> + : dom_walker (CDI_DOMINATORS), m_lv (lv)
> +{
> +}
> +
> +/* Process BB before processing the blocks it dominates. */
> +
> +edge
> +loop_versioning::lv_dom_walker::before_dom_children (basic_block bb)
> +{
> + m_range_analyzer.enter (bb);
> +
> + if (bb == bb->loop_father->header)
> + m_lv.prune_loop_conditions (bb->loop_father,
> + m_range_analyzer.get_vr_values ());
> +
> + for (gimple_stmt_iterator si = gsi_start_bb (bb); !gsi_end_p (si);
> + gsi_next (&si))
> + m_range_analyzer.record_ranges_from_stmt (gsi_stmt (si), false);
> +
> + return NULL;
> +}
> +
> +/* Process BB after processing the blocks it dominates. */
> +
> +void
> +loop_versioning::lv_dom_walker::after_dom_children (basic_block bb)
> +{
> + m_range_analyzer.leave (bb);
> +}
> +
> +/* Decide whether to replace VAL with a new value in a versioned loop.
> + Return the new value if so, otherwise return null. */
> +
> +tree
> +loop_versioning::name_prop::get_value (tree val)
> +{
> + if (TREE_CODE (val) == SSA_NAME
> + && bitmap_bit_p (&m_li.unity_name_ids, SSA_NAME_VERSION (val)))
> + return build_one_cst (TREE_TYPE (val));
> + return NULL_TREE;
> +}
> +
> +/* Initialize the structure to optimize FN. */
> +
> +loop_versioning::loop_versioning (function *fn)
> + : m_fn (fn),
> + m_nloops (number_of_loops (fn)),
> + m_num_conditions (0)
> +{
> + bitmap_obstack_initialize (&m_obstack);
> +
> + m_loops.safe_grow_cleared (m_nloops);
> + for (unsigned int i = 0; i < m_nloops; ++i)
> + {
> + m_loops[i].outermost = get_loop (m_fn, 0);
> + bitmap_initialize (&m_loops[i].unity_name_ids, &m_obstack);
> + }
> +}
> +
> +loop_versioning::~loop_versioning ()
> +{
> + for (unsigned int i = 0; i < m_nloops; ++i)
> + m_loops[i].unity_names.release ();
> + bitmap_obstack_release (&m_obstack);
> +}
> +
> +/* Return the maximum number of instructions allowed in LOOP before
> + it becomes too big for versioning.
> +
> + There are separate limits for inner and outer loops. The limit for
> + inner loops applies only to loops that benefit directly from versioning.
> + The limit for outer loops applies to all code in the outer loop and
> + its subloops that *doesn't* benefit directly from versioning; such code
> + would be "taken along for the ride". The idea is that if the cost of
> + the latter is small, it is better to version outer loops rather than
> + inner loops, both to reduce the number of repeated checks and to enable
> + more of the loop nest to be optimized as a natural nest (e.g. by loop
> + interchange or outer-loop vectorization). */
> +
> +unsigned int
> +loop_versioning::max_insns_for_loop (struct loop *loop)
> +{
> + return (loop->inner
> + ? PARAM_VALUE (PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS)
> + : PARAM_VALUE (PARAM_LOOP_VERSIONING_MAX_INNER_INSNS));
> +}
> +
> +/* Return true if for cost reasons we should avoid versioning any loop
> + that contains STMT.
> +
> + Note that we don't need to check whether versioning is invalid for
> + correctness reasons, since the versioning process does that for us.
> + The conditions involved are too rare to be worth duplicating here. */
> +
> +bool
> +loop_versioning::expensive_stmt_p (gimple *stmt)
> +{
> + if (gcall *call = dyn_cast <gcall *> (stmt))
> + /* Assume for now that the time spent in an "expensive" call would
> + overwhelm any saving from versioning. */
> + return !gimple_inexpensive_call_p (call);
> + return false;
> +}
> +
> +/* Record that we want to version LOOP for the case in which SSA name NAME
> + is equal to 1. We already know that NAME is invariant in LOOP. */
> +
> +void
> +loop_versioning::version_for_unity (struct loop *loop, tree name)
> +{
> + loop_info &li = get_loop_info (loop);
> +
> + if (bitmap_set_bit (&li.unity_name_ids, SSA_NAME_VERSION (name)))
> + {
> + /* This is the first time we've wanted to version LOOP for NAME. */
> + li.unity_names.safe_push (name);
> +
> + /* Keep track of the outermost loop that can handle all versioning
> + checks in LI. */
> + struct loop *outermost
> + = outermost_invariant_loop_for_expr (loop, name);
> + if (loop_depth (li.outermost) < loop_depth (outermost))
> + li.outermost = outermost;
> +
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + {
> + fprintf (dump_file, ";; Want to version loop %d (depth %d)"
> + " for when ", loop->num, loop_depth (loop));
> + print_generic_expr (dump_file, name, TDF_SLIM);
> + fprintf (dump_file, " == 1");
Since you are writing a new pass you want to use the new dump interface.
if (dump_enabled_p ())
dump_printf (MSG_NOTE, ";; Want to version loop %d (depth %d)"
" for when %E == 1", loop->num, loop_depth (loop), name);
...
it's much nicer to be able to use %E/%G than separate calls for the
tree parts.
> + if (outermost == loop)
> + fprintf (dump_file, "; cannot hoist check further");
> + else
> + {
> + fprintf (dump_file, "; could hoist check to loop %d (depth %d)",
> + outermost->num, loop_depth (outermost));
> + if (loop_depth (li.outermost) > loop_depth (outermost))
> + fprintf (dump_file, ", but that's further than"
> + " other checks allow");
> + }
> + fprintf (dump_file, "\n");
> + }
> +
> + m_num_conditions += 1;
> + }
> + else
> + {
> + /* This is a duplicate request. */
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + {
> + fprintf (dump_file, ";; Already want to version loop for when ");
> + print_generic_expr (dump_file, name, TDF_SLIM);
> + fprintf (dump_file, " == 1\n");
> + }
> + }
> +}
> +
> +/* Return true if in principle it is worth versioning an index fragment of
> + the form:
> +
> + (i * b * SCALE) / FACTOR
> +
> + for the case in which b == 1. */
> +
> +bool
> +loop_versioning::acceptable_scale_p (tree scale, poly_uint64 factor)
> +{
> + /* See whether SCALE is a constant multiple of FACTOR, and if the
> + multiple is small enough for us to treat it as a potential grouped
> + access. For example:
> +
> + for (auto i : ...)
> + y[i] = f (x[4 * i * stride],
> + x[4 * i * stride + 1],
> + x[4 * i * stride + 2]);
> +
> + would benefit from versioning for the case in which stride == 1.
> + High multiples of i * stride are less likely to benefit, and could
> + indicate a simulated multi-dimensional array.
> +
> + This is just a heuristic, to avoid having to do expensive group
> + analysis of the data references in a loop. */
> + poly_uint64 const_scale;
> + unsigned int multiple;
> + if (poly_int_tree_p (scale, &const_scale)
> + && constant_multiple_p (const_scale, factor, &multiple))
> + {
> + unsigned int maxval = PARAM_VALUE (PARAM_LOOP_VERSIONING_GROUP_SIZE);
> + return IN_RANGE (multiple, 1, maxval);
Hmm. So you _do_ want to version sth like
struct X { int i; int j; } a[2048];
for (int i = start; i < end; ++i)
a[i*s].i = 1;
? That is, even with s == 1 the accesses will not become contiguous?
OK, I suppose that's because you are looking at a stmt in isolation
and another stmt may access a[i*s].j here.
That is, would it be a future improvement to run sth like the
vectorizers group access analysis on the references and perform
this check on whole groups then, possibly better being able to
constrain what is now the magic parameter PARAM_LOOP_VERSIONING_GROUP_SIZE?
I guess you tried to constrain it to the stmts access size but of course
that fails short of handling later SLP vectorized cases.
> + }
> +
> + return false;
> +}
> +
> +/* Decide whether an index fragment of the form:
> +
> + (i * STEP) / FACTOR
> +
> + is likely to be for an innermost dimension. If we think it is,
> + return one of the constant values that it could have (returning 1 if
> + that's a possibility). If think it isn't, return null. Otherwise
> + return STEP, to indicate that it might or might not be an inner
> + dimension. */
> +
> +tree
> +loop_versioning::get_step_if_innermost (tree step, poly_uint64 factor)
> +{
> + const unsigned int MAX_NITERS = 8;
> +
> + tree likely = NULL_TREE;
> + tree unlikely = NULL_TREE;
> + tree worklist[MAX_NITERS];
> + unsigned int length = 0;
> + worklist[length++] = step;
> + for (unsigned int i = 0; i < length; ++i)
> + {
> + tree expr = worklist[i];
> +
> + if (TREE_CONSTANT (expr))
? CONSTANT_CLASS_P (expr)?
> + {
> + /* See if multiplying by EXPR applies a scale that would be
> + consistent with an individual access or a small grouped
> + access. */
> + if (acceptable_scale_p (expr, factor))
> + {
> + likely = expr;
> + if (integer_onep (expr))
> + break;
> + }
> + else
> + unlikely = expr;
> + continue;
> + }
> +
> + /* Otherwise we can only handle SSA names. */
> + gimple *stmt = maybe_get_stmt (expr);
> + if (!stmt)
> + continue;
> +
> + /* If EXPR is set by a PHI node, queue its arguments in case
> + we find one that is consistent with an inner dimension.
> +
> + An important instance of this is the Fortran handling of array
> + descriptors, which calculates the stride of the inner dimension
> + using a PHI equivalent of:
> +
> + raw_stride = a.dim[0].stride;
> + stride = raw_stride != 0 ? raw_stride : 1;
> +
> + (Strides for outer dimensions do not treat 0 specially.) */
> + if (gphi *phi = dyn_cast <gphi *> (stmt))
> + {
> + unsigned int nargs = gimple_phi_num_args (phi);
> + for (unsigned int j = 0; j < nargs && length < MAX_NITERS; ++j)
> + worklist[length++] = gimple_phi_arg_def (phi, j);
> + continue;
> + }
So only PHIs seed the worklist.
> + /* If the value is set by an assignment, expect it to be read from
> + memory (such as an array descriptor) rather than be calculated. */
> + if (gassign *assign = dyn_cast <gassign *> (stmt))
> + {
> + if (gimple_assign_single_p (assign)
> + && is_gimple_lvalue (gimple_assign_rhs1 (assign)))
Please do not use is_gimple_lvalue, that's supposed to be a gimplifier-only
predicate. Do you want to use gimple_assign_load_p here?
> + continue;
> +
> + unlikely = expr;
> + }
> +
> + /* Things like calls don't really tell us anything. */
> + }
So I don't understand the above fully but it looks like plain
function arguments STEP will never be 'likely'?
That is, I'd appreciate some more comments of what SSA def
chains we walk and what we look for in the end.
> + if (likely)
> + {
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + fprintf (dump_file, "likely to be innermost dimension\n");
> + return likely;
> + }
> +
> + if (unlikely)
> + {
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + fprintf (dump_file, "probably not innermost dimension\n");
> + return NULL_TREE;
> + }
> +
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + {
> + if (maybe_ne (factor, 1U))
> + fprintf (dump_file, "didn't find expected scaling factor\n");
> + else
> + fprintf (dump_file, "no information about value\n");
> + }
> + return step;
> +}
> +
> +/* STEP appears in an index fragment of the form:
> +
> + {..., +, STEP}_n / FACTOR
> +
> + Remove any conversions and grouping or scaling factors from STEP and
> + return the underlying index step. Set *STEP_IF_INNERMOST to the value
> + returned by get_step_if_innermost. */
> +
> +tree
> +loop_versioning::extract_step (tree step, poly_uint64 factor,
> + tree *step_if_innermost)
> +{
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + {
> + fprintf (dump_file, ";; step ");
> + print_generic_expr (dump_file, step, TDF_SLIM);
> + fprintf (dump_file, ": ");
> + }
> +
> + step = strip_casts (step);
> +
> + /* Peel any scaling, which generally happens after conversion to
> + pointer width. For example, on LP64 systems:
> +
> + int *x, i, stride;
> + ... x[4 * i * stride] ...;
> +
> + multiplies i * stride by 4 using ints, then widens the result
> + to pointer width before multiplying by sizeof (int). */
> + poly_uint64 scale;
> + if (TREE_CODE (step) == MULT_EXPR
> + && poly_int_tree_p (TREE_OPERAND (step, 1), &scale)
> + && known_eq (scale, factor))
> + {
> + step = strip_casts (TREE_OPERAND (step, 0));
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + fprintf (dump_file, "scaled, ");
> + factor = 1;
> + }
So I think the extra constant scale may or may not be in CHREC_RIGHT
depending on whether the expression was hoisted out of the loop as
invariant or implicit in sth like an ARRAY_REF.
> + /* Peel any grouping multiple; see acceptable_scale_p for details. */
> + if (TREE_CODE (step) == MULT_EXPR
> + && acceptable_scale_p (TREE_OPERAND (step, 1), factor))
> + {
> + step = strip_casts (TREE_OPERAND (step, 0));
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + fprintf (dump_file, "%sgrouped, ",
> + maybe_ne (factor, 1U) ? "scaled, " : "");
> + factor = 1;
> + }
Which means I'm not sure why there are (only) two MULTs being
looked for and only one has acceptable_scale_p applied...
> + *step_if_innermost = get_step_if_innermost (step, factor);
I know some people like to factor things but get_step_if_innermost
is called once if I see correctly - it makes understanding and
review harder to dissect things :/
> + return step;
> +}
> +
> +/* Analyze the evolution of index fragment EXPR / FACTOR in LOOP and its
> + containing loops to see whether any part of it could be simplified
> + by versioning. Register the versioning opportunities if so. */
> +
> +void
> +loop_versioning::analyze_evolution (struct loop *loop, tree expr,
> + poly_uint64 factor)
I find the name of this function confusing given it's nearly
the same as analyze_scalar_evolution.
> +{
> + const unsigned int MAX_NSPLIT = 8;
> +
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + {
> + fprintf (dump_file, ";; Analyzing use of ");
> + print_generic_expr (dump_file, expr, TDF_SLIM);
> + if (maybe_ne (factor, 1U))
> + {
> + fprintf (dump_file, " (which addresses ");
> + print_dec (factor, dump_file);
> + fprintf (dump_file, " bytes)");
> + }
> + fprintf (dump_file, " in loop %d (depth %d)\n",
> + loop->num, loop_depth (loop));
> + }
> +
> + /* The main problem we have here is that we cannot assume that the
> + innermost loop iterates over the innermost dimension of an array.
> + Accidentally adding versioning checks for outer dimensions would
> + cause the version condition to be false, which as well as bloating
> + the code would defeat loop versioning benefits for other accesses.
> +
> + Unfortunately all we usually see at this stage is general address
> + arithmetic, with no positive way of identifying how many dimensions
> + an array access has and which multiplication factors in the address
> + expression correspond to which array dimensions. In C code this is
> + often not even explicit in the source, since variable-sized multi-
> + dimensional arrays are often simulated using one-dimensional arrays.
> +
> + The three main ways in which we deal with this are:
> +
> + - use heuristics that positively identify steps that are likely
> + to represent the inner dimension.
> +
> + - use heuristics that positively identify steps that are unlikely
> + to represent the inner dimension.
> +
> + - if a part of EXPR is invariant in LOOP, analyze its evolution in
> + the outer loops to see whether we can positively identify any of
> + it as iterating over the inner dimension. */
> + tree best_step = NULL_TREE;
> + auto_vec<tree, MAX_NSPLIT> worklist;
> + worklist.quick_push (expr);
> + unsigned int nsplit = 0;
> + while (!worklist.is_empty ())
> + {
> + expr = strip_casts (worklist.pop ());
> + tree_code code = TREE_CODE (expr);
> +
> + if (code == POLYNOMIAL_CHREC)
> + {
> + /* Analyze the CHREC_RIGHT as the step in an index fragment. */
> + tree step_if_innermost;
> + tree step = extract_step (CHREC_RIGHT (expr), factor,
> + &step_if_innermost);
> + if (!best_step)
> + {
> + /* This is the outermost chrec for the original expression.
> + It's not worth carrying on if the step isn't versionable,
> + or if we're pretty sure it's not for the inner dimension. */
> + if (!step_if_innermost
> + || TREE_CODE (step) != SSA_NAME
> + || !expr_invariant_in_loop_p (loop, step))
> + return;
> +
> + best_step = step;
> +
> + /* We should version for STEP == 1 if we know that that can be
> + true under some circumstances. */
> + if (integer_onep (step_if_innermost))
> + break;
> +
> + /* Bail out if this appears to be the step for the innermost
> + dimension, but isn't likely to be 1.
> +
> + ??? We could instead version for when it equals
> + STEP_IF_INNERMOST, but it's not likely to have as much
> + benefit as versioning for 1. */
> + if (step_if_innermost != step)
> + return;
> + }
> + else
> + {
> + /* This is an inner chrec. If it looks like it iterates over
> + the innermost dimension, abort any attempt to version for
> + the outermost chrec (which if we reach here wasn't itself
> + obviously iterating over the innermost dimension). */
> + if (step_if_innermost && TREE_CONSTANT (step_if_innermost))
> + return;
> + }
> + worklist.quick_push (CHREC_LEFT (expr));
> + continue;
> + }
> +
> + /* Analyze parts of a queued CHREC_LEFT. This is better than just
> + analyzing the evolution of the whole expression since the value
> + could include a mixture of analyzable and unanalyzable elements.
> + Use NSPLIT to count cases in which we add more expressions to
> + analyze, as opposed to just simplifying the existing one. */
> + if (code == PLUS_EXPR || code == POINTER_PLUS_EXPR || code == MINUS_EXPR)
> + {
> + worklist.quick_push (TREE_OPERAND (expr, 0));
> + if (nsplit++ < MAX_NSPLIT)
> + worklist.quick_push (TREE_OPERAND (expr, 1));
> + continue;
> + }
> + if (code == MULT_EXPR)
> + {
> + tree op0 = strip_casts (TREE_OPERAND (expr, 0));
> + tree op1 = TREE_OPERAND (expr, 1);
> + if (TREE_CODE (op0) == PLUS_EXPR || TREE_CODE (op0) == MINUS_EXPR)
> + {
> + tree type = TREE_TYPE (expr);
> + tree op00 = fold_convert (type, TREE_OPERAND (op0, 0));
> + worklist.quick_push (fold_build2 (MULT_EXPR, type, op00, op1));
> + if (nsplit++ < MAX_NSPLIT)
> + {
> + tree op01 = fold_convert (type, TREE_OPERAND (op0, 1));
> + worklist.quick_push (fold_build2 (MULT_EXPR, type,
> + op01, op1));
> + }
> + continue;
> + }
> + }
> +
> + /* If EXPR is invariant in LOOP, analyze it wrt the innermost loop
> + for which it could evolve (i.e. the loop containing the outermost
> + one for which EXPR is invariant). */
This isn't how analyze_scalar_evolution works - you _always_ have to
feed the innermost loop that the expression is used in (the context).
Then you instantiate the result in the outermost loop of the nest you
are interested in. Otherwise you get garbage.
It looks like you are re-analyzing SSA names in the evolution - that's
odd and shouldn't be necessary (but you forget to instantiate, so...).
I suggest to move the analyze_scalar_evolution part out of the worklist
loop where you are sure you have an SSA name.
> + struct loop *wrt_loop = outermost_invariant_loop_for_expr (loop, expr);
> + if (wrt_loop)
> + {
> + wrt_loop = loop_outer (wrt_loop);
> + if (!wrt_loop)
> + continue;
> + }
> + else
> + wrt_loop = loop;
> + tree evolution = analyze_scalar_evolution (wrt_loop, expr);
> + tree chrec = strip_casts (evolution);
> + if (TREE_CODE (chrec) == POLYNOMIAL_CHREC)
> + {
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + {
> + fprintf (dump_file, ";; evolution of ");
> + print_generic_expr (dump_file, expr, TDF_SLIM);
> + fprintf (dump_file, " in loop %d (depth %d): ",
> + wrt_loop->num, loop_depth (wrt_loop));
> + print_generic_expr (dump_file, evolution, TDF_SLIM);
> + fprintf (dump_file, "\n");
> + }
> + worklist.quick_push (chrec);
> + }
> + else
> + {
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + {
> + fprintf (dump_file, ";; cannot analyze ");
> + print_generic_expr (dump_file, expr, TDF_SLIM);
> + fprintf (dump_file, " any further\n");
> + }
> + }
> + }
> + if (best_step)
> + version_for_unity (loop, best_step);
> +}
> +
> +/* Analyze multiplication MULT to see whether we can identify "gather-like"
> + versioning opportunities such as:
> +
> + for (int i = 0; i < n; ++i)
> + res += a[index[i] * stride];
> +
> + Return true if there was a versioning opportunity.
> +
> + LOOP is the loop that contains MULT. Dividing the value of MULT
> + by FACTOR converts it into an element index. */
> +
> +bool
> +loop_versioning::analyze_product (struct loop *loop, gassign *mult,
> + poly_uint64 factor)
> +{
> + /* Record the original LHS for the dump message below. */
> + tree lhs = gimple_assign_lhs (mult);
> +
> + /* Peel any scaling, which generally happens after conversion to
> + pointer width. For example, on LP64 systems:
> +
> + int *x, i, stride;
> + ... x[4 * i * stride] ...;
> +
> + multiplies i * stride by 4 using ints, then widens the result
> + to pointer width before multiplying by sizeof (int). */
> + poly_uint64 scale;
> + if (poly_int_tree_p (gimple_assign_rhs2 (mult), &scale)
> + && known_eq (scale, factor))
> + {
> + mult = maybe_get_assign_strip_casts (loop, gimple_assign_rhs1 (mult));
> + if (!mult || gimple_assign_rhs_code (mult) != MULT_EXPR)
> + return false;
> + factor = 1;
> + }
> +
> + /* Peel any grouping multiple; see acceptable_scale_p for details. */
> + if (acceptable_scale_p (gimple_assign_rhs2 (mult), factor))
> + {
> + mult = maybe_get_assign_strip_casts (loop, gimple_assign_rhs1 (mult));
> + if (!mult || gimple_assign_rhs_code (mult) != MULT_EXPR)
> + return false;
> + }
> + else if (maybe_ne (factor, 1U))
> + /* We expect to see a scaling multiplication when FACTOR is > 1. */
> + return false;
> +
> + /* See whether this multiplication involves a loop-invariant SSA name
> + and a non-invariant SSA name. */
> + tree op1 = gimple_assign_rhs1 (mult);
> + tree op2 = gimple_assign_rhs2 (mult);
> + if (TREE_CODE (op1) != SSA_NAME || TREE_CODE (op2) != SSA_NAME)
> + return false;
> +
> + bool invariant1_p = expr_invariant_in_loop_p (loop, op1);
> + bool invariant2_p = expr_invariant_in_loop_p (loop, op2);
> + if (invariant1_p == invariant2_p)
> + return false;
> +
> + /* Make sure that the invariant is OP1 and the other operand is OP2. */
> + if (invariant2_p)
> + std::swap (op1, op2);
> +
> + /* Bail out if OP2 can be analyzed as an evolution; we want to use
> + analyze_evolution in that case instead. There's no point trying
> + hard to avoid repeating the call to analyze_scalar_evolution since
> + that function does its own caching. */
> + if (tree_contains_chrecs (analyze_scalar_evolution (loop, op2), NULL))
Don't you want !chrec_contains_undetermined (...) instead? I wonder what
is missing to allow all interesting expressions to be analyzed by
SCEV?
> + return false;
> +
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + {
> + fprintf (dump_file, ";; Address fragment ");
> + print_generic_expr (dump_file, lhs, TDF_SLIM);
> + fprintf (dump_file, " multiplies invariant ");
> + print_generic_expr (dump_file, op1, TDF_SLIM);
> + fprintf (dump_file, " by ");
> + print_generic_expr (dump_file, op2, TDF_SLIM);
> + fprintf (dump_file, ", which isn't a scalar evolution\n");
> + }
> +
> + version_for_unity (loop, op1);
> + return true;
> +}
> +
> +/* Treat EXPR as a sum of products and apply analyze_product to each of the
> + products. Return true if one of the products provides a versioning
> + opportunity. FACTOR is as for analyze_product. */
> +
> +bool
> +loop_versioning::analyze_sum_of_products (struct loop *loop, tree expr,
> + poly_uint64 factor)
> +{
This looks awfully close to what tree-affine.c does apart from more
aggressively stripping conversions? I see all the analysis parts
are limited and thus "O(1)" but still there's going to be a lot of
redundancy involved for repeated use of (derived) "IVs"? Wouldn't
it be good to have a bitmap of already handled SSA_NAMEs to stop
processing early?
> + const unsigned int MAX_NITERS = 8;
> +
> + tree worklist[MAX_NITERS];
> + unsigned int length = 0;
> + worklist[length++] = expr;
> + for (unsigned int i = 0; i < length; ++i)
> + {
> + expr = worklist[i];
> + gassign *assign = maybe_get_assign_strip_casts (loop, expr);
> + if (!assign)
> + continue;
> +
> + tree_code code = gimple_assign_rhs_code (assign);
> + if (code == PLUS_EXPR || code == POINTER_PLUS_EXPR || code == MINUS_EXPR)
POINTER_MINUS_EXPR?
> + {
> + if (length < MAX_NITERS)
> + worklist[length++] = gimple_assign_rhs1 (assign);
> + if (length < MAX_NITERS)
> + worklist[length++] = gimple_assign_rhs2 (assign);
> + }
> + else if (code == MULT_EXPR && analyze_product (loop, assign, factor))
> + return true;
> + }
> + return false;
> +}
> +
> +/* Analyze pointer expression EXPR, which occurs in loop LOOP and which
> + is used to address a value of type TYPE. */
> +
> +void
> +loop_versioning::analyze_pointer (struct loop *loop, tree expr, tree type)
> +{
Inline into single caller
> + poly_uint64 factor;
> + if (poly_int_tree_p (TYPE_SIZE_UNIT (type), &factor))
> + {
> + if (!analyze_sum_of_products (loop, expr, factor))
> + analyze_evolution (loop, expr, factor);
> + }
> +}
> +
> +/* Analyze expression EXPR, which occurs in loop LOOP. */
> +
> +void
> +loop_versioning::analyze_expr (struct loop *loop, tree expr)
> +{
> + while (handled_component_p (expr))
> + {
> + /* See whether we can use versioning to avoid a multiplication
> + in the array index. */
> + if (TREE_CODE (expr) == ARRAY_REF)
> + {
> + if (!analyze_sum_of_products (loop, TREE_OPERAND (expr, 1), 1))
> + analyze_evolution (loop, TREE_OPERAND (expr, 1), 1);
> + }
> + expr = TREE_OPERAND (expr, 0);
> + }
> +
> + if (TREE_CODE (expr) == MEM_REF)
> + {
> + tree addr = TREE_OPERAND (expr, 0);
> + /* See whether we can use versioning to avoid a multiplication
> + in the pointer calculation. This is generally only worth
> + doing if the multiplication occurs in this loop rather than
> + an outer loop. */
Why's that so here but not above for ARRAY_REF? That is, what is
the difference between a[i] and ptr = &a[i]; *ptr?
> + if (!expr_invariant_in_loop_p (loop, addr))
> + analyze_pointer (loop, addr, TREE_TYPE (expr));
> + }
> +
> + /* These would be easy to handle if they existed at this stage. */
> + gcc_checking_assert (TREE_CODE (expr) != TARGET_MEM_REF);
> +}
> +
> +/* Analyze STMT looking for useful version checks. */
> +
> +void
> +loop_versioning::analyze_stmt (gimple *stmt)
> +{
Please inline into single caller to make flow easily visible.
> + struct loop *loop = gimple_bb (stmt)->loop_father;
> +
> + unsigned int nops = gimple_num_ops (stmt);
> + for (unsigned int i = 0; i < nops; ++i)
> + if (tree op = gimple_op (stmt, i))
> + analyze_expr (loop, op);
I think you instead want to use gimple_walk_load_store_ops ().
> +}
> +
> +/* Analyze all the statements in BB looking for useful version checks. */
> +
> +void
> +loop_versioning::analyze_block (basic_block bb)
> +{
> + struct loop *loop = bb->loop_father;
> + loop_info &li = get_loop_info (loop);
> + if (li.rejected_p)
> + return;
> +
> + for (gimple_stmt_iterator gsi = gsi_start_bb (bb); !gsi_end_p (gsi);
> + gsi_next (&gsi))
> + {
> + gimple *stmt = gsi_stmt (gsi);
> + if (expensive_stmt_p (stmt))
> + {
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + {
> + struct loop *loop = gimple_bb (stmt)->loop_father;
> + fprintf (dump_file, ";; Loop %d (depth %d) contains expensive"
> + " stmt: ", loop->num, loop_depth (loop));
> + print_gimple_stmt (dump_file, stmt, 0, TDF_SLIM);
> + }
> + li.rejected_p = true;
I assume that once a loop is rejected this or another way there's
no reason to look at any outer loop of it, thus ...
> + break;
> + }
> +
> + /* Only look for direct versioning opportunities in inner loops
> + since the benefit tends to be much smaller for outer loops. */
> + if (!loop->inner)
> + analyze_stmt (stmt);
> +
> + /* The point of the instruction limit is to prevent excessive
> + code growth, so this is a size-based estimate even though
> + the optimization is aimed at speed. */
> + li.num_insns += estimate_num_insns (stmt, &eni_size_weights);
> + }
> +}
> +
> +/* Analyze all the blocks in the function looking for useful version checks.
> + Return true if we found one. */
> +
> +bool
> +loop_versioning::analyze_blocks ()
> +{
> + /* For now we don't try to version the whole function, although
> + versioning at that level could be useful in some cases. */
> + get_loop_info (get_loop (m_fn, 0)).rejected_p = true;
> +
> + basic_block bb;
> + FOR_EACH_BB_FN (bb, m_fn)
> + if (loop_outer (bb->loop_father))
> + analyze_block (bb);
.... I'd structure this as a
FOR_EACH_LOOP (... LI_FROM_INNERMOST)
look at loop body, only analyze stmts belonging to loop (not subloops)
walk eventually even open-coding this as recursion so you can quickly
finish off outer loop processing once an inner loop got disabled.
> +
> + return m_num_conditions != 0;
> +}
> +
> +/* Use the ranges in VRS to remove impossible versioning conditions from
> + LOOP. */
Does it actually happen to make it worthwhile? ;)
> +
> +void
> +loop_versioning::prune_loop_conditions (struct loop *loop, vr_values *vrs)
> +{
> + loop_info &li = get_loop_info (loop);
> +
> + unsigned int i = li.unity_names.length ();
> + while (i > 0)
> + {
> + i -= 1;
> + tree name = li.unity_names[i];
> + value_range *vr = vrs->get_value_range (name);
> + if (vr && !range_includes_p (vr, 1))
> + {
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + {
> + fprintf (dump_file, ";; ");
> + print_generic_expr (dump_file, name, TDF_SLIM);
> + fprintf (dump_file, " can never be 1 in loop %d\n", loop->num);
> + }
> +
> + li.unity_names.unordered_remove (i);
> + bitmap_clear_bit (&li.unity_name_ids, SSA_NAME_VERSION (name));
> + m_num_conditions -= 1;
> + }
> + }
> +}
> +
> +/* Remove any scheduled loop version conditions that will never be true.
> + Return true if any remain. */
> +
> +bool
> +loop_versioning::prune_conditions ()
> +{
> + calculate_dominance_info (CDI_DOMINATORS);
> + lv_dom_walker dom_walker (*this);
> + dom_walker.walk (ENTRY_BLOCK_PTR_FOR_FN (m_fn));
> + return m_num_conditions != 0;
> +}
> +
> +/* Merge the version checks for INNER into immediately-enclosing loop
> + OUTER. */
> +
> +void
> +loop_versioning::merge_loop_info (struct loop *outer, struct loop *inner)
> +{
> + loop_info &inner_li = get_loop_info (inner);
> + loop_info &outer_li = get_loop_info (outer);
> +
> + tree name;
> + unsigned int i;
> + FOR_EACH_VEC_ELT (inner_li.unity_names, i, name)
> + if (bitmap_set_bit (&outer_li.unity_name_ids, SSA_NAME_VERSION (name)))
> + {
> + outer_li.unity_names.safe_push (name);
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + {
> + fprintf (dump_file, ";; Hoisting check that ");
> + print_generic_expr (dump_file, name, TDF_SLIM);
> + fprintf (dump_file, " == 1 from loop %d (depth %d) to loop %d\n",
> + inner->num, loop_depth (inner), outer->num);
> + }
> + }
> +
> + if (loop_depth (outer_li.outermost) < loop_depth (inner_li.outermost))
> + outer_li.outermost = inner_li.outermost;
> +}
> +
> +/* Add LOOP to the queue of loops to version. */
> +
> +void
> +loop_versioning::add_loop_to_queue (struct loop *loop)
> +{
> + loop_info &li = get_loop_info (loop);
> +
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + fprintf (dump_file, ";; Queuing loop %d (depth %d) for versioning\n",
> + loop->num, loop_depth (loop));
> + m_loops_to_version.safe_push (loop);
> +
> + /* Don't try to version superloops. */
> + li.rejected_p = true;
> +}
> +
> +/* Decide whether the cost model would allow us to version LOOP,
> + either directly or as part of a parent loop, and return true if so.
> + This does not imply that the loop is actually worth versioning in its
> + own right, just that it would be valid to version it if something
> + benefited.
> +
> + We have already made this decision for all inner loops of LOOP. */
> +
> +bool
> +loop_versioning::decide_whether_loop_is_versionable (struct loop *loop)
> +{
> + loop_info &li = get_loop_info (loop);
> +
> + if (li.rejected_p)
> + return false;
> +
> + /* Examine the decisions made for inner loops. */
> + for (struct loop *inner = loop->inner; inner; inner = inner->next)
> + {
> + loop_info &inner_li = get_loop_info (inner);
> + if (inner_li.rejected_p)
> + {
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + fprintf (dump_file, ";; Not versioning loop %d (depth %d)"
> + " because inner loop %d should not be versioned\n",
> + loop->num, loop_depth (loop), inner->num);
> + return false;
> + }
> +
> + if (inner_li.worth_versioning_p ())
> + li.subloops_benefit_p = true;
> +
> + /* Accumulate the number of instructions from subloops that are not
> + the innermost, or that don't benefit from versioning. Only the
> + instructions from innermost loops that benefit from versioning
> + should be weighed against loop-versioning-max-inner-insns;
> + everything else should be weighed against
> + loop-versioning-max-outer-insns. */
> + if (!inner_li.worth_versioning_p () || inner->inner)
> + {
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + fprintf (dump_file, ";; Counting %d instructions from"
> + " loop %d (depth %d) against parent loop %d\n",
> + inner_li.num_insns, inner->num, loop_depth (inner),
> + loop->num);
> + li.num_insns += inner_li.num_insns;
> + }
> + }
> +
> + /* Enforce the size limits. */
> + if (li.worth_versioning_p ())
> + {
> + unsigned int max_num_insns = max_insns_for_loop (loop);
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + fprintf (dump_file, ";; Loop %d (depth %d) has %d instructions,"
> + " against a limit of %d\n", loop->num, loop_depth (loop),
> + li.num_insns, max_num_insns);
> + if (li.num_insns > max_num_insns)
> + {
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + fprintf (dump_file, ";; Not versioning loop %d (depth %d)"
> + " because it is too big\n", loop->num, loop_depth (loop));
> + return false;
> + }
> + }
> +
> + /* Hoist all version checks for subloops to this loop. */
> + for (struct loop *subloop = loop->inner; subloop; subloop = subloop->next)
> + merge_loop_info (loop, subloop);
> +
> + return true;
> +}
> +
> +/* Decide which loops to version and add them to the versioning queue.
> + Return true if there are any loops to version. */
> +
> +bool
> +loop_versioning::make_versioning_decisions ()
> +{
> + struct loop *loop;
> + FOR_EACH_LOOP (loop, LI_FROM_INNERMOST)
> + {
> + loop_info &linfo = get_loop_info (loop);
> + if (decide_whether_loop_is_versionable (loop))
> + {
> + /* Commit to versioning LOOP directly if we can't hoist the
> + version checks any further. */
> + if (linfo.worth_versioning_p ()
> + && (loop_depth (loop) == 1 || linfo.outermost == loop))
> + add_loop_to_queue (loop);
> + }
> + else
> + {
> + /* We can't version this loop, so individually version any
> + subloops that would benefit and haven't been versioned yet. */
> + linfo.rejected_p = true;
> + for (struct loop *subloop = loop->inner; subloop;
> + subloop = subloop->next)
> + if (get_loop_info (subloop).worth_versioning_p ())
> + add_loop_to_queue (subloop);
> + }
> + }
> +
> + return !m_loops_to_version.is_empty ();
> +}
> +
> +/* Attempt to implement loop versioning for LOOP, using the information
> + cached in the associated loop_info. Return true on success. */
> +
> +bool
> +loop_versioning::version_loop (struct loop *loop)
> +{
> + loop_info &li = get_loop_info (loop);
> +
> + /* Not protected by TDF_DETAILS since this is the main piece of
> + information. */
> + if (dump_file)
> + fprintf (dump_file, ";; Versioning loop %d (depth %d)\n",
> + loop->num, loop_depth (loop));
> +
> + /* Build up a condition that selects the original loop instead of
> + the simplified loop. */
> + tree cond = boolean_false_node;
> + tree name;
> + unsigned int i;
> + FOR_EACH_VEC_ELT (li.unity_names, i, name)
> + {
> + tree ne_one = fold_build2 (NE_EXPR, boolean_type_node, name,
> + build_one_cst (TREE_TYPE (name)));
> + cond = fold_build2 (TRUTH_OR_EXPR, boolean_type_node, cond, ne_one);
> + }
> +
> + /* Convert the condition into a suitable gcond. */
> + gimple_seq stmts = NULL;
> + cond = force_gimple_operand_1 (cond, &stmts, is_gimple_condexpr, NULL_TREE);
> +
> + /* Version the loop. */
> + initialize_original_copy_tables ();
> + basic_block cond_bb;
> + struct loop *else_loop
> + = loop_version (loop, cond, &cond_bb,
> + profile_probability::unlikely (),
> + profile_probability::likely (),
> + profile_probability::unlikely (),
> + profile_probability::likely (), true);
> + free_original_copy_tables ();
> + if (!else_loop)
> + {
> + if (dump_file)
> + fprintf (dump_file, ";; Versioning of loop %d failed\n", loop->num);
> + return false;
> + }
> +
> + /* Insert the statements that feed COND. */
> + if (stmts)
> + {
> + gimple_stmt_iterator gsi = gsi_last_bb (cond_bb);
> + gsi_insert_seq_before (&gsi, stmts, GSI_SAME_STMT);
> + }
> +
> + /* Simplify the new loop, which is used when COND is false. */
> + name_prop (li).substitute_and_fold (else_loop->header);
> + return true;
> +}
> +
> +/* Attempt to version all loops in the versioning queue. Return true
> + if we succeeded for at least one loop. */
> +
> +bool
> +loop_versioning::implement_versioning_decisions ()
> +{
> + bool any_succeeded_p = false;
> +
> + struct loop *loop;
> + unsigned int i;
> + FOR_EACH_VEC_ELT (m_loops_to_version, i, loop)
> + if (version_loop (loop))
> + any_succeeded_p = true;
> +
> + return any_succeeded_p;
> +}
> +
> +/* Run the pass and return a set of TODO_* flags. */
> +
> +unsigned int
> +loop_versioning::run ()
> +{
> + gcc_assert (scev_initialized_p ());
> +
> + if (!analyze_blocks ()
> + || !prune_conditions ()
> + || !make_versioning_decisions ()
> + || !implement_versioning_decisions ())
> + return 0;
> +
> + return TODO_update_ssa;
> +}
> +
> +/* Loop versioningting pass. */
> +
> +namespace {
> +
> +const pass_data pass_data_loop_versioning =
> +{
> + GIMPLE_PASS, /* type */
> + "lversion", /* name */
> + OPTGROUP_LOOP, /* optinfo_flags */
> + TV_LOOP_VERSIONING, /* tv_id */
> + PROP_cfg, /* properties_required */
> + 0, /* properties_provided */
> + 0, /* properties_destroyed */
> + 0, /* todo_flags_start */
> + 0, /* todo_flags_finish */
> +};
> +
> +class pass_loop_versioning : public gimple_opt_pass
> +{
> +public:
> + pass_loop_versioning (gcc::context *ctxt)
> + : gimple_opt_pass (pass_data_loop_versioning, ctxt)
> + {}
> +
> + /* opt_pass methods: */
> + virtual bool gate (function *) { return flag_version_loops_for_strides; }
> + virtual unsigned int execute (function *);
> +};
> +
> +unsigned int
> +pass_loop_versioning::execute (function *fn)
> +{
> + if (number_of_loops (fn) <= 1)
> + return 0;
> +
> + return loop_versioning (fn).run ();
> +}
> +
> +} // anon namespace
> +
> +gimple_opt_pass *
> +make_pass_loop_versioning (gcc::context *ctxt)
> +{
> + return new pass_loop_versioning (ctxt);
> +}
> Index: gcc/testsuite/gcc.dg/loop-versioning-1.c
> ===================================================================
> --- /dev/null 2018-09-14 11:16:31.122530289 +0100
> +++ gcc/testsuite/gcc.dg/loop-versioning-1.c 2018-10-24 14:02:15.184152693 +0100
> @@ -0,0 +1,20 @@
> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
> +
> +/* The simplest IV case. */
> +
> +void
> +f1 (double *x, int stepx, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + x[stepx * i] = 100;
> +}
> +
> +void
> +f2 (double *x, int stepx, int limit)
> +{
> + for (int i = 0; i < limit; i += stepx)
> + x[i] = 100;
> +}
> +
> +/* { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 1\)} 2 "lversion" } } */
> +/* { dg-final { scan-tree-dump-times {Versioning loop} 2 "lversion" } } */
> Index: gcc/testsuite/gcc.dg/loop-versioning-10.c
> ===================================================================
> --- /dev/null 2018-09-14 11:16:31.122530289 +0100
> +++ gcc/testsuite/gcc.dg/loop-versioning-10.c 2018-10-24 14:02:15.184152693 +0100
> @@ -0,0 +1,17 @@
> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
> +
> +/* Test that we can version a gather-like operation in which a variable
> + stride is applied to the index. */
> +
> +int
> +f1 (int *x, int *index, int step, int n)
> +{
> + int res = 0;
> + for (int i = 0; i < n; ++i)
> + res += x[index[i] * step];
> + return res;
> +}
> +
> +/* { dg-final { scan-tree-dump-times {Address[^\n]*invariant} 1 "lversion" } } */
> +/* { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 1\)} 1 "lversion" } } */
> +/* { dg-final { scan-tree-dump-times {Versioning loop} 1 "lversion" } } */
> Index: gcc/testsuite/gcc.dg/loop-versioning-11.c
> ===================================================================
> --- /dev/null 2018-09-14 11:16:31.122530289 +0100
> +++ gcc/testsuite/gcc.dg/loop-versioning-11.c 2018-10-24 14:02:15.188152659 +0100
> @@ -0,0 +1,29 @@
> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
> +
> +/* Test that we don't try to version for something that is never 1. */
> +
> +void
> +f1 (double *x, int stepx, int n)
> +{
> + if (stepx == 1)
> + for (int i = 0; i < n; ++i)
> + x[i] = 100;
> + else
> + for (int i = 0; i < n; ++i)
> + x[stepx * i] = 100;
> +}
> +
> +void
> +f2 (double *x, int stepx, int n)
> +{
> + if (stepx <= 1)
> + for (int i = 0; i < n; ++i)
> + x[i] = 100;
> + else
> + for (int i = 0; i < n; ++i)
> + x[stepx * i] = 100;
> +}
> +
> +/* { dg-final { scan-tree-dump-times {Want to version loop} 2 "lversion" } } */
> +/* { dg-final { scan-tree-dump-times {can never be 1} 2 "lversion" } } */
> +/* { dg-final { scan-tree-dump-not {Versioning loop} "lversion" } } */
> Index: gcc/testsuite/gcc.dg/loop-versioning-2.c
> ===================================================================
> --- /dev/null 2018-09-14 11:16:31.122530289 +0100
> +++ gcc/testsuite/gcc.dg/loop-versioning-2.c 2018-10-24 14:02:15.188152659 +0100
> @@ -0,0 +1,39 @@
> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
> +
> +/* Versioning these loops allows loop interchange. */
> +
> +void
> +f1 (double x[][100], int step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + x[j * step][i] = 100;
> +}
> +
> +void
> +f2 (double x[][100], int step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + x[j][i * step] = 100;
> +}
> +
> +void
> +f3 (double x[][100], int step, int limit)
> +{
> + for (int i = 0; i < 100; ++i)
> + for (int j = 0; j < limit; j += step)
> + x[j][i] = 100;
> +}
> +
> +void
> +f4 (double x[][100], int step, int limit)
> +{
> + for (int i = 0; i < limit; i += step)
> + for (int j = 0; j < 100; ++j)
> + x[j][i] = 100;
> +}
> +
> +/* { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 2\)} 4 "lversion" } } */
> +/* { dg-final { scan-tree-dump-times {Hoisting check} 4 "lversion" } } */
> +/* { dg-final { scan-tree-dump-times {Versioning loop [0-9]+ \(depth 1\)} 4 "lversion" } } */
> Index: gcc/testsuite/gcc.dg/loop-versioning-3.c
> ===================================================================
> --- /dev/null 2018-09-14 11:16:31.122530289 +0100
> +++ gcc/testsuite/gcc.dg/loop-versioning-3.c 2018-10-24 14:02:15.188152659 +0100
> @@ -0,0 +1,24 @@
> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
> +
> +/* Versioning these loops for when both steps are 1 allows loop
> + interchange. */
> +
> +void
> +f1 (double x[][100], int step1, int step2, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + x[j * step1][i * step2] = 100;
> +}
> +
> +void
> +f2 (double x[][100], int step1, int step2, int limit)
> +{
> + for (int i = 0; i < limit; i += step1)
> + for (int j = 0; j < limit; j += step2)
> + x[j][i] = 100;
> +}
> +
> +/* { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 2\)} 4 "lversion" } } */
> +/* { dg-final { scan-tree-dump-times {Hoisting check} 4 "lversion" } } */
> +/* { dg-final { scan-tree-dump-times {Versioning loop} 2 "lversion" } } */
> Index: gcc/testsuite/gcc.dg/loop-versioning-4.c
> ===================================================================
> --- /dev/null 2018-09-14 11:16:31.122530289 +0100
> +++ gcc/testsuite/gcc.dg/loop-versioning-4.c 2018-10-24 14:02:15.188152659 +0100
> @@ -0,0 +1,38 @@
> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
> +
> +/* These shouldn't be versioned; it's extremely likely that the code
> + is emulating two-dimensional arrays. */
> +
> +void
> +f1 (double *x, int step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + x[i * step + j] = 100;
> +}
> +
> +void
> +f2 (double *x, int step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + x[j * step + i] = 100;
> +}
> +
> +void
> +f3 (double *x, int *offsets, int step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + x[i * step + j + offsets[i]] = 100;
> +}
> +
> +void
> +f4 (double *x, int *offsets, int step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + x[j * step + i + offsets[i]] = 100;
> +}
> +
> +/* { dg-final { scan-tree-dump-not {Versioning loop} "lversion" } } */
> Index: gcc/testsuite/gcc.dg/loop-versioning-5.c
> ===================================================================
> --- /dev/null 2018-09-14 11:16:31.122530289 +0100
> +++ gcc/testsuite/gcc.dg/loop-versioning-5.c 2018-10-24 14:02:15.188152659 +0100
> @@ -0,0 +1,17 @@
> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
> +
> +/* There's no information about whether STEP1 or STEP2 is innermost,
> + so we should assume the code is sensible and version for the inner
> + evolution, i.e. when STEP2 is 1. */
> +
> +void
> +f1 (double *x, int step1, int step2, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + x[i * step1 + j * step2] = 100;
> +}
> +
> +/* { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 2\) for when step2} 1 "lversion" } } */
> +/* { dg-final { scan-tree-dump-times {Want to version loop} 1 "lversion" } } */
> +/* { dg-final { scan-tree-dump-times {Versioning loop} 1 "lversion" } } */
> Index: gcc/testsuite/gcc.dg/loop-versioning-6.c
> ===================================================================
> --- /dev/null 2018-09-14 11:16:31.122530289 +0100
> +++ gcc/testsuite/gcc.dg/loop-versioning-6.c 2018-10-24 14:02:15.188152659 +0100
> @@ -0,0 +1,30 @@
> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
> +
> +/* The read from y in f1 will be hoisted to the outer loop. In general
> + it's not worth versioning outer loops when the inner loops don't also
> + benefit.
> +
> + This test is meant to be a slight counterexample, since versioning
> + does lead to cheaper outer-loop vectorization. However, the benefit
> + isn't enough to justify the cost. */
> +
> +void
> +f1 (double *restrict x, double *restrict y, int step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + x[i + j] = y[i * step];
> +}
> +
> +/* A similar example in which the read can't be hoisted, but could
> + for example be handled by vectorizer alias checks. */
> +
> +void
> +f2 (double *x, double *y, int step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + x[i + j] = y[i * step];
> +}
> +
> +/* { dg-final { scan-tree-dump-not {Versioning loop} "lversion" } } */
> Index: gcc/testsuite/gcc.dg/loop-versioning-7.c
> ===================================================================
> --- /dev/null 2018-09-14 11:16:31.122530289 +0100
> +++ gcc/testsuite/gcc.dg/loop-versioning-7.c 2018-10-24 14:02:15.188152659 +0100
> @@ -0,0 +1,32 @@
> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
> +
> +/* Check that versioning can handle arrays of structures. */
> +
> +struct foo {
> + int a, b, c;
> +};
> +
> +void
> +f1 (struct foo *x, int stepx, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + {
> + x[stepx * i].a = 1;
> + x[stepx * i].b = 2;
> + x[stepx * i].c = 3;
> + }
> +}
> +
> +void
> +f2 (struct foo *x, int stepx, int limit)
> +{
> + for (int i = 0; i < limit; i += stepx)
> + {
> + x[i].a = 1;
> + x[i].b = 2;
> + x[i].c = 3;
> + }
> +}
> +
> +/* { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 1\)} 2 "lversion" } } */
> +/* { dg-final { scan-tree-dump-times {Versioning loop} 2 "lversion" } } */
> Index: gcc/testsuite/gcc.dg/loop-versioning-8.c
> ===================================================================
> --- /dev/null 2018-09-14 11:16:31.122530289 +0100
> +++ gcc/testsuite/gcc.dg/loop-versioning-8.c 2018-10-24 14:02:15.188152659 +0100
> @@ -0,0 +1,44 @@
> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
> +
> +/* Check that versioning can handle arrays of structures that wrap
> + subarrays. */
> +
> +struct foo {
> + int a[100];
> +};
> +
> +void
> +f1 (struct foo *x, int step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + x[j * step].a[i] = 100;
> +}
> +
> +void
> +f2 (struct foo *x, int step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + x[j].a[i * step] = 100;
> +}
> +
> +void
> +f3 (struct foo *x, int step, int limit)
> +{
> + for (int i = 0; i < 100; ++i)
> + for (int j = 0; j < limit; j += step)
> + x[j].a[i] = 100;
> +}
> +
> +void
> +f4 (struct foo *x, int step, int limit)
> +{
> + for (int i = 0; i < limit; i += step)
> + for (int j = 0; j < 100; ++j)
> + x[j].a[i] = 100;
> +}
> +
> +/* { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 2\)} 4 "lversion" } } */
> +/* { dg-final { scan-tree-dump-times {Hoisting check} 4 "lversion" } } */
> +/* { dg-final { scan-tree-dump-times {Versioning loop [0-9]+ \(depth 1\)} 4 "lversion" } } */
> Index: gcc/testsuite/gcc.dg/loop-versioning-9.c
> ===================================================================
> --- /dev/null 2018-09-14 11:16:31.122530289 +0100
> +++ gcc/testsuite/gcc.dg/loop-versioning-9.c 2018-10-24 14:02:15.188152659 +0100
> @@ -0,0 +1,48 @@
> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
> +
> +/* Check that versioning can handle small groups of accesses. */
> +
> +void
> +f1 (int *x, int *y, int step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + x[i] = y[i * step * 2] + y[i * step * 2 + 1];
> +}
> +
> +void
> +f2 (int *x, int *y, __INTPTR_TYPE__ step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + x[i] = y[i * step * 2] + y[i * step * 2 + 1];
> +}
> +
> +void
> +f3 (int *x, int *y, int step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + x[i] = y[i * step * 3] + y[i * step * 3 + 2];
> +}
> +
> +void
> +f4 (int *x, int *y, __INTPTR_TYPE__ step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + x[i] = y[i * step * 3] + y[i * step * 3 + 2];
> +}
> +
> +void
> +f5 (int *x, int *y, int step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + x[i] = y[i * step * 4] + y[i * step * 4 + 3];
> +}
> +
> +void
> +f6 (int *x, int *y, __INTPTR_TYPE__ step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + x[i] = y[i * step * 4] + y[i * step * 4 + 3];
> +}
> +
> +/* { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 1\)} 6 "lversion" } } */
> +/* { dg-final { scan-tree-dump-times {Versioning loop} 6 "lversion" } } */
> Index: gcc/testsuite/gfortran.dg/loop_versioning_1.f90
> ===================================================================
> --- /dev/null 2018-09-14 11:16:31.122530289 +0100
> +++ gcc/testsuite/gfortran.dg/loop_versioning_1.f90 2018-10-24 14:02:15.188152659 +0100
> @@ -0,0 +1,28 @@
> +! { dg-options "-O3 -fdump-tree-lversion-details" }
> +
> +! The simplest IV case.
> +
> +subroutine f1(x)
> + real :: x(:)
> + x(:) = 100
> +end subroutine f1
> +
> +subroutine f2(x, n, step)
> + integer :: n, step
> + real :: x(n * step)
> + do i = 1, n
> + x(i * step) = 100
> + end do
> +end subroutine f2
> +
> +subroutine f3(x, limit, step)
> + integer :: limit, step
> + real :: x(limit)
> + do i = 1, limit, step
> + x(i) = 100
> + end do
> +end subroutine f3
> +
> +! { dg-final { scan-tree-dump-times {likely to be innermost dimension} 1 "lversion" } }
> +! { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 1\)} 3 "lversion" } }
> +! { dg-final { scan-tree-dump-times {Versioning loop} 3 "lversion" } }
> Index: gcc/testsuite/gfortran.dg/loop_versioning_2.f90
> ===================================================================
> --- /dev/null 2018-09-14 11:16:31.122530289 +0100
> +++ gcc/testsuite/gfortran.dg/loop_versioning_2.f90 2018-10-24 14:02:15.188152659 +0100
> @@ -0,0 +1,38 @@
> +! { dg-options "-O3 -fdump-tree-lversion-details -fno-frontend-loop-interchange" }
> +
> +! We could version the loop for when the first dimension has a stride
> +! of 1, but at present there's no real benefit. The gimple loop
> +! interchange pass couldn't handle the versioned loop, and interchange
> +! is instead done by the frontend (but disabled by the options above).
> +
> +subroutine f1(x)
> + real :: x(:, :)
> + do i = lbound(x, 1), ubound(x, 1)
> + do j = lbound(x, 2), ubound(x, 2)
> + x(i, j) = 100
> + end do
> + end do
> +end subroutine f1
> +
> +subroutine f2(x, n, step)
> + integer :: n, step
> + real :: x(100, 100)
> + do i = 1, n
> + do j = 1, n
> + x(i * step, j) = 100
> + end do
> + end do
> +end subroutine f2
> +
> +subroutine f3(x, n, step)
> + integer :: n, step
> + real :: x(n * step, n)
> + do i = 1, n
> + do j = 1, n
> + x(i * step, j) = 100
> + end do
> + end do
> +end subroutine f3
> +
> +! { dg-final { scan-tree-dump-times {likely to be innermost dimension} 1 "lversion" } }
> +! { dg-final { scan-tree-dump-not {Versioning loop} "lversion" } }
> Index: gcc/testsuite/gfortran.dg/loop_versioning_3.f90
> ===================================================================
> --- /dev/null 2018-09-14 11:16:31.122530289 +0100
> +++ gcc/testsuite/gfortran.dg/loop_versioning_3.f90 2018-10-24 14:02:15.188152659 +0100
> @@ -0,0 +1,29 @@
> +! { dg-options "-O3 -fdump-tree-lversion-details -fno-frontend-loop-interchange" }
> +
> +! Test a case in which the outer loop iterates over the inner dimension.
> +! The options above prevent the frontend from interchanging the loops.
> +
> +subroutine f1(x, limit, step, n)
> + integer :: limit, step, n
> + real :: x(limit, n)
> + do i = 1, limit, step
> + do j = 1, n
> + x(i, j) = 100
> + end do
> + end do
> +end subroutine f1
> +
> +subroutine f2(x, n, limit, step)
> + integer :: n, limit, step
> + real :: x(limit, n)
> + do i = 1, n
> + do j = 1, limit, step
> + x(j, i) = 100
> + end do
> + end do
> +end subroutine f2
> +
> +! FIXME: The frontend doesn't give us enough information to tell which loop
> +! is iterating over the innermost dimension, so we optimistically
> +! assume the inner one is.
> +! { dg-final { scan-tree-dump-not {Versioning loop} "lversion" { xfail *-*-* } } }
> Index: gcc/testsuite/gfortran.dg/loop_versioning_4.f90
> ===================================================================
> --- /dev/null 2018-09-14 11:16:31.122530289 +0100
> +++ gcc/testsuite/gfortran.dg/loop_versioning_4.f90 2018-10-24 14:02:15.188152659 +0100
> @@ -0,0 +1,52 @@
> +! { dg-options "-O3 -fdump-tree-lversion-details -fno-frontend-loop-interchange" }
> +
> +! Test cases in which versioning is useful for a two-dimensional array.
> +
> +subroutine f1(x)
> + real :: x(:, :)
> + x(:, :) = 100
> +end subroutine f1
> +
> +subroutine f2(x)
> + real :: x(:, :)
> + do i = lbound(x, 1), ubound(x, 1)
> + do j = lbound(x, 2), ubound(x, 2)
> + x(j, i) = 100
> + end do
> + end do
> +end subroutine f2
> +
> +subroutine f3(x, n, step)
> + integer :: n, step
> + real :: x(100, 100)
> + do i = 1, n
> + do j = 1, n
> + x(j * step, i) = 100
> + end do
> + end do
> +end subroutine f3
> +
> +subroutine f4(x, n, step)
> + integer :: n, step
> + real :: x(n * step, n)
> + do i = 1, n
> + do j = 1, n
> + x(j * step, i) = 100
> + end do
> + end do
> +end subroutine f4
> +
> +subroutine f5(x, n, limit, step)
> + integer :: n, limit, step
> + real :: x(limit, n)
> + do i = 1, n
> + do j = 1, limit, step
> + x(j, i) = 100
> + end do
> + end do
> +end subroutine f5
> +
> +! { dg-final { scan-tree-dump-times {likely to be innermost dimension} 2 "lversion" } }
> +! { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 2\)} 5 "lversion" } }
> +! { dg-final { scan-tree-dump-times {Hoisting check} 5 "lversion" } }
> +! { dg-final { scan-tree-dump-times {Versioning loop [0-9]+ \(depth 1\)} 5 "lversion" } }
> Index: gcc/testsuite/gfortran.dg/loop_versioning_5.f90
> ===================================================================
> --- /dev/null 2018-09-14 11:16:31.122530289 +0100
> +++ gcc/testsuite/gfortran.dg/loop_versioning_5.f90 2018-10-24 14:02:15.188152659 +0100
> @@ -0,0 +1,56 @@
> +! { dg-options "-O3 -fdump-tree-lversion-details -fno-frontend-loop-interchange" }
> +
> +! Make sure that in a "badly nested" loop, we don't treat the inner loop
> +! as iterating over the inner dimension with a variable stride.
> +
> +subroutine f1(x, n)
> + integer :: n
> + real :: x(100, 100)
> + do i = 1, n
> + do j = 1, n
> + x(i, j) = 100
> + end do
> + end do
> +end subroutine f1
> +
> +subroutine f2(x, n, step)
> + integer :: n, step
> + real :: x(100, 100)
> + do i = 1, n
> + do j = 1, n
> + x(i, j * step) = 100
> + end do
> + end do
> +end subroutine f2
> +
> +subroutine f3(x, n)
> + integer :: n
> + real :: x(n, n)
> + do i = 1, n
> + do j = 1, n
> + x(i, j) = 100
> + end do
> + end do
> +end subroutine f3
> +
> +subroutine f4(x, n, step)
> + integer :: n, step
> + real :: x(n, n * step)
> + do i = 1, n
> + do j = 1, n
> + x(i, j * step) = 100
> + end do
> + end do
> +end subroutine f4
> +
> +subroutine f5(x, n, limit, step)
> + integer :: n, limit, step
> + real :: x(n, limit)
> + do i = 1, n
> + do j = 1, limit, step
> + x(i, j) = 100
> + end do
> + end do
> +end subroutine f5
> +
> +! { dg-final { scan-tree-dump-not {Versioning loop} "lversion" } }
> Index: gcc/testsuite/gfortran.dg/loop_versioning_6.f90
> ===================================================================
> --- /dev/null 2018-09-14 11:16:31.122530289 +0100
> +++ gcc/testsuite/gfortran.dg/loop_versioning_6.f90 2018-10-24 14:02:15.188152659 +0100
> @@ -0,0 +1,93 @@
> +! { dg-options "-O3 -fdump-tree-lversion-details" }
> +
> +! Check that versioning can handle small groups of accesses.
> +
> +subroutine f1(x)
> + real :: x(:)
> + do i = lbound(x, 1), ubound(x, 1) / 2
> + x(i * 2) = 100
> + x(i * 2 + 1) = 101
> + end do
> +end subroutine f1
> +
> +subroutine f2(x, n, step)
> + integer :: n, step
> + real :: x(n * step * 2)
> + do i = 1, n
> + x(i * step * 2) = 100
> + x(i * step * 2 + 1) = 101
> + end do
> +end subroutine f2
> +
> +subroutine f3(x, limit, step)
> + integer :: limit, step
> + real :: x(limit * 2)
> + do i = 1, limit, step
> + x(i * 2) = 100
> + x(i * 2 + 1) = 101
> + end do
> +end subroutine f3
> +
> +subroutine f4(x)
> + real :: x(:)
> + do i = lbound(x, 1), ubound(x, 1) / 3
> + x(i * 3) = 100
> + x(i * 3 + 1) = 101
> + x(i * 3 + 2) = 102
> + end do
> +end subroutine f4
> +
> +subroutine f5(x, n, step)
> + integer :: n, step
> + real :: x(n * step * 3)
> + do i = 1, n
> + x(i * step * 3) = 100
> + x(i * step * 3 + 1) = 101
> + x(i * step * 3 + 2) = 102
> + end do
> +end subroutine f5
> +
> +subroutine f6(x, limit, step)
> + integer :: limit, step
> + real :: x(limit * 3)
> + do i = 1, limit, step
> + x(i * 3) = 100
> + x(i * 3 + 1) = 101
> + x(i * 3 + 2) = 102
> + end do
> +end subroutine f6
> +
> +subroutine f7(x)
> + real :: x(:)
> + do i = lbound(x, 1), ubound(x, 1) / 4
> + x(i * 4) = 100
> + x(i * 4 + 1) = 101
> + x(i * 4 + 2) = 102
> + x(i * 4 + 3) = 103
> + end do
> +end subroutine f7
> +
> +subroutine f8(x, n, step)
> + integer :: n, step
> + real :: x(n * step * 4)
> + do i = 1, n
> + x(i * step * 4) = 100
> + x(i * step * 4 + 1) = 101
> + x(i * step * 4 + 2) = 102
> + x(i * step * 4 + 3) = 103
> + end do
> +end subroutine f8
> +
> +subroutine f9(x, limit, step)
> + integer :: limit, step
> + real :: x(limit * 4)
> + do i = 1, limit, step
> + x(i * 4) = 100
> + x(i * 4 + 1) = 101
> + x(i * 4 + 2) = 102
> + x(i * 4 + 3) = 103
> + end do
> +end subroutine f9
> +
> +! { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 1\)} 9 "lversion" } }
> +! { dg-final { scan-tree-dump-times {Versioning loop} 9 "lversion" } }
> Index: gcc/testsuite/gfortran.dg/loop_versioning_7.f90
> ===================================================================
> --- /dev/null 2018-09-14 11:16:31.122530289 +0100
> +++ gcc/testsuite/gfortran.dg/loop_versioning_7.f90 2018-10-24 14:02:15.188152659 +0100
> @@ -0,0 +1,67 @@
> +! { dg-options "-O3 -fdump-tree-lversion-details" }
> +
> +! Check that versioning can handle small groups of accesses, with the
> +! group being a separate array dimension.
> +
> +subroutine f1(x, n, step)
> + integer :: n, step
> + real :: x(2, n * step)
> + do i = 1, n
> + x(1, i * step) = 100
> + x(2, i * step) = 101
> + end do
> +end subroutine f1
> +
> +subroutine f2(x, limit, step)
> + integer :: limit, step
> + real :: x(2, limit)
> + do i = 1, limit, step
> + x(1, i) = 100
> + x(2, i) = 101
> + end do
> +end subroutine f2
> +
> +subroutine f3(x, n, step)
> + integer :: n, step
> + real :: x(3, n * step)
> + do i = 1, n
> + x(1, i * step) = 100
> + x(2, i * step) = 101
> + x(3, i * step) = 102
> + end do
> +end subroutine f3
> +
> +subroutine f4(x, limit, step)
> + integer :: limit, step
> + real :: x(3, limit)
> + do i = 1, limit, step
> + x(1, i) = 100
> + x(2, i) = 101
> + x(3, i) = 102
> + end do
> +end subroutine f4
> +
> +subroutine f5(x, n, step)
> + integer :: n, step
> + real :: x(4, n * step)
> + do i = 1, n
> + x(1, i * step) = 100
> + x(2, i * step) = 101
> + x(3, i * step) = 102
> + x(4, i * step) = 103
> + end do
> +end subroutine f5
> +
> +subroutine f6(x, limit, step)
> + integer :: limit, step
> + real :: x(4, limit)
> + do i = 1, limit, step
> + x(1, i) = 100
> + x(2, i) = 101
> + x(3, i) = 102
> + x(4, i) = 103
> + end do
> +end subroutine f6
> +
> +! { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 1\)} 6 "lversion" } }
> +! { dg-final { scan-tree-dump-times {Versioning loop} 6 "lversion" } }
> Index: gcc/testsuite/gfortran.dg/loop_versioning_8.f90
> ===================================================================
> --- /dev/null 2018-09-14 11:16:31.122530289 +0100
> +++ gcc/testsuite/gfortran.dg/loop_versioning_8.f90 2018-10-24 14:02:15.188152659 +0100
> @@ -0,0 +1,13 @@
> +! { dg-options "-O3 -fdump-tree-lversion-details" }
> +
> +! Check that versioning is applied to a gather-like reduction operation.
> +
> +function f(x, index, n)
> + integer :: n
> + real :: x(:)
> + integer :: index(n)
> + f = sum(x(index(:)))
> +end function f
> +
> +! { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 1\)} 1 "lversion" } }
> +! { dg-final { scan-tree-dump-times {Versioning loop} 1 "lversion" } }
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Add a loop versioning pass
2018-10-30 10:55 Add a loop versioning pass Richard Biener
@ 2018-11-09 10:46 ` Kyrill Tkachov
2018-11-28 16:48 ` Richard Sandiford
1 sibling, 0 replies; 17+ messages in thread
From: Kyrill Tkachov @ 2018-11-09 10:46 UTC (permalink / raw)
To: Richard Biener, gcc-patches, Richard Sandiford
[-- Attachment #1: Type: text/plain, Size: 71560 bytes --]
Hi Richard,
On 30/10/18 10:10, Richard Biener wrote:
>
> (sorry for breaking threading -- I composed a review mail offline but
> gmail has no way of nicely sending that neither has it a way to bounce
> messages...)
>
While Richard is away I've attempted to address some of your comments.
I've tackled the simpler ones as I was not involved in the design and benchmarking of this pass
and don't have experience with some of the APIs used (scalar evolution, in particular).
Nevertheless, hopefully this cleans up some of the simpler concerns before close of stage 1
(though I hope Richard will address the more complex ones once he's back).
> > This patch adds a pass that versions loops with variable index strides
> > for the case in which the stride is 1. E.g.:
> >
> > for (int i = 0; i < n; ++i)
> > x[i * stride] = ...;
> >
> > becomes:
> >
> > if (stepx == 1)
> > for (int i = 0; i < n; ++i)
> > x[i] = ...;
> > else
> > for (int i = 0; i < n; ++i)
> > x[i * stride] = ...;
> >
> > This is useful for both vector code and scalar code, and in some cases
> > can enable further optimisations like loop interchange or pattern
> > recognition.
> >
> > The pass gives a 7.6% improvement on Cortex-A72 for 554.roms_r at -O3
> > and a 2.4% improvement for 465.tonto. I haven't found any SPEC tests
> > that regress.
> >
> > Sizewise, there's a 10% increase in .text for both 554.roms_r and
> > 465.tonto. That's obviously a lot, but in tonto's case it's because
> > the whole program is written using assumed-shape arrays and pointers,
> > so a large number of functions really do benefit from versioning.
> > roms likewise makes heavy use of assumed-shape arrays, and that
> > improvement in performance IMO justifies the code growth.
>
> Ouch. I know that at least with LTO IPA-CP can do "quite" some
> propagation of constant strides. Not sure if we're aggressive
> enough in actually doing the cloning for all cases we figure out
> strides though. But my question is how we can avoid doing the
> versioning for loops in the copy that did not have the IPA-CPed
> stride of one? Ideally we'd be able to mark individual references
> as {definitely,likely,unlikely,not}-unit-stride?
>
> > The next biggest .text increase is 4.5% for 548.exchange2_r. I did see
> > a small (0.4%) speed improvement there, but although both 3-iteration runs
> > produced stable results, that might still be noise. There was a slightly
> > larger (non-noise) improvement for a 256-bit SVE model.
> >
> > 481.wrf and 521.wrf_r .text grew by 2.8% and 2.5% respectively, but
> > without any noticeable improvement in performance. No other test grew
> > by more than 2%.
> >
> > Although the main SPEC beneficiaries are all Fortran tests, the
> > benchmarks we use for SVE also include some C and C++ tests that
> > benefit.
>
> Did you see any slowdown, for example because versioning was forced
> to be on an innermost loop? I'm thinking of the testcase in
> PR87561 where we do have strided accesses in the innermost loop.
>
> Since you cite performance numbers how did you measure them?
> I assume -Ofast -march=native but did you check with -flto?
>
> > Using -frepack-arrays gives the same benefits in many Fortran cases.
> > The problem is that using that option inappropriately can force a full
> > array copy for arguments that the function only reads once, and so it
> > isn't really something we can turn on by default. The new pass is
> > supposed to give most of the benefits of -frepack-arrays without
> > the risk of unnecessary repacking.
> >
> > The patch therefore enables the pass by default at -O3.
>
> I think that's reasonable.
>
> One possible enhancement would be to add a value-profile for the
> strides so we can guide this optimization better.
>
> The pass falls foul of C++ class make small methods of everything.
> That makes following the code very hard. Please inline single-used
> methods in callers wherever possible to make the code read
> more like GCC code (using GCC API).
I've inlined the two single-use methods you pointed out.
>
> The pass contains an awful lot of heuristics :/ Like last year
> with the interchange pass I would suggest to rip most of it out
> and first lay infrastructure with the cases you can positively
> identify without applying heuristics or "hacks" like stripping
> semantically required casts. That makes it also clear which
> testcases test which code-path. That said, all the analyze
> multiplications/plusses/factors stuff was extremely hard to review
> and I have no overall picture why this is all so complicated or
> necessary.
>
> > Tested on aarch64-linux-gnu and x86_64-linux-gnu. OK to install?
> >
> > Richard
> >
> >
> > 2018-10-24 Richard Sandiford <richard.sandiford@arm.com>
> >
> > gcc/
> > * doc/invoke.texi (-fversion-loops-for-strides): Document
> > (loop-versioning-group-size, loop-versioning-max-inner-insns)
> > (loop-versioning-max-outer-insns): Document new --params.
> > * Makefile.in (OBJS): Add gimple-loop-versioning.o.
> > * common.opt (fversion-loops-for-strides): New option.
> > * opts.c (default_options_table): Enable fversion-loops-for-strides
> > at -O3.
> > * params.def (PARAM_LOOP_VERSIONING_GROUP_SIZE)
> > (PARAM_LOOP_VERSIONING_MAX_INNER_INSNS)
> > (PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS): New parameters.
> > * passes.def: Add pass_loop_versioning.
> > * timevar.def (TV_LOOP_VERSIONING): New time variable.
> > * tree-ssa-propagate.h
> > (substitute_and_fold_engine::substitute_and_fold): Add an optional
> > block parameter.
> > * tree-ssa-propagate.c
> > (substitute_and_fold_engine::substitute_and_fold): Likewise.
> > When passed, only walk blocks dominated by that block.
> > * tree-vrp.h (range_includes_p): Declare.
> > (range_includes_zero_p): Turn into an inline wrapper around
> > range_includes_p.
> > * tree-vrp.c (range_includes_p): New function, generalizing...
> > (range_includes_zero_p): ...this.
> > * tree-pass.h (make_pass_loop_versioning): Declare.
> > * gimple-loop-versioning.cc: New file.
> >
> > gcc/testsuite/
> > * gcc.dg/loop-versioning-1.c: New test.
> > * gcc.dg/loop-versioning-10.c: Likewise.
> > * gcc.dg/loop-versioning-11.c: Likewise.
> > * gcc.dg/loop-versioning-2.c: Likewise.
> > * gcc.dg/loop-versioning-3.c: Likewise.
> > * gcc.dg/loop-versioning-4.c: Likewise.
> > * gcc.dg/loop-versioning-5.c: Likewise.
> > * gcc.dg/loop-versioning-6.c: Likewise.
> > * gcc.dg/loop-versioning-7.c: Likewise.
> > * gcc.dg/loop-versioning-8.c: Likewise.
> > * gcc.dg/loop-versioning-9.c: Likewise.
> > * gfortran.dg/loop_versioning_1.f90: Likewise.
> > * gfortran.dg/loop_versioning_2.f90: Likewise.
> > * gfortran.dg/loop_versioning_3.f90: Likewise.
> > * gfortran.dg/loop_versioning_4.f90: Likewise.
> > * gfortran.dg/loop_versioning_5.f90: Likewise.
> > * gfortran.dg/loop_versioning_6.f90: Likewise.
> > * gfortran.dg/loop_versioning_7.f90: Likewise.
> > * gfortran.dg/loop_versioning_8.f90: Likewise.
> >
> > Index: gcc/doc/invoke.texi
> > ===================================================================
> > --- gcc/doc/invoke.texi 2018-10-24 14:02:14.000000000 +0100
> > +++ gcc/doc/invoke.texi 2018-10-24 14:02:15.184152693 +0100
> > @@ -7934,7 +7934,8 @@ by @option{-O2} and also turns on the fo
> > -fvect-cost-model @gol
> > -ftree-partial-pre @gol
> > -fpeel-loops @gol
> > --fipa-cp-clone}
> > +-fipa-cp-clone @gol
> > +-fversion-loops-for-strides}
> >
> > @item -O0
> > @opindex O0
> > @@ -10358,6 +10359,29 @@ for one side of the iteration space and
> > Move branches with loop invariant conditions out of the loop, with duplicates
> > of the loop on both branches (modified according to result of the condition).
> >
> > +@item -fversion-loops-for-strides
> > +@opindex fversion-loops-for-strides
> > +If a loop iterates over an array with a variable stride, create another
> > +version of the loop that assumes the stride is always 1. For example:
> > +
> > +@smallexample
> > +for (int i = 0; i < n; ++i)
> > + x[i * stride] = @dots{};
> > +@end smallexample
> > +
> > +becomes:
> > +
> > +@smallexample
> > +if (stride == 1)
> > + for (int i = 0; i < n; ++i)
> > + x[i] = @dots{};
> > +else
> > + for (int i = 0; i < n; ++i)
> > + x[i * stride] = @dots{};
> > +@end smallexample
> > +
> > +This is particularly useful for assumed-shape arrays in Fortran.
>
> "where it allows better vectorization assuming contiguous accesses."
>
Done.
> > +
> > @item -ffunction-sections
> > @itemx -fdata-sections
> > @opindex ffunction-sections
> > @@ -11567,6 +11591,20 @@ Hardware autoprefetcher scheduler model
> > Number of lookahead cycles the model looks into; at '
> > ' only enable instruction sorting heuristic.
> >
> > +@item loop-versioning-group-size
> > +Make the loop versioning pass optimize @samp{a[i * index * @var{N}]}
> > +in the same way as it would optimize @samp{a[i * index]} when @var{N}
> > +is less than or equal to this value.
> > +
> > +@item loop-versioning-max-inner-insns
> > +The maximum number of instructions that an inner loop can have
> > +before the loop versioning pass considers it too big to copy.
> > +
> > +@item loop-versioning-max-outer-insns
> > +The maximum number of instructions that an outer loop can have
> > +before the loop versioning pass considers it too big to copy,
> > +discounting any instructions in inner loops that directly benefit
> > +from versioning.
> >
> > @end table
> > @end table
> > Index: gcc/Makefile.in
> > ===================================================================
> > --- gcc/Makefile.in 2018-10-24 14:02:14.000000000 +0100
> > +++ gcc/Makefile.in 2018-10-24 14:02:15.180152727 +0100
> > @@ -1312,6 +1312,7 @@ OBJS = \
> > gimple-laddress.o \
> > gimple-loop-interchange.o \
> > gimple-loop-jam.o \
> > + gimple-loop-versioning.o \
> > gimple-low.o \
> > gimple-pretty-print.o \
> > gimple-ssa-backprop.o \
> > Index: gcc/common.opt
> > ===================================================================
> > --- gcc/common.opt 2018-10-24 14:02:14.000000000 +0100
> > +++ gcc/common.opt 2018-10-24 14:02:15.180152727 +0100
> > @@ -2712,6 +2712,10 @@ fsplit-loops
> > Common Report Var(flag_split_loops) Optimization
> > Perform loop splitting.
> >
> > +fversion-loops-for-strides
> > +Common Report Var(flag_version_loops_for_strides) Optimization
> > +Version loops based on whether indices have a stride of 1.
>
> stride of one.
>
Done.
> > +
> > funwind-tables
> > Common Report Var(flag_unwind_tables) Optimization
> > Just generate unwind tables for exception handling.
> > Index: gcc/opts.c
> > ===================================================================
> > --- gcc/opts.c 2018-10-24 14:02:14.000000000 +0100
> > +++ gcc/opts.c 2018-10-24 14:02:15.184152693 +0100
> > @@ -544,6 +544,7 @@ static const struct default_options defa
> > { OPT_LEVELS_3_PLUS, OPT_fipa_cp_clone, NULL, 1 },
> > { OPT_LEVELS_3_PLUS, OPT_ftree_partial_pre, NULL, 1 },
> > { OPT_LEVELS_3_PLUS, OPT_fpeel_loops, NULL, 1 },
> > + { OPT_LEVELS_3_PLUS, OPT_fversion_loops_for_strides, NULL, 1 },
> >
> > /* -Ofast adds optimizations to -O3. */
> > { OPT_LEVELS_FAST, OPT_ffast_math, NULL, 1 },
> > Index: gcc/params.def
> > ===================================================================
> > --- gcc/params.def 2018-10-24 14:02:14.000000000 +0100
> > +++ gcc/params.def 2018-10-24 14:02:15.184152693 +0100
> > @@ -1360,6 +1360,25 @@ DEFPARAM(PARAM_AVOID_FMA_MAX_BITS,
> > "Maximum number of bits for which we avoid creating FMAs.",
> > 0, 0, 512)
> >
> > +DEFPARAM(PARAM_LOOP_VERSIONING_GROUP_SIZE,
> > + "loop-versioning-group-size",
> > + "The maximum constant N for which accesses of the form x[N * step]"
> > + " are worth versioning for the case in which step is 1",
> > + 4, 1, 0)
> > +
> > +DEFPARAM(PARAM_LOOP_VERSIONING_MAX_INNER_INSNS,
> > + "loop-versioning-max-inner-insns",
> > + "The maximum number of instructions in an inner loop that is being"
> > + " considered for versioning",
> > + 200, 0, 0)
> > +
> > +DEFPARAM(PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS,
> > + "loop-versioning-max-outer-insns",
> > + "The maximum number of instructions in an outer loop that is being"
> > + " considered for versioning, on top of the instructions in inner"
> > + " loops",
> > + 100, 0, 0)
> > +
> > /*
> >
> > Local variables:
> > Index: gcc/passes.def
> > ===================================================================
> > --- gcc/passes.def 2018-10-24 14:02:14.000000000 +0100
> > +++ gcc/passes.def 2018-10-24 14:02:15.184152693 +0100
> > @@ -260,6 +260,7 @@ along with GCC; see the file COPYING3.
> > NEXT_PASS (pass_tree_loop);
> > PUSH_INSERT_PASSES_WITHIN (pass_tree_loop)
> > NEXT_PASS (pass_tree_loop_init);
> > + NEXT_PASS (pass_loop_versioning);
>
> I think neither of the following visible passes benefit from the
> versioning but they might need to duplicate work (and code).
> pass_loop_jam benefits because it needs to do dependence analysis.
> Can you move the pass after loop splitting please?
Done.
>
> > NEXT_PASS (pass_tree_unswitch);
> > NEXT_PASS (pass_scev_cprop);
> > NEXT_PASS (pass_loop_split);
> > Index: gcc/timevar.def
> > ===================================================================
> > --- gcc/timevar.def 2018-10-24 14:02:14.000000000 +0100
> > +++ gcc/timevar.def 2018-10-24 14:02:15.188152659 +0100
> > @@ -234,6 +234,7 @@ DEFTIMEVAR (TV_DSE1 , "
> > DEFTIMEVAR (TV_DSE2 , "dead store elim2")
> > DEFTIMEVAR (TV_LOOP , "loop analysis")
> > DEFTIMEVAR (TV_LOOP_INIT , "loop init")
> > +DEFTIMEVAR (TV_LOOP_VERSIONING , "loop versioning")
> > DEFTIMEVAR (TV_LOOP_MOVE_INVARIANTS , "loop invariant motion")
> > DEFTIMEVAR (TV_LOOP_UNROLL , "loop unrolling")
> > DEFTIMEVAR (TV_LOOP_DOLOOP , "loop doloop")
> > Index: gcc/tree-ssa-propagate.h
> > ===================================================================
> > --- gcc/tree-ssa-propagate.h 2018-10-24 14:02:14.000000000 +0100
> > +++ gcc/tree-ssa-propagate.h 2018-10-24 14:02:15.188152659 +0100
> > @@ -104,7 +104,7 @@ extern void propagate_tree_value_into_st
> > virtual bool fold_stmt (gimple_stmt_iterator *) { return false; }
> > virtual tree get_value (tree) { return NULL_TREE; }
> >
> > - bool substitute_and_fold (void);
> > + bool substitute_and_fold (basic_block = NULL);
>
> I'm not sure I approve of your use of the SSA propagator but
> at least dominance constrains the versioned loop body appropriately...
>
> > bool replace_uses_in (gimple *);
> > bool replace_phi_args_in (gphi *);
> > };
> > Index: gcc/tree-ssa-propagate.c
> > ===================================================================
> > --- gcc/tree-ssa-propagate.c 2018-10-24 14:02:14.000000000 +0100
> > +++ gcc/tree-ssa-propagate.c 2018-10-24 14:02:15.188152659 +0100
> > @@ -1152,6 +1152,10 @@ substitute_and_fold_dom_walker::before_d
> >
> >
> > /* Perform final substitution and folding of propagated values.
> > + Process the whole function if BLOCK is null, otherwise only
> > + process the blocks that BLOCK dominates. In the latter case,
> > + it is the caller's responsibility to ensure that dominator
> > + information is available and up-to-date.
> >
> > PROP_VALUE[I] contains the single value that should be substituted
> > at every use of SSA name N_I. If PROP_VALUE is NULL, no values are
> > @@ -1168,16 +1172,24 @@ substitute_and_fold_dom_walker::before_d
> > Return TRUE when something changed. */
> >
> > bool
> > -substitute_and_fold_engine::substitute_and_fold (void)
> > +substitute_and_fold_engine::substitute_and_fold (basic_block block)
> > {
> > if (dump_file && (dump_flags & TDF_DETAILS))
> > fprintf (dump_file, "\nSubstituting values and folding statements\n\n");
> >
> > memset (&prop_stats, 0, sizeof (prop_stats));
> >
> > - calculate_dominance_info (CDI_DOMINATORS);
> > + /* Don't call calculate_dominance_info when iterating over a subgraph.
> > + Callers that are using the interface this way are likely to want to
> > + iterate over several disjoint subgraphs, and it would be expensive
> > + in enable-checking builds to revalidate the whole dominance tree
> > + each time. */
> > + if (block)
> > + gcc_assert (dom_info_state (CDI_DOMINATORS));
> > + else
> > + calculate_dominance_info (CDI_DOMINATORS);
> > substitute_and_fold_dom_walker walker (CDI_DOMINATORS, this);
> > - walker.walk (ENTRY_BLOCK_PTR_FOR_FN (cfun));
> > + walker.walk (block ? block : ENTRY_BLOCK_PTR_FOR_FN (cfun));
> >
> > /* We cannot remove stmts during the BB walk, especially not release
> > SSA names there as that destroys the lattice of our callers.
> > Index: gcc/tree-vrp.h
> > ===================================================================
> > --- gcc/tree-vrp.h 2018-10-24 14:02:14.000000000 +0100
> > +++ gcc/tree-vrp.h 2018-10-24 14:02:15.188152659 +0100
> > @@ -86,7 +86,7 @@ extern void register_edge_assert_for (tr
> > tree, tree, vec<assert_info> &);
> > extern bool stmt_interesting_for_vrp (gimple *);
> > extern void set_value_range_to_varying (value_range *);
> > -extern bool range_includes_zero_p (const value_range *);
> > +extern bool range_includes_p (const value_range *, HOST_WIDE_INT);
> > extern bool infer_value_range (gimple *, tree, tree_code *, tree *);
> >
> > extern void set_value_range_to_nonnull (value_range *, tree);
> > @@ -122,4 +122,13 @@ extern int value_inside_range (tree, tre
> > extern tree get_single_symbol (tree, bool *, tree *);
> > extern void maybe_set_nonzero_bits (edge, tree);
> > extern value_range_type determine_value_range (tree, wide_int *, wide_int *);
> > +
> > +/* Return TRUE if *VR includes the value zero. */
> > +
> > +inline bool
> > +range_includes_zero_p (const value_range *vr)
> > +{
> > + return range_includes_p (vr, 0);
> > +}
> > +
> > #endif /* GCC_TREE_VRP_H */
> > Index: gcc/tree-vrp.c
> > ===================================================================
> > --- gcc/tree-vrp.c 2018-10-24 14:02:14.000000000 +0100
> > +++ gcc/tree-vrp.c 2018-10-24 14:02:15.188152659 +0100
> > @@ -844,10 +844,10 @@ value_inside_range (tree val, tree min,
> > }
> >
> >
> > -/* Return TRUE if *VR includes the value zero. */
> > +/* Return TRUE if *VR includes the value X. */
> >
> > bool
> > -range_includes_zero_p (const value_range *vr)
> > +range_includes_p (const value_range *vr, HOST_WIDE_INT x)
> > {
> > if (vr->type == VR_VARYING)
> > return true;
> > @@ -856,13 +856,13 @@ range_includes_zero_p (const value_range
> > if (vr->type == VR_UNDEFINED)
> > return true;
> >
> > - tree zero = build_int_cst (TREE_TYPE (vr->min), 0);
> > + tree x_cst = build_int_cst (TREE_TYPE (vr->min), x);
> > if (vr->type == VR_ANTI_RANGE)
> > {
> > - int res = value_inside_range (zero, vr->min, vr->max);
> > + int res = value_inside_range (x_cst, vr->min, vr->max);
> > return res == 0 || res == -2;
> > }
> > - return value_inside_range (zero, vr->min, vr->max) != 0;
> > + return value_inside_range (x_cst, vr->min, vr->max) != 0;
> > }
> >
> > /* If *VR has a value rante that is a single constant value return that,
> > Index: gcc/tree-pass.h
> > ===================================================================
> > --- gcc/tree-pass.h 2018-10-24 14:02:14.000000000 +0100
> > +++ gcc/tree-pass.h 2018-10-24 14:02:15.188152659 +0100
> > @@ -362,6 +362,7 @@ extern gimple_opt_pass *make_pass_fix_lo
> > extern gimple_opt_pass *make_pass_tree_loop (gcc::context *ctxt);
> > extern gimple_opt_pass *make_pass_tree_no_loop (gcc::context *ctxt);
> > extern gimple_opt_pass *make_pass_tree_loop_init (gcc::context *ctxt);
> > +extern gimple_opt_pass *make_pass_loop_versioning (gcc::context *ctxt);
> > extern gimple_opt_pass *make_pass_lim (gcc::context *ctxt);
> > extern gimple_opt_pass *make_pass_linterchange (gcc::context *ctxt);
> > extern gimple_opt_pass *make_pass_tree_unswitch (gcc::context *ctxt);
> > Index: gcc/gimple-loop-versioning.cc
> > ===================================================================
> > --- /dev/null 2018-09-14 11:16:31.122530289 +0100
> > +++ gcc/gimple-loop-versioning.cc 2018-10-24 14:02:15.184152693 +0100
> > @@ -0,0 +1,1417 @@
> > +/* Loop versioning pass.
> > + Copyright (C) 2018 Free Software Foundation, Inc.
> > +
> > +This file is part of GCC.
> > +
> > +GCC is free software; you can redistribute it and/or modify it
> > +under the terms of the GNU General Public License as published by the
> > +Free Software Foundation; either version 3, or (at your option) any
> > +later version.
> > +
> > +GCC is distributed in the hope that it will be useful, but WITHOUT
> > +ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
> > +FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
> > +for more details.
> > +
> > +You should have received a copy of the GNU General Public License
> > +along with GCC; see the file COPYING3. If not see
> > +<http://www.gnu.org/licenses/>. */
> > +
> > +#include "config.h"
> > +#include "system.h"
> > +#include "coretypes.h"
> > +#include "backend.h"
> > +#include "tree.h"
> > +#include "gimple.h"
> > +#include "gimple-iterator.h"
> > +#include "tree-pass.h"
> > +#include "gimplify-me.h"
> > +#include "cfgloop.h"
> > +#include "tree-ssa-loop.h"
> > +#include "ssa.h"
> > +#include "tree-scalar-evolution.h"
> > +#include "tree-chrec.h"
> > +#include "tree-ssa-loop-ivopts.h"
> > +#include "fold-const.h"
> > +#include "tree-ssa-propagate.h"
> > +#include "tree-inline.h"
> > +#include "domwalk.h"
> > +#include "alloc-pool.h"
> > +#include "vr-values.h"
> > +#include "gimple-ssa-evrp-analyze.h"
> > +#include "gimple-pretty-print.h"
> > +#include "params.h"
> > +
> > +/* This pass looks for loops that could be simplified if certain loop
> > + invariant conditions were true. It is effectively a form of loop
> > + splitting in which the pass produces the split conditions itself,
> > + instead of using ones that are already present in the IL.
> > +
> > + Versioning for when strides are 1
> > + ---------------------------------
> > +
> > + At the moment the only thing the pass looks for are memory references
> > + like:
> > +
> > + for (auto i : ...)
> > + ...x[i * stride]...
> > +
> > + It considers changing such loops to:
> > +
> > + if (stride == 1)
> > + for (auto i : ...) [A]
> > + ...x[i]...
> > + else
> > + for (auto i : ...) [B]
> > + ...x[i * stride]...
> > +
> > + This can have several benefits:
> > +
> > + (1) [A] is often easier or cheaper to vectorize than [B].
> > +
> > + (2) The scalar code in [A] is simpler than the scalar code in [B]
> > + (if the loops cannot be vectorized or need an epilogue loop).
> > +
> > + (3) We might recognize [A] as a pattern, such as a memcpy or memset.
> > +
> > + (4) [A] has simpler address evolutions, which can help other passes
> > + like loop interchange.
> > +
> > + The optimization is particularly useful for assumed-shape arrays in
> > + Fortran, where the stride of the innermost dimension depends on the
> > + array descriptor but is often equal to 1 in practice. For example:
> > +
> > + subroutine f1(x)
> > + real :: x(:)
> > + x(:) = 100
> > + end subroutine f1
> > +
> > + generates the equivalent of:
> > +
> > + raw_stride = *x.dim[0].stride;
> > + stride = raw_stride != 0 ? raw_stride : 1;
> > + x_base = *x.data;
> > + ...
> > + tmp1 = stride * S;
> > + tmp2 = tmp1 - stride;
> > + *x_base[tmp2] = 1.0e+2;
> > +
> > + but in the common case that stride == 1, the last three statements
> > + simplify to:
> > +
> > + tmp3 = S + -1;
> > + *x_base[tmp3] = 1.0e+2;
> > +
> > + The optimization is in principle very simple. The difficult parts are:
> > +
> > + (a) deciding which parts of a general address calculation correspond
> > + to the inner dimension of an array, since this usually isn't explicit
> > + in the IL, and for C often isn't even explicit in the source code
> > +
> > + (b) estimating when the transformation is worthwhile
> > +
> > + Structure
> > + ---------
> > +
> > + The pass has four phases:
> > +
> > + (1) Walk through the statements looking for and recording potential
> > + versioning opportunities. Stop if there are none.
> > +
> > + (2) Use context-sensitive range information to see whether any versioning
> > + conditions are impossible in practice. Remove them if so, and stop
> > + if no opportunities remain.
> > +
> > + (We do this only after (1) to keep compile time down when no
> > + versioning opportunities exist.)
> > +
> > + (3) Apply the cost model. Decide which versioning opportunities are
> > + worthwhile and at which nesting level they should be applied.
> > +
> > + (4) Attempt to version all the loops selected by (3), so that:
> > +
> > + for (...)
> > + ...
> > +
> > + becomes:
> > +
> > + if (!cond)
> > + for (...) // Original loop
> > + ...
> > + else
> > + for (...) // New loop
> > + ...
> > +
> > + Use the version condition COND to simplify the new loop. */
> > +class loop_versioning {
> > +public:
> > + loop_versioning (function *);
> > + ~loop_versioning ();
> > + unsigned int run ();
> > +
> > +private:
> > + /* Information about the versioning we'd like to apply to a loop. */
> > + struct loop_info {
> > + bool worth_versioning_p () const;
> > +
> > + /* True if we've decided not to version this loop. The remaining
> > + fields are meaningless if so. */
> > + bool rejected_p;
> > +
> > + /* True if at least one subloop of this loop benefits from versioning. */
> > + bool subloops_benefit_p;
> > +
> > + /* An estimate of the total number of instructions in the loop,
> > + excluding those in subloops that benefit from versioning. */
> > + unsigned int num_insns;
> > +
> > + /* The outermost loop that can handle all the version checks
> > + described below. */
> > + struct loop *outermost;
> > +
> > + /* We'd like to version the loop for the case in which these
> > + SSA_NAMEs are all equal to 1 at runtime. */
> > + vec<tree> unity_names;
> > +
> > + /* The set of SSA_NAMEs in UNITY_NAMES, keyed off the SSA_NAME_VERSION. */
> > + bitmap_head unity_name_ids;
>
> I wonder why you need both, a vector and a bitmap since you can
> cheaply iterate over the bitmap and get the SSA name via ssa_name (version)?
>
> > + };
> > +
> > + /* Used to walk the dominator tree to find loop versioning conditions
> > + that are always false. */
> > + class lv_dom_walker : public dom_walker
> > + {
> > + public:
> > + lv_dom_walker (loop_versioning &);
> > +
> > + edge before_dom_children (basic_block) FINAL OVERRIDE;
> > + void after_dom_children (basic_block) FINAL OVERRIDE;
> > +
> > + private:
> > + /* The parent pass. */
> > + loop_versioning &m_lv;
> > +
> > + /* Used to build context-dependent range information. */
> > + evrp_range_analyzer m_range_analyzer;
> > + };
> > +
> > + /* Used to simplify statements based on conditions that are established
> > + by the version checks. */
> > + class name_prop : public substitute_and_fold_engine
> > + {
> > + public:
> > + name_prop (loop_info &li) : m_li (li) {}
> > + tree get_value (tree) FINAL OVERRIDE;
> > +
> > + private:
> > + /* Information about the versioning we've performed on the loop. */
> > + loop_info &m_li;
> > + };
> > +
> > + loop_info &get_loop_info (struct loop *loop) { return m_loops[loop->num]; }
> > +
> > + unsigned int max_insns_for_loop (struct loop *);
> > + bool expensive_stmt_p (gimple *);
> > +
> > + void version_for_unity (struct loop *, tree);
> > + bool acceptable_scale_p (tree, poly_uint64);
> > + tree get_step_if_innermost (tree, poly_uint64);
> > + tree extract_step (tree, poly_uint64, tree *);
> > + void analyze_evolution (struct loop *, tree, poly_uint64);
> > + bool analyze_product (struct loop *, gassign *, poly_uint64);
> > + bool analyze_sum_of_products (struct loop *, tree, poly_uint64);
> > + void analyze_pointer (struct loop *, tree, tree);
> > + void analyze_expr (struct loop *, tree);
> > + void analyze_stmt (gimple *);
> > + void analyze_block (basic_block);
> > + bool analyze_blocks ();
> > +
> > + void prune_loop_conditions (struct loop *, vr_values *);
> > + bool prune_conditions ();
> > +
> > + void merge_loop_info (struct loop *, struct loop *);
> > + void add_loop_to_queue (struct loop *);
> > + bool decide_whether_loop_is_versionable (struct loop *);
> > + bool make_versioning_decisions ();
> > +
> > + bool version_loop (struct loop *);
> > + bool implement_versioning_decisions ();
> > +
> > + /* The function we're optimizing. */
> > + function *m_fn;
> > +
> > + /* The obstack to use for all pass-specific bitmaps. */
> > + bitmap_obstack m_obstack;
> > +
> > + /* The number of loops in the function. */
> > + unsigned int m_nloops;
> > +
> > + /* The total number of loop version conditions we've found. */
> > + unsigned int m_num_conditions;
> > +
> > + /* Information about each loop. */
> > + auto_vec<loop_info> m_loops;
> > +
> > + /* The list of loops that we've decided to version. */
> > + auto_vec<struct loop *> m_loops_to_version;
> > +};
> > +
> > +/* If EXPR is an SSA name and not a default definition, return the
> > + defining statement, otherwise return null. */
> > +
> > +static gimple *
> > +maybe_get_stmt (tree expr)
> > +{
> > + if (TREE_CODE (expr) == SSA_NAME && !SSA_NAME_IS_DEFAULT_DEF (expr))
> > + return SSA_NAME_DEF_STMT (expr);
> > + return NULL;
> > +}
> > +
> > +/* Like maybe_get_stmt, but also return null if the defining
> > + statement isn't an assignment. */
> > +
> > +static gassign *
> > +maybe_get_assign (tree expr)
> > +{
> > + return safe_dyn_cast <gassign *> (maybe_get_stmt (expr));
> > +}
> > +
> > +/* If EXPR is an SSA name, look through any casts to see whether the
> > + unconverted value is defined in LOOP by a gassign. Return the
> > + gassign if so, otherwise return null. */
> > +
> > +gassign *
> > +maybe_get_assign_strip_casts (struct loop *loop, tree expr)
> > +{
> > + const unsigned int MAX_NITERS = 4;
> > +
> > + tree type = TREE_TYPE (expr);
> > + for (unsigned int niters = 0; niters < MAX_NITERS; ++niters)
> > + {
> > + gassign *assign = maybe_get_assign (expr);
> > + if (!assign || gimple_bb (assign)->loop_father != loop)
> > + return NULL;
> > + expr = gimple_assign_rhs1 (assign);
> > + if (CONVERT_EXPR_CODE_P (gimple_assign_rhs_code (assign))
> > + && INTEGRAL_TYPE_P (TREE_TYPE (expr)) == INTEGRAL_TYPE_P (type)
> > + && POINTER_TYPE_P (TREE_TYPE (expr)) == POINTER_TYPE_P (type))
> > + ;
> > + else
> > + return assign;
> > + }
> > + return NULL;
> > +}
> > +
> > +/* Strip all conversions of integers from EXPR, regardless of whether
> > + the conversions are nops. This is useful in the context of this pass
> > + because we're not trying to fold or simulate the expression; we just
> > + want to see how it's structured. */
> > +
> > +static tree
> > +strip_casts (tree expr)
> > +{
> > + while (CONVERT_EXPR_P (expr)
> > + && INTEGRAL_TYPE_P (TREE_TYPE (TREE_OPERAND (expr, 0))))
> > + expr = TREE_OPERAND (expr, 0);
> > + return expr;
> > +}
> > +
> > +/* Return true if we want to version the loop, i.e. if we have a
> > + specific reason for doing so and no specific reason not to. */
> > +
> > +bool
> > +loop_versioning::loop_info::worth_versioning_p () const
> > +{
> > + return !rejected_p && (!unity_names.is_empty () || subloops_benefit_p);
> > +}
> > +
> > +loop_versioning::lv_dom_walker::lv_dom_walker (loop_versioning &lv)
> > + : dom_walker (CDI_DOMINATORS), m_lv (lv)
> > +{
> > +}
> > +
> > +/* Process BB before processing the blocks it dominates. */
> > +
> > +edge
> > +loop_versioning::lv_dom_walker::before_dom_children (basic_block bb)
> > +{
> > + m_range_analyzer.enter (bb);
> > +
> > + if (bb == bb->loop_father->header)
> > + m_lv.prune_loop_conditions (bb->loop_father,
> > + m_range_analyzer.get_vr_values ());
> > +
> > + for (gimple_stmt_iterator si = gsi_start_bb (bb); !gsi_end_p (si);
> > + gsi_next (&si))
> > + m_range_analyzer.record_ranges_from_stmt (gsi_stmt (si), false);
> > +
> > + return NULL;
> > +}
> > +
> > +/* Process BB after processing the blocks it dominates. */
> > +
> > +void
> > +loop_versioning::lv_dom_walker::after_dom_children (basic_block bb)
> > +{
> > + m_range_analyzer.leave (bb);
> > +}
> > +
> > +/* Decide whether to replace VAL with a new value in a versioned loop.
> > + Return the new value if so, otherwise return null. */
> > +
> > +tree
> > +loop_versioning::name_prop::get_value (tree val)
> > +{
> > + if (TREE_CODE (val) == SSA_NAME
> > + && bitmap_bit_p (&m_li.unity_name_ids, SSA_NAME_VERSION (val)))
> > + return build_one_cst (TREE_TYPE (val));
> > + return NULL_TREE;
> > +}
> > +
> > +/* Initialize the structure to optimize FN. */
> > +
> > +loop_versioning::loop_versioning (function *fn)
> > + : m_fn (fn),
> > + m_nloops (number_of_loops (fn)),
> > + m_num_conditions (0)
> > +{
> > + bitmap_obstack_initialize (&m_obstack);
> > +
> > + m_loops.safe_grow_cleared (m_nloops);
> > + for (unsigned int i = 0; i < m_nloops; ++i)
> > + {
> > + m_loops[i].outermost = get_loop (m_fn, 0);
> > + bitmap_initialize (&m_loops[i].unity_name_ids, &m_obstack);
> > + }
> > +}
> > +
> > +loop_versioning::~loop_versioning ()
> > +{
> > + for (unsigned int i = 0; i < m_nloops; ++i)
> > + m_loops[i].unity_names.release ();
> > + bitmap_obstack_release (&m_obstack);
> > +}
> > +
> > +/* Return the maximum number of instructions allowed in LOOP before
> > + it becomes too big for versioning.
> > +
> > + There are separate limits for inner and outer loops. The limit for
> > + inner loops applies only to loops that benefit directly from versioning.
> > + The limit for outer loops applies to all code in the outer loop and
> > + its subloops that *doesn't* benefit directly from versioning; such code
> > + would be "taken along for the ride". The idea is that if the cost of
> > + the latter is small, it is better to version outer loops rather than
> > + inner loops, both to reduce the number of repeated checks and to enable
> > + more of the loop nest to be optimized as a natural nest (e.g. by loop
> > + interchange or outer-loop vectorization). */
> > +
> > +unsigned int
> > +loop_versioning::max_insns_for_loop (struct loop *loop)
> > +{
> > + return (loop->inner
> > + ? PARAM_VALUE (PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS)
> > + : PARAM_VALUE (PARAM_LOOP_VERSIONING_MAX_INNER_INSNS));
> > +}
> > +
> > +/* Return true if for cost reasons we should avoid versioning any loop
> > + that contains STMT.
> > +
> > + Note that we don't need to check whether versioning is invalid for
> > + correctness reasons, since the versioning process does that for us.
> > + The conditions involved are too rare to be worth duplicating here. */
> > +
> > +bool
> > +loop_versioning::expensive_stmt_p (gimple *stmt)
> > +{
> > + if (gcall *call = dyn_cast <gcall *> (stmt))
> > + /* Assume for now that the time spent in an "expensive" call would
> > + overwhelm any saving from versioning. */
> > + return !gimple_inexpensive_call_p (call);
> > + return false;
> > +}
> > +
> > +/* Record that we want to version LOOP for the case in which SSA name NAME
> > + is equal to 1. We already know that NAME is invariant in LOOP. */
> > +
> > +void
> > +loop_versioning::version_for_unity (struct loop *loop, tree name)
> > +{
> > + loop_info &li = get_loop_info (loop);
> > +
> > + if (bitmap_set_bit (&li.unity_name_ids, SSA_NAME_VERSION (name)))
> > + {
> > + /* This is the first time we've wanted to version LOOP for NAME. */
> > + li.unity_names.safe_push (name);
> > +
> > + /* Keep track of the outermost loop that can handle all versioning
> > + checks in LI. */
> > + struct loop *outermost
> > + = outermost_invariant_loop_for_expr (loop, name);
> > + if (loop_depth (li.outermost) < loop_depth (outermost))
> > + li.outermost = outermost;
> > +
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + {
> > + fprintf (dump_file, ";; Want to version loop %d (depth %d)"
> > + " for when ", loop->num, loop_depth (loop));
> > + print_generic_expr (dump_file, name, TDF_SLIM);
> > + fprintf (dump_file, " == 1");
>
> Since you are writing a new pass you want to use the new dump interface.
>
> if (dump_enabled_p ())
> dump_printf (MSG_NOTE, ";; Want to version loop %d (depth %d)"
> " for when %E == 1", loop->num, loop_depth (loop), name);
> ...
>
> it's much nicer to be able to use %E/%G than separate calls for the
> tree parts.
Done. I've used the new dump interface throughout the new file, it is much cleaner
and more pleasant to use. I've used %T for printing tree operands.
>
> > + if (outermost == loop)
> > + fprintf (dump_file, "; cannot hoist check further");
> > + else
> > + {
> > + fprintf (dump_file, "; could hoist check to loop %d (depth %d)",
> > + outermost->num, loop_depth (outermost));
> > + if (loop_depth (li.outermost) > loop_depth (outermost))
> > + fprintf (dump_file, ", but that's further than"
> > + " other checks allow");
> > + }
> > + fprintf (dump_file, "\n");
> > + }
> > +
> > + m_num_conditions += 1;
> > + }
> > + else
> > + {
> > + /* This is a duplicate request. */
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + {
> > + fprintf (dump_file, ";; Already want to version loop for when ");
> > + print_generic_expr (dump_file, name, TDF_SLIM);
> > + fprintf (dump_file, " == 1\n");
> > + }
> > + }
> > +}
> > +
> > +/* Return true if in principle it is worth versioning an index fragment of
> > + the form:
> > +
> > + (i * b * SCALE) / FACTOR
> > +
> > + for the case in which b == 1. */
> > +
> > +bool
> > +loop_versioning::acceptable_scale_p (tree scale, poly_uint64 factor)
> > +{
> > + /* See whether SCALE is a constant multiple of FACTOR, and if the
> > + multiple is small enough for us to treat it as a potential grouped
> > + access. For example:
> > +
> > + for (auto i : ...)
> > + y[i] = f (x[4 * i * stride],
> > + x[4 * i * stride + 1],
> > + x[4 * i * stride + 2]);
> > +
> > + would benefit from versioning for the case in which stride == 1.
> > + High multiples of i * stride are less likely to benefit, and could
> > + indicate a simulated multi-dimensional array.
> > +
> > + This is just a heuristic, to avoid having to do expensive group
> > + analysis of the data references in a loop. */
> > + poly_uint64 const_scale;
> > + unsigned int multiple;
> > + if (poly_int_tree_p (scale, &const_scale)
> > + && constant_multiple_p (const_scale, factor, &multiple))
> > + {
> > + unsigned int maxval = PARAM_VALUE (PARAM_LOOP_VERSIONING_GROUP_SIZE);
> > + return IN_RANGE (multiple, 1, maxval);
>
> Hmm. So you _do_ want to version sth like
>
> struct X { int i; int j; } a[2048];
>
> for (int i = start; i < end; ++i)
> a[i*s].i = 1;
>
> ? That is, even with s == 1 the accesses will not become contiguous?
> OK, I suppose that's because you are looking at a stmt in isolation
> and another stmt may access a[i*s].j here.
>
> That is, would it be a future improvement to run sth like the
> vectorizers group access analysis on the references and perform
> this check on whole groups then, possibly better being able to
> constrain what is now the magic parameter PARAM_LOOP_VERSIONING_GROUP_SIZE?
>
> I guess you tried to constrain it to the stmts access size but of course
> that fails short of handling later SLP vectorized cases.
>
> > + }
> > +
> > + return false;
> > +}
> > +
> > +/* Decide whether an index fragment of the form:
> > +
> > + (i * STEP) / FACTOR
> > +
> > + is likely to be for an innermost dimension. If we think it is,
> > + return one of the constant values that it could have (returning 1 if
> > + that's a possibility). If think it isn't, return null. Otherwise
> > + return STEP, to indicate that it might or might not be an inner
> > + dimension. */
> > +
> > +tree
> > +loop_versioning::get_step_if_innermost (tree step, poly_uint64 factor)
> > +{
> > + const unsigned int MAX_NITERS = 8;
> > +
> > + tree likely = NULL_TREE;
> > + tree unlikely = NULL_TREE;
> > + tree worklist[MAX_NITERS];
> > + unsigned int length = 0;
> > + worklist[length++] = step;
> > + for (unsigned int i = 0; i < length; ++i)
> > + {
> > + tree expr = worklist[i];
> > +
> > + if (TREE_CONSTANT (expr))
>
> ? CONSTANT_CLASS_P (expr)?
I've done that and it seems to work fine.
>
> > + {
> > + /* See if multiplying by EXPR applies a scale that would be
> > + consistent with an individual access or a small grouped
> > + access. */
> > + if (acceptable_scale_p (expr, factor))
> > + {
> > + likely = expr;
> > + if (integer_onep (expr))
> > + break;
> > + }
> > + else
> > + unlikely = expr;
> > + continue;
> > + }
> > +
> > + /* Otherwise we can only handle SSA names. */
> > + gimple *stmt = maybe_get_stmt (expr);
> > + if (!stmt)
> > + continue;
> > +
> > + /* If EXPR is set by a PHI node, queue its arguments in case
> > + we find one that is consistent with an inner dimension.
> > +
> > + An important instance of this is the Fortran handling of array
> > + descriptors, which calculates the stride of the inner dimension
> > + using a PHI equivalent of:
> > +
> > + raw_stride = a.dim[0].stride;
> > + stride = raw_stride != 0 ? raw_stride : 1;
> > +
> > + (Strides for outer dimensions do not treat 0 specially.) */
> > + if (gphi *phi = dyn_cast <gphi *> (stmt))
> > + {
> > + unsigned int nargs = gimple_phi_num_args (phi);
> > + for (unsigned int j = 0; j < nargs && length < MAX_NITERS; ++j)
> > + worklist[length++] = gimple_phi_arg_def (phi, j);
> > + continue;
> > + }
>
> So only PHIs seed the worklist.
>
> > + /* If the value is set by an assignment, expect it to be read from
> > + memory (such as an array descriptor) rather than be calculated. */
> > + if (gassign *assign = dyn_cast <gassign *> (stmt))
> > + {
> > + if (gimple_assign_single_p (assign)
> > + && is_gimple_lvalue (gimple_assign_rhs1 (assign)))
>
> Please do not use is_gimple_lvalue, that's supposed to be a gimplifier-only
> predicate. Do you want to use gimple_assign_load_p here?
gimple_assign_load_p works fine here. I've used that.
>
> > + continue;
> > +
> > + unlikely = expr;
> > + }
> > +
> > + /* Things like calls don't really tell us anything. */
> > + }
>
> So I don't understand the above fully but it looks like plain
> function arguments STEP will never be 'likely'?
>
> That is, I'd appreciate some more comments of what SSA def
> chains we walk and what we look for in the end.
>
> > + if (likely)
> > + {
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + fprintf (dump_file, "likely to be innermost dimension\n");
> > + return likely;
> > + }
> > +
> > + if (unlikely)
> > + {
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + fprintf (dump_file, "probably not innermost dimension\n");
> > + return NULL_TREE;
> > + }
> > +
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + {
> > + if (maybe_ne (factor, 1U))
> > + fprintf (dump_file, "didn't find expected scaling factor\n");
> > + else
> > + fprintf (dump_file, "no information about value\n");
> > + }
> > + return step;
> > +}
> > +
> > +/* STEP appears in an index fragment of the form:
> > +
> > + {..., +, STEP}_n / FACTOR
> > +
> > + Remove any conversions and grouping or scaling factors from STEP and
> > + return the underlying index step. Set *STEP_IF_INNERMOST to the value
> > + returned by get_step_if_innermost. */
> > +
> > +tree
> > +loop_versioning::extract_step (tree step, poly_uint64 factor,
> > + tree *step_if_innermost)
> > +{
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + {
> > + fprintf (dump_file, ";; step ");
> > + print_generic_expr (dump_file, step, TDF_SLIM);
> > + fprintf (dump_file, ": ");
> > + }
> > +
> > + step = strip_casts (step);
> > +
> > + /* Peel any scaling, which generally happens after conversion to
> > + pointer width. For example, on LP64 systems:
> > +
> > + int *x, i, stride;
> > + ... x[4 * i * stride] ...;
> > +
> > + multiplies i * stride by 4 using ints, then widens the result
> > + to pointer width before multiplying by sizeof (int). */
> > + poly_uint64 scale;
> > + if (TREE_CODE (step) == MULT_EXPR
> > + && poly_int_tree_p (TREE_OPERAND (step, 1), &scale)
> > + && known_eq (scale, factor))
> > + {
> > + step = strip_casts (TREE_OPERAND (step, 0));
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + fprintf (dump_file, "scaled, ");
> > + factor = 1;
> > + }
>
> So I think the extra constant scale may or may not be in CHREC_RIGHT
> depending on whether the expression was hoisted out of the loop as
> invariant or implicit in sth like an ARRAY_REF.
>
> > + /* Peel any grouping multiple; see acceptable_scale_p for details. */
> > + if (TREE_CODE (step) == MULT_EXPR
> > + && acceptable_scale_p (TREE_OPERAND (step, 1), factor))
> > + {
> > + step = strip_casts (TREE_OPERAND (step, 0));
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + fprintf (dump_file, "%sgrouped, ",
> > + maybe_ne (factor, 1U) ? "scaled, " : "");
> > + factor = 1;
> > + }
>
> Which means I'm not sure why there are (only) two MULTs being
> looked for and only one has acceptable_scale_p applied...
>
> > + *step_if_innermost = get_step_if_innermost (step, factor);
>
> I know some people like to factor things but get_step_if_innermost
> is called once if I see correctly - it makes understanding and
> review harder to dissect things :/
>
> > + return step;
> > +}
> > +
> > +/* Analyze the evolution of index fragment EXPR / FACTOR in LOOP and its
> > + containing loops to see whether any part of it could be simplified
> > + by versioning. Register the versioning opportunities if so. */
> > +
> > +void
> > +loop_versioning::analyze_evolution (struct loop *loop, tree expr,
> > + poly_uint64 factor)
>
> I find the name of this function confusing given it's nearly
> the same as analyze_scalar_evolution.
I've renamed it to analyze_index_evolution (though not sure if it's a much better name, naming things is notoriously hard).
>
> > +{
> > + const unsigned int MAX_NSPLIT = 8;
> > +
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + {
> > + fprintf (dump_file, ";; Analyzing use of ");
> > + print_generic_expr (dump_file, expr, TDF_SLIM);
> > + if (maybe_ne (factor, 1U))
> > + {
> > + fprintf (dump_file, " (which addresses ");
> > + print_dec (factor, dump_file);
> > + fprintf (dump_file, " bytes)");
> > + }
> > + fprintf (dump_file, " in loop %d (depth %d)\n",
> > + loop->num, loop_depth (loop));
> > + }
> > +
> > + /* The main problem we have here is that we cannot assume that the
> > + innermost loop iterates over the innermost dimension of an array.
> > + Accidentally adding versioning checks for outer dimensions would
> > + cause the version condition to be false, which as well as bloating
> > + the code would defeat loop versioning benefits for other accesses.
> > +
> > + Unfortunately all we usually see at this stage is general address
> > + arithmetic, with no positive way of identifying how many dimensions
> > + an array access has and which multiplication factors in the address
> > + expression correspond to which array dimensions. In C code this is
> > + often not even explicit in the source, since variable-sized multi-
> > + dimensional arrays are often simulated using one-dimensional arrays.
> > +
> > + The three main ways in which we deal with this are:
> > +
> > + - use heuristics that positively identify steps that are likely
> > + to represent the inner dimension.
> > +
> > + - use heuristics that positively identify steps that are unlikely
> > + to represent the inner dimension.
> > +
> > + - if a part of EXPR is invariant in LOOP, analyze its evolution in
> > + the outer loops to see whether we can positively identify any of
> > + it as iterating over the inner dimension. */
> > + tree best_step = NULL_TREE;
> > + auto_vec<tree, MAX_NSPLIT> worklist;
> > + worklist.quick_push (expr);
> > + unsigned int nsplit = 0;
> > + while (!worklist.is_empty ())
> > + {
> > + expr = strip_casts (worklist.pop ());
> > + tree_code code = TREE_CODE (expr);
> > +
> > + if (code == POLYNOMIAL_CHREC)
> > + {
> > + /* Analyze the CHREC_RIGHT as the step in an index fragment. */
> > + tree step_if_innermost;
> > + tree step = extract_step (CHREC_RIGHT (expr), factor,
> > + &step_if_innermost);
> > + if (!best_step)
> > + {
> > + /* This is the outermost chrec for the original expression.
> > + It's not worth carrying on if the step isn't versionable,
> > + or if we're pretty sure it's not for the inner dimension. */
> > + if (!step_if_innermost
> > + || TREE_CODE (step) != SSA_NAME
> > + || !expr_invariant_in_loop_p (loop, step))
> > + return;
> > +
> > + best_step = step;
> > +
> > + /* We should version for STEP == 1 if we know that that can be
> > + true under some circumstances. */
> > + if (integer_onep (step_if_innermost))
> > + break;
> > +
> > + /* Bail out if this appears to be the step for the innermost
> > + dimension, but isn't likely to be 1.
> > +
> > + ??? We could instead version for when it equals
> > + STEP_IF_INNERMOST, but it's not likely to have as much
> > + benefit as versioning for 1. */
> > + if (step_if_innermost != step)
> > + return;
> > + }
> > + else
> > + {
> > + /* This is an inner chrec. If it looks like it iterates over
> > + the innermost dimension, abort any attempt to version for
> > + the outermost chrec (which if we reach here wasn't itself
> > + obviously iterating over the innermost dimension). */
> > + if (step_if_innermost && TREE_CONSTANT (step_if_innermost))
> > + return;
> > + }
> > + worklist.quick_push (CHREC_LEFT (expr));
> > + continue;
> > + }
> > +
> > + /* Analyze parts of a queued CHREC_LEFT. This is better than just
> > + analyzing the evolution of the whole expression since the value
> > + could include a mixture of analyzable and unanalyzable elements.
> > + Use NSPLIT to count cases in which we add more expressions to
> > + analyze, as opposed to just simplifying the existing one. */
> > + if (code == PLUS_EXPR || code == POINTER_PLUS_EXPR || code == MINUS_EXPR)
> > + {
> > + worklist.quick_push (TREE_OPERAND (expr, 0));
> > + if (nsplit++ < MAX_NSPLIT)
> > + worklist.quick_push (TREE_OPERAND (expr, 1));
> > + continue;
> > + }
> > + if (code == MULT_EXPR)
> > + {
> > + tree op0 = strip_casts (TREE_OPERAND (expr, 0));
> > + tree op1 = TREE_OPERAND (expr, 1);
> > + if (TREE_CODE (op0) == PLUS_EXPR || TREE_CODE (op0) == MINUS_EXPR)
> > + {
> > + tree type = TREE_TYPE (expr);
> > + tree op00 = fold_convert (type, TREE_OPERAND (op0, 0));
> > + worklist.quick_push (fold_build2 (MULT_EXPR, type, op00, op1));
> > + if (nsplit++ < MAX_NSPLIT)
> > + {
> > + tree op01 = fold_convert (type, TREE_OPERAND (op0, 1));
> > + worklist.quick_push (fold_build2 (MULT_EXPR, type,
> > + op01, op1));
> > + }
> > + continue;
> > + }
> > + }
> > +
> > + /* If EXPR is invariant in LOOP, analyze it wrt the innermost loop
> > + for which it could evolve (i.e. the loop containing the outermost
> > + one for which EXPR is invariant). */
>
> This isn't how analyze_scalar_evolution works - you _always_ have to
> feed the innermost loop that the expression is used in (the context).
> Then you instantiate the result in the outermost loop of the nest you
> are interested in. Otherwise you get garbage.
>
> It looks like you are re-analyzing SSA names in the evolution - that's
> odd and shouldn't be necessary (but you forget to instantiate, so...).
>
> I suggest to move the analyze_scalar_evolution part out of the worklist
> loop where you are sure you have an SSA name.
>
> > + struct loop *wrt_loop = outermost_invariant_loop_for_expr (loop, expr);
> > + if (wrt_loop)
> > + {
> > + wrt_loop = loop_outer (wrt_loop);
> > + if (!wrt_loop)
> > + continue;
> > + }
> > + else
> > + wrt_loop = loop;
> > + tree evolution = analyze_scalar_evolution (wrt_loop, expr);
> > + tree chrec = strip_casts (evolution);
> > + if (TREE_CODE (chrec) == POLYNOMIAL_CHREC)
> > + {
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + {
> > + fprintf (dump_file, ";; evolution of ");
> > + print_generic_expr (dump_file, expr, TDF_SLIM);
> > + fprintf (dump_file, " in loop %d (depth %d): ",
> > + wrt_loop->num, loop_depth (wrt_loop));
> > + print_generic_expr (dump_file, evolution, TDF_SLIM);
> > + fprintf (dump_file, "\n");
> > + }
> > + worklist.quick_push (chrec);
> > + }
> > + else
> > + {
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + {
> > + fprintf (dump_file, ";; cannot analyze ");
> > + print_generic_expr (dump_file, expr, TDF_SLIM);
> > + fprintf (dump_file, " any further\n");
> > + }
> > + }
> > + }
> > + if (best_step)
> > + version_for_unity (loop, best_step);
> > +}
> > +
> > +/* Analyze multiplication MULT to see whether we can identify "gather-like"
> > + versioning opportunities such as:
> > +
> > + for (int i = 0; i < n; ++i)
> > + res += a[index[i] * stride];
> > +
> > + Return true if there was a versioning opportunity.
> > +
> > + LOOP is the loop that contains MULT. Dividing the value of MULT
> > + by FACTOR converts it into an element index. */
> > +
> > +bool
> > +loop_versioning::analyze_product (struct loop *loop, gassign *mult,
> > + poly_uint64 factor)
> > +{
> > + /* Record the original LHS for the dump message below. */
> > + tree lhs = gimple_assign_lhs (mult);
> > +
> > + /* Peel any scaling, which generally happens after conversion to
> > + pointer width. For example, on LP64 systems:
> > +
> > + int *x, i, stride;
> > + ... x[4 * i * stride] ...;
> > +
> > + multiplies i * stride by 4 using ints, then widens the result
> > + to pointer width before multiplying by sizeof (int). */
> > + poly_uint64 scale;
> > + if (poly_int_tree_p (gimple_assign_rhs2 (mult), &scale)
> > + && known_eq (scale, factor))
> > + {
> > + mult = maybe_get_assign_strip_casts (loop, gimple_assign_rhs1 (mult));
> > + if (!mult || gimple_assign_rhs_code (mult) != MULT_EXPR)
> > + return false;
> > + factor = 1;
> > + }
> > +
> > + /* Peel any grouping multiple; see acceptable_scale_p for details. */
> > + if (acceptable_scale_p (gimple_assign_rhs2 (mult), factor))
> > + {
> > + mult = maybe_get_assign_strip_casts (loop, gimple_assign_rhs1 (mult));
> > + if (!mult || gimple_assign_rhs_code (mult) != MULT_EXPR)
> > + return false;
> > + }
> > + else if (maybe_ne (factor, 1U))
> > + /* We expect to see a scaling multiplication when FACTOR is > 1. */
> > + return false;
> > +
> > + /* See whether this multiplication involves a loop-invariant SSA name
> > + and a non-invariant SSA name. */
> > + tree op1 = gimple_assign_rhs1 (mult);
> > + tree op2 = gimple_assign_rhs2 (mult);
> > + if (TREE_CODE (op1) != SSA_NAME || TREE_CODE (op2) != SSA_NAME)
> > + return false;
> > +
> > + bool invariant1_p = expr_invariant_in_loop_p (loop, op1);
> > + bool invariant2_p = expr_invariant_in_loop_p (loop, op2);
> > + if (invariant1_p == invariant2_p)
> > + return false;
> > +
> > + /* Make sure that the invariant is OP1 and the other operand is OP2. */
> > + if (invariant2_p)
> > + std::swap (op1, op2);
> > +
> > + /* Bail out if OP2 can be analyzed as an evolution; we want to use
> > + analyze_evolution in that case instead. There's no point trying
> > + hard to avoid repeating the call to analyze_scalar_evolution since
> > + that function does its own caching. */
> > + if (tree_contains_chrecs (analyze_scalar_evolution (loop, op2), NULL))
>
> Don't you want !chrec_contains_undetermined (...) instead? I wonder what
> is missing to allow all interesting expressions to be analyzed by
> SCEV?
I've tried that and with !chrec_contains_undetermined we fail to version the loop-versioning-10.c testcase:
for (int i = 0; i < n; ++i)
res += x[index[i] * step];
>
> > + return false;
> > +
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + {
> > + fprintf (dump_file, ";; Address fragment ");
> > + print_generic_expr (dump_file, lhs, TDF_SLIM);
> > + fprintf (dump_file, " multiplies invariant ");
> > + print_generic_expr (dump_file, op1, TDF_SLIM);
> > + fprintf (dump_file, " by ");
> > + print_generic_expr (dump_file, op2, TDF_SLIM);
> > + fprintf (dump_file, ", which isn't a scalar evolution\n");
> > + }
> > +
> > + version_for_unity (loop, op1);
> > + return true;
> > +}
> > +
> > +/* Treat EXPR as a sum of products and apply analyze_product to each of the
> > + products. Return true if one of the products provides a versioning
> > + opportunity. FACTOR is as for analyze_product. */
> > +
> > +bool
> > +loop_versioning::analyze_sum_of_products (struct loop *loop, tree expr,
> > + poly_uint64 factor)
> > +{
>
> This looks awfully close to what tree-affine.c does apart from more
> aggressively stripping conversions? I see all the analysis parts
> are limited and thus "O(1)" but still there's going to be a lot of
> redundancy involved for repeated use of (derived) "IVs"? Wouldn't
> it be good to have a bitmap of already handled SSA_NAMEs to stop
> processing early?
I've used a bitmap of SSA names to limit this.
>
> > + const unsigned int MAX_NITERS = 8;
> > +
> > + tree worklist[MAX_NITERS];
> > + unsigned int length = 0;
> > + worklist[length++] = expr;
> > + for (unsigned int i = 0; i < length; ++i)
> > + {
> > + expr = worklist[i];
> > + gassign *assign = maybe_get_assign_strip_casts (loop, expr);
> > + if (!assign)
> > + continue;
> > +
> > + tree_code code = gimple_assign_rhs_code (assign);
> > + if (code == PLUS_EXPR || code == POINTER_PLUS_EXPR || code == MINUS_EXPR)
>
> POINTER_MINUS_EXPR?
>
> > + {
> > + if (length < MAX_NITERS)
> > + worklist[length++] = gimple_assign_rhs1 (assign);
> > + if (length < MAX_NITERS)
> > + worklist[length++] = gimple_assign_rhs2 (assign);
> > + }
> > + else if (code == MULT_EXPR && analyze_product (loop, assign, factor))
> > + return true;
> > + }
> > + return false;
> > +}
> > +
> > +/* Analyze pointer expression EXPR, which occurs in loop LOOP and which
> > + is used to address a value of type TYPE. */
> > +
> > +void
> > +loop_versioning::analyze_pointer (struct loop *loop, tree expr, tree type)
> > +{
>
> Inline into single caller
Done.
>
> > + poly_uint64 factor;
> > + if (poly_int_tree_p (TYPE_SIZE_UNIT (type), &factor))
> > + {
> > + if (!analyze_sum_of_products (loop, expr, factor))
> > + analyze_evolution (loop, expr, factor);
> > + }
> > +}
> > +
> > +/* Analyze expression EXPR, which occurs in loop LOOP. */
> > +
> > +void
> > +loop_versioning::analyze_expr (struct loop *loop, tree expr)
> > +{
> > + while (handled_component_p (expr))
> > + {
> > + /* See whether we can use versioning to avoid a multiplication
> > + in the array index. */
> > + if (TREE_CODE (expr) == ARRAY_REF)
> > + {
> > + if (!analyze_sum_of_products (loop, TREE_OPERAND (expr, 1), 1))
> > + analyze_evolution (loop, TREE_OPERAND (expr, 1), 1);
> > + }
> > + expr = TREE_OPERAND (expr, 0);
> > + }
> > +
> > + if (TREE_CODE (expr) == MEM_REF)
> > + {
> > + tree addr = TREE_OPERAND (expr, 0);
> > + /* See whether we can use versioning to avoid a multiplication
> > + in the pointer calculation. This is generally only worth
> > + doing if the multiplication occurs in this loop rather than
> > + an outer loop. */
>
> Why's that so here but not above for ARRAY_REF? That is, what is
> the difference between a[i] and ptr = &a[i]; *ptr?
>
> > + if (!expr_invariant_in_loop_p (loop, addr))
> > + analyze_pointer (loop, addr, TREE_TYPE (expr));
> > + }
> > +
> > + /* These would be easy to handle if they existed at this stage. */
> > + gcc_checking_assert (TREE_CODE (expr) != TARGET_MEM_REF);
> > +}
> > +
> > +/* Analyze STMT looking for useful version checks. */
> > +
> > +void
> > +loop_versioning::analyze_stmt (gimple *stmt)
> > +{
>
> Please inline into single caller to make flow easily visible.
Done.
>
> > + struct loop *loop = gimple_bb (stmt)->loop_father;
> > +
> > + unsigned int nops = gimple_num_ops (stmt);
> > + for (unsigned int i = 0; i < nops; ++i)
> > + if (tree op = gimple_op (stmt, i))
> > + analyze_expr (loop, op);
>
> I think you instead want to use gimple_walk_load_store_ops ().
Do you mean walk_stmt_load_store_ops ?
Doing that was a bit awkward as analyze_expr is a member function and I couldn't find a neat way of
passing a callback to walk_stmt_load_store_ops with the appropriate object state.
Maybe my C++ is lacking here, but this loop looks easier to read IMO.
>
> > +}
> > +
> > +/* Analyze all the statements in BB looking for useful version checks. */
> > +
> > +void
> > +loop_versioning::analyze_block (basic_block bb)
> > +{
> > + struct loop *loop = bb->loop_father;
> > + loop_info &li = get_loop_info (loop);
> > + if (li.rejected_p)
> > + return;
> > +
> > + for (gimple_stmt_iterator gsi = gsi_start_bb (bb); !gsi_end_p (gsi);
> > + gsi_next (&gsi))
> > + {
> > + gimple *stmt = gsi_stmt (gsi);
> > + if (expensive_stmt_p (stmt))
> > + {
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + {
> > + struct loop *loop = gimple_bb (stmt)->loop_father;
> > + fprintf (dump_file, ";; Loop %d (depth %d) contains expensive"
> > + " stmt: ", loop->num, loop_depth (loop));
> > + print_gimple_stmt (dump_file, stmt, 0, TDF_SLIM);
> > + }
> > + li.rejected_p = true;
>
> I assume that once a loop is rejected this or another way there's
> no reason to look at any outer loop of it, thus ...
>
> > + break;
> > + }
> > +
> > + /* Only look for direct versioning opportunities in inner loops
> > + since the benefit tends to be much smaller for outer loops. */
> > + if (!loop->inner)
> > + analyze_stmt (stmt);
> > +
> > + /* The point of the instruction limit is to prevent excessive
> > + code growth, so this is a size-based estimate even though
> > + the optimization is aimed at speed. */
> > + li.num_insns += estimate_num_insns (stmt, &eni_size_weights);
> > + }
> > +}
> > +
> > +/* Analyze all the blocks in the function looking for useful version checks.
> > + Return true if we found one. */
> > +
> > +bool
> > +loop_versioning::analyze_blocks ()
> > +{
> > + /* For now we don't try to version the whole function, although
> > + versioning at that level could be useful in some cases. */
> > + get_loop_info (get_loop (m_fn, 0)).rejected_p = true;
> > +
> > + basic_block bb;
> > + FOR_EACH_BB_FN (bb, m_fn)
> > + if (loop_outer (bb->loop_father))
> > + analyze_block (bb);
>
> .... I'd structure this as a
>
> FOR_EACH_LOOP (... LI_FROM_INNERMOST)
> look at loop body, only analyze stmts belonging to loop (not subloops)
>
> walk eventually even open-coding this as recursion so you can quickly
> finish off outer loop processing once an inner loop got disabled.
>
> > +
> > + return m_num_conditions != 0;
> > +}
> > +
> > +/* Use the ranges in VRS to remove impossible versioning conditions from
> > + LOOP. */
>
> Does it actually happen to make it worthwhile? ;)
>
> > +
> > +void
> > +loop_versioning::prune_loop_conditions (struct loop *loop, vr_values *vrs)
> > +{
> > + loop_info &li = get_loop_info (loop);
> > +
> > + unsigned int i = li.unity_names.length ();
> > + while (i > 0)
> > + {
> > + i -= 1;
> > + tree name = li.unity_names[i];
> > + value_range *vr = vrs->get_value_range (name);
> > + if (vr && !range_includes_p (vr, 1))
> > + {
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + {
> > + fprintf (dump_file, ";; ");
> > + print_generic_expr (dump_file, name, TDF_SLIM);
> > + fprintf (dump_file, " can never be 1 in loop %d\n", loop->num);
> > + }
> > +
> > + li.unity_names.unordered_remove (i);
> > + bitmap_clear_bit (&li.unity_name_ids, SSA_NAME_VERSION (name));
> > + m_num_conditions -= 1;
> > + }
> > + }
> > +}
<snip>
2018-11-08 Richard Sandiford <richard.sandiford@arm.com>
Ramana Radhakrishnan <ramana.radhakrishnan@arm.com>
Kyrylo Tkachov <kyrylo.tkachov@arm.com>
gcc/
* Makefile.in (OBJS): Add gimple-loop-versioning.o.
* common.opt (fversion-loops-for-strides): New option.
* opts.c (default_options_table): Enable fversion-loops-for-strides
at -O3.
* params.def (PARAM_LOOP_VERSIONING_GROUP_SIZE)
(PARAM_LOOP_VERSIONING_MAX_INNER_INSNS)
(PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS): New parameters.
* passes.def: Add pass_loop_versioning.
* timevar.def (TV_LOOP_VERSIONING): New time variable.
* tree-ssa-propagate.h
(substitute_and_fold_engine::substitute_and_fold): Add an optional
block parameter.
* tree-ssa-propagate.c
(substitute_and_fold_engine::substitute_and_fold): Likewise.
When passed, only walk blocks dominated by that block.
* tree-vrp.h (range_includes_p): Declare.
(range_includes_zero_p): Turn into an inline wrapper around
range_includes_p.
* tree-vrp.c (range_includes_p): New function, generalizing...
(range_includes_zero_p): ...this.
* tree-pass.h (make_pass_loop_versioning): Declare.
* gimple-loop-versioning.cc: New file.
gcc/testsuite/
* gcc.dg/loop-versioning-1.c: New test.
* gcc.dg/loop-versioning-10.c: Likewise.
* gcc.dg/loop-versioning-11.c: Likewise.
* gcc.dg/loop-versioning-2.c: Likewise.
* gcc.dg/loop-versioning-3.c: Likewise.
* gcc.dg/loop-versioning-4.c: Likewise.
* gcc.dg/loop-versioning-5.c: Likewise.
* gcc.dg/loop-versioning-6.c: Likewise.
* gcc.dg/loop-versioning-7.c: Likewise.
* gcc.dg/loop-versioning-8.c: Likewise.
* gcc.dg/loop-versioning-9.c: Likewise.
* gfortran.dg/loop_versioning_1.f90: Likewise.
* gfortran.dg/loop_versioning_2.f90: Likewise.
* gfortran.dg/loop_versioning_3.f90: Likewise.
* gfortran.dg/loop_versioning_4.f90: Likewise.
* gfortran.dg/loop_versioning_5.f90: Likewise.
* gfortran.dg/loop_versioning_6.f90: Likewise.
* gfortran.dg/loop_versioning_7.f90: Likewise.
* gfortran.dg/loop_versioning_8.f90: Likewise.
[-- Attachment #2: lversion.patch --]
[-- Type: text/x-patch, Size: 78378 bytes --]
diff --git a/gcc/Makefile.in b/gcc/Makefile.in
index 719a516c356d21a285aa4f65fb6ca7608e19481a..65474fbfdc71c80863e1e8876d90d36f6a4c360b 100644
--- a/gcc/Makefile.in
+++ b/gcc/Makefile.in
@@ -1320,6 +1320,7 @@ OBJS = \
gimple-laddress.o \
gimple-loop-interchange.o \
gimple-loop-jam.o \
+ gimple-loop-versioning.o \
gimple-low.o \
gimple-pretty-print.o \
gimple-ssa-backprop.o \
diff --git a/gcc/common.opt b/gcc/common.opt
index 2971dc21b1fee55e8aee9a313023635c093b23a4..7658eeec8c87a26387040e2c1a0331dd16216611 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -2716,6 +2716,10 @@ fsplit-loops
Common Report Var(flag_split_loops) Optimization
Perform loop splitting.
+fversion-loops-for-strides
+Common Report Var(flag_version_loops_for_strides) Optimization
+Version loops based on whether indices have a stride of one.
+
funwind-tables
Common Report Var(flag_unwind_tables) Optimization
Just generate unwind tables for exception handling.
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index d2e6639dad4b5ec7dddfddf0ea78034d115534cd..d403b0d05d9cd733e2bd04de3360800ed71e3944 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -7962,7 +7962,8 @@ by @option{-O2} and also turns on the following optimization flags:
-fvect-cost-model @gol
-ftree-partial-pre @gol
-fpeel-loops @gol
--fipa-cp-clone}
+-fipa-cp-clone @gol
+-fversion-loops-for-strides}
@item -O0
@opindex O0
@@ -10386,6 +10387,30 @@ for one side of the iteration space and false for the other.
Move branches with loop invariant conditions out of the loop, with duplicates
of the loop on both branches (modified according to result of the condition).
+@item -fversion-loops-for-strides
+@opindex fversion-loops-for-strides
+If a loop iterates over an array with a variable stride, create another
+version of the loop that assumes the stride is always one. For example:
+
+@smallexample
+for (int i = 0; i < n; ++i)
+ x[i * stride] = @dots{};
+@end smallexample
+
+becomes:
+
+@smallexample
+if (stride == 1)
+ for (int i = 0; i < n; ++i)
+ x[i] = @dots{};
+else
+ for (int i = 0; i < n; ++i)
+ x[i * stride] = @dots{};
+@end smallexample
+
+This is particularly useful for assumed-shape arrays in Fortran where
+it allows better vectorization assuming contiguous accesses.
+
@item -ffunction-sections
@itemx -fdata-sections
@opindex ffunction-sections
@@ -11598,6 +11623,20 @@ Hardware autoprefetcher scheduler model control flag.
Number of lookahead cycles the model looks into; at '
' only enable instruction sorting heuristic.
+@item loop-versioning-group-size
+Make the loop versioning pass optimize @samp{a[i * index * @var{N}]}
+in the same way as it would optimize @samp{a[i * index]} when @var{N}
+is less than or equal to this value.
+
+@item loop-versioning-max-inner-insns
+The maximum number of instructions that an inner loop can have
+before the loop versioning pass considers it too big to copy.
+
+@item loop-versioning-max-outer-insns
+The maximum number of instructions that an outer loop can have
+before the loop versioning pass considers it too big to copy,
+discounting any instructions in inner loops that directly benefit
+from versioning.
@end table
@end table
diff --git a/gcc/gimple-loop-versioning.cc b/gcc/gimple-loop-versioning.cc
new file mode 100644
index 0000000000000000000000000000000000000000..e104440403b7042b9139baa861ec787bf7f0b8f6
--- /dev/null
+++ b/gcc/gimple-loop-versioning.cc
@@ -0,0 +1,1380 @@
+/* Loop versioning pass.
+ Copyright (C) 2018 Free Software Foundation, Inc.
+
+This file is part of GCC.
+
+GCC is free software; you can redistribute it and/or modify it
+under the terms of the GNU General Public License as published by the
+Free Software Foundation; either version 3, or (at your option) any
+later version.
+
+GCC is distributed in the hope that it will be useful, but WITHOUT
+ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
+for more details.
+
+You should have received a copy of the GNU General Public License
+along with GCC; see the file COPYING3. If not see
+<http://www.gnu.org/licenses/>. */
+
+#include "config.h"
+#include "system.h"
+#include "coretypes.h"
+#include "backend.h"
+#include "tree.h"
+#include "gimple.h"
+#include "gimple-iterator.h"
+#include "tree-pass.h"
+#include "gimplify-me.h"
+#include "cfgloop.h"
+#include "tree-ssa-loop.h"
+#include "ssa.h"
+#include "tree-scalar-evolution.h"
+#include "tree-chrec.h"
+#include "tree-ssa-loop-ivopts.h"
+#include "fold-const.h"
+#include "tree-ssa-propagate.h"
+#include "tree-inline.h"
+#include "domwalk.h"
+#include "alloc-pool.h"
+#include "vr-values.h"
+#include "gimple-ssa-evrp-analyze.h"
+#include "params.h"
+
+namespace {
+
+/* This pass looks for loops that could be simplified if certain loop
+ invariant conditions were true. It is effectively a form of loop
+ splitting in which the pass produces the split conditions itself,
+ instead of using ones that are already present in the IL.
+
+ Versioning for when strides are 1
+ ---------------------------------
+
+ At the moment the only thing the pass looks for are memory references
+ like:
+
+ for (auto i : ...)
+ ...x[i * stride]...
+
+ It considers changing such loops to:
+
+ if (stride == 1)
+ for (auto i : ...) [A]
+ ...x[i]...
+ else
+ for (auto i : ...) [B]
+ ...x[i * stride]...
+
+ This can have several benefits:
+
+ (1) [A] is often easier or cheaper to vectorize than [B].
+
+ (2) The scalar code in [A] is simpler than the scalar code in [B]
+ (if the loops cannot be vectorized or need an epilogue loop).
+
+ (3) We might recognize [A] as a pattern, such as a memcpy or memset.
+
+ (4) [A] has simpler address evolutions, which can help other passes
+ like loop interchange.
+
+ The optimization is particularly useful for assumed-shape arrays in
+ Fortran, where the stride of the innermost dimension depends on the
+ array descriptor but is often equal to 1 in practice. For example:
+
+ subroutine f1(x)
+ real :: x(:)
+ x(:) = 100
+ end subroutine f1
+
+ generates the equivalent of:
+
+ raw_stride = *x.dim[0].stride;
+ stride = raw_stride != 0 ? raw_stride : 1;
+ x_base = *x.data;
+ ...
+ tmp1 = stride * S;
+ tmp2 = tmp1 - stride;
+ *x_base[tmp2] = 1.0e+2;
+
+ but in the common case that stride == 1, the last three statements
+ simplify to:
+
+ tmp3 = S + -1;
+ *x_base[tmp3] = 1.0e+2;
+
+ The optimization is in principle very simple. The difficult parts are:
+
+ (a) deciding which parts of a general address calculation correspond
+ to the inner dimension of an array, since this usually isn't explicit
+ in the IL, and for C often isn't even explicit in the source code
+
+ (b) estimating when the transformation is worthwhile
+
+ Structure
+ ---------
+
+ The pass has four phases:
+
+ (1) Walk through the statements looking for and recording potential
+ versioning opportunities. Stop if there are none.
+
+ (2) Use context-sensitive range information to see whether any versioning
+ conditions are impossible in practice. Remove them if so, and stop
+ if no opportunities remain.
+
+ (We do this only after (1) to keep compile time down when no
+ versioning opportunities exist.)
+
+ (3) Apply the cost model. Decide which versioning opportunities are
+ worthwhile and at which nesting level they should be applied.
+
+ (4) Attempt to version all the loops selected by (3), so that:
+
+ for (...)
+ ...
+
+ becomes:
+
+ if (!cond)
+ for (...) // Original loop
+ ...
+ else
+ for (...) // New loop
+ ...
+
+ Use the version condition COND to simplify the new loop. */
+class loop_versioning {
+public:
+ loop_versioning (function *);
+ ~loop_versioning ();
+ unsigned int run ();
+
+private:
+ /* Information about the versioning we'd like to apply to a loop. */
+ struct loop_info {
+ bool worth_versioning_p () const;
+
+ /* True if we've decided not to version this loop. The remaining
+ fields are meaningless if so. */
+ bool rejected_p;
+
+ /* True if at least one subloop of this loop benefits from versioning. */
+ bool subloops_benefit_p;
+
+ /* An estimate of the total number of instructions in the loop,
+ excluding those in subloops that benefit from versioning. */
+ unsigned int num_insns;
+
+ /* The outermost loop that can handle all the version checks
+ described below. */
+ struct loop *outermost;
+
+ /* We'd like to version the loop for the case in which these
+ SSA_NAMEs are all equal to 1 at runtime. */
+ vec<tree> unity_names;
+
+ /* The set of SSA_NAMEs in UNITY_NAMES, keyed off the SSA_NAME_VERSION. */
+ bitmap_head unity_name_ids;
+ };
+
+ /* Used to walk the dominator tree to find loop versioning conditions
+ that are always false. */
+ class lv_dom_walker : public dom_walker
+ {
+ public:
+ lv_dom_walker (loop_versioning &);
+
+ edge before_dom_children (basic_block) FINAL OVERRIDE;
+ void after_dom_children (basic_block) FINAL OVERRIDE;
+
+ private:
+ /* The parent pass. */
+ loop_versioning &m_lv;
+
+ /* Used to build context-dependent range information. */
+ evrp_range_analyzer m_range_analyzer;
+ };
+
+ /* Used to simplify statements based on conditions that are established
+ by the version checks. */
+ class name_prop : public substitute_and_fold_engine
+ {
+ public:
+ name_prop (loop_info &li) : m_li (li) {}
+ tree get_value (tree) FINAL OVERRIDE;
+
+ private:
+ /* Information about the versioning we've performed on the loop. */
+ loop_info &m_li;
+ };
+
+ loop_info &get_loop_info (struct loop *loop) { return m_loops[loop->num]; }
+
+ unsigned int max_insns_for_loop (struct loop *);
+ bool expensive_stmt_p (gimple *);
+
+ void version_for_unity (struct loop *, tree);
+ bool acceptable_scale_p (tree, poly_uint64);
+ tree get_step_if_innermost (tree, poly_uint64);
+ tree extract_step (tree, poly_uint64, tree *);
+ void analyze_index_evolution (struct loop *, tree, poly_uint64);
+ bool analyze_product (struct loop *, gassign *, poly_uint64);
+ bool analyze_sum_of_products (struct loop *, tree, poly_uint64);
+ void analyze_expr (struct loop *, tree);
+ void analyze_block (basic_block);
+ bool analyze_blocks ();
+
+ void prune_loop_conditions (struct loop *, vr_values *);
+ bool prune_conditions ();
+
+ void merge_loop_info (struct loop *, struct loop *);
+ void add_loop_to_queue (struct loop *);
+ bool decide_whether_loop_is_versionable (struct loop *);
+ bool make_versioning_decisions ();
+
+ bool version_loop (struct loop *);
+ bool implement_versioning_decisions ();
+
+ /* The function we're optimizing. */
+ function *m_fn;
+
+ /* The obstack to use for all pass-specific bitmaps. */
+ bitmap_obstack m_obstack;
+
+ /* The number of loops in the function. */
+ unsigned int m_nloops;
+
+ /* The total number of loop version conditions we've found. */
+ unsigned int m_num_conditions;
+
+ /* Information about each loop. */
+ auto_vec<loop_info> m_loops;
+
+ /* The list of loops that we've decided to version. */
+ auto_vec<struct loop *> m_loops_to_version;
+};
+
+/* If EXPR is an SSA name and not a default definition, return the
+ defining statement, otherwise return null. */
+
+static gimple *
+maybe_get_stmt (tree expr)
+{
+ if (TREE_CODE (expr) == SSA_NAME && !SSA_NAME_IS_DEFAULT_DEF (expr))
+ return SSA_NAME_DEF_STMT (expr);
+ return NULL;
+}
+
+/* Like maybe_get_stmt, but also return null if the defining
+ statement isn't an assignment. */
+
+static gassign *
+maybe_get_assign (tree expr)
+{
+ return safe_dyn_cast <gassign *> (maybe_get_stmt (expr));
+}
+
+/* If EXPR is an SSA name, look through any casts to see whether the
+ unconverted value is defined in LOOP by a gassign. Return the
+ gassign if so, otherwise return null. */
+
+gassign *
+maybe_get_assign_strip_casts (struct loop *loop, tree expr)
+{
+ const unsigned int MAX_NITERS = 4;
+
+ tree type = TREE_TYPE (expr);
+ for (unsigned int niters = 0; niters < MAX_NITERS; ++niters)
+ {
+ gassign *assign = maybe_get_assign (expr);
+ if (!assign || gimple_bb (assign)->loop_father != loop)
+ return NULL;
+ expr = gimple_assign_rhs1 (assign);
+ if (CONVERT_EXPR_CODE_P (gimple_assign_rhs_code (assign))
+ && INTEGRAL_TYPE_P (TREE_TYPE (expr)) == INTEGRAL_TYPE_P (type)
+ && POINTER_TYPE_P (TREE_TYPE (expr)) == POINTER_TYPE_P (type))
+ ;
+ else
+ return assign;
+ }
+ return NULL;
+}
+
+/* Strip all conversions of integers from EXPR, regardless of whether
+ the conversions are nops. This is useful in the context of this pass
+ because we're not trying to fold or simulate the expression; we just
+ want to see how it's structured. */
+
+static tree
+strip_casts (tree expr)
+{
+ while (CONVERT_EXPR_P (expr)
+ && INTEGRAL_TYPE_P (TREE_TYPE (TREE_OPERAND (expr, 0))))
+ expr = TREE_OPERAND (expr, 0);
+ return expr;
+}
+
+/* Return true if we want to version the loop, i.e. if we have a
+ specific reason for doing so and no specific reason not to. */
+
+bool
+loop_versioning::loop_info::worth_versioning_p () const
+{
+ return !rejected_p && (!unity_names.is_empty () || subloops_benefit_p);
+}
+
+loop_versioning::lv_dom_walker::lv_dom_walker (loop_versioning &lv)
+ : dom_walker (CDI_DOMINATORS), m_lv (lv)
+{
+}
+
+/* Process BB before processing the blocks it dominates. */
+
+edge
+loop_versioning::lv_dom_walker::before_dom_children (basic_block bb)
+{
+ m_range_analyzer.enter (bb);
+
+ if (bb == bb->loop_father->header)
+ m_lv.prune_loop_conditions (bb->loop_father,
+ m_range_analyzer.get_vr_values ());
+
+ for (gimple_stmt_iterator si = gsi_start_bb (bb); !gsi_end_p (si);
+ gsi_next (&si))
+ m_range_analyzer.record_ranges_from_stmt (gsi_stmt (si), false);
+
+ return NULL;
+}
+
+/* Process BB after processing the blocks it dominates. */
+
+void
+loop_versioning::lv_dom_walker::after_dom_children (basic_block bb)
+{
+ m_range_analyzer.leave (bb);
+}
+
+/* Decide whether to replace VAL with a new value in a versioned loop.
+ Return the new value if so, otherwise return null. */
+
+tree
+loop_versioning::name_prop::get_value (tree val)
+{
+ if (TREE_CODE (val) == SSA_NAME
+ && bitmap_bit_p (&m_li.unity_name_ids, SSA_NAME_VERSION (val)))
+ return build_one_cst (TREE_TYPE (val));
+ return NULL_TREE;
+}
+
+/* Initialize the structure to optimize FN. */
+
+loop_versioning::loop_versioning (function *fn)
+ : m_fn (fn),
+ m_nloops (number_of_loops (fn)),
+ m_num_conditions (0)
+{
+ bitmap_obstack_initialize (&m_obstack);
+
+ m_loops.safe_grow_cleared (m_nloops);
+ for (unsigned int i = 0; i < m_nloops; ++i)
+ {
+ m_loops[i].outermost = get_loop (m_fn, 0);
+ bitmap_initialize (&m_loops[i].unity_name_ids, &m_obstack);
+ }
+}
+
+loop_versioning::~loop_versioning ()
+{
+ for (unsigned int i = 0; i < m_nloops; ++i)
+ m_loops[i].unity_names.release ();
+ bitmap_obstack_release (&m_obstack);
+}
+
+/* Return the maximum number of instructions allowed in LOOP before
+ it becomes too big for versioning.
+
+ There are separate limits for inner and outer loops. The limit for
+ inner loops applies only to loops that benefit directly from versioning.
+ The limit for outer loops applies to all code in the outer loop and
+ its subloops that *doesn't* benefit directly from versioning; such code
+ would be "taken along for the ride". The idea is that if the cost of
+ the latter is small, it is better to version outer loops rather than
+ inner loops, both to reduce the number of repeated checks and to enable
+ more of the loop nest to be optimized as a natural nest (e.g. by loop
+ interchange or outer-loop vectorization). */
+
+unsigned int
+loop_versioning::max_insns_for_loop (struct loop *loop)
+{
+ return (loop->inner
+ ? PARAM_VALUE (PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS)
+ : PARAM_VALUE (PARAM_LOOP_VERSIONING_MAX_INNER_INSNS));
+}
+
+/* Return true if for cost reasons we should avoid versioning any loop
+ that contains STMT.
+
+ Note that we don't need to check whether versioning is invalid for
+ correctness reasons, since the versioning process does that for us.
+ The conditions involved are too rare to be worth duplicating here. */
+
+bool
+loop_versioning::expensive_stmt_p (gimple *stmt)
+{
+ if (gcall *call = dyn_cast <gcall *> (stmt))
+ /* Assume for now that the time spent in an "expensive" call would
+ overwhelm any saving from versioning. */
+ return !gimple_inexpensive_call_p (call);
+ return false;
+}
+
+/* Record that we want to version LOOP for the case in which SSA name NAME
+ is equal to 1. We already know that NAME is invariant in LOOP. */
+
+void
+loop_versioning::version_for_unity (struct loop *loop, tree name)
+{
+ loop_info &li = get_loop_info (loop);
+
+ if (bitmap_set_bit (&li.unity_name_ids, SSA_NAME_VERSION (name)))
+ {
+ /* This is the first time we've wanted to version LOOP for NAME. */
+ li.unity_names.safe_push (name);
+
+ /* Keep track of the outermost loop that can handle all versioning
+ checks in LI. */
+ struct loop *outermost
+ = outermost_invariant_loop_for_expr (loop, name);
+ if (loop_depth (li.outermost) < loop_depth (outermost))
+ li.outermost = outermost;
+
+ if (dump_enabled_p ())
+ {
+ dump_printf (MSG_NOTE, ";; Want to version loop %d (depth %d)"
+ " for when %T == 1", loop->num, loop_depth (loop), name);
+ if (outermost == loop)
+ dump_printf (MSG_NOTE, "; cannot hoist check further");
+ else
+ {
+ dump_printf (MSG_NOTE,
+ "; could hoist check to loop %d (depth %d)",
+ outermost->num, loop_depth (outermost));
+ if (loop_depth (li.outermost) > loop_depth (outermost))
+ dump_printf (MSG_NOTE, ", but that's further than"
+ " other checks allow");
+ }
+ dump_printf (MSG_NOTE, "\n");
+ }
+
+ m_num_conditions += 1;
+ }
+ else
+ {
+ /* This is a duplicate request. */
+ if (dump_enabled_p ())
+ dump_printf (MSG_NOTE,
+ ";; Already want to version loop for when %T == 1\n",
+ name);
+
+ }
+}
+
+/* Return true if in principle it is worth versioning an index fragment of
+ the form:
+
+ (i * b * SCALE) / FACTOR
+
+ for the case in which b == 1. */
+
+bool
+loop_versioning::acceptable_scale_p (tree scale, poly_uint64 factor)
+{
+ /* See whether SCALE is a constant multiple of FACTOR, and if the
+ multiple is small enough for us to treat it as a potential grouped
+ access. For example:
+
+ for (auto i : ...)
+ y[i] = f (x[4 * i * stride],
+ x[4 * i * stride + 1],
+ x[4 * i * stride + 2]);
+
+ would benefit from versioning for the case in which stride == 1.
+ High multiples of i * stride are less likely to benefit, and could
+ indicate a simulated multi-dimensional array.
+
+ This is just a heuristic, to avoid having to do expensive group
+ analysis of the data references in a loop. */
+ poly_uint64 const_scale;
+ unsigned int multiple;
+ if (poly_int_tree_p (scale, &const_scale)
+ && constant_multiple_p (const_scale, factor, &multiple))
+ {
+ unsigned int maxval = PARAM_VALUE (PARAM_LOOP_VERSIONING_GROUP_SIZE);
+ return IN_RANGE (multiple, 1, maxval);
+ }
+
+ return false;
+}
+
+/* Decide whether an index fragment of the form:
+
+ (i * STEP) / FACTOR
+
+ is likely to be for an innermost dimension. If we think it is,
+ return one of the constant values that it could have (returning 1 if
+ that's a possibility). If think it isn't, return null. Otherwise
+ return STEP, to indicate that it might or might not be an inner
+ dimension. */
+
+tree
+loop_versioning::get_step_if_innermost (tree step, poly_uint64 factor)
+{
+ const unsigned int MAX_NITERS = 8;
+
+ tree likely = NULL_TREE;
+ tree unlikely = NULL_TREE;
+ tree worklist[MAX_NITERS];
+ unsigned int length = 0;
+ worklist[length++] = step;
+ for (unsigned int i = 0; i < length; ++i)
+ {
+ tree expr = worklist[i];
+
+ if (CONSTANT_CLASS_P (expr))
+ {
+ /* See if multiplying by EXPR applies a scale that would be
+ consistent with an individual access or a small grouped
+ access. */
+ if (acceptable_scale_p (expr, factor))
+ {
+ likely = expr;
+ if (integer_onep (expr))
+ break;
+ }
+ else
+ unlikely = expr;
+ continue;
+ }
+
+ /* Otherwise we can only handle SSA names. */
+ gimple *stmt = maybe_get_stmt (expr);
+ if (!stmt)
+ continue;
+
+ /* If EXPR is set by a PHI node, queue its arguments in case
+ we find one that is consistent with an inner dimension.
+
+ An important instance of this is the Fortran handling of array
+ descriptors, which calculates the stride of the inner dimension
+ using a PHI equivalent of:
+
+ raw_stride = a.dim[0].stride;
+ stride = raw_stride != 0 ? raw_stride : 1;
+
+ (Strides for outer dimensions do not treat 0 specially.) */
+ if (gphi *phi = dyn_cast <gphi *> (stmt))
+ {
+ unsigned int nargs = gimple_phi_num_args (phi);
+ for (unsigned int j = 0; j < nargs && length < MAX_NITERS; ++j)
+ worklist[length++] = gimple_phi_arg_def (phi, j);
+ continue;
+ }
+
+ /* If the value is set by an assignment, expect it to be read from
+ memory (such as an array descriptor) rather than be calculated. */
+ if (gassign *assign = dyn_cast <gassign *> (stmt))
+ {
+ if (gimple_assign_load_p (assign))
+ continue;
+
+ unlikely = expr;
+ }
+
+ /* Things like calls don't really tell us anything. */
+ }
+
+ if (likely)
+ {
+ if (dump_enabled_p ())
+ dump_printf (MSG_NOTE, "likely to be innermost dimension\n");
+ return likely;
+ }
+
+ if (unlikely)
+ {
+ if (dump_enabled_p ())
+ dump_printf (MSG_NOTE, "probably not innermost dimension\n");
+ return NULL_TREE;
+ }
+
+ if (dump_enabled_p ())
+ {
+ if (maybe_ne (factor, 1U))
+ dump_printf (MSG_NOTE, "didn't find expected scaling factor\n");
+ else
+ dump_printf (MSG_NOTE, "no information about value\n");
+ }
+ return step;
+}
+
+/* STEP appears in an index fragment of the form:
+
+ {..., +, STEP}_n / FACTOR
+
+ Remove any conversions and grouping or scaling factors from STEP and
+ return the underlying index step. Set *STEP_IF_INNERMOST to the value
+ returned by get_step_if_innermost. */
+
+tree
+loop_versioning::extract_step (tree step, poly_uint64 factor,
+ tree *step_if_innermost)
+{
+ if (dump_enabled_p ())
+ dump_printf (MSG_NOTE, ";; step %T: ", step);
+
+ step = strip_casts (step);
+
+ /* Peel any scaling, which generally happens after conversion to
+ pointer width. For example, on LP64 systems:
+
+ int *x, i, stride;
+ ... x[4 * i * stride] ...;
+
+ multiplies i * stride by 4 using ints, then widens the result
+ to pointer width before multiplying by sizeof (int). */
+ poly_uint64 scale;
+ if (TREE_CODE (step) == MULT_EXPR
+ && poly_int_tree_p (TREE_OPERAND (step, 1), &scale)
+ && known_eq (scale, factor))
+ {
+ step = strip_casts (TREE_OPERAND (step, 0));
+ if (dump_enabled_p ())
+ dump_printf (MSG_NOTE, "scaled, ");
+ factor = 1;
+ }
+
+ /* Peel any grouping multiple; see acceptable_scale_p for details. */
+ if (TREE_CODE (step) == MULT_EXPR
+ && acceptable_scale_p (TREE_OPERAND (step, 1), factor))
+ {
+ step = strip_casts (TREE_OPERAND (step, 0));
+ if (dump_enabled_p ())
+ dump_printf (MSG_NOTE, "%sgrouped, ",
+ maybe_ne (factor, 1U) ? "scaled, " : "");
+ factor = 1;
+ }
+
+ *step_if_innermost = get_step_if_innermost (step, factor);
+ return step;
+}
+
+/* Analyze the evolution of index fragment EXPR / FACTOR in LOOP and its
+ containing loops to see whether any part of it could be simplified
+ by versioning. Register the versioning opportunities if so. */
+
+void
+loop_versioning::analyze_index_evolution (struct loop *loop, tree expr,
+ poly_uint64 factor)
+{
+ const unsigned int MAX_NSPLIT = 8;
+
+ if (dump_enabled_p ())
+ {
+ dump_printf (MSG_NOTE, ";; Analyzing use of %T", expr);
+ if (maybe_ne (factor, 1U))
+ {
+ dump_printf (MSG_NOTE, " (which addresses ");
+ dump_dec (MSG_NOTE, factor);
+ dump_printf (MSG_NOTE, " bytes)");
+ }
+ dump_printf (MSG_NOTE, " in loop %d (depth %d)\n",
+ loop->num, loop_depth (loop));
+ }
+
+ /* The main problem we have here is that we cannot assume that the
+ innermost loop iterates over the innermost dimension of an array.
+ Accidentally adding versioning checks for outer dimensions would
+ cause the version condition to be false, which as well as bloating
+ the code would defeat loop versioning benefits for other accesses.
+
+ Unfortunately all we usually see at this stage is general address
+ arithmetic, with no positive way of identifying how many dimensions
+ an array access has and which multiplication factors in the address
+ expression correspond to which array dimensions. In C code this is
+ often not even explicit in the source, since variable-sized multi-
+ dimensional arrays are often simulated using one-dimensional arrays.
+
+ The three main ways in which we deal with this are:
+
+ - use heuristics that positively identify steps that are likely
+ to represent the inner dimension.
+
+ - use heuristics that positively identify steps that are unlikely
+ to represent the inner dimension.
+
+ - if a part of EXPR is invariant in LOOP, analyze its evolution in
+ the outer loops to see whether we can positively identify any of
+ it as iterating over the inner dimension. */
+ tree best_step = NULL_TREE;
+ auto_vec<tree, MAX_NSPLIT> worklist;
+ worklist.quick_push (expr);
+ unsigned int nsplit = 0;
+ while (!worklist.is_empty ())
+ {
+ expr = strip_casts (worklist.pop ());
+ tree_code code = TREE_CODE (expr);
+
+ if (code == POLYNOMIAL_CHREC)
+ {
+ /* Analyze the CHREC_RIGHT as the step in an index fragment. */
+ tree step_if_innermost;
+ tree step = extract_step (CHREC_RIGHT (expr), factor,
+ &step_if_innermost);
+ if (!best_step)
+ {
+ /* This is the outermost chrec for the original expression.
+ It's not worth carrying on if the step isn't versionable,
+ or if we're pretty sure it's not for the inner dimension. */
+ if (!step_if_innermost
+ || TREE_CODE (step) != SSA_NAME
+ || !expr_invariant_in_loop_p (loop, step))
+ return;
+
+ best_step = step;
+
+ /* We should version for STEP == 1 if we know that that can be
+ true under some circumstances. */
+ if (integer_onep (step_if_innermost))
+ break;
+
+ /* Bail out if this appears to be the step for the innermost
+ dimension, but isn't likely to be 1.
+
+ ??? We could instead version for when it equals
+ STEP_IF_INNERMOST, but it's not likely to have as much
+ benefit as versioning for 1. */
+ if (step_if_innermost != step)
+ return;
+ }
+ else
+ {
+ /* This is an inner chrec. If it looks like it iterates over
+ the innermost dimension, abort any attempt to version for
+ the outermost chrec (which if we reach here wasn't itself
+ obviously iterating over the innermost dimension). */
+ if (step_if_innermost && TREE_CONSTANT (step_if_innermost))
+ return;
+ }
+ worklist.quick_push (CHREC_LEFT (expr));
+ continue;
+ }
+
+ /* Analyze parts of a queued CHREC_LEFT. This is better than just
+ analyzing the evolution of the whole expression since the value
+ could include a mixture of analyzable and unanalyzable elements.
+ Use NSPLIT to count cases in which we add more expressions to
+ analyze, as opposed to just simplifying the existing one. */
+ if (code == PLUS_EXPR || code == POINTER_PLUS_EXPR || code == MINUS_EXPR)
+ {
+ worklist.quick_push (TREE_OPERAND (expr, 0));
+ if (nsplit++ < MAX_NSPLIT)
+ worklist.quick_push (TREE_OPERAND (expr, 1));
+ continue;
+ }
+ if (code == MULT_EXPR)
+ {
+ tree op0 = strip_casts (TREE_OPERAND (expr, 0));
+ tree op1 = TREE_OPERAND (expr, 1);
+ if (TREE_CODE (op0) == PLUS_EXPR || TREE_CODE (op0) == MINUS_EXPR)
+ {
+ tree type = TREE_TYPE (expr);
+ tree op00 = fold_convert (type, TREE_OPERAND (op0, 0));
+ worklist.quick_push (fold_build2 (MULT_EXPR, type, op00, op1));
+ if (nsplit++ < MAX_NSPLIT)
+ {
+ tree op01 = fold_convert (type, TREE_OPERAND (op0, 1));
+ worklist.quick_push (fold_build2 (MULT_EXPR, type,
+ op01, op1));
+ }
+ continue;
+ }
+ }
+
+ /* If EXPR is invariant in LOOP, analyze it wrt the innermost loop
+ for which it could evolve (i.e. the loop containing the outermost
+ one for which EXPR is invariant). */
+ struct loop *wrt_loop = outermost_invariant_loop_for_expr (loop, expr);
+ if (wrt_loop)
+ {
+ wrt_loop = loop_outer (wrt_loop);
+ if (!wrt_loop)
+ continue;
+ }
+ else
+ wrt_loop = loop;
+ tree evolution = analyze_scalar_evolution (wrt_loop, expr);
+ tree chrec = strip_casts (evolution);
+ if (TREE_CODE (chrec) == POLYNOMIAL_CHREC)
+ {
+ if (dump_enabled_p ())
+ dump_printf (MSG_NOTE,
+ ";; evolution of %T in loop %d (depth %d):\n",
+ expr, wrt_loop->num, loop_depth (wrt_loop));
+ worklist.quick_push (chrec);
+ }
+ else
+ {
+ if (dump_enabled_p ())
+ dump_printf (MSG_NOTE, ";; cannot analyze %T any further\n",
+ expr);
+ }
+ }
+ if (best_step)
+ version_for_unity (loop, best_step);
+}
+
+/* Analyze multiplication MULT to see whether we can identify "gather-like"
+ versioning opportunities such as:
+
+ for (int i = 0; i < n; ++i)
+ res += a[index[i] * stride];
+
+ Return true if there was a versioning opportunity.
+
+ LOOP is the loop that contains MULT. Dividing the value of MULT
+ by FACTOR converts it into an element index. */
+
+bool
+loop_versioning::analyze_product (struct loop *loop, gassign *mult,
+ poly_uint64 factor)
+{
+ /* Record the original LHS for the dump message below. */
+ tree lhs = gimple_assign_lhs (mult);
+
+ /* Peel any scaling, which generally happens after conversion to
+ pointer width. For example, on LP64 systems:
+
+ int *x, i, stride;
+ ... x[4 * i * stride] ...;
+
+ multiplies i * stride by 4 using ints, then widens the result
+ to pointer width before multiplying by sizeof (int). */
+ poly_uint64 scale;
+ if (poly_int_tree_p (gimple_assign_rhs2 (mult), &scale)
+ && known_eq (scale, factor))
+ {
+ mult = maybe_get_assign_strip_casts (loop, gimple_assign_rhs1 (mult));
+ if (!mult || gimple_assign_rhs_code (mult) != MULT_EXPR)
+ return false;
+ factor = 1;
+ }
+
+ /* Peel any grouping multiple; see acceptable_scale_p for details. */
+ if (acceptable_scale_p (gimple_assign_rhs2 (mult), factor))
+ {
+ mult = maybe_get_assign_strip_casts (loop, gimple_assign_rhs1 (mult));
+ if (!mult || gimple_assign_rhs_code (mult) != MULT_EXPR)
+ return false;
+ }
+ else if (maybe_ne (factor, 1U))
+ /* We expect to see a scaling multiplication when FACTOR is > 1. */
+ return false;
+
+ /* See whether this multiplication involves a loop-invariant SSA name
+ and a non-invariant SSA name. */
+ tree op1 = gimple_assign_rhs1 (mult);
+ tree op2 = gimple_assign_rhs2 (mult);
+ if (TREE_CODE (op1) != SSA_NAME || TREE_CODE (op2) != SSA_NAME)
+ return false;
+
+ bool invariant1_p = expr_invariant_in_loop_p (loop, op1);
+ bool invariant2_p = expr_invariant_in_loop_p (loop, op2);
+ if (invariant1_p == invariant2_p)
+ return false;
+
+ /* Make sure that the invariant is OP1 and the other operand is OP2. */
+ if (invariant2_p)
+ std::swap (op1, op2);
+
+ /* Bail out if OP2 can be analyzed as an evolution; we want to use
+ analyze_index_evolution in that case instead. There's no point trying
+ hard to avoid repeating the call to analyze_scalar_evolution since
+ that function does its own caching. */
+ if (tree_contains_chrecs (analyze_scalar_evolution (loop, op2), NULL))
+ return false;
+
+ if (dump_enabled_p ())
+ dump_printf (MSG_NOTE,
+ ";; Address fragment %T multiplies invariant %T by %T,"
+ "which isn't a scalar evolution\n", lhs, op1, op2);
+
+ version_for_unity (loop, op1);
+ return true;
+}
+
+/* Treat EXPR as a sum of products and apply analyze_product to each of the
+ products. Return true if one of the products provides a versioning
+ opportunity. FACTOR is as for analyze_product. */
+
+bool
+loop_versioning::analyze_sum_of_products (struct loop *loop, tree expr,
+ poly_uint64 factor)
+{
+ const unsigned int MAX_NITERS = 8;
+
+ auto_bitmap ssa_names;
+ tree worklist[MAX_NITERS];
+ unsigned int length = 0;
+ worklist[length++] = expr;
+ for (unsigned int i = 0; i < length; ++i)
+ {
+ expr = worklist[i];
+ if (TREE_CODE (expr) == SSA_NAME)
+ {
+ if (bitmap_bit_p (ssa_names, SSA_NAME_VERSION (expr)))
+ continue;
+ else
+ bitmap_set_bit (ssa_names, SSA_NAME_VERSION (expr));
+ }
+ gassign *assign = maybe_get_assign_strip_casts (loop, expr);
+ if (!assign)
+ continue;
+
+ tree_code code = gimple_assign_rhs_code (assign);
+ if (code == PLUS_EXPR || code == POINTER_PLUS_EXPR || code == MINUS_EXPR)
+ {
+ if (length < MAX_NITERS)
+ worklist[length++] = gimple_assign_rhs1 (assign);
+ if (length < MAX_NITERS)
+ worklist[length++] = gimple_assign_rhs2 (assign);
+ }
+ else if (code == MULT_EXPR && analyze_product (loop, assign, factor))
+ return true;
+ }
+ return false;
+}
+
+/* Analyze expression EXPR, which occurs in loop LOOP. */
+
+void
+loop_versioning::analyze_expr (struct loop *loop, tree expr)
+{
+ while (handled_component_p (expr))
+ {
+ /* See whether we can use versioning to avoid a multiplication
+ in the array index. */
+ if (TREE_CODE (expr) == ARRAY_REF)
+ {
+ if (!analyze_sum_of_products (loop, TREE_OPERAND (expr, 1), 1))
+ analyze_index_evolution (loop, TREE_OPERAND (expr, 1), 1);
+ }
+ expr = TREE_OPERAND (expr, 0);
+ }
+
+ if (TREE_CODE (expr) == MEM_REF)
+ {
+ tree addr = TREE_OPERAND (expr, 0);
+ /* See whether we can use versioning to avoid a multiplication
+ in the pointer calculation. This is generally only worth
+ doing if the multiplication occurs in this loop rather than
+ an outer loop. */
+ if (!expr_invariant_in_loop_p (loop, addr))
+ {
+ poly_uint64 factor;
+ if (poly_int_tree_p (TYPE_SIZE_UNIT (TREE_TYPE (expr)), &factor)
+ && !analyze_sum_of_products (loop, addr, factor))
+ analyze_index_evolution (loop, addr, factor);
+ }
+ }
+
+ /* These would be easy to handle if they existed at this stage. */
+ gcc_checking_assert (TREE_CODE (expr) != TARGET_MEM_REF);
+}
+
+
+/* Analyze all the statements in BB looking for useful version checks. */
+
+void
+loop_versioning::analyze_block (basic_block bb)
+{
+ struct loop *loop = bb->loop_father;
+ loop_info &li = get_loop_info (loop);
+ if (li.rejected_p)
+ return;
+
+ for (gimple_stmt_iterator gsi = gsi_start_bb (bb); !gsi_end_p (gsi);
+ gsi_next (&gsi))
+ {
+ gimple *stmt = gsi_stmt (gsi);
+ if (expensive_stmt_p (stmt))
+ {
+ if (dump_enabled_p ())
+ {
+ struct loop *loop = gimple_bb (stmt)->loop_father;
+ dump_printf (MSG_NOTE, ";; Loop %d (depth %d) contains expensive"
+ " stmt: %G", loop->num, loop_depth (loop), stmt);
+ }
+ li.rejected_p = true;
+ break;
+ }
+
+ /* Only look for direct versioning opportunities in inner loops
+ since the benefit tends to be much smaller for outer loops. */
+ if (!loop->inner)
+ {
+ unsigned int nops = gimple_num_ops (stmt);
+ for (unsigned int i = 0; i < nops; ++i)
+ if (tree op = gimple_op (stmt, i))
+ analyze_expr (loop, op);
+ }
+
+ /* The point of the instruction limit is to prevent excessive
+ code growth, so this is a size-based estimate even though
+ the optimization is aimed at speed. */
+ li.num_insns += estimate_num_insns (stmt, &eni_size_weights);
+ }
+}
+
+/* Analyze all the blocks in the function looking for useful version checks.
+ Return true if we found one. */
+
+bool
+loop_versioning::analyze_blocks ()
+{
+ /* For now we don't try to version the whole function, although
+ versioning at that level could be useful in some cases. */
+ get_loop_info (get_loop (m_fn, 0)).rejected_p = true;
+
+ basic_block bb;
+ FOR_EACH_BB_FN (bb, m_fn)
+ if (loop_outer (bb->loop_father))
+ analyze_block (bb);
+
+ return m_num_conditions != 0;
+}
+
+/* Use the ranges in VRS to remove impossible versioning conditions from
+ LOOP. */
+
+void
+loop_versioning::prune_loop_conditions (struct loop *loop, vr_values *vrs)
+{
+ loop_info &li = get_loop_info (loop);
+
+ unsigned int i = li.unity_names.length ();
+ while (i > 0)
+ {
+ i -= 1;
+ tree name = li.unity_names[i];
+ value_range *vr = vrs->get_value_range (name);
+ if (vr && !range_includes_p (vr, 1))
+ {
+ if (dump_enabled_p ())
+ dump_printf (MSG_NOTE, ";; %T can never be 1 in loop %d\n",
+ name, loop->num);
+
+ li.unity_names.unordered_remove (i);
+ bitmap_clear_bit (&li.unity_name_ids, SSA_NAME_VERSION (name));
+ m_num_conditions -= 1;
+ }
+ }
+}
+
+/* Remove any scheduled loop version conditions that will never be true.
+ Return true if any remain. */
+
+bool
+loop_versioning::prune_conditions ()
+{
+ calculate_dominance_info (CDI_DOMINATORS);
+ lv_dom_walker dom_walker (*this);
+ dom_walker.walk (ENTRY_BLOCK_PTR_FOR_FN (m_fn));
+ return m_num_conditions != 0;
+}
+
+/* Merge the version checks for INNER into immediately-enclosing loop
+ OUTER. */
+
+void
+loop_versioning::merge_loop_info (struct loop *outer, struct loop *inner)
+{
+ loop_info &inner_li = get_loop_info (inner);
+ loop_info &outer_li = get_loop_info (outer);
+
+ tree name;
+ unsigned int i;
+ FOR_EACH_VEC_ELT (inner_li.unity_names, i, name)
+ if (bitmap_set_bit (&outer_li.unity_name_ids, SSA_NAME_VERSION (name)))
+ {
+ outer_li.unity_names.safe_push (name);
+ if (dump_enabled_p ())
+ dump_printf (MSG_NOTE,
+ ";; Hoisting check that %T == 1 from loop"
+ " %d (depth %d) to loop %d\n",
+ name, inner->num, loop_depth (inner), outer->num);
+ }
+
+ if (loop_depth (outer_li.outermost) < loop_depth (inner_li.outermost))
+ outer_li.outermost = inner_li.outermost;
+}
+
+/* Add LOOP to the queue of loops to version. */
+
+void
+loop_versioning::add_loop_to_queue (struct loop *loop)
+{
+ loop_info &li = get_loop_info (loop);
+
+ if (dump_enabled_p ())
+ dump_printf (MSG_NOTE, ";; Queuing loop %d (depth %d) for versioning\n",
+ loop->num, loop_depth (loop));
+ m_loops_to_version.safe_push (loop);
+
+ /* Don't try to version superloops. */
+ li.rejected_p = true;
+}
+
+/* Decide whether the cost model would allow us to version LOOP,
+ either directly or as part of a parent loop, and return true if so.
+ This does not imply that the loop is actually worth versioning in its
+ own right, just that it would be valid to version it if something
+ benefited.
+
+ We have already made this decision for all inner loops of LOOP. */
+
+bool
+loop_versioning::decide_whether_loop_is_versionable (struct loop *loop)
+{
+ loop_info &li = get_loop_info (loop);
+
+ if (li.rejected_p)
+ return false;
+
+ /* Examine the decisions made for inner loops. */
+ for (struct loop *inner = loop->inner; inner; inner = inner->next)
+ {
+ loop_info &inner_li = get_loop_info (inner);
+ if (inner_li.rejected_p)
+ {
+ if (dump_enabled_p ())
+ dump_printf (MSG_NOTE, ";; Not versioning loop %d (depth %d)"
+ " because inner loop %d should not be versioned\n",
+ loop->num, loop_depth (loop), inner->num);
+ return false;
+ }
+
+ if (inner_li.worth_versioning_p ())
+ li.subloops_benefit_p = true;
+
+ /* Accumulate the number of instructions from subloops that are not
+ the innermost, or that don't benefit from versioning. Only the
+ instructions from innermost loops that benefit from versioning
+ should be weighed against loop-versioning-max-inner-insns;
+ everything else should be weighed against
+ loop-versioning-max-outer-insns. */
+ if (!inner_li.worth_versioning_p () || inner->inner)
+ {
+ if (dump_enabled_p ())
+ dump_printf (MSG_NOTE, ";; Counting %d instructions from"
+ " loop %d (depth %d) against parent loop %d\n",
+ inner_li.num_insns, inner->num, loop_depth (inner),
+ loop->num);
+ li.num_insns += inner_li.num_insns;
+ }
+ }
+
+ /* Enforce the size limits. */
+ if (li.worth_versioning_p ())
+ {
+ unsigned int max_num_insns = max_insns_for_loop (loop);
+ if (dump_enabled_p ())
+ dump_printf (MSG_NOTE, ";; Loop %d (depth %d) has %d instructions,"
+ " against a limit of %d\n", loop->num, loop_depth (loop),
+ li.num_insns, max_num_insns);
+ if (li.num_insns > max_num_insns)
+ {
+ if (dump_enabled_p ())
+ dump_printf (MSG_NOTE, ";; Not versioning loop %d (depth %d)"
+ " because it is too big\n", loop->num,
+ loop_depth (loop));
+ return false;
+ }
+ }
+
+ /* Hoist all version checks for subloops to this loop. */
+ for (struct loop *subloop = loop->inner; subloop; subloop = subloop->next)
+ merge_loop_info (loop, subloop);
+
+ return true;
+}
+
+/* Decide which loops to version and add them to the versioning queue.
+ Return true if there are any loops to version. */
+
+bool
+loop_versioning::make_versioning_decisions ()
+{
+ struct loop *loop;
+ FOR_EACH_LOOP (loop, LI_FROM_INNERMOST)
+ {
+ loop_info &linfo = get_loop_info (loop);
+ if (decide_whether_loop_is_versionable (loop))
+ {
+ /* Commit to versioning LOOP directly if we can't hoist the
+ version checks any further. */
+ if (linfo.worth_versioning_p ()
+ && (loop_depth (loop) == 1 || linfo.outermost == loop))
+ add_loop_to_queue (loop);
+ }
+ else
+ {
+ /* We can't version this loop, so individually version any
+ subloops that would benefit and haven't been versioned yet. */
+ linfo.rejected_p = true;
+ for (struct loop *subloop = loop->inner; subloop;
+ subloop = subloop->next)
+ if (get_loop_info (subloop).worth_versioning_p ())
+ add_loop_to_queue (subloop);
+ }
+ }
+
+ return !m_loops_to_version.is_empty ();
+}
+
+/* Attempt to implement loop versioning for LOOP, using the information
+ cached in the associated loop_info. Return true on success. */
+
+bool
+loop_versioning::version_loop (struct loop *loop)
+{
+ loop_info &li = get_loop_info (loop);
+
+ /* Not protected by TDF_DETAILS since this is the main piece of
+ information. */
+ if (dump_enabled_p ())
+ dump_printf (MSG_OPTIMIZED_LOCATIONS, ";; Versioning loop %d (depth %d)\n",
+ loop->num, loop_depth (loop));
+
+ /* Build up a condition that selects the original loop instead of
+ the simplified loop. */
+ tree cond = boolean_false_node;
+ tree name;
+ unsigned int i;
+ FOR_EACH_VEC_ELT (li.unity_names, i, name)
+ {
+ tree ne_one = fold_build2 (NE_EXPR, boolean_type_node, name,
+ build_one_cst (TREE_TYPE (name)));
+ cond = fold_build2 (TRUTH_OR_EXPR, boolean_type_node, cond, ne_one);
+ }
+
+ /* Convert the condition into a suitable gcond. */
+ gimple_seq stmts = NULL;
+ cond = force_gimple_operand_1 (cond, &stmts, is_gimple_condexpr, NULL_TREE);
+
+ /* Version the loop. */
+ initialize_original_copy_tables ();
+ basic_block cond_bb;
+ struct loop *else_loop
+ = loop_version (loop, cond, &cond_bb,
+ profile_probability::unlikely (),
+ profile_probability::likely (),
+ profile_probability::unlikely (),
+ profile_probability::likely (), true);
+ free_original_copy_tables ();
+ if (!else_loop)
+ {
+ if (dump_enabled_p ())
+ dump_printf (MSG_OPTIMIZED_LOCATIONS,
+ ";; Versioning of loop %d failed\n", loop->num);
+ return false;
+ }
+
+ /* Insert the statements that feed COND. */
+ if (stmts)
+ {
+ gimple_stmt_iterator gsi = gsi_last_bb (cond_bb);
+ gsi_insert_seq_before (&gsi, stmts, GSI_SAME_STMT);
+ }
+
+ /* Simplify the new loop, which is used when COND is false. */
+ name_prop (li).substitute_and_fold (else_loop->header);
+ return true;
+}
+
+/* Attempt to version all loops in the versioning queue. Return true
+ if we succeeded for at least one loop. */
+
+bool
+loop_versioning::implement_versioning_decisions ()
+{
+ bool any_succeeded_p = false;
+
+ struct loop *loop;
+ unsigned int i;
+ FOR_EACH_VEC_ELT (m_loops_to_version, i, loop)
+ if (version_loop (loop))
+ any_succeeded_p = true;
+
+ return any_succeeded_p;
+}
+
+/* Run the pass and return a set of TODO_* flags. */
+
+unsigned int
+loop_versioning::run ()
+{
+ gcc_assert (scev_initialized_p ());
+
+ if (!analyze_blocks ()
+ || !prune_conditions ()
+ || !make_versioning_decisions ()
+ || !implement_versioning_decisions ())
+ return 0;
+
+ return TODO_update_ssa;
+}
+
+/* Loop versioning pass. */
+
+const pass_data pass_data_loop_versioning =
+{
+ GIMPLE_PASS, /* type */
+ "lversion", /* name */
+ OPTGROUP_LOOP, /* optinfo_flags */
+ TV_LOOP_VERSIONING, /* tv_id */
+ PROP_cfg, /* properties_required */
+ 0, /* properties_provided */
+ 0, /* properties_destroyed */
+ 0, /* todo_flags_start */
+ 0, /* todo_flags_finish */
+};
+
+class pass_loop_versioning : public gimple_opt_pass
+{
+public:
+ pass_loop_versioning (gcc::context *ctxt)
+ : gimple_opt_pass (pass_data_loop_versioning, ctxt)
+ {}
+
+ /* opt_pass methods: */
+ virtual bool gate (function *) { return flag_version_loops_for_strides; }
+ virtual unsigned int execute (function *);
+};
+
+unsigned int
+pass_loop_versioning::execute (function *fn)
+{
+ if (number_of_loops (fn) <= 1)
+ return 0;
+
+ return loop_versioning (fn).run ();
+}
+
+} // anon namespace
+
+gimple_opt_pass *
+make_pass_loop_versioning (gcc::context *ctxt)
+{
+ return new pass_loop_versioning (ctxt);
+}
diff --git a/gcc/opts.c b/gcc/opts.c
index 34c283dd76557b714d1c4abc1962b5bda537058e..43b77908307a3caf26c8381c41f27bc5d72d5363 100644
--- a/gcc/opts.c
+++ b/gcc/opts.c
@@ -544,6 +544,7 @@ static const struct default_options default_options_table[] =
{ OPT_LEVELS_3_PLUS, OPT_fipa_cp_clone, NULL, 1 },
{ OPT_LEVELS_3_PLUS, OPT_ftree_partial_pre, NULL, 1 },
{ OPT_LEVELS_3_PLUS, OPT_fpeel_loops, NULL, 1 },
+ { OPT_LEVELS_3_PLUS, OPT_fversion_loops_for_strides, NULL, 1 },
/* -Ofast adds optimizations to -O3. */
{ OPT_LEVELS_FAST, OPT_ffast_math, NULL, 1 },
diff --git a/gcc/params.def b/gcc/params.def
index 092116f7538ad23841c91c69ae3397d048f6c643..b4690507faf299dbb292c11a0e4305ca5ce45803 100644
--- a/gcc/params.def
+++ b/gcc/params.def
@@ -1365,6 +1365,25 @@ DEFPARAM(PARAM_AVOID_FMA_MAX_BITS,
"Maximum number of bits for which we avoid creating FMAs.",
0, 0, 512)
+DEFPARAM(PARAM_LOOP_VERSIONING_GROUP_SIZE,
+ "loop-versioning-group-size",
+ "The maximum constant N for which accesses of the form x[N * step]"
+ " are worth versioning for the case in which step is 1",
+ 4, 1, 0)
+
+DEFPARAM(PARAM_LOOP_VERSIONING_MAX_INNER_INSNS,
+ "loop-versioning-max-inner-insns",
+ "The maximum number of instructions in an inner loop that is being"
+ " considered for versioning",
+ 200, 0, 0)
+
+DEFPARAM(PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS,
+ "loop-versioning-max-outer-insns",
+ "The maximum number of instructions in an outer loop that is being"
+ " considered for versioning, on top of the instructions in inner"
+ " loops",
+ 100, 0, 0)
+
/*
Local variables:
diff --git a/gcc/passes.def b/gcc/passes.def
index 24f212c8e31b3def4948838f87824fd10fa3ca31..0c8416fb63b828165768386a9757b995ed38a3ca 100644
--- a/gcc/passes.def
+++ b/gcc/passes.def
@@ -264,6 +264,7 @@ along with GCC; see the file COPYING3. If not see
NEXT_PASS (pass_tree_unswitch);
NEXT_PASS (pass_scev_cprop);
NEXT_PASS (pass_loop_split);
+ NEXT_PASS (pass_loop_versioning);
NEXT_PASS (pass_loop_jam);
/* All unswitching, final value replacement and splitting can expose
empty loops. Remove them now. */
diff --git a/gcc/testsuite/gcc.dg/loop-versioning-1.c b/gcc/testsuite/gcc.dg/loop-versioning-1.c
new file mode 100644
index 0000000000000000000000000000000000000000..4a6b5a994857814048b0a320c68e96d828f0694e
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/loop-versioning-1.c
@@ -0,0 +1,20 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* The simplest IV case. */
+
+void
+f1 (double *x, int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ x[stepx * i] = 100;
+}
+
+void
+f2 (double *x, int stepx, int limit)
+{
+ for (int i = 0; i < limit; i += stepx)
+ x[i] = 100;
+}
+
+/* { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 1\)} 2 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {Versioning loop} 2 "lversion" } } */
diff --git a/gcc/testsuite/gcc.dg/loop-versioning-10.c b/gcc/testsuite/gcc.dg/loop-versioning-10.c
new file mode 100644
index 0000000000000000000000000000000000000000..6c1763ce094827a8b6ba7637d9b3c05f44b051ba
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/loop-versioning-10.c
@@ -0,0 +1,17 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Test that we can version a gather-like operation in which a variable
+ stride is applied to the index. */
+
+int
+f1 (int *x, int *index, int step, int n)
+{
+ int res = 0;
+ for (int i = 0; i < n; ++i)
+ res += x[index[i] * step];
+ return res;
+}
+
+/* { dg-final { scan-tree-dump-times {Address[^\n]*invariant} 1 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 1\)} 1 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {Versioning loop} 1 "lversion" } } */
diff --git a/gcc/testsuite/gcc.dg/loop-versioning-11.c b/gcc/testsuite/gcc.dg/loop-versioning-11.c
new file mode 100644
index 0000000000000000000000000000000000000000..897c0a9ec0f06a3f21789b946d40cb016b29b190
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/loop-versioning-11.c
@@ -0,0 +1,29 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Test that we don't try to version for something that is never 1. */
+
+void
+f1 (double *x, int stepx, int n)
+{
+ if (stepx == 1)
+ for (int i = 0; i < n; ++i)
+ x[i] = 100;
+ else
+ for (int i = 0; i < n; ++i)
+ x[stepx * i] = 100;
+}
+
+void
+f2 (double *x, int stepx, int n)
+{
+ if (stepx <= 1)
+ for (int i = 0; i < n; ++i)
+ x[i] = 100;
+ else
+ for (int i = 0; i < n; ++i)
+ x[stepx * i] = 100;
+}
+
+/* { dg-final { scan-tree-dump-times {Want to version loop} 2 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {can never be 1} 2 "lversion" } } */
+/* { dg-final { scan-tree-dump-not {Versioning loop} "lversion" } } */
diff --git a/gcc/testsuite/gcc.dg/loop-versioning-2.c b/gcc/testsuite/gcc.dg/loop-versioning-2.c
new file mode 100644
index 0000000000000000000000000000000000000000..c105b64878647e166de745edda3eade03a70d03b
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/loop-versioning-2.c
@@ -0,0 +1,39 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Versioning these loops allows loop interchange. */
+
+void
+f1 (double x[][100], int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j * step][i] = 100;
+}
+
+void
+f2 (double x[][100], int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j][i * step] = 100;
+}
+
+void
+f3 (double x[][100], int step, int limit)
+{
+ for (int i = 0; i < 100; ++i)
+ for (int j = 0; j < limit; j += step)
+ x[j][i] = 100;
+}
+
+void
+f4 (double x[][100], int step, int limit)
+{
+ for (int i = 0; i < limit; i += step)
+ for (int j = 0; j < 100; ++j)
+ x[j][i] = 100;
+}
+
+/* { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 2\)} 4 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {Hoisting check} 4 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {Versioning loop [0-9]+ \(depth 1\)} 4 "lversion" } } */
diff --git a/gcc/testsuite/gcc.dg/loop-versioning-3.c b/gcc/testsuite/gcc.dg/loop-versioning-3.c
new file mode 100644
index 0000000000000000000000000000000000000000..3e9c9f391e7a8d2428de8e0b8876c39cbdcfaf60
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/loop-versioning-3.c
@@ -0,0 +1,24 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Versioning these loops for when both steps are 1 allows loop
+ interchange. */
+
+void
+f1 (double x[][100], int step1, int step2, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j * step1][i * step2] = 100;
+}
+
+void
+f2 (double x[][100], int step1, int step2, int limit)
+{
+ for (int i = 0; i < limit; i += step1)
+ for (int j = 0; j < limit; j += step2)
+ x[j][i] = 100;
+}
+
+/* { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 2\)} 4 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {Hoisting check} 4 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {Versioning loop} 2 "lversion" } } */
diff --git a/gcc/testsuite/gcc.dg/loop-versioning-4.c b/gcc/testsuite/gcc.dg/loop-versioning-4.c
new file mode 100644
index 0000000000000000000000000000000000000000..4083952e7f1f022efd40feeda6fa4ff31a07352d
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/loop-versioning-4.c
@@ -0,0 +1,38 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* These shouldn't be versioned; it's extremely likely that the code
+ is emulating two-dimensional arrays. */
+
+void
+f1 (double *x, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[i * step + j] = 100;
+}
+
+void
+f2 (double *x, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j * step + i] = 100;
+}
+
+void
+f3 (double *x, int *offsets, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[i * step + j + offsets[i]] = 100;
+}
+
+void
+f4 (double *x, int *offsets, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j * step + i + offsets[i]] = 100;
+}
+
+/* { dg-final { scan-tree-dump-not {Versioning loop} "lversion" } } */
diff --git a/gcc/testsuite/gcc.dg/loop-versioning-5.c b/gcc/testsuite/gcc.dg/loop-versioning-5.c
new file mode 100644
index 0000000000000000000000000000000000000000..c2a0ab621fa56a92ce2fed75fb9bc6aa94e6d532
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/loop-versioning-5.c
@@ -0,0 +1,17 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* There's no information about whether STEP1 or STEP2 is innermost,
+ so we should assume the code is sensible and version for the inner
+ evolution, i.e. when STEP2 is 1. */
+
+void
+f1 (double *x, int step1, int step2, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[i * step1 + j * step2] = 100;
+}
+
+/* { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 2\) for when step2} 1 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {Want to version loop} 1 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {Versioning loop} 1 "lversion" } } */
diff --git a/gcc/testsuite/gcc.dg/loop-versioning-6.c b/gcc/testsuite/gcc.dg/loop-versioning-6.c
new file mode 100644
index 0000000000000000000000000000000000000000..b4f9b82e169ab35a6b924988ddcb99691c79f9bc
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/loop-versioning-6.c
@@ -0,0 +1,30 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* The read from y in f1 will be hoisted to the outer loop. In general
+ it's not worth versioning outer loops when the inner loops don't also
+ benefit.
+
+ This test is meant to be a slight counterexample, since versioning
+ does lead to cheaper outer-loop vectorization. However, the benefit
+ isn't enough to justify the cost. */
+
+void
+f1 (double *restrict x, double *restrict y, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[i + j] = y[i * step];
+}
+
+/* A similar example in which the read can't be hoisted, but could
+ for example be handled by vectorizer alias checks. */
+
+void
+f2 (double *x, double *y, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[i + j] = y[i * step];
+}
+
+/* { dg-final { scan-tree-dump-not {Versioning loop} "lversion" } } */
diff --git a/gcc/testsuite/gcc.dg/loop-versioning-7.c b/gcc/testsuite/gcc.dg/loop-versioning-7.c
new file mode 100644
index 0000000000000000000000000000000000000000..53ca2dec0c7e84d0860c4b63a0076f8f333b14a4
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/loop-versioning-7.c
@@ -0,0 +1,32 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Check that versioning can handle arrays of structures. */
+
+struct foo {
+ int a, b, c;
+};
+
+void
+f1 (struct foo *x, int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[stepx * i].a = 1;
+ x[stepx * i].b = 2;
+ x[stepx * i].c = 3;
+ }
+}
+
+void
+f2 (struct foo *x, int stepx, int limit)
+{
+ for (int i = 0; i < limit; i += stepx)
+ {
+ x[i].a = 1;
+ x[i].b = 2;
+ x[i].c = 3;
+ }
+}
+
+/* { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 1\)} 2 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {Versioning loop} 2 "lversion" } } */
diff --git a/gcc/testsuite/gcc.dg/loop-versioning-8.c b/gcc/testsuite/gcc.dg/loop-versioning-8.c
new file mode 100644
index 0000000000000000000000000000000000000000..a487106b1e2bbcec8ebb3e42a5a7bae27c021f4b
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/loop-versioning-8.c
@@ -0,0 +1,44 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Check that versioning can handle arrays of structures that wrap
+ subarrays. */
+
+struct foo {
+ int a[100];
+};
+
+void
+f1 (struct foo *x, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j * step].a[i] = 100;
+}
+
+void
+f2 (struct foo *x, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j].a[i * step] = 100;
+}
+
+void
+f3 (struct foo *x, int step, int limit)
+{
+ for (int i = 0; i < 100; ++i)
+ for (int j = 0; j < limit; j += step)
+ x[j].a[i] = 100;
+}
+
+void
+f4 (struct foo *x, int step, int limit)
+{
+ for (int i = 0; i < limit; i += step)
+ for (int j = 0; j < 100; ++j)
+ x[j].a[i] = 100;
+}
+
+/* { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 2\)} 4 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {Hoisting check} 4 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {Versioning loop [0-9]+ \(depth 1\)} 4 "lversion" } } */
diff --git a/gcc/testsuite/gcc.dg/loop-versioning-9.c b/gcc/testsuite/gcc.dg/loop-versioning-9.c
new file mode 100644
index 0000000000000000000000000000000000000000..e93a7009c86823a57479d171fe68c93f22c45aab
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/loop-versioning-9.c
@@ -0,0 +1,48 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Check that versioning can handle small groups of accesses. */
+
+void
+f1 (int *x, int *y, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ x[i] = y[i * step * 2] + y[i * step * 2 + 1];
+}
+
+void
+f2 (int *x, int *y, __INTPTR_TYPE__ step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ x[i] = y[i * step * 2] + y[i * step * 2 + 1];
+}
+
+void
+f3 (int *x, int *y, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ x[i] = y[i * step * 3] + y[i * step * 3 + 2];
+}
+
+void
+f4 (int *x, int *y, __INTPTR_TYPE__ step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ x[i] = y[i * step * 3] + y[i * step * 3 + 2];
+}
+
+void
+f5 (int *x, int *y, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ x[i] = y[i * step * 4] + y[i * step * 4 + 3];
+}
+
+void
+f6 (int *x, int *y, __INTPTR_TYPE__ step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ x[i] = y[i * step * 4] + y[i * step * 4 + 3];
+}
+
+/* { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 1\)} 6 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {Versioning loop} 6 "lversion" } } */
diff --git a/gcc/testsuite/gfortran.dg/loop_versioning_1.f90 b/gcc/testsuite/gfortran.dg/loop_versioning_1.f90
new file mode 100644
index 0000000000000000000000000000000000000000..68617f0f31a884e3380bcaefe899fe1c0084d5c5
--- /dev/null
+++ b/gcc/testsuite/gfortran.dg/loop_versioning_1.f90
@@ -0,0 +1,28 @@
+! { dg-options "-O3 -fdump-tree-lversion-details" }
+
+! The simplest IV case.
+
+subroutine f1(x)
+ real :: x(:)
+ x(:) = 100
+end subroutine f1
+
+subroutine f2(x, n, step)
+ integer :: n, step
+ real :: x(n * step)
+ do i = 1, n
+ x(i * step) = 100
+ end do
+end subroutine f2
+
+subroutine f3(x, limit, step)
+ integer :: limit, step
+ real :: x(limit)
+ do i = 1, limit, step
+ x(i) = 100
+ end do
+end subroutine f3
+
+! { dg-final { scan-tree-dump-times {likely to be innermost dimension} 1 "lversion" } }
+! { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 1\)} 3 "lversion" } }
+! { dg-final { scan-tree-dump-times {Versioning loop} 3 "lversion" } }
diff --git a/gcc/testsuite/gfortran.dg/loop_versioning_2.f90 b/gcc/testsuite/gfortran.dg/loop_versioning_2.f90
new file mode 100644
index 0000000000000000000000000000000000000000..8839022e3cc2ddc7d7aa13428d6c5d2d877848be
--- /dev/null
+++ b/gcc/testsuite/gfortran.dg/loop_versioning_2.f90
@@ -0,0 +1,38 @@
+! { dg-options "-O3 -fdump-tree-lversion-details -fno-frontend-loop-interchange" }
+
+! We could version the loop for when the first dimension has a stride
+! of 1, but at present there's no real benefit. The gimple loop
+! interchange pass couldn't handle the versioned loop, and interchange
+! is instead done by the frontend (but disabled by the options above).
+
+subroutine f1(x)
+ real :: x(:, :)
+ do i = lbound(x, 1), ubound(x, 1)
+ do j = lbound(x, 2), ubound(x, 2)
+ x(i, j) = 100
+ end do
+ end do
+end subroutine f1
+
+subroutine f2(x, n, step)
+ integer :: n, step
+ real :: x(100, 100)
+ do i = 1, n
+ do j = 1, n
+ x(i * step, j) = 100
+ end do
+ end do
+end subroutine f2
+
+subroutine f3(x, n, step)
+ integer :: n, step
+ real :: x(n * step, n)
+ do i = 1, n
+ do j = 1, n
+ x(i * step, j) = 100
+ end do
+ end do
+end subroutine f3
+
+! { dg-final { scan-tree-dump-times {likely to be innermost dimension} 1 "lversion" } }
+! { dg-final { scan-tree-dump-not {Versioning loop} "lversion" } }
diff --git a/gcc/testsuite/gfortran.dg/loop_versioning_3.f90 b/gcc/testsuite/gfortran.dg/loop_versioning_3.f90
new file mode 100644
index 0000000000000000000000000000000000000000..19332f88a79c4f75d08291274cca7b3591b44b4b
--- /dev/null
+++ b/gcc/testsuite/gfortran.dg/loop_versioning_3.f90
@@ -0,0 +1,29 @@
+! { dg-options "-O3 -fdump-tree-lversion-details -fno-frontend-loop-interchange" }
+
+! Test a case in which the outer loop iterates over the inner dimension.
+! The options above prevent the frontend from interchanging the loops.
+
+subroutine f1(x, limit, step, n)
+ integer :: limit, step, n
+ real :: x(limit, n)
+ do i = 1, limit, step
+ do j = 1, n
+ x(i, j) = 100
+ end do
+ end do
+end subroutine f1
+
+subroutine f2(x, n, limit, step)
+ integer :: n, limit, step
+ real :: x(limit, n)
+ do i = 1, n
+ do j = 1, limit, step
+ x(j, i) = 100
+ end do
+ end do
+end subroutine f2
+
+! FIXME: The frontend doesn't give us enough information to tell which loop
+! is iterating over the innermost dimension, so we optimistically
+! assume the inner one is.
+! { dg-final { scan-tree-dump-not {Versioning loop} "lversion" { xfail *-*-* } } }
diff --git a/gcc/testsuite/gfortran.dg/loop_versioning_4.f90 b/gcc/testsuite/gfortran.dg/loop_versioning_4.f90
new file mode 100644
index 0000000000000000000000000000000000000000..90d7769431cf09ec7591252c1cbe2b6c368e7c1f
--- /dev/null
+++ b/gcc/testsuite/gfortran.dg/loop_versioning_4.f90
@@ -0,0 +1,52 @@
+! { dg-options "-O3 -fdump-tree-lversion-details -fno-frontend-loop-interchange" }
+
+! Test cases in which versioning is useful for a two-dimensional array.
+
+subroutine f1(x)
+ real :: x(:, :)
+ x(:, :) = 100
+end subroutine f1
+
+subroutine f2(x)
+ real :: x(:, :)
+ do i = lbound(x, 1), ubound(x, 1)
+ do j = lbound(x, 2), ubound(x, 2)
+ x(j, i) = 100
+ end do
+ end do
+end subroutine f2
+
+subroutine f3(x, n, step)
+ integer :: n, step
+ real :: x(100, 100)
+ do i = 1, n
+ do j = 1, n
+ x(j * step, i) = 100
+ end do
+ end do
+end subroutine f3
+
+subroutine f4(x, n, step)
+ integer :: n, step
+ real :: x(n * step, n)
+ do i = 1, n
+ do j = 1, n
+ x(j * step, i) = 100
+ end do
+ end do
+end subroutine f4
+
+subroutine f5(x, n, limit, step)
+ integer :: n, limit, step
+ real :: x(limit, n)
+ do i = 1, n
+ do j = 1, limit, step
+ x(j, i) = 100
+ end do
+ end do
+end subroutine f5
+
+! { dg-final { scan-tree-dump-times {likely to be innermost dimension} 2 "lversion" } }
+! { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 2\)} 5 "lversion" } }
+! { dg-final { scan-tree-dump-times {Hoisting check} 5 "lversion" } }
+! { dg-final { scan-tree-dump-times {Versioning loop [0-9]+ \(depth 1\)} 5 "lversion" } }
diff --git a/gcc/testsuite/gfortran.dg/loop_versioning_5.f90 b/gcc/testsuite/gfortran.dg/loop_versioning_5.f90
new file mode 100644
index 0000000000000000000000000000000000000000..6b6235446aa19b54b6e6fb455e04dcfc77139126
--- /dev/null
+++ b/gcc/testsuite/gfortran.dg/loop_versioning_5.f90
@@ -0,0 +1,56 @@
+! { dg-options "-O3 -fdump-tree-lversion-details -fno-frontend-loop-interchange" }
+
+! Make sure that in a "badly nested" loop, we don't treat the inner loop
+! as iterating over the inner dimension with a variable stride.
+
+subroutine f1(x, n)
+ integer :: n
+ real :: x(100, 100)
+ do i = 1, n
+ do j = 1, n
+ x(i, j) = 100
+ end do
+ end do
+end subroutine f1
+
+subroutine f2(x, n, step)
+ integer :: n, step
+ real :: x(100, 100)
+ do i = 1, n
+ do j = 1, n
+ x(i, j * step) = 100
+ end do
+ end do
+end subroutine f2
+
+subroutine f3(x, n)
+ integer :: n
+ real :: x(n, n)
+ do i = 1, n
+ do j = 1, n
+ x(i, j) = 100
+ end do
+ end do
+end subroutine f3
+
+subroutine f4(x, n, step)
+ integer :: n, step
+ real :: x(n, n * step)
+ do i = 1, n
+ do j = 1, n
+ x(i, j * step) = 100
+ end do
+ end do
+end subroutine f4
+
+subroutine f5(x, n, limit, step)
+ integer :: n, limit, step
+ real :: x(n, limit)
+ do i = 1, n
+ do j = 1, limit, step
+ x(i, j) = 100
+ end do
+ end do
+end subroutine f5
+
+! { dg-final { scan-tree-dump-not {Versioning loop} "lversion" } }
diff --git a/gcc/testsuite/gfortran.dg/loop_versioning_6.f90 b/gcc/testsuite/gfortran.dg/loop_versioning_6.f90
new file mode 100644
index 0000000000000000000000000000000000000000..16dda837ee657aa44899aeedefe674fff02832f6
--- /dev/null
+++ b/gcc/testsuite/gfortran.dg/loop_versioning_6.f90
@@ -0,0 +1,93 @@
+! { dg-options "-O3 -fdump-tree-lversion-details" }
+
+! Check that versioning can handle small groups of accesses.
+
+subroutine f1(x)
+ real :: x(:)
+ do i = lbound(x, 1), ubound(x, 1) / 2
+ x(i * 2) = 100
+ x(i * 2 + 1) = 101
+ end do
+end subroutine f1
+
+subroutine f2(x, n, step)
+ integer :: n, step
+ real :: x(n * step * 2)
+ do i = 1, n
+ x(i * step * 2) = 100
+ x(i * step * 2 + 1) = 101
+ end do
+end subroutine f2
+
+subroutine f3(x, limit, step)
+ integer :: limit, step
+ real :: x(limit * 2)
+ do i = 1, limit, step
+ x(i * 2) = 100
+ x(i * 2 + 1) = 101
+ end do
+end subroutine f3
+
+subroutine f4(x)
+ real :: x(:)
+ do i = lbound(x, 1), ubound(x, 1) / 3
+ x(i * 3) = 100
+ x(i * 3 + 1) = 101
+ x(i * 3 + 2) = 102
+ end do
+end subroutine f4
+
+subroutine f5(x, n, step)
+ integer :: n, step
+ real :: x(n * step * 3)
+ do i = 1, n
+ x(i * step * 3) = 100
+ x(i * step * 3 + 1) = 101
+ x(i * step * 3 + 2) = 102
+ end do
+end subroutine f5
+
+subroutine f6(x, limit, step)
+ integer :: limit, step
+ real :: x(limit * 3)
+ do i = 1, limit, step
+ x(i * 3) = 100
+ x(i * 3 + 1) = 101
+ x(i * 3 + 2) = 102
+ end do
+end subroutine f6
+
+subroutine f7(x)
+ real :: x(:)
+ do i = lbound(x, 1), ubound(x, 1) / 4
+ x(i * 4) = 100
+ x(i * 4 + 1) = 101
+ x(i * 4 + 2) = 102
+ x(i * 4 + 3) = 103
+ end do
+end subroutine f7
+
+subroutine f8(x, n, step)
+ integer :: n, step
+ real :: x(n * step * 4)
+ do i = 1, n
+ x(i * step * 4) = 100
+ x(i * step * 4 + 1) = 101
+ x(i * step * 4 + 2) = 102
+ x(i * step * 4 + 3) = 103
+ end do
+end subroutine f8
+
+subroutine f9(x, limit, step)
+ integer :: limit, step
+ real :: x(limit * 4)
+ do i = 1, limit, step
+ x(i * 4) = 100
+ x(i * 4 + 1) = 101
+ x(i * 4 + 2) = 102
+ x(i * 4 + 3) = 103
+ end do
+end subroutine f9
+
+! { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 1\)} 9 "lversion" } }
+! { dg-final { scan-tree-dump-times {Versioning loop} 9 "lversion" } }
diff --git a/gcc/testsuite/gfortran.dg/loop_versioning_7.f90 b/gcc/testsuite/gfortran.dg/loop_versioning_7.f90
new file mode 100644
index 0000000000000000000000000000000000000000..0f004b57138c737b1f8ad75e5fc29439244eeb1f
--- /dev/null
+++ b/gcc/testsuite/gfortran.dg/loop_versioning_7.f90
@@ -0,0 +1,67 @@
+! { dg-options "-O3 -fdump-tree-lversion-details" }
+
+! Check that versioning can handle small groups of accesses, with the
+! group being a separate array dimension.
+
+subroutine f1(x, n, step)
+ integer :: n, step
+ real :: x(2, n * step)
+ do i = 1, n
+ x(1, i * step) = 100
+ x(2, i * step) = 101
+ end do
+end subroutine f1
+
+subroutine f2(x, limit, step)
+ integer :: limit, step
+ real :: x(2, limit)
+ do i = 1, limit, step
+ x(1, i) = 100
+ x(2, i) = 101
+ end do
+end subroutine f2
+
+subroutine f3(x, n, step)
+ integer :: n, step
+ real :: x(3, n * step)
+ do i = 1, n
+ x(1, i * step) = 100
+ x(2, i * step) = 101
+ x(3, i * step) = 102
+ end do
+end subroutine f3
+
+subroutine f4(x, limit, step)
+ integer :: limit, step
+ real :: x(3, limit)
+ do i = 1, limit, step
+ x(1, i) = 100
+ x(2, i) = 101
+ x(3, i) = 102
+ end do
+end subroutine f4
+
+subroutine f5(x, n, step)
+ integer :: n, step
+ real :: x(4, n * step)
+ do i = 1, n
+ x(1, i * step) = 100
+ x(2, i * step) = 101
+ x(3, i * step) = 102
+ x(4, i * step) = 103
+ end do
+end subroutine f5
+
+subroutine f6(x, limit, step)
+ integer :: limit, step
+ real :: x(4, limit)
+ do i = 1, limit, step
+ x(1, i) = 100
+ x(2, i) = 101
+ x(3, i) = 102
+ x(4, i) = 103
+ end do
+end subroutine f6
+
+! { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 1\)} 6 "lversion" } }
+! { dg-final { scan-tree-dump-times {Versioning loop} 6 "lversion" } }
diff --git a/gcc/testsuite/gfortran.dg/loop_versioning_8.f90 b/gcc/testsuite/gfortran.dg/loop_versioning_8.f90
new file mode 100644
index 0000000000000000000000000000000000000000..98ec30cbb81028353364ee12ebe64a14d7ffb23e
--- /dev/null
+++ b/gcc/testsuite/gfortran.dg/loop_versioning_8.f90
@@ -0,0 +1,13 @@
+! { dg-options "-O3 -fdump-tree-lversion-details" }
+
+! Check that versioning is applied to a gather-like reduction operation.
+
+function f(x, index, n)
+ integer :: n
+ real :: x(:)
+ integer :: index(n)
+ f = sum(x(index(:)))
+end function f
+
+! { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 1\)} 1 "lversion" } }
+! { dg-final { scan-tree-dump-times {Versioning loop} 1 "lversion" } }
diff --git a/gcc/timevar.def b/gcc/timevar.def
index 91221ae5b0ef23746d7b7dad4b2f631ef33b482d..033b48040bb07307fb77db593105da00dfa12082 100644
--- a/gcc/timevar.def
+++ b/gcc/timevar.def
@@ -234,6 +234,7 @@ DEFTIMEVAR (TV_DSE1 , "dead store elim1")
DEFTIMEVAR (TV_DSE2 , "dead store elim2")
DEFTIMEVAR (TV_LOOP , "loop analysis")
DEFTIMEVAR (TV_LOOP_INIT , "loop init")
+DEFTIMEVAR (TV_LOOP_VERSIONING , "loop versioning")
DEFTIMEVAR (TV_LOOP_MOVE_INVARIANTS , "loop invariant motion")
DEFTIMEVAR (TV_LOOP_UNROLL , "loop unrolling")
DEFTIMEVAR (TV_LOOP_DOLOOP , "loop doloop")
diff --git a/gcc/tree-pass.h b/gcc/tree-pass.h
index af15adc8e0cf39246791fd87a50e3f9502eb99a1..d9aed62b26f68965e0f54a2d3a252dee22c83907 100644
--- a/gcc/tree-pass.h
+++ b/gcc/tree-pass.h
@@ -362,6 +362,7 @@ extern gimple_opt_pass *make_pass_fix_loops (gcc::context *ctxt);
extern gimple_opt_pass *make_pass_tree_loop (gcc::context *ctxt);
extern gimple_opt_pass *make_pass_tree_no_loop (gcc::context *ctxt);
extern gimple_opt_pass *make_pass_tree_loop_init (gcc::context *ctxt);
+extern gimple_opt_pass *make_pass_loop_versioning (gcc::context *ctxt);
extern gimple_opt_pass *make_pass_lim (gcc::context *ctxt);
extern gimple_opt_pass *make_pass_linterchange (gcc::context *ctxt);
extern gimple_opt_pass *make_pass_tree_unswitch (gcc::context *ctxt);
diff --git a/gcc/tree-ssa-propagate.h b/gcc/tree-ssa-propagate.h
index 56e1b1c13794b51543c8a7a1cc113471fb4de82a..73512f65f28e21a73daec9d3d49560ecdd4468d9 100644
--- a/gcc/tree-ssa-propagate.h
+++ b/gcc/tree-ssa-propagate.h
@@ -104,7 +104,7 @@ class substitute_and_fold_engine
virtual bool fold_stmt (gimple_stmt_iterator *) { return false; }
virtual tree get_value (tree) { return NULL_TREE; }
- bool substitute_and_fold (void);
+ bool substitute_and_fold (basic_block = NULL);
bool replace_uses_in (gimple *);
bool replace_phi_args_in (gphi *);
};
diff --git a/gcc/tree-ssa-propagate.c b/gcc/tree-ssa-propagate.c
index 1f3ccafa00e3782e43439f95489fadaa906c9bb1..8225ebee49e72d0c5c211a2e9341ff50cb50b314 100644
--- a/gcc/tree-ssa-propagate.c
+++ b/gcc/tree-ssa-propagate.c
@@ -1154,6 +1154,10 @@ substitute_and_fold_dom_walker::before_dom_children (basic_block bb)
/* Perform final substitution and folding of propagated values.
+ Process the whole function if BLOCK is null, otherwise only
+ process the blocks that BLOCK dominates. In the latter case,
+ it is the caller's responsibility to ensure that dominator
+ information is available and up-to-date.
PROP_VALUE[I] contains the single value that should be substituted
at every use of SSA name N_I. If PROP_VALUE is NULL, no values are
@@ -1170,16 +1174,24 @@ substitute_and_fold_dom_walker::before_dom_children (basic_block bb)
Return TRUE when something changed. */
bool
-substitute_and_fold_engine::substitute_and_fold (void)
+substitute_and_fold_engine::substitute_and_fold (basic_block block)
{
if (dump_file && (dump_flags & TDF_DETAILS))
fprintf (dump_file, "\nSubstituting values and folding statements\n\n");
memset (&prop_stats, 0, sizeof (prop_stats));
- calculate_dominance_info (CDI_DOMINATORS);
+ /* Don't call calculate_dominance_info when iterating over a subgraph.
+ Callers that are using the interface this way are likely to want to
+ iterate over several disjoint subgraphs, and it would be expensive
+ in enable-checking builds to revalidate the whole dominance tree
+ each time. */
+ if (block)
+ gcc_assert (dom_info_state (CDI_DOMINATORS));
+ else
+ calculate_dominance_info (CDI_DOMINATORS);
substitute_and_fold_dom_walker walker (CDI_DOMINATORS, this);
- walker.walk (ENTRY_BLOCK_PTR_FOR_FN (cfun));
+ walker.walk (block ? block : ENTRY_BLOCK_PTR_FOR_FN (cfun));
/* We cannot remove stmts during the BB walk, especially not release
SSA names there as that destroys the lattice of our callers.
diff --git a/gcc/tree-vrp.h b/gcc/tree-vrp.h
index c251329a195fe8def91998562c2dccd2f09bf8fa..601a3734208a40493029cdbb36b7c3c10b93edce 100644
--- a/gcc/tree-vrp.h
+++ b/gcc/tree-vrp.h
@@ -187,7 +187,7 @@ extern void register_edge_assert_for (tree, edge, enum tree_code,
tree, tree, vec<assert_info> &);
extern bool stmt_interesting_for_vrp (gimple *);
extern void set_value_range_to_varying (value_range *);
-extern bool range_includes_zero_p (const value_range *);
+extern bool range_includes_p (const value_range *, HOST_WIDE_INT);
extern bool infer_value_range (gimple *, tree, tree_code *, tree *);
extern void set_value_range_to_nonnull (value_range *, tree);
@@ -218,4 +218,12 @@ extern int value_inside_range (tree, tree, tree);
extern tree get_single_symbol (tree, bool *, tree *);
extern void maybe_set_nonzero_bits (edge, tree);
extern value_range_kind determine_value_range (tree, wide_int *, wide_int *);
+
+/* Return TRUE if *VR includes the value zero. */
+inline bool
+range_includes_zero_p (const value_range *vr)
+{
+ return range_includes_p (vr, 0);
+}
+
#endif /* GCC_TREE_VRP_H */
diff --git a/gcc/tree-vrp.c b/gcc/tree-vrp.c
index 17b0b6c60378f5b6e9c2e7d34632bd0f9fe8faa3..b53349cb30f543bedd8cec0b603acb3a0b248e19 100644
--- a/gcc/tree-vrp.c
+++ b/gcc/tree-vrp.c
@@ -1081,15 +1081,15 @@ value_inside_range (tree val, tree min, tree max)
}
-/* Return TRUE if *VR includes the value zero. */
+/* Return TRUE if *VR includes the value X. */
bool
-range_includes_zero_p (const value_range *vr)
+range_includes_p (const value_range *vr, HOST_WIDE_INT x)
{
if (vr->varying_p () || vr->undefined_p ())
return true;
- tree zero = build_int_cst (vr->type (), 0);
- return vr->may_contain_p (zero);
+ tree x_cst = build_int_cst (vr->type (), x);
+ return vr->may_contain_p (x_cst);
}
/* If *VR has a value range that is a single constant value return that,
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Add a loop versioning pass
2018-10-30 10:55 Add a loop versioning pass Richard Biener
2018-11-09 10:46 ` Kyrill Tkachov
@ 2018-11-28 16:48 ` Richard Sandiford
2018-11-29 11:31 ` Martin Jambor
2018-12-03 13:16 ` Richard Biener
1 sibling, 2 replies; 17+ messages in thread
From: Richard Sandiford @ 2018-11-28 16:48 UTC (permalink / raw)
To: Richard Biener; +Cc: gcc-patches
Thanks for the review and sorry for the long time replying.
Richard Biener <rguenther@suse.de> writes:
>> This patch adds a pass that versions loops with variable index strides
>> for the case in which the stride is 1. E.g.:
>>
>> for (int i = 0; i < n; ++i)
>> x[i * stride] = ...;
>>
>> becomes:
>>
>> if (stepx == 1)
>> for (int i = 0; i < n; ++i)
>> x[i] = ...;
>> else
>> for (int i = 0; i < n; ++i)
>> x[i * stride] = ...;
>>
>> This is useful for both vector code and scalar code, and in some cases
>> can enable further optimisations like loop interchange or pattern
>> recognition.
>>
>> The pass gives a 7.6% improvement on Cortex-A72 for 554.roms_r at -O3
>> and a 2.4% improvement for 465.tonto. I haven't found any SPEC tests
>> that regress.
>>
>> Sizewise, there's a 10% increase in .text for both 554.roms_r and
>> 465.tonto. That's obviously a lot, but in tonto's case it's because
>> the whole program is written using assumed-shape arrays and pointers,
>> so a large number of functions really do benefit from versioning.
>> roms likewise makes heavy use of assumed-shape arrays, and that
>> improvement in performance IMO justifies the code growth.
>
> Ouch. I know that at least with LTO IPA-CP can do "quite" some
> propagation of constant strides. Not sure if we're aggressive
> enough in actually doing the cloning for all cases we figure out
> strides though. But my question is how we can avoid doing the
> versioning for loops in the copy that did not have the IPA-CPed
> stride of one? Ideally we'd be able to mark individual references
> as {definitely,likely,unlikely,not}-unit-stride?
This is a more elaborate version of what I was trying to do with
the IFN I'd mentioned on IRC a few weeks back. It would be a bit
like a long-living ASSERT_EXPR that provides hints about the value
of the SSA name. Another alternative would be to annotate the
SSA name itself, but then we'd easily lose the information during
optimisation.
But will IPA-CP conditionally use a clone for a stride of 1 for
calls that pass a variable stride that might or might not be 1?
E.g. if it clones foo as foo.1 for calls in which a stride parameter
is 1 at compile time, does it also make foo call foo.1 when the stride
parameter is 1 at runtime? If not, that sounds like a missed optimisation.
If so, prune_conditions should stop us from versioning.
>> The next biggest .text increase is 4.5% for 548.exchange2_r. I did see
>> a small (0.4%) speed improvement there, but although both 3-iteration runs
>> produced stable results, that might still be noise. There was a slightly
>> larger (non-noise) improvement for a 256-bit SVE model.
>>
>> 481.wrf and 521.wrf_r .text grew by 2.8% and 2.5% respectively, but
>> without any noticeable improvement in performance. No other test grew
>> by more than 2%.
>>
>> Although the main SPEC beneficiaries are all Fortran tests, the
>> benchmarks we use for SVE also include some C and C++ tests that
>> benefit.
>
> Did you see any slowdown, for example because versioning was forced
> to be on an innermost loop? I'm thinking of the testcase in
> PR87561 where we do have strided accesses in the innermost loop.
Not so far.
> Since you cite performance numbers how did you measure them?
> I assume -Ofast -march=native but did you check with -flto?
-O3 -march=native. No, didn't check LTO, will do that.
>> Using -frepack-arrays gives the same benefits in many Fortran cases.
>> The problem is that using that option inappropriately can force a full
>> array copy for arguments that the function only reads once, and so it
>> isn't really something we can turn on by default. The new pass is
>> supposed to give most of the benefits of -frepack-arrays without
>> the risk of unnecessary repacking.
>>
>> The patch therefore enables the pass by default at -O3.
>
> I think that's reasonable.
>
> One possible enhancement would be to add a value-profile for the
> strides so we can guide this optimization better.
>
> The pass falls foul of C++ class make small methods of everything.
> That makes following the code very hard. Please inline single-used
> methods in callers wherever possible to make the code read
> more like GCC code (using GCC API).
I'll bite my tongue on this one :-)
> The pass contains an awful lot of heuristics :/ Like last year
> with the interchange pass I would suggest to rip most of it out
> and first lay infrastructure with the cases you can positively
> identify without applying heuristics or "hacks" like stripping
> semantically required casts. That makes it also clear which
> testcases test which code-path. That said, all the analyze
> multiplications/plusses/factors stuff was extremely hard to review
> and I have no overall picture why this is all so complicated or
> necessary.
I think the tests do cover most of this -- was glad to see that
when Kyrill tried taking something out, one of the tests started
to fail :-). And the comments in the tests as well as the code
are supposed to explain why this is desirable. But I can try to
spruce up the comments in the code if they're not detailed enough.
The problem is that the motivating (Fortran) case can't be identified
without applying heuristics. All we see is a collection of adds and
multiplies, with no semantic information to say which multiplies are for
the inner dimension and which aren't. I agree it would be nicer not
to have them, which is why I'd originally suggested the IFN to provide
information about dimensions.
>> +@item -fversion-loops-for-strides
>> +@opindex fversion-loops-for-strides
>> +If a loop iterates over an array with a variable stride, create another
>> +version of the loop that assumes the stride is always 1. For example:
>> +
>> +@smallexample
>> +for (int i = 0; i < n; ++i)
>> + x[i * stride] = @dots{};
>> +@end smallexample
>> +
>> +becomes:
>> +
>> +@smallexample
>> +if (stride == 1)
>> + for (int i = 0; i < n; ++i)
>> + x[i] = @dots{};
>> +else
>> + for (int i = 0; i < n; ++i)
>> + x[i * stride] = @dots{};
>> +@end smallexample
>> +
>> +This is particularly useful for assumed-shape arrays in Fortran.
>
> "where it allows better vectorization assuming contiguous accesses."
Mind if I make it "where, for example, ..."? Vectorisation isn't the
only target here.
>> + /* We'd like to version the loop for the case in which these
>> + SSA_NAMEs are all equal to 1 at runtime. */
>> + vec<tree> unity_names;
>> +
>> + /* The set of SSA_NAMEs in UNITY_NAMES, keyed off the SSA_NAME_VERSION. */
>> + bitmap_head unity_name_ids;
>
> I wonder why you need both, a vector and a bitmap since you can
> cheaply iterate over the bitmap and get the SSA name via ssa_name (version)?
Ah, yeah, that would be better.
>> + if (dump_file && (dump_flags & TDF_DETAILS))
>> + {
>> + fprintf (dump_file, ";; Want to version loop %d (depth %d)"
>> + " for when ", loop->num, loop_depth (loop));
>> + print_generic_expr (dump_file, name, TDF_SLIM);
>> + fprintf (dump_file, " == 1");
>
> Since you are writing a new pass you want to use the new dump interface.
>
> if (dump_enabled_p ())
> dump_printf (MSG_NOTE, ";; Want to version loop %d (depth %d)"
> " for when %E == 1", loop->num, loop_depth (loop), name);
> ...
>
> it's much nicer to be able to use %E/%G than separate calls for the
> tree parts.
Thanks to Kyrill and Ramana for fixing this.
>> +/* Return true if in principle it is worth versioning an index fragment of
>> + the form:
>> +
>> + (i * b * SCALE) / FACTOR
>> +
>> + for the case in which b == 1. */
>> +
>> +bool
>> +loop_versioning::acceptable_scale_p (tree scale, poly_uint64 factor)
>> +{
>> + /* See whether SCALE is a constant multiple of FACTOR, and if the
>> + multiple is small enough for us to treat it as a potential grouped
>> + access. For example:
>> +
>> + for (auto i : ...)
>> + y[i] = f (x[4 * i * stride],
>> + x[4 * i * stride + 1],
>> + x[4 * i * stride + 2]);
>> +
>> + would benefit from versioning for the case in which stride == 1.
>> + High multiples of i * stride are less likely to benefit, and could
>> + indicate a simulated multi-dimensional array.
>> +
>> + This is just a heuristic, to avoid having to do expensive group
>> + analysis of the data references in a loop. */
>> + poly_uint64 const_scale;
>> + unsigned int multiple;
>> + if (poly_int_tree_p (scale, &const_scale)
>> + && constant_multiple_p (const_scale, factor, &multiple))
>> + {
>> + unsigned int maxval = PARAM_VALUE (PARAM_LOOP_VERSIONING_GROUP_SIZE);
>> + return IN_RANGE (multiple, 1, maxval);
>
> Hmm. So you _do_ want to version sth like
>
> struct X { int i; int j; } a[2048];
>
> for (int i = start; i < end; ++i)
> a[i*s].i = 1;
>
> ? That is, even with s == 1 the accesses will not become contiguous?
> OK, I suppose that's because you are looking at a stmt in isolation
> and another stmt may access a[i*s].j here.
>
> That is, would it be a future improvement to run sth like the
> vectorizers group access analysis on the references and perform
> this check on whole groups then, possibly better being able to
> constrain what is now the magic parameter PARAM_LOOP_VERSIONING_GROUP_SIZE?
Yeah, possibly. The problem is that we might end up having to reproduce
vectoriser heuristics. E.g. stores with gaps should be vectorisable
in future with SVE, using contiguous predicated stores in which some
lanes are inactive. So we don't necessarily need the .j access for
this to be worthwhile.
But perhaps the argument for versioning for vectorisation is stronger
for grouped accesses than contiguous ones.
Note that the above example doesn't rely on the grouping heuristic
since we just strip the COMPONENT_REF and then analyze the ARRAY_REF
with a factor of 1. The heuristic instead handles things like:
a[i * stride * 2]
etc., and the param is more there to decide when a constant factor is
small enough to be a realistic group size instead of an outer dimension.
E.g. if we see:
a[i * stride * 20000]
the problem is that i * stride * 20000 is likely to be applying
an outer dimension index.
> I guess you tried to constrain it to the stmts access size but of course
> that fails short of handling later SLP vectorized cases.
Right.
>> +/* Decide whether an index fragment of the form:
>> +
>> + (i * STEP) / FACTOR
>> +
>> + is likely to be for an innermost dimension. If we think it is,
>> + return one of the constant values that it could have (returning 1 if
>> + that's a possibility). If think it isn't, return null. Otherwise
>> + return STEP, to indicate that it might or might not be an inner
>> + dimension. */
>> +
>> +tree
>> +loop_versioning::get_step_if_innermost (tree step, poly_uint64 factor)
>> +{
>> + const unsigned int MAX_NITERS = 8;
>> +
>> + tree likely = NULL_TREE;
>> + tree unlikely = NULL_TREE;
>> + tree worklist[MAX_NITERS];
>> + unsigned int length = 0;
>> + worklist[length++] = step;
>> + for (unsigned int i = 0; i < length; ++i)
>> + {
>> + tree expr = worklist[i];
>> +
>> + if (TREE_CONSTANT (expr))
>
> ? CONSTANT_CLASS_P (expr)?
OK.
>> + {
>> + /* See if multiplying by EXPR applies a scale that would be
>> + consistent with an individual access or a small grouped
>> + access. */
>> + if (acceptable_scale_p (expr, factor))
>> + {
>> + likely = expr;
>> + if (integer_onep (expr))
>> + break;
>> + }
>> + else
>> + unlikely = expr;
>> + continue;
>> + }
>> +
>> + /* Otherwise we can only handle SSA names. */
>> + gimple *stmt = maybe_get_stmt (expr);
>> + if (!stmt)
>> + continue;
>> +
>> + /* If EXPR is set by a PHI node, queue its arguments in case
>> + we find one that is consistent with an inner dimension.
>> +
>> + An important instance of this is the Fortran handling of array
>> + descriptors, which calculates the stride of the inner dimension
>> + using a PHI equivalent of:
>> +
>> + raw_stride = a.dim[0].stride;
>> + stride = raw_stride != 0 ? raw_stride : 1;
>> +
>> + (Strides for outer dimensions do not treat 0 specially.) */
>> + if (gphi *phi = dyn_cast <gphi *> (stmt))
>> + {
>> + unsigned int nargs = gimple_phi_num_args (phi);
>> + for (unsigned int j = 0; j < nargs && length < MAX_NITERS; ++j)
>> + worklist[length++] = gimple_phi_arg_def (phi, j);
>> + continue;
>> + }
>
> So only PHIs seed the worklist.
Yeah.
>> + /* If the value is set by an assignment, expect it to be read from
>> + memory (such as an array descriptor) rather than be calculated. */
>> + if (gassign *assign = dyn_cast <gassign *> (stmt))
>> + {
>> + if (gimple_assign_single_p (assign)
>> + && is_gimple_lvalue (gimple_assign_rhs1 (assign)))
>
> Please do not use is_gimple_lvalue, that's supposed to be a gimplifier-only
> predicate. Do you want to use gimple_assign_load_p here?
OK.
>> + continue;
>> +
>> + unlikely = expr;
>> + }
>> +
>> + /* Things like calls don't really tell us anything. */
>> + }
>
> So I don't understand the above fully but it looks like plain
> function arguments STEP will never be 'likely'?
Right. They're neither likely not unlikely, just like a value
loaded from memory.
> That is, I'd appreciate some more comments of what SSA def
> chains we walk and what we look for in the end.
OK, will try.
>> + if (likely)
>> + {
>> + if (dump_file && (dump_flags & TDF_DETAILS))
>> + fprintf (dump_file, "likely to be innermost dimension\n");
>> + return likely;
>> + }
>> +
>> + if (unlikely)
>> + {
>> + if (dump_file && (dump_flags & TDF_DETAILS))
>> + fprintf (dump_file, "probably not innermost dimension\n");
>> + return NULL_TREE;
>> + }
>> +
>> + if (dump_file && (dump_flags & TDF_DETAILS))
>> + {
>> + if (maybe_ne (factor, 1U))
>> + fprintf (dump_file, "didn't find expected scaling factor\n");
>> + else
>> + fprintf (dump_file, "no information about value\n");
>> + }
>> + return step;
>> +}
>> +
>> +/* STEP appears in an index fragment of the form:
>> +
>> + {..., +, STEP}_n / FACTOR
>> +
>> + Remove any conversions and grouping or scaling factors from STEP and
>> + return the underlying index step. Set *STEP_IF_INNERMOST to the value
>> + returned by get_step_if_innermost. */
>> +
>> +tree
>> +loop_versioning::extract_step (tree step, poly_uint64 factor,
>> + tree *step_if_innermost)
>> +{
>> + if (dump_file && (dump_flags & TDF_DETAILS))
>> + {
>> + fprintf (dump_file, ";; step ");
>> + print_generic_expr (dump_file, step, TDF_SLIM);
>> + fprintf (dump_file, ": ");
>> + }
>> +
>> + step = strip_casts (step);
>> +
>> + /* Peel any scaling, which generally happens after conversion to
>> + pointer width. For example, on LP64 systems:
>> +
>> + int *x, i, stride;
>> + ... x[4 * i * stride] ...;
>> +
>> + multiplies i * stride by 4 using ints, then widens the result
>> + to pointer width before multiplying by sizeof (int). */
>> + poly_uint64 scale;
>> + if (TREE_CODE (step) == MULT_EXPR
>> + && poly_int_tree_p (TREE_OPERAND (step, 1), &scale)
>> + && known_eq (scale, factor))
>> + {
>> + step = strip_casts (TREE_OPERAND (step, 0));
>> + if (dump_file && (dump_flags & TDF_DETAILS))
>> + fprintf (dump_file, "scaled, ");
>> + factor = 1;
>> + }
>
> So I think the extra constant scale may or may not be in CHREC_RIGHT
> depending on whether the expression was hoisted out of the loop as
> invariant or implicit in sth like an ARRAY_REF.
For ARRAY_REF the factor is always 1.
>> + /* Peel any grouping multiple; see acceptable_scale_p for details. */
>> + if (TREE_CODE (step) == MULT_EXPR
>> + && acceptable_scale_p (TREE_OPERAND (step, 1), factor))
>> + {
>> + step = strip_casts (TREE_OPERAND (step, 0));
>> + if (dump_file && (dump_flags & TDF_DETAILS))
>> + fprintf (dump_file, "%sgrouped, ",
>> + maybe_ne (factor, 1U) ? "scaled, " : "");
>> + factor = 1;
>> + }
>
> Which means I'm not sure why there are (only) two MULTs being
> looked for and only one has acceptable_scale_p applied...
The first one checks for multiplications by the access size in pointer
width, i.e. for the case in which a[i*b] becomes:
a + ((cast) (i * b) * sizeof (a[0]))
It also happens to catch the case in which sizeof (a[0]) == 1 and i*b is
calculated in pointer width, which if either condition didn't hold would
instead be handled by the second multiplication.
Only the second multiplication has the grouping heuristic applied
because the grouping heuristic is aimed at explicit multiplications
by a constant, as above, and those would normally be done in the same
width as the multiplication by the variable stride. The multiplication
by the group size and the multiplication by the access size (n/a for
ARRAY_REFs) would be folded into one if the multiplications all happen
in pointer width.
>> +{
>> + const unsigned int MAX_NSPLIT = 8;
>> +
>> + if (dump_file && (dump_flags & TDF_DETAILS))
>> + {
>> + fprintf (dump_file, ";; Analyzing use of ");
>> + print_generic_expr (dump_file, expr, TDF_SLIM);
>> + if (maybe_ne (factor, 1U))
>> + {
>> + fprintf (dump_file, " (which addresses ");
>> + print_dec (factor, dump_file);
>> + fprintf (dump_file, " bytes)");
>> + }
>> + fprintf (dump_file, " in loop %d (depth %d)\n",
>> + loop->num, loop_depth (loop));
>> + }
>> +
>> + /* The main problem we have here is that we cannot assume that the
>> + innermost loop iterates over the innermost dimension of an array.
>> + Accidentally adding versioning checks for outer dimensions would
>> + cause the version condition to be false, which as well as bloating
>> + the code would defeat loop versioning benefits for other accesses.
>> +
>> + Unfortunately all we usually see at this stage is general address
>> + arithmetic, with no positive way of identifying how many dimensions
>> + an array access has and which multiplication factors in the address
>> + expression correspond to which array dimensions. In C code this is
>> + often not even explicit in the source, since variable-sized multi-
>> + dimensional arrays are often simulated using one-dimensional arrays.
>> +
>> + The three main ways in which we deal with this are:
>> +
>> + - use heuristics that positively identify steps that are likely
>> + to represent the inner dimension.
>> +
>> + - use heuristics that positively identify steps that are unlikely
>> + to represent the inner dimension.
>> +
>> + - if a part of EXPR is invariant in LOOP, analyze its evolution in
>> + the outer loops to see whether we can positively identify any of
>> + it as iterating over the inner dimension. */
>> + tree best_step = NULL_TREE;
>> + auto_vec<tree, MAX_NSPLIT> worklist;
>> + worklist.quick_push (expr);
>> + unsigned int nsplit = 0;
>> + while (!worklist.is_empty ())
>> + {
>> + expr = strip_casts (worklist.pop ());
>> + tree_code code = TREE_CODE (expr);
>> +
>> + if (code == POLYNOMIAL_CHREC)
>> + {
>> + /* Analyze the CHREC_RIGHT as the step in an index fragment. */
>> + tree step_if_innermost;
>> + tree step = extract_step (CHREC_RIGHT (expr), factor,
>> + &step_if_innermost);
>> + if (!best_step)
>> + {
>> + /* This is the outermost chrec for the original expression.
>> + It's not worth carrying on if the step isn't versionable,
>> + or if we're pretty sure it's not for the inner dimension. */
>> + if (!step_if_innermost
>> + || TREE_CODE (step) != SSA_NAME
>> + || !expr_invariant_in_loop_p (loop, step))
>> + return;
>> +
>> + best_step = step;
>> +
>> + /* We should version for STEP == 1 if we know that that can be
>> + true under some circumstances. */
>> + if (integer_onep (step_if_innermost))
>> + break;
>> +
>> + /* Bail out if this appears to be the step for the innermost
>> + dimension, but isn't likely to be 1.
>> +
>> + ??? We could instead version for when it equals
>> + STEP_IF_INNERMOST, but it's not likely to have as much
>> + benefit as versioning for 1. */
>> + if (step_if_innermost != step)
>> + return;
>> + }
>> + else
>> + {
>> + /* This is an inner chrec. If it looks like it iterates over
>> + the innermost dimension, abort any attempt to version for
>> + the outermost chrec (which if we reach here wasn't itself
>> + obviously iterating over the innermost dimension). */
>> + if (step_if_innermost && TREE_CONSTANT (step_if_innermost))
>> + return;
>> + }
>> + worklist.quick_push (CHREC_LEFT (expr));
>> + continue;
>> + }
>> +
>> + /* Analyze parts of a queued CHREC_LEFT. This is better than just
>> + analyzing the evolution of the whole expression since the value
>> + could include a mixture of analyzable and unanalyzable elements.
>> + Use NSPLIT to count cases in which we add more expressions to
>> + analyze, as opposed to just simplifying the existing one. */
>> + if (code == PLUS_EXPR || code == POINTER_PLUS_EXPR || code == MINUS_EXPR)
>> + {
>> + worklist.quick_push (TREE_OPERAND (expr, 0));
>> + if (nsplit++ < MAX_NSPLIT)
>> + worklist.quick_push (TREE_OPERAND (expr, 1));
>> + continue;
>> + }
>> + if (code == MULT_EXPR)
>> + {
>> + tree op0 = strip_casts (TREE_OPERAND (expr, 0));
>> + tree op1 = TREE_OPERAND (expr, 1);
>> + if (TREE_CODE (op0) == PLUS_EXPR || TREE_CODE (op0) == MINUS_EXPR)
>> + {
>> + tree type = TREE_TYPE (expr);
>> + tree op00 = fold_convert (type, TREE_OPERAND (op0, 0));
>> + worklist.quick_push (fold_build2 (MULT_EXPR, type, op00, op1));
>> + if (nsplit++ < MAX_NSPLIT)
>> + {
>> + tree op01 = fold_convert (type, TREE_OPERAND (op0, 1));
>> + worklist.quick_push (fold_build2 (MULT_EXPR, type,
>> + op01, op1));
>> + }
>> + continue;
>> + }
>> + }
>> +
>> + /* If EXPR is invariant in LOOP, analyze it wrt the innermost loop
>> + for which it could evolve (i.e. the loop containing the outermost
>> + one for which EXPR is invariant). */
>
> This isn't how analyze_scalar_evolution works - you _always_ have to
> feed the innermost loop that the expression is used in (the context).
> Then you instantiate the result in the outermost loop of the nest you
> are interested in. Otherwise you get garbage.
I dispute this. :-) You might remember we had a similar discussion
about the use in get_base_for_alignment_1.
analyze_scalar_evolution describes how the supplied value V evolves,
which isn't garbage in itself. Then instantiating the SCEV tells
you what this means for the use in:
RES = V; // A
...
use of RES // B
in terms of values that are available at B. But we're not interested in
the second bit here. We only want to analyze *how* V evolves, not use
or compute its value in a given context. Instantiating the SCEV would
lose useful information.
> It looks like you are re-analyzing SSA names in the evolution - that's
> odd and shouldn't be necessary (but you forget to instantiate, so...).
>
> I suggest to move the analyze_scalar_evolution part out of the worklist
> loop where you are sure you have an SSA name.
This too is because we don't care whether the whole address can
be expressed as a SCEV, we just want to search for parts of the
calculation that are more likely to be inner dimensions.
This isn't (just) because we don't instantiate. It's because we're
more aggressive than SCEV can be in looking through casts.
>> + /* Bail out if OP2 can be analyzed as an evolution; we want to use
>> + analyze_evolution in that case instead. There's no point trying
>> + hard to avoid repeating the call to analyze_scalar_evolution since
>> + that function does its own caching. */
>> + if (tree_contains_chrecs (analyze_scalar_evolution (loop, op2), NULL))
>
> Don't you want !chrec_contains_undetermined (...) instead? I wonder what
> is missing to allow all interesting expressions to be analyzed by
> SCEV?
This code is handling cases that vary arbitrarily between iterations,
such as a histogram update. I don't think SCEV could ever provide
anything useful here.
analyze_scalar_evolution returns op2 if it can't analyze op2 as a SCEV.
>> +/* Treat EXPR as a sum of products and apply analyze_product to each of the
>> + products. Return true if one of the products provides a versioning
>> + opportunity. FACTOR is as for analyze_product. */
>> +
>> +bool
>> +loop_versioning::analyze_sum_of_products (struct loop *loop, tree expr,
>> + poly_uint64 factor)
>> +{
>
> This looks awfully close to what tree-affine.c does apart from more
> aggressively stripping conversions? I see all the analysis parts
> are limited and thus "O(1)" but still there's going to be a lot of
> redundancy involved for repeated use of (derived) "IVs"? Wouldn't
> it be good to have a bitmap of already handled SSA_NAMEs to stop
> processing early?
I did consider this but think it'll do more harm than good.
Bitmap lookup is O(N) in the worst case, and the PR you were
working on a month or so ago shows that that really can bite.
Whereas like you say the limits make this O(1).
So although Kyrill and Ramana's patch added this check, I'd prefer to
leave it out. It seems like premature optimisation and I think it'll
make things worse in practice.
>> + const unsigned int MAX_NITERS = 8;
>> +
>> + tree worklist[MAX_NITERS];
>> + unsigned int length = 0;
>> + worklist[length++] = expr;
>> + for (unsigned int i = 0; i < length; ++i)
>> + {
>> + expr = worklist[i];
>> + gassign *assign = maybe_get_assign_strip_casts (loop, expr);
>> + if (!assign)
>> + continue;
>> +
>> + tree_code code = gimple_assign_rhs_code (assign);
>> + if (code == PLUS_EXPR || code == POINTER_PLUS_EXPR || code == MINUS_EXPR)
>
> POINTER_MINUS_EXPR?
Do you mean POINTER_DIFF_EXPR? I wouldn't expect that to be interesting
since it would give information about the two pointers being subtracted.
>> +/* Analyze expression EXPR, which occurs in loop LOOP. */
>> +
>> +void
>> +loop_versioning::analyze_expr (struct loop *loop, tree expr)
>> +{
>> + while (handled_component_p (expr))
>> + {
>> + /* See whether we can use versioning to avoid a multiplication
>> + in the array index. */
>> + if (TREE_CODE (expr) == ARRAY_REF)
>> + {
>> + if (!analyze_sum_of_products (loop, TREE_OPERAND (expr, 1), 1))
>> + analyze_evolution (loop, TREE_OPERAND (expr, 1), 1);
>> + }
>> + expr = TREE_OPERAND (expr, 0);
>> + }
>> +
>> + if (TREE_CODE (expr) == MEM_REF)
>> + {
>> + tree addr = TREE_OPERAND (expr, 0);
>> + /* See whether we can use versioning to avoid a multiplication
>> + in the pointer calculation. This is generally only worth
>> + doing if the multiplication occurs in this loop rather than
>> + an outer loop. */
>
> Why's that so here but not above for ARRAY_REF? That is, what is
> the difference between a[i] and ptr = &a[i]; *ptr?
Good question. I think there was a (probably poor) reason, but clearly
I didn't write it down.
> > + struct loop *loop = gimple_bb (stmt)->loop_father;
> > +
> > + unsigned int nops = gimple_num_ops (stmt);
> > + for (unsigned int i = 0; i < nops; ++i)
> > + if (tree op = gimple_op (stmt, i))
> > + analyze_expr (loop, op);
>
> I think you instead want to use gimple_walk_load_store_ops ().
I agree with Kyrill's response to this. In addition, I'd originally
wondered whether we should also version for ADDR_EXPRs, and that might
make sense if we add more types of versioning (not just unit strides)
in future.
>> +/* Analyze all the statements in BB looking for useful version checks. */
>> +
>> +void
>> +loop_versioning::analyze_block (basic_block bb)
>> +{
>> + struct loop *loop = bb->loop_father;
>> + loop_info &li = get_loop_info (loop);
>> + if (li.rejected_p)
>> + return;
>> +
>> + for (gimple_stmt_iterator gsi = gsi_start_bb (bb); !gsi_end_p (gsi);
>> + gsi_next (&gsi))
>> + {
>> + gimple *stmt = gsi_stmt (gsi);
>> + if (expensive_stmt_p (stmt))
>> + {
>> + if (dump_file && (dump_flags & TDF_DETAILS))
>> + {
>> + struct loop *loop = gimple_bb (stmt)->loop_father;
>> + fprintf (dump_file, ";; Loop %d (depth %d) contains expensive"
>> + " stmt: ", loop->num, loop_depth (loop));
>> + print_gimple_stmt (dump_file, stmt, 0, TDF_SLIM);
>> + }
>> + li.rejected_p = true;
>
> I assume that once a loop is rejected this or another way there's
> no reason to look at any outer loop of it, thus ...
>
>> + break;
>> + }
>> +
>> + /* Only look for direct versioning opportunities in inner loops
>> + since the benefit tends to be much smaller for outer loops. */
>> + if (!loop->inner)
>> + analyze_stmt (stmt);
>> +
>> + /* The point of the instruction limit is to prevent excessive
>> + code growth, so this is a size-based estimate even though
>> + the optimization is aimed at speed. */
>> + li.num_insns += estimate_num_insns (stmt, &eni_size_weights);
>> + }
>> +}
>> +
>> +/* Analyze all the blocks in the function looking for useful version checks.
>> + Return true if we found one. */
>> +
>> +bool
>> +loop_versioning::analyze_blocks ()
>> +{
>> + /* For now we don't try to version the whole function, although
>> + versioning at that level could be useful in some cases. */
>> + get_loop_info (get_loop (m_fn, 0)).rejected_p = true;
>> +
>> + basic_block bb;
>> + FOR_EACH_BB_FN (bb, m_fn)
>> + if (loop_outer (bb->loop_father))
>> + analyze_block (bb);
>
> .... I'd structure this as a
>
> FOR_EACH_LOOP (... LI_FROM_INNERMOST)
> look at loop body, only analyze stmts belonging to loop (not subloops)
>
> walk eventually even open-coding this as recursion so you can quickly
> finish off outer loop processing once an inner loop got disabled.
OK. Thought about doing that, but it seemed relatively hard to get the
list of blocks that are in a loop but not in subloops. (Let me know if
there's an easy way.) But I guess it's probably worth an extra ad-hoc
walk over the blocks (not the contents) to construct a list of blocks
for each loop.
>> +/* Use the ranges in VRS to remove impossible versioning conditions from
>> + LOOP. */
>
> Does it actually happen to make it worthwhile? ;)
Yeah. E.g. classic netlib daxpy had this.
Thanks,
Richard
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Add a loop versioning pass
2018-11-28 16:48 ` Richard Sandiford
@ 2018-11-29 11:31 ` Martin Jambor
2018-12-03 13:16 ` Richard Biener
1 sibling, 0 replies; 17+ messages in thread
From: Martin Jambor @ 2018-11-29 11:31 UTC (permalink / raw)
To: Richard Sandiford, Richard Biener; +Cc: gcc-patches
Hi,
On Wed, Nov 28 2018, Richard Sandiford wrote:
>>> The pass gives a 7.6% improvement on Cortex-A72 for 554.roms_r at -O3
>>> and a 2.4% improvement for 465.tonto. I haven't found any SPEC tests
>>> that regress.
>>>
>>> Sizewise, there's a 10% increase in .text for both 554.roms_r and
>>> 465.tonto. That's obviously a lot, but in tonto's case it's because
>>> the whole program is written using assumed-shape arrays and pointers,
>>> so a large number of functions really do benefit from versioning.
>>> roms likewise makes heavy use of assumed-shape arrays, and that
>>> improvement in performance IMO justifies the code growth.
>>
>> Ouch. I know that at least with LTO IPA-CP can do "quite" some
>> propagation of constant strides. Not sure if we're aggressive
>> enough in actually doing the cloning for all cases we figure out
>> strides though. But my question is how we can avoid doing the
>> versioning for loops in the copy that did not have the IPA-CPed
>> stride of one? Ideally we'd be able to mark individual references
>> as {definitely,likely,unlikely,not}-unit-stride?
>
> This is a more elaborate version of what I was trying to do with
> the IFN I'd mentioned on IRC a few weeks back. It would be a bit
> like a long-living ASSERT_EXPR that provides hints about the value
> of the SSA name. Another alternative would be to annotate the
> SSA name itself, but then we'd easily lose the information during
> optimisation.
>
> But will IPA-CP conditionally use a clone for a stride of 1 for
> calls that pass a variable stride that might or might not be 1?
> E.g. if it clones foo as foo.1 for calls in which a stride parameter
> is 1 at compile time, does it also make foo call foo.1 when the stride
> parameter is 1 at runtime? If not, that sounds like a missed optimisation.
> If so, prune_conditions should stop us from versioning.
IPA-CP only creates clones for compile-time constants, it does not add
any run-time check to the callers. It is an interesting idea that it
could add these for some cases when it decides to do the cloning
anyway.
Martin
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Add a loop versioning pass
2018-11-28 16:48 ` Richard Sandiford
2018-11-29 11:31 ` Martin Jambor
@ 2018-12-03 13:16 ` Richard Biener
2018-12-06 13:19 ` Richard Sandiford
1 sibling, 1 reply; 17+ messages in thread
From: Richard Biener @ 2018-12-03 13:16 UTC (permalink / raw)
To: Richard Sandiford; +Cc: gcc-patches
On Wed, 28 Nov 2018, Richard Sandiford wrote:
> Thanks for the review and sorry for the long time replying.
Likewise ...
> Richard Biener <rguenther@suse.de> writes:
> >> This patch adds a pass that versions loops with variable index strides
> >> for the case in which the stride is 1. E.g.:
> >>
> >> for (int i = 0; i < n; ++i)
> >> x[i * stride] = ...;
> >>
> >> becomes:
> >>
> >> if (stepx == 1)
> >> for (int i = 0; i < n; ++i)
> >> x[i] = ...;
> >> else
> >> for (int i = 0; i < n; ++i)
> >> x[i * stride] = ...;
> >>
> >> This is useful for both vector code and scalar code, and in some cases
> >> can enable further optimisations like loop interchange or pattern
> >> recognition.
> >>
> >> The pass gives a 7.6% improvement on Cortex-A72 for 554.roms_r at -O3
> >> and a 2.4% improvement for 465.tonto. I haven't found any SPEC tests
> >> that regress.
> >>
> >> Sizewise, there's a 10% increase in .text for both 554.roms_r and
> >> 465.tonto. That's obviously a lot, but in tonto's case it's because
> >> the whole program is written using assumed-shape arrays and pointers,
> >> so a large number of functions really do benefit from versioning.
> >> roms likewise makes heavy use of assumed-shape arrays, and that
> >> improvement in performance IMO justifies the code growth.
> >
> > Ouch. I know that at least with LTO IPA-CP can do "quite" some
> > propagation of constant strides. Not sure if we're aggressive
> > enough in actually doing the cloning for all cases we figure out
> > strides though. But my question is how we can avoid doing the
> > versioning for loops in the copy that did not have the IPA-CPed
> > stride of one? Ideally we'd be able to mark individual references
> > as {definitely,likely,unlikely,not}-unit-stride?
>
> This is a more elaborate version of what I was trying to do with
> the IFN I'd mentioned on IRC a few weeks back. It would be a bit
> like a long-living ASSERT_EXPR that provides hints about the value
> of the SSA name. Another alternative would be to annotate the
> SSA name itself, but then we'd easily lose the information during
> optimisation.
>
> But will IPA-CP conditionally use a clone for a stride of 1 for
> calls that pass a variable stride that might or might not be 1?
> E.g. if it clones foo as foo.1 for calls in which a stride parameter
> is 1 at compile time, does it also make foo call foo.1 when the stride
> parameter is 1 at runtime? If not, that sounds like a missed optimisation.
> If so, prune_conditions should stop us from versioning.
Martin answered that.
> >> The next biggest .text increase is 4.5% for 548.exchange2_r. I did see
> >> a small (0.4%) speed improvement there, but although both 3-iteration runs
> >> produced stable results, that might still be noise. There was a slightly
> >> larger (non-noise) improvement for a 256-bit SVE model.
> >>
> >> 481.wrf and 521.wrf_r .text grew by 2.8% and 2.5% respectively, but
> >> without any noticeable improvement in performance. No other test grew
> >> by more than 2%.
> >>
> >> Although the main SPEC beneficiaries are all Fortran tests, the
> >> benchmarks we use for SVE also include some C and C++ tests that
> >> benefit.
> >
> > Did you see any slowdown, for example because versioning was forced
> > to be on an innermost loop? I'm thinking of the testcase in
> > PR87561 where we do have strided accesses in the innermost loop.
>
> Not so far.
>
> > Since you cite performance numbers how did you measure them?
> > I assume -Ofast -march=native but did you check with -flto?
>
> -O3 -march=native. No, didn't check LTO, will do that.
>
> >> Using -frepack-arrays gives the same benefits in many Fortran cases.
> >> The problem is that using that option inappropriately can force a full
> >> array copy for arguments that the function only reads once, and so it
> >> isn't really something we can turn on by default. The new pass is
> >> supposed to give most of the benefits of -frepack-arrays without
> >> the risk of unnecessary repacking.
> >>
> >> The patch therefore enables the pass by default at -O3.
> >
> > I think that's reasonable.
> >
> > One possible enhancement would be to add a value-profile for the
> > strides so we can guide this optimization better.
> >
> > The pass falls foul of C++ class make small methods of everything.
> > That makes following the code very hard. Please inline single-used
> > methods in callers wherever possible to make the code read
> > more like GCC code (using GCC API).
>
> I'll bite my tongue on this one :-)
>
> > The pass contains an awful lot of heuristics :/ Like last year
> > with the interchange pass I would suggest to rip most of it out
> > and first lay infrastructure with the cases you can positively
> > identify without applying heuristics or "hacks" like stripping
> > semantically required casts. That makes it also clear which
> > testcases test which code-path. That said, all the analyze
> > multiplications/plusses/factors stuff was extremely hard to review
> > and I have no overall picture why this is all so complicated or
> > necessary.
>
> I think the tests do cover most of this -- was glad to see that
> when Kyrill tried taking something out, one of the tests started
> to fail :-). And the comments in the tests as well as the code
> are supposed to explain why this is desirable. But I can try to
> spruce up the comments in the code if they're not detailed enough.
>
> The problem is that the motivating (Fortran) case can't be identified
> without applying heuristics. All we see is a collection of adds and
> multiplies, with no semantic information to say which multiplies are for
> the inner dimension and which aren't. I agree it would be nicer not
> to have them, which is why I'd originally suggested the IFN to provide
> information about dimensions.
Sure, still a new pass having _all_ the heuristic built in with
10s of testcases do not make it easy to review those heuristics...
> >> +@item -fversion-loops-for-strides
> >> +@opindex fversion-loops-for-strides
> >> +If a loop iterates over an array with a variable stride, create another
> >> +version of the loop that assumes the stride is always 1. For example:
> >> +
> >> +@smallexample
> >> +for (int i = 0; i < n; ++i)
> >> + x[i * stride] = @dots{};
> >> +@end smallexample
> >> +
> >> +becomes:
> >> +
> >> +@smallexample
> >> +if (stride == 1)
> >> + for (int i = 0; i < n; ++i)
> >> + x[i] = @dots{};
> >> +else
> >> + for (int i = 0; i < n; ++i)
> >> + x[i * stride] = @dots{};
> >> +@end smallexample
> >> +
> >> +This is particularly useful for assumed-shape arrays in Fortran.
> >
> > "where it allows better vectorization assuming contiguous accesses."
>
> Mind if I make it "where, for example, ..."? Vectorisation isn't the
> only target here.
Sure.
> >> + /* We'd like to version the loop for the case in which these
> >> + SSA_NAMEs are all equal to 1 at runtime. */
> >> + vec<tree> unity_names;
> >> +
> >> + /* The set of SSA_NAMEs in UNITY_NAMES, keyed off the SSA_NAME_VERSION. */
> >> + bitmap_head unity_name_ids;
> >
> > I wonder why you need both, a vector and a bitmap since you can
> > cheaply iterate over the bitmap and get the SSA name via ssa_name (version)?
>
> Ah, yeah, that would be better.
>
> >> + if (dump_file && (dump_flags & TDF_DETAILS))
> >> + {
> >> + fprintf (dump_file, ";; Want to version loop %d (depth %d)"
> >> + " for when ", loop->num, loop_depth (loop));
> >> + print_generic_expr (dump_file, name, TDF_SLIM);
> >> + fprintf (dump_file, " == 1");
> >
> > Since you are writing a new pass you want to use the new dump interface.
> >
> > if (dump_enabled_p ())
> > dump_printf (MSG_NOTE, ";; Want to version loop %d (depth %d)"
> > " for when %E == 1", loop->num, loop_depth (loop), name);
> > ...
> >
> > it's much nicer to be able to use %E/%G than separate calls for the
> > tree parts.
>
> Thanks to Kyrill and Ramana for fixing this.
Note I didn't look at the updated patch yet.
> >> +/* Return true if in principle it is worth versioning an index fragment of
> >> + the form:
> >> +
> >> + (i * b * SCALE) / FACTOR
> >> +
> >> + for the case in which b == 1. */
> >> +
> >> +bool
> >> +loop_versioning::acceptable_scale_p (tree scale, poly_uint64 factor)
> >> +{
> >> + /* See whether SCALE is a constant multiple of FACTOR, and if the
> >> + multiple is small enough for us to treat it as a potential grouped
> >> + access. For example:
> >> +
> >> + for (auto i : ...)
> >> + y[i] = f (x[4 * i * stride],
> >> + x[4 * i * stride + 1],
> >> + x[4 * i * stride + 2]);
> >> +
> >> + would benefit from versioning for the case in which stride == 1.
> >> + High multiples of i * stride are less likely to benefit, and could
> >> + indicate a simulated multi-dimensional array.
> >> +
> >> + This is just a heuristic, to avoid having to do expensive group
> >> + analysis of the data references in a loop. */
> >> + poly_uint64 const_scale;
> >> + unsigned int multiple;
> >> + if (poly_int_tree_p (scale, &const_scale)
> >> + && constant_multiple_p (const_scale, factor, &multiple))
> >> + {
> >> + unsigned int maxval = PARAM_VALUE (PARAM_LOOP_VERSIONING_GROUP_SIZE);
> >> + return IN_RANGE (multiple, 1, maxval);
> >
> > Hmm. So you _do_ want to version sth like
> >
> > struct X { int i; int j; } a[2048];
> >
> > for (int i = start; i < end; ++i)
> > a[i*s].i = 1;
> >
> > ? That is, even with s == 1 the accesses will not become contiguous?
> > OK, I suppose that's because you are looking at a stmt in isolation
> > and another stmt may access a[i*s].j here.
> >
> > That is, would it be a future improvement to run sth like the
> > vectorizers group access analysis on the references and perform
> > this check on whole groups then, possibly better being able to
> > constrain what is now the magic parameter PARAM_LOOP_VERSIONING_GROUP_SIZE?
>
> Yeah, possibly. The problem is that we might end up having to reproduce
> vectoriser heuristics. E.g. stores with gaps should be vectorisable
> in future with SVE, using contiguous predicated stores in which some
> lanes are inactive. So we don't necessarily need the .j access for
> this to be worthwhile.
>
> But perhaps the argument for versioning for vectorisation is stronger
> for grouped accesses than contiguous ones.
>
> Note that the above example doesn't rely on the grouping heuristic
> since we just strip the COMPONENT_REF and then analyze the ARRAY_REF
> with a factor of 1. The heuristic instead handles things like:
>
> a[i * stride * 2]
>
> etc., and the param is more there to decide when a constant factor is
> small enough to be a realistic group size instead of an outer dimension.
> E.g. if we see:
>
> a[i * stride * 20000]
>
> the problem is that i * stride * 20000 is likely to be applying
> an outer dimension index.
>
> > I guess you tried to constrain it to the stmts access size but of course
> > that fails short of handling later SLP vectorized cases.
>
> Right.
So I think we want to constrain it to sth like the maximum targets
vector size. We want consecutive iterations to bring two memory
accesses to the same vector load, otherwise we can use strided
stores (or gathers) anyways. This means the distance between
accesses should be at most max_vectsize / 2? (non-power-of-two
interleaving is also quite limited, sth. to be considered)
> >> +/* Decide whether an index fragment of the form:
> >> +
> >> + (i * STEP) / FACTOR
> >> +
> >> + is likely to be for an innermost dimension. If we think it is,
> >> + return one of the constant values that it could have (returning 1 if
> >> + that's a possibility). If think it isn't, return null. Otherwise
> >> + return STEP, to indicate that it might or might not be an inner
> >> + dimension. */
> >> +
> >> +tree
> >> +loop_versioning::get_step_if_innermost (tree step, poly_uint64 factor)
> >> +{
> >> + const unsigned int MAX_NITERS = 8;
> >> +
> >> + tree likely = NULL_TREE;
> >> + tree unlikely = NULL_TREE;
> >> + tree worklist[MAX_NITERS];
> >> + unsigned int length = 0;
> >> + worklist[length++] = step;
> >> + for (unsigned int i = 0; i < length; ++i)
> >> + {
> >> + tree expr = worklist[i];
> >> +
> >> + if (TREE_CONSTANT (expr))
> >
> > ? CONSTANT_CLASS_P (expr)?
>
> OK.
>
> >> + {
> >> + /* See if multiplying by EXPR applies a scale that would be
> >> + consistent with an individual access or a small grouped
> >> + access. */
> >> + if (acceptable_scale_p (expr, factor))
> >> + {
> >> + likely = expr;
> >> + if (integer_onep (expr))
> >> + break;
> >> + }
> >> + else
> >> + unlikely = expr;
> >> + continue;
> >> + }
> >> +
> >> + /* Otherwise we can only handle SSA names. */
> >> + gimple *stmt = maybe_get_stmt (expr);
> >> + if (!stmt)
> >> + continue;
> >> +
> >> + /* If EXPR is set by a PHI node, queue its arguments in case
> >> + we find one that is consistent with an inner dimension.
> >> +
> >> + An important instance of this is the Fortran handling of array
> >> + descriptors, which calculates the stride of the inner dimension
> >> + using a PHI equivalent of:
> >> +
> >> + raw_stride = a.dim[0].stride;
> >> + stride = raw_stride != 0 ? raw_stride : 1;
> >> +
> >> + (Strides for outer dimensions do not treat 0 specially.) */
> >> + if (gphi *phi = dyn_cast <gphi *> (stmt))
> >> + {
> >> + unsigned int nargs = gimple_phi_num_args (phi);
> >> + for (unsigned int j = 0; j < nargs && length < MAX_NITERS; ++j)
> >> + worklist[length++] = gimple_phi_arg_def (phi, j);
> >> + continue;
> >> + }
> >
> > So only PHIs seed the worklist.
>
> Yeah.
>
> >> + /* If the value is set by an assignment, expect it to be read from
> >> + memory (such as an array descriptor) rather than be calculated. */
> >> + if (gassign *assign = dyn_cast <gassign *> (stmt))
> >> + {
> >> + if (gimple_assign_single_p (assign)
> >> + && is_gimple_lvalue (gimple_assign_rhs1 (assign)))
> >
> > Please do not use is_gimple_lvalue, that's supposed to be a gimplifier-only
> > predicate. Do you want to use gimple_assign_load_p here?
>
> OK.
>
> >> + continue;
> >> +
> >> + unlikely = expr;
> >> + }
> >> +
> >> + /* Things like calls don't really tell us anything. */
> >> + }
> >
> > So I don't understand the above fully but it looks like plain
> > function arguments STEP will never be 'likely'?
>
> Right. They're neither likely not unlikely, just like a value
> loaded from memory.
>
> > That is, I'd appreciate some more comments of what SSA def
> > chains we walk and what we look for in the end.
>
> OK, will try.
>
> >> + if (likely)
> >> + {
> >> + if (dump_file && (dump_flags & TDF_DETAILS))
> >> + fprintf (dump_file, "likely to be innermost dimension\n");
> >> + return likely;
> >> + }
> >> +
> >> + if (unlikely)
> >> + {
> >> + if (dump_file && (dump_flags & TDF_DETAILS))
> >> + fprintf (dump_file, "probably not innermost dimension\n");
> >> + return NULL_TREE;
> >> + }
> >> +
> >> + if (dump_file && (dump_flags & TDF_DETAILS))
> >> + {
> >> + if (maybe_ne (factor, 1U))
> >> + fprintf (dump_file, "didn't find expected scaling factor\n");
> >> + else
> >> + fprintf (dump_file, "no information about value\n");
> >> + }
> >> + return step;
> >> +}
> >> +
> >> +/* STEP appears in an index fragment of the form:
> >> +
> >> + {..., +, STEP}_n / FACTOR
> >> +
> >> + Remove any conversions and grouping or scaling factors from STEP and
> >> + return the underlying index step. Set *STEP_IF_INNERMOST to the value
> >> + returned by get_step_if_innermost. */
> >> +
> >> +tree
> >> +loop_versioning::extract_step (tree step, poly_uint64 factor,
> >> + tree *step_if_innermost)
> >> +{
> >> + if (dump_file && (dump_flags & TDF_DETAILS))
> >> + {
> >> + fprintf (dump_file, ";; step ");
> >> + print_generic_expr (dump_file, step, TDF_SLIM);
> >> + fprintf (dump_file, ": ");
> >> + }
> >> +
> >> + step = strip_casts (step);
> >> +
> >> + /* Peel any scaling, which generally happens after conversion to
> >> + pointer width. For example, on LP64 systems:
> >> +
> >> + int *x, i, stride;
> >> + ... x[4 * i * stride] ...;
> >> +
> >> + multiplies i * stride by 4 using ints, then widens the result
> >> + to pointer width before multiplying by sizeof (int). */
> >> + poly_uint64 scale;
> >> + if (TREE_CODE (step) == MULT_EXPR
> >> + && poly_int_tree_p (TREE_OPERAND (step, 1), &scale)
> >> + && known_eq (scale, factor))
> >> + {
> >> + step = strip_casts (TREE_OPERAND (step, 0));
> >> + if (dump_file && (dump_flags & TDF_DETAILS))
> >> + fprintf (dump_file, "scaled, ");
> >> + factor = 1;
> >> + }
> >
> > So I think the extra constant scale may or may not be in CHREC_RIGHT
> > depending on whether the expression was hoisted out of the loop as
> > invariant or implicit in sth like an ARRAY_REF.
>
> For ARRAY_REF the factor is always 1.
>
> >> + /* Peel any grouping multiple; see acceptable_scale_p for details. */
> >> + if (TREE_CODE (step) == MULT_EXPR
> >> + && acceptable_scale_p (TREE_OPERAND (step, 1), factor))
> >> + {
> >> + step = strip_casts (TREE_OPERAND (step, 0));
> >> + if (dump_file && (dump_flags & TDF_DETAILS))
> >> + fprintf (dump_file, "%sgrouped, ",
> >> + maybe_ne (factor, 1U) ? "scaled, " : "");
> >> + factor = 1;
> >> + }
> >
> > Which means I'm not sure why there are (only) two MULTs being
> > looked for and only one has acceptable_scale_p applied...
>
> The first one checks for multiplications by the access size in pointer
> width, i.e. for the case in which a[i*b] becomes:
>
> a + ((cast) (i * b) * sizeof (a[0]))
>
> It also happens to catch the case in which sizeof (a[0]) == 1 and i*b is
> calculated in pointer width, which if either condition didn't hold would
> instead be handled by the second multiplication.
>
> Only the second multiplication has the grouping heuristic applied
> because the grouping heuristic is aimed at explicit multiplications
> by a constant, as above, and those would normally be done in the same
> width as the multiplication by the variable stride. The multiplication
> by the group size and the multiplication by the access size (n/a for
> ARRAY_REFs) would be folded into one if the multiplications all happen
> in pointer width.
>
> >> +{
> >> + const unsigned int MAX_NSPLIT = 8;
> >> +
> >> + if (dump_file && (dump_flags & TDF_DETAILS))
> >> + {
> >> + fprintf (dump_file, ";; Analyzing use of ");
> >> + print_generic_expr (dump_file, expr, TDF_SLIM);
> >> + if (maybe_ne (factor, 1U))
> >> + {
> >> + fprintf (dump_file, " (which addresses ");
> >> + print_dec (factor, dump_file);
> >> + fprintf (dump_file, " bytes)");
> >> + }
> >> + fprintf (dump_file, " in loop %d (depth %d)\n",
> >> + loop->num, loop_depth (loop));
> >> + }
> >> +
> >> + /* The main problem we have here is that we cannot assume that the
> >> + innermost loop iterates over the innermost dimension of an array.
> >> + Accidentally adding versioning checks for outer dimensions would
> >> + cause the version condition to be false, which as well as bloating
> >> + the code would defeat loop versioning benefits for other accesses.
> >> +
> >> + Unfortunately all we usually see at this stage is general address
> >> + arithmetic, with no positive way of identifying how many dimensions
> >> + an array access has and which multiplication factors in the address
> >> + expression correspond to which array dimensions. In C code this is
> >> + often not even explicit in the source, since variable-sized multi-
> >> + dimensional arrays are often simulated using one-dimensional arrays.
> >> +
> >> + The three main ways in which we deal with this are:
> >> +
> >> + - use heuristics that positively identify steps that are likely
> >> + to represent the inner dimension.
> >> +
> >> + - use heuristics that positively identify steps that are unlikely
> >> + to represent the inner dimension.
> >> +
> >> + - if a part of EXPR is invariant in LOOP, analyze its evolution in
> >> + the outer loops to see whether we can positively identify any of
> >> + it as iterating over the inner dimension. */
> >> + tree best_step = NULL_TREE;
> >> + auto_vec<tree, MAX_NSPLIT> worklist;
> >> + worklist.quick_push (expr);
> >> + unsigned int nsplit = 0;
> >> + while (!worklist.is_empty ())
> >> + {
> >> + expr = strip_casts (worklist.pop ());
> >> + tree_code code = TREE_CODE (expr);
> >> +
> >> + if (code == POLYNOMIAL_CHREC)
> >> + {
> >> + /* Analyze the CHREC_RIGHT as the step in an index fragment. */
> >> + tree step_if_innermost;
> >> + tree step = extract_step (CHREC_RIGHT (expr), factor,
> >> + &step_if_innermost);
> >> + if (!best_step)
> >> + {
> >> + /* This is the outermost chrec for the original expression.
> >> + It's not worth carrying on if the step isn't versionable,
> >> + or if we're pretty sure it's not for the inner dimension. */
> >> + if (!step_if_innermost
> >> + || TREE_CODE (step) != SSA_NAME
> >> + || !expr_invariant_in_loop_p (loop, step))
> >> + return;
> >> +
> >> + best_step = step;
> >> +
> >> + /* We should version for STEP == 1 if we know that that can be
> >> + true under some circumstances. */
> >> + if (integer_onep (step_if_innermost))
> >> + break;
> >> +
> >> + /* Bail out if this appears to be the step for the innermost
> >> + dimension, but isn't likely to be 1.
> >> +
> >> + ??? We could instead version for when it equals
> >> + STEP_IF_INNERMOST, but it's not likely to have as much
> >> + benefit as versioning for 1. */
> >> + if (step_if_innermost != step)
> >> + return;
> >> + }
> >> + else
> >> + {
> >> + /* This is an inner chrec. If it looks like it iterates over
> >> + the innermost dimension, abort any attempt to version for
> >> + the outermost chrec (which if we reach here wasn't itself
> >> + obviously iterating over the innermost dimension). */
> >> + if (step_if_innermost && TREE_CONSTANT (step_if_innermost))
> >> + return;
> >> + }
> >> + worklist.quick_push (CHREC_LEFT (expr));
> >> + continue;
> >> + }
> >> +
> >> + /* Analyze parts of a queued CHREC_LEFT. This is better than just
> >> + analyzing the evolution of the whole expression since the value
> >> + could include a mixture of analyzable and unanalyzable elements.
> >> + Use NSPLIT to count cases in which we add more expressions to
> >> + analyze, as opposed to just simplifying the existing one. */
> >> + if (code == PLUS_EXPR || code == POINTER_PLUS_EXPR || code == MINUS_EXPR)
> >> + {
> >> + worklist.quick_push (TREE_OPERAND (expr, 0));
> >> + if (nsplit++ < MAX_NSPLIT)
> >> + worklist.quick_push (TREE_OPERAND (expr, 1));
> >> + continue;
> >> + }
> >> + if (code == MULT_EXPR)
> >> + {
> >> + tree op0 = strip_casts (TREE_OPERAND (expr, 0));
> >> + tree op1 = TREE_OPERAND (expr, 1);
> >> + if (TREE_CODE (op0) == PLUS_EXPR || TREE_CODE (op0) == MINUS_EXPR)
> >> + {
> >> + tree type = TREE_TYPE (expr);
> >> + tree op00 = fold_convert (type, TREE_OPERAND (op0, 0));
> >> + worklist.quick_push (fold_build2 (MULT_EXPR, type, op00, op1));
> >> + if (nsplit++ < MAX_NSPLIT)
> >> + {
> >> + tree op01 = fold_convert (type, TREE_OPERAND (op0, 1));
> >> + worklist.quick_push (fold_build2 (MULT_EXPR, type,
> >> + op01, op1));
> >> + }
> >> + continue;
> >> + }
> >> + }
> >> +
> >> + /* If EXPR is invariant in LOOP, analyze it wrt the innermost loop
> >> + for which it could evolve (i.e. the loop containing the outermost
> >> + one for which EXPR is invariant). */
> >
> > This isn't how analyze_scalar_evolution works - you _always_ have to
> > feed the innermost loop that the expression is used in (the context).
> > Then you instantiate the result in the outermost loop of the nest you
> > are interested in. Otherwise you get garbage.
>
> I dispute this. :-) You might remember we had a similar discussion
> about the use in get_base_for_alignment_1.
I don't remember ;)
> analyze_scalar_evolution describes how the supplied value V evolves,
> which isn't garbage in itself. Then instantiating the SCEV tells
Then analyze_scalar_evolution wouldn't need a 'loop' parameter.
It probably should get a stmt (or a use_operand) instead.
> you what this means for the use in:
>
> RES = V; // A
> ...
> use of RES // B
>
> in terms of values that are available at B. But we're not interested in
> the second bit here. We only want to analyze *how* V evolves, not use
> or compute its value in a given context. Instantiating the SCEV would
> lose useful information.
Instantiating only "loses" information in case there's an overall
effect of a loop to consider. I guess in your case it only brings
in not useful information.
That said, you can indeed pass analyze_scalar_evolution outer loops
of the loop the use is in (it's also in the outer loops).
>
> > It looks like you are re-analyzing SSA names in the evolution - that's
> > odd and shouldn't be necessary (but you forget to instantiate, so...).
> >
> > I suggest to move the analyze_scalar_evolution part out of the worklist
> > loop where you are sure you have an SSA name.
>
> This too is because we don't care whether the whole address can
> be expressed as a SCEV, we just want to search for parts of the
> calculation that are more likely to be inner dimensions.
>
> This isn't (just) because we don't instantiate. It's because we're
> more aggressive than SCEV can be in looking through casts.
:/
> >> + /* Bail out if OP2 can be analyzed as an evolution; we want to use
> >> + analyze_evolution in that case instead. There's no point trying
> >> + hard to avoid repeating the call to analyze_scalar_evolution since
> >> + that function does its own caching. */
> >> + if (tree_contains_chrecs (analyze_scalar_evolution (loop, op2), NULL))
> >
> > Don't you want !chrec_contains_undetermined (...) instead? I wonder what
> > is missing to allow all interesting expressions to be analyzed by
> > SCEV?
>
> This code is handling cases that vary arbitrarily between iterations,
> such as a histogram update. I don't think SCEV could ever provide
> anything useful here.
>
> analyze_scalar_evolution returns op2 if it can't analyze op2 as a SCEV.
Hmm, right. Looks inconsistent to instantiation...
> >> +/* Treat EXPR as a sum of products and apply analyze_product to each of the
> >> + products. Return true if one of the products provides a versioning
> >> + opportunity. FACTOR is as for analyze_product. */
> >> +
> >> +bool
> >> +loop_versioning::analyze_sum_of_products (struct loop *loop, tree expr,
> >> + poly_uint64 factor)
> >> +{
> >
> > This looks awfully close to what tree-affine.c does apart from more
> > aggressively stripping conversions? I see all the analysis parts
> > are limited and thus "O(1)" but still there's going to be a lot of
> > redundancy involved for repeated use of (derived) "IVs"? Wouldn't
> > it be good to have a bitmap of already handled SSA_NAMEs to stop
> > processing early?
>
> I did consider this but think it'll do more harm than good.
> Bitmap lookup is O(N) in the worst case, and the PR you were
> working on a month or so ago shows that that really can bite.
> Whereas like you say the limits make this O(1).
>
> So although Kyrill and Ramana's patch added this check, I'd prefer to
> leave it out. It seems like premature optimisation and I think it'll
> make things worse in practice.
OK.
> >> + const unsigned int MAX_NITERS = 8;
> >> +
> >> + tree worklist[MAX_NITERS];
> >> + unsigned int length = 0;
> >> + worklist[length++] = expr;
> >> + for (unsigned int i = 0; i < length; ++i)
> >> + {
> >> + expr = worklist[i];
> >> + gassign *assign = maybe_get_assign_strip_casts (loop, expr);
> >> + if (!assign)
> >> + continue;
> >> +
> >> + tree_code code = gimple_assign_rhs_code (assign);
> >> + if (code == PLUS_EXPR || code == POINTER_PLUS_EXPR || code == MINUS_EXPR)
> >
> > POINTER_MINUS_EXPR?
>
> Do you mean POINTER_DIFF_EXPR? I wouldn't expect that to be interesting
> since it would give information about the two pointers being subtracted.
>
> >> +/* Analyze expression EXPR, which occurs in loop LOOP. */
> >> +
> >> +void
> >> +loop_versioning::analyze_expr (struct loop *loop, tree expr)
> >> +{
> >> + while (handled_component_p (expr))
> >> + {
> >> + /* See whether we can use versioning to avoid a multiplication
> >> + in the array index. */
> >> + if (TREE_CODE (expr) == ARRAY_REF)
> >> + {
> >> + if (!analyze_sum_of_products (loop, TREE_OPERAND (expr, 1), 1))
> >> + analyze_evolution (loop, TREE_OPERAND (expr, 1), 1);
> >> + }
> >> + expr = TREE_OPERAND (expr, 0);
> >> + }
> >> +
> >> + if (TREE_CODE (expr) == MEM_REF)
> >> + {
> >> + tree addr = TREE_OPERAND (expr, 0);
> >> + /* See whether we can use versioning to avoid a multiplication
> >> + in the pointer calculation. This is generally only worth
> >> + doing if the multiplication occurs in this loop rather than
> >> + an outer loop. */
> >
> > Why's that so here but not above for ARRAY_REF? That is, what is
> > the difference between a[i] and ptr = &a[i]; *ptr?
>
> Good question. I think there was a (probably poor) reason, but clearly
> I didn't write it down.
>
> > > + struct loop *loop = gimple_bb (stmt)->loop_father;
> > > +
> > > + unsigned int nops = gimple_num_ops (stmt);
> > > + for (unsigned int i = 0; i < nops; ++i)
> > > + if (tree op = gimple_op (stmt, i))
> > > + analyze_expr (loop, op);
> >
> > I think you instead want to use gimple_walk_load_store_ops ().
>
> I agree with Kyrill's response to this.
C++ bites you? ;) I suppose marshalling a pmf through the void *
data and having the callback invoke the pmf might be possible?
But yeah - GCC is C and most callbacks are just function pointers
and not pointer-to-member fns.
You are at least also walking debug-stmts here I think
(unless they are expensive_stmt_p ...).
> In addition, I'd originally
> wondered whether we should also version for ADDR_EXPRs, and that might
> make sense if we add more types of versioning (not just unit strides)
> in future.
There's gimple_walk_load_store_addr_ops () of course.
> >> +/* Analyze all the statements in BB looking for useful version checks. */
> >> +
> >> +void
> >> +loop_versioning::analyze_block (basic_block bb)
> >> +{
> >> + struct loop *loop = bb->loop_father;
> >> + loop_info &li = get_loop_info (loop);
> >> + if (li.rejected_p)
> >> + return;
> >> +
> >> + for (gimple_stmt_iterator gsi = gsi_start_bb (bb); !gsi_end_p (gsi);
> >> + gsi_next (&gsi))
> >> + {
> >> + gimple *stmt = gsi_stmt (gsi);
> >> + if (expensive_stmt_p (stmt))
> >> + {
> >> + if (dump_file && (dump_flags & TDF_DETAILS))
> >> + {
> >> + struct loop *loop = gimple_bb (stmt)->loop_father;
> >> + fprintf (dump_file, ";; Loop %d (depth %d) contains expensive"
> >> + " stmt: ", loop->num, loop_depth (loop));
> >> + print_gimple_stmt (dump_file, stmt, 0, TDF_SLIM);
> >> + }
> >> + li.rejected_p = true;
> >
> > I assume that once a loop is rejected this or another way there's
> > no reason to look at any outer loop of it, thus ...
> >
> >> + break;
> >> + }
> >> +
> >> + /* Only look for direct versioning opportunities in inner loops
> >> + since the benefit tends to be much smaller for outer loops. */
> >> + if (!loop->inner)
> >> + analyze_stmt (stmt);
> >> +
> >> + /* The point of the instruction limit is to prevent excessive
> >> + code growth, so this is a size-based estimate even though
> >> + the optimization is aimed at speed. */
> >> + li.num_insns += estimate_num_insns (stmt, &eni_size_weights);
> >> + }
> >> +}
> >> +
> >> +/* Analyze all the blocks in the function looking for useful version checks.
> >> + Return true if we found one. */
> >> +
> >> +bool
> >> +loop_versioning::analyze_blocks ()
> >> +{
> >> + /* For now we don't try to version the whole function, although
> >> + versioning at that level could be useful in some cases. */
> >> + get_loop_info (get_loop (m_fn, 0)).rejected_p = true;
> >> +
> >> + basic_block bb;
> >> + FOR_EACH_BB_FN (bb, m_fn)
> >> + if (loop_outer (bb->loop_father))
> >> + analyze_block (bb);
> >
> > .... I'd structure this as a
> >
> > FOR_EACH_LOOP (... LI_FROM_INNERMOST)
> > look at loop body, only analyze stmts belonging to loop (not subloops)
> >
> > walk eventually even open-coding this as recursion so you can quickly
> > finish off outer loop processing once an inner loop got disabled.
>
> OK. Thought about doing that, but it seemed relatively hard to get the
> list of blocks that are in a loop but not in subloops. (Let me know if
> there's an easy way.)
Simply skip bbs with bb->loop_father != loop.
> But I guess it's probably worth an extra ad-hoc
> walk over the blocks (not the contents) to construct a list of blocks
> for each loop.
Well, constructing a list of blocks in an order that visits inner loop
blocks first (recursively) would be useful here and in other places
I guess. That can be done via get_loop_body (outermost-interesting-loop);
and then sorting the blocks after loop-father loop-dfs order or so.
But as said just skipping bb->loop_father != loop would work for me
(at the expense of looking at bb->loop_father N^2 times).
> >> +/* Use the ranges in VRS to remove impossible versioning conditions from
> >> + LOOP. */
> >
> > Does it actually happen to make it worthwhile? ;)
>
> Yeah. E.g. classic netlib daxpy had this.
I see.
Richard.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Add a loop versioning pass
2018-12-03 13:16 ` Richard Biener
@ 2018-12-06 13:19 ` Richard Sandiford
2018-12-12 12:06 ` Richard Biener
0 siblings, 1 reply; 17+ messages in thread
From: Richard Sandiford @ 2018-12-06 13:19 UTC (permalink / raw)
To: Richard Biener; +Cc: gcc-patches
Richard Biener <rguenther@suse.de> writes:
>> > The pass contains an awful lot of heuristics :/ Like last year
>> > with the interchange pass I would suggest to rip most of it out
>> > and first lay infrastructure with the cases you can positively
>> > identify without applying heuristics or "hacks" like stripping
>> > semantically required casts. That makes it also clear which
>> > testcases test which code-path. That said, all the analyze
>> > multiplications/plusses/factors stuff was extremely hard to review
>> > and I have no overall picture why this is all so complicated or
>> > necessary.
>>
>> I think the tests do cover most of this -- was glad to see that
>> when Kyrill tried taking something out, one of the tests started
>> to fail :-). And the comments in the tests as well as the code
>> are supposed to explain why this is desirable. But I can try to
>> spruce up the comments in the code if they're not detailed enough.
>>
>> The problem is that the motivating (Fortran) case can't be identified
>> without applying heuristics. All we see is a collection of adds and
>> multiplies, with no semantic information to say which multiplies are for
>> the inner dimension and which aren't. I agree it would be nicer not
>> to have them, which is why I'd originally suggested the IFN to provide
>> information about dimensions.
>
> Sure, still a new pass having _all_ the heuristic built in with
> 10s of testcases do not make it easy to review those heuristics...
Fair :-) In the patch below I've tried to cut down on the special cases
and tried to add more comments explaining which patterns the code is
trying to detect/reject. Probably the key part is the one beginning:
+ The main difficulty here isn't finding strides that could be used
+ in a version check (that's pretty easy). The problem instead is to
+ avoid versioning for some stride S that is unlikely ever to be 1 at
+ runtime. Versioning for S == 1 on its own would lead to unnecessary
+ code bloat, while adding S == 1 to more realistic version conditions
+ would lose the optimisation opportunity offered by those other conditions.
>> >> +/* Return true if in principle it is worth versioning an index fragment of
>> >> + the form:
>> >> +
>> >> + (i * b * SCALE) / FACTOR
>> >> +
>> >> + for the case in which b == 1. */
>> >> +
>> >> +bool
>> >> +loop_versioning::acceptable_scale_p (tree scale, poly_uint64 factor)
>> >> +{
>> >> + /* See whether SCALE is a constant multiple of FACTOR, and if the
>> >> + multiple is small enough for us to treat it as a potential grouped
>> >> + access. For example:
>> >> +
>> >> + for (auto i : ...)
>> >> + y[i] = f (x[4 * i * stride],
>> >> + x[4 * i * stride + 1],
>> >> + x[4 * i * stride + 2]);
>> >> +
>> >> + would benefit from versioning for the case in which stride == 1.
>> >> + High multiples of i * stride are less likely to benefit, and could
>> >> + indicate a simulated multi-dimensional array.
>> >> +
>> >> + This is just a heuristic, to avoid having to do expensive group
>> >> + analysis of the data references in a loop. */
>> >> + poly_uint64 const_scale;
>> >> + unsigned int multiple;
>> >> + if (poly_int_tree_p (scale, &const_scale)
>> >> + && constant_multiple_p (const_scale, factor, &multiple))
>> >> + {
>> >> + unsigned int maxval = PARAM_VALUE (PARAM_LOOP_VERSIONING_GROUP_SIZE);
>> >> + return IN_RANGE (multiple, 1, maxval);
>> >
>> > Hmm. So you _do_ want to version sth like
>> >
>> > struct X { int i; int j; } a[2048];
>> >
>> > for (int i = start; i < end; ++i)
>> > a[i*s].i = 1;
>> >
>> > ? That is, even with s == 1 the accesses will not become contiguous?
>> > OK, I suppose that's because you are looking at a stmt in isolation
>> > and another stmt may access a[i*s].j here.
>> >
>> > That is, would it be a future improvement to run sth like the
>> > vectorizers group access analysis on the references and perform
>> > this check on whole groups then, possibly better being able to
>> > constrain what is now the magic parameter PARAM_LOOP_VERSIONING_GROUP_SIZE?
>>
>> Yeah, possibly. The problem is that we might end up having to reproduce
>> vectoriser heuristics. E.g. stores with gaps should be vectorisable
>> in future with SVE, using contiguous predicated stores in which some
>> lanes are inactive. So we don't necessarily need the .j access for
>> this to be worthwhile.
>>
>> But perhaps the argument for versioning for vectorisation is stronger
>> for grouped accesses than contiguous ones.
>>
>> Note that the above example doesn't rely on the grouping heuristic
>> since we just strip the COMPONENT_REF and then analyze the ARRAY_REF
>> with a factor of 1. The heuristic instead handles things like:
>>
>> a[i * stride * 2]
>>
>> etc., and the param is more there to decide when a constant factor is
>> small enough to be a realistic group size instead of an outer dimension.
>> E.g. if we see:
>>
>> a[i * stride * 20000]
>>
>> the problem is that i * stride * 20000 is likely to be applying
>> an outer dimension index.
>>
>> > I guess you tried to constrain it to the stmts access size but of course
>> > that fails short of handling later SLP vectorized cases.
>>
>> Right.
>
> So I think we want to constrain it to sth like the maximum targets
> vector size. We want consecutive iterations to bring two memory
> accesses to the same vector load, otherwise we can use strided
> stores (or gathers) anyways. This means the distance between
> accesses should be at most max_vectsize / 2? (non-power-of-two
> interleaving is also quite limited, sth. to be considered)
OK. The updated patch removes the new param and instead uses the maximum of:
- omp_max_vf () (maximum vector size in bytes, or 1 if loop vectorisation
is disabled)
- MAX_FIXED_MODE_SIZE, which should be a good limit for scalar code.
>> >> +{
>> >> + const unsigned int MAX_NSPLIT = 8;
>> >> +
>> >> + if (dump_file && (dump_flags & TDF_DETAILS))
>> >> + {
>> >> + fprintf (dump_file, ";; Analyzing use of ");
>> >> + print_generic_expr (dump_file, expr, TDF_SLIM);
>> >> + if (maybe_ne (factor, 1U))
>> >> + {
>> >> + fprintf (dump_file, " (which addresses ");
>> >> + print_dec (factor, dump_file);
>> >> + fprintf (dump_file, " bytes)");
>> >> + }
>> >> + fprintf (dump_file, " in loop %d (depth %d)\n",
>> >> + loop->num, loop_depth (loop));
>> >> + }
>> >> +
>> >> + /* The main problem we have here is that we cannot assume that the
>> >> + innermost loop iterates over the innermost dimension of an array.
>> >> + Accidentally adding versioning checks for outer dimensions would
>> >> + cause the version condition to be false, which as well as bloating
>> >> + the code would defeat loop versioning benefits for other accesses.
>> >> +
>> >> + Unfortunately all we usually see at this stage is general address
>> >> + arithmetic, with no positive way of identifying how many dimensions
>> >> + an array access has and which multiplication factors in the address
>> >> + expression correspond to which array dimensions. In C code this is
>> >> + often not even explicit in the source, since variable-sized multi-
>> >> + dimensional arrays are often simulated using one-dimensional arrays.
>> >> +
>> >> + The three main ways in which we deal with this are:
>> >> +
>> >> + - use heuristics that positively identify steps that are likely
>> >> + to represent the inner dimension.
>> >> +
>> >> + - use heuristics that positively identify steps that are unlikely
>> >> + to represent the inner dimension.
>> >> +
>> >> + - if a part of EXPR is invariant in LOOP, analyze its evolution in
>> >> + the outer loops to see whether we can positively identify any of
>> >> + it as iterating over the inner dimension. */
>> >> + tree best_step = NULL_TREE;
>> >> + auto_vec<tree, MAX_NSPLIT> worklist;
>> >> + worklist.quick_push (expr);
>> >> + unsigned int nsplit = 0;
>> >> + while (!worklist.is_empty ())
>> >> + {
>> >> + expr = strip_casts (worklist.pop ());
>> >> + tree_code code = TREE_CODE (expr);
>> >> +
>> >> + if (code == POLYNOMIAL_CHREC)
>> >> + {
>> >> + /* Analyze the CHREC_RIGHT as the step in an index fragment. */
>> >> + tree step_if_innermost;
>> >> + tree step = extract_step (CHREC_RIGHT (expr), factor,
>> >> + &step_if_innermost);
>> >> + if (!best_step)
>> >> + {
>> >> + /* This is the outermost chrec for the original expression.
>> >> + It's not worth carrying on if the step isn't versionable,
>> >> + or if we're pretty sure it's not for the inner dimension. */
>> >> + if (!step_if_innermost
>> >> + || TREE_CODE (step) != SSA_NAME
>> >> + || !expr_invariant_in_loop_p (loop, step))
>> >> + return;
>> >> +
>> >> + best_step = step;
>> >> +
>> >> + /* We should version for STEP == 1 if we know that that can be
>> >> + true under some circumstances. */
>> >> + if (integer_onep (step_if_innermost))
>> >> + break;
>> >> +
>> >> + /* Bail out if this appears to be the step for the innermost
>> >> + dimension, but isn't likely to be 1.
>> >> +
>> >> + ??? We could instead version for when it equals
>> >> + STEP_IF_INNERMOST, but it's not likely to have as much
>> >> + benefit as versioning for 1. */
>> >> + if (step_if_innermost != step)
>> >> + return;
>> >> + }
>> >> + else
>> >> + {
>> >> + /* This is an inner chrec. If it looks like it iterates over
>> >> + the innermost dimension, abort any attempt to version for
>> >> + the outermost chrec (which if we reach here wasn't itself
>> >> + obviously iterating over the innermost dimension). */
>> >> + if (step_if_innermost && TREE_CONSTANT (step_if_innermost))
>> >> + return;
>> >> + }
>> >> + worklist.quick_push (CHREC_LEFT (expr));
>> >> + continue;
>> >> + }
>> >> +
>> >> + /* Analyze parts of a queued CHREC_LEFT. This is better than just
>> >> + analyzing the evolution of the whole expression since the value
>> >> + could include a mixture of analyzable and unanalyzable elements.
>> >> + Use NSPLIT to count cases in which we add more expressions to
>> >> + analyze, as opposed to just simplifying the existing one. */
>> >> + if (code == PLUS_EXPR || code == POINTER_PLUS_EXPR || code == MINUS_EXPR)
>> >> + {
>> >> + worklist.quick_push (TREE_OPERAND (expr, 0));
>> >> + if (nsplit++ < MAX_NSPLIT)
>> >> + worklist.quick_push (TREE_OPERAND (expr, 1));
>> >> + continue;
>> >> + }
>> >> + if (code == MULT_EXPR)
>> >> + {
>> >> + tree op0 = strip_casts (TREE_OPERAND (expr, 0));
>> >> + tree op1 = TREE_OPERAND (expr, 1);
>> >> + if (TREE_CODE (op0) == PLUS_EXPR || TREE_CODE (op0) == MINUS_EXPR)
>> >> + {
>> >> + tree type = TREE_TYPE (expr);
>> >> + tree op00 = fold_convert (type, TREE_OPERAND (op0, 0));
>> >> + worklist.quick_push (fold_build2 (MULT_EXPR, type, op00, op1));
>> >> + if (nsplit++ < MAX_NSPLIT)
>> >> + {
>> >> + tree op01 = fold_convert (type, TREE_OPERAND (op0, 1));
>> >> + worklist.quick_push (fold_build2 (MULT_EXPR, type,
>> >> + op01, op1));
>> >> + }
>> >> + continue;
>> >> + }
>> >> + }
>> >> +
>> >> + /* If EXPR is invariant in LOOP, analyze it wrt the innermost loop
>> >> + for which it could evolve (i.e. the loop containing the outermost
>> >> + one for which EXPR is invariant). */
>> >
>> > This isn't how analyze_scalar_evolution works - you _always_ have to
>> > feed the innermost loop that the expression is used in (the context).
>> > Then you instantiate the result in the outermost loop of the nest you
>> > are interested in. Otherwise you get garbage.
>>
>> I dispute this. :-) You might remember we had a similar discussion
>> about the use in get_base_for_alignment_1.
>
> I don't remember ;)
>
>> analyze_scalar_evolution describes how the supplied value V evolves,
>> which isn't garbage in itself. Then instantiating the SCEV tells
>
> Then analyze_scalar_evolution wouldn't need a 'loop' parameter.
> It probably should get a stmt (or a use_operand) instead.
Maybe :-)
>> you what this means for the use in:
>>
>> RES = V; // A
>> ...
>> use of RES // B
>>
>> in terms of values that are available at B. But we're not interested in
>> the second bit here. We only want to analyze *how* V evolves, not use
>> or compute its value in a given context. Instantiating the SCEV would
>> lose useful information.
>
> Instantiating only "loses" information in case there's an overall
> effect of a loop to consider. I guess in your case it only brings
> in not useful information.
Well, any information we can get is useful here. Even if something
is never going to be a versioning opportunity itself, it can still
help to stop us adding pointless version checks for other parts
of the address calculation.
> That said, you can indeed pass analyze_scalar_evolution outer loops
> of the loop the use is in (it's also in the outer loops).
>
>> > It looks like you are re-analyzing SSA names in the evolution - that's
>> > odd and shouldn't be necessary (but you forget to instantiate, so...).
>> >
>> > I suggest to move the analyze_scalar_evolution part out of the worklist
>> > loop where you are sure you have an SSA name.
>>
>> This too is because we don't care whether the whole address can
>> be expressed as a SCEV, we just want to search for parts of the
>> calculation that are more likely to be inner dimensions.
>>
>> This isn't (just) because we don't instantiate. It's because we're
>> more aggressive than SCEV can be in looking through casts.
>
> :/
The updated patch avoids this.
>> >> + /* Bail out if OP2 can be analyzed as an evolution; we want to use
>> >> + analyze_evolution in that case instead. There's no point trying
>> >> + hard to avoid repeating the call to analyze_scalar_evolution since
>> >> + that function does its own caching. */
>> >> + if (tree_contains_chrecs (analyze_scalar_evolution (loop, op2), NULL))
>> >
>> > Don't you want !chrec_contains_undetermined (...) instead? I wonder what
>> > is missing to allow all interesting expressions to be analyzed by
>> > SCEV?
>>
>> This code is handling cases that vary arbitrarily between iterations,
>> such as a histogram update. I don't think SCEV could ever provide
>> anything useful here.
>>
>> analyze_scalar_evolution returns op2 if it can't analyze op2 as a SCEV.
>
> Hmm, right. Looks inconsistent to instantiation...
>
>> >> +/* Treat EXPR as a sum of products and apply analyze_product to each of the
>> >> + products. Return true if one of the products provides a versioning
>> >> + opportunity. FACTOR is as for analyze_product. */
>> >> +
>> >> +bool
>> >> +loop_versioning::analyze_sum_of_products (struct loop *loop, tree expr,
>> >> + poly_uint64 factor)
>> >> +{
>> >
>> > This looks awfully close to what tree-affine.c does apart from more
>> > aggressively stripping conversions? I see all the analysis parts
>> > are limited and thus "O(1)" but still there's going to be a lot of
>> > redundancy involved for repeated use of (derived) "IVs"? Wouldn't
>> > it be good to have a bitmap of already handled SSA_NAMEs to stop
>> > processing early?
>>
>> I did consider this but think it'll do more harm than good.
>> Bitmap lookup is O(N) in the worst case, and the PR you were
>> working on a month or so ago shows that that really can bite.
>> Whereas like you say the limits make this O(1).
>>
>> So although Kyrill and Ramana's patch added this check, I'd prefer to
>> leave it out. It seems like premature optimisation and I think it'll
>> make things worse in practice.
>
> OK.
>
>> >> + const unsigned int MAX_NITERS = 8;
>> >> +
>> >> + tree worklist[MAX_NITERS];
>> >> + unsigned int length = 0;
>> >> + worklist[length++] = expr;
>> >> + for (unsigned int i = 0; i < length; ++i)
>> >> + {
>> >> + expr = worklist[i];
>> >> + gassign *assign = maybe_get_assign_strip_casts (loop, expr);
>> >> + if (!assign)
>> >> + continue;
>> >> +
>> >> + tree_code code = gimple_assign_rhs_code (assign);
>> >> + if (code == PLUS_EXPR || code == POINTER_PLUS_EXPR || code == MINUS_EXPR)
>> >
>> > POINTER_MINUS_EXPR?
>>
>> Do you mean POINTER_DIFF_EXPR? I wouldn't expect that to be interesting
>> since it would give information about the two pointers being subtracted.
>>
>> >> +/* Analyze expression EXPR, which occurs in loop LOOP. */
>> >> +
>> >> +void
>> >> +loop_versioning::analyze_expr (struct loop *loop, tree expr)
>> >> +{
>> >> + while (handled_component_p (expr))
>> >> + {
>> >> + /* See whether we can use versioning to avoid a multiplication
>> >> + in the array index. */
>> >> + if (TREE_CODE (expr) == ARRAY_REF)
>> >> + {
>> >> + if (!analyze_sum_of_products (loop, TREE_OPERAND (expr, 1), 1))
>> >> + analyze_evolution (loop, TREE_OPERAND (expr, 1), 1);
>> >> + }
>> >> + expr = TREE_OPERAND (expr, 0);
>> >> + }
>> >> +
>> >> + if (TREE_CODE (expr) == MEM_REF)
>> >> + {
>> >> + tree addr = TREE_OPERAND (expr, 0);
>> >> + /* See whether we can use versioning to avoid a multiplication
>> >> + in the pointer calculation. This is generally only worth
>> >> + doing if the multiplication occurs in this loop rather than
>> >> + an outer loop. */
>> >
>> > Why's that so here but not above for ARRAY_REF? That is, what is
>> > the difference between a[i] and ptr = &a[i]; *ptr?
>>
>> Good question. I think there was a (probably poor) reason, but clearly
>> I didn't write it down.
>>
>> > > + struct loop *loop = gimple_bb (stmt)->loop_father;
>> > > +
>> > > + unsigned int nops = gimple_num_ops (stmt);
>> > > + for (unsigned int i = 0; i < nops; ++i)
>> > > + if (tree op = gimple_op (stmt, i))
>> > > + analyze_expr (loop, op);
>> >
>> > I think you instead want to use gimple_walk_load_store_ops ().
>>
>> I agree with Kyrill's response to this.
>
> C++ bites you? ;) I suppose marshalling a pmf through the void *
> data and having the callback invoke the pmf might be possible?
> But yeah - GCC is C and most callbacks are just function pointers
> and not pointer-to-member fns.
>
> You are at least also walking debug-stmts here I think
> (unless they are expensive_stmt_p ...).
Bah, yes. I remember this only about half the time :-(
>> In addition, I'd originally
>> wondered whether we should also version for ADDR_EXPRs, and that might
>> make sense if we add more types of versioning (not just unit strides)
>> in future.
>
> There's gimple_walk_load_store_addr_ops () of course.
>
>> >> +/* Analyze all the statements in BB looking for useful version checks. */
>> >> +
>> >> +void
>> >> +loop_versioning::analyze_block (basic_block bb)
>> >> +{
>> >> + struct loop *loop = bb->loop_father;
>> >> + loop_info &li = get_loop_info (loop);
>> >> + if (li.rejected_p)
>> >> + return;
>> >> +
>> >> + for (gimple_stmt_iterator gsi = gsi_start_bb (bb); !gsi_end_p (gsi);
>> >> + gsi_next (&gsi))
>> >> + {
>> >> + gimple *stmt = gsi_stmt (gsi);
>> >> + if (expensive_stmt_p (stmt))
>> >> + {
>> >> + if (dump_file && (dump_flags & TDF_DETAILS))
>> >> + {
>> >> + struct loop *loop = gimple_bb (stmt)->loop_father;
>> >> + fprintf (dump_file, ";; Loop %d (depth %d) contains expensive"
>> >> + " stmt: ", loop->num, loop_depth (loop));
>> >> + print_gimple_stmt (dump_file, stmt, 0, TDF_SLIM);
>> >> + }
>> >> + li.rejected_p = true;
>> >
>> > I assume that once a loop is rejected this or another way there's
>> > no reason to look at any outer loop of it, thus ...
>> >
>> >> + break;
>> >> + }
>> >> +
>> >> + /* Only look for direct versioning opportunities in inner loops
>> >> + since the benefit tends to be much smaller for outer loops. */
>> >> + if (!loop->inner)
>> >> + analyze_stmt (stmt);
>> >> +
>> >> + /* The point of the instruction limit is to prevent excessive
>> >> + code growth, so this is a size-based estimate even though
>> >> + the optimization is aimed at speed. */
>> >> + li.num_insns += estimate_num_insns (stmt, &eni_size_weights);
>> >> + }
>> >> +}
>> >> +
>> >> +/* Analyze all the blocks in the function looking for useful version checks.
>> >> + Return true if we found one. */
>> >> +
>> >> +bool
>> >> +loop_versioning::analyze_blocks ()
>> >> +{
>> >> + /* For now we don't try to version the whole function, although
>> >> + versioning at that level could be useful in some cases. */
>> >> + get_loop_info (get_loop (m_fn, 0)).rejected_p = true;
>> >> +
>> >> + basic_block bb;
>> >> + FOR_EACH_BB_FN (bb, m_fn)
>> >> + if (loop_outer (bb->loop_father))
>> >> + analyze_block (bb);
>> >
>> > .... I'd structure this as a
>> >
>> > FOR_EACH_LOOP (... LI_FROM_INNERMOST)
>> > look at loop body, only analyze stmts belonging to loop (not subloops)
>> >
>> > walk eventually even open-coding this as recursion so you can quickly
>> > finish off outer loop processing once an inner loop got disabled.
>>
>> OK. Thought about doing that, but it seemed relatively hard to get the
>> list of blocks that are in a loop but not in subloops. (Let me know if
>> there's an easy way.)
>
> Simply skip bbs with bb->loop_father != loop.
>
>> But I guess it's probably worth an extra ad-hoc
>> walk over the blocks (not the contents) to construct a list of blocks
>> for each loop.
>
> Well, constructing a list of blocks in an order that visits inner loop
> blocks first (recursively) would be useful here and in other places
> I guess. That can be done via get_loop_body (outermost-interesting-loop);
> and then sorting the blocks after loop-father loop-dfs order or so.
> But as said just skipping bb->loop_father != loop would work for me
> (at the expense of looking at bb->loop_father N^2 times).
Yeah, but I wanted to avoid even that quadraticness. The updated patch
uses a linear walk to create a linked list of blocks for each loop.
Changes from last time (to address comments that I might have snipped):
- We now decompose the address fragments into a sum of the form:
[base + ]var1 * const1 + var2 * const2 + ... + [CONST_MIN, CONST_MAX]
then analyse each var<i> * const<i> for versioning opportunities.
First we check whether var<i> is set by a multiplication by a
loop invariant, and only if that fails fall back to SCEV analysis.
There's no longer any processing of the CHREC_LEFT; only the
CHREC_RIGHT matters. The processsing of SCEVs is generally
much simpler than before.
- We now collect grouping information for accesses in a loop by
pooling any of the decomposed addresses fragments described above
that differ only in [CONST_MIN, CONST_MAX]. So for example:
a[i * stride] + ...;
a[i * stride + 1] + ...;
is treated as a single access of sizeof(a[0]) * 2 and:
a[i * stride] + ...;
a[i * stride + 3] + ...;
is treated as a single access of sizeof(a[0]) * 4 (ignoring the gap
in the middle).
- We only version loops for conditions that would lead to consecutive groups,
rather than cases that would leave gaps between groups.
- We process loops innermost-first, and only analyse address fragments
if we get to the end of the loop without finding something that stops
us from versioning.
- The handling of pointers and array indices is consistent.
- This version drops some of the heuristics related to enabling more
loop interchange opportunities. I've kept the tests but changed
the expected result.
- I think the only heuristic-heavy function left is get_inner_likelihood,
which is very difficult to avoid.
Tested on x86_64-linux-gnu, aarch64-linux-gnu and aarch64_be-elf.
Also repeated the performance testing (but haven't yet tried an
LTO variant; will do that over the weekend).
Thanks,
Richard
2018-12-06 Richard Sandiford <richard.sandiford@arm.com>
Ramana Radhakrishnan <ramana.radhakrishnan@arm.com>
Kyrylo Tkachov <kyrylo.tkachov@arm.com>
gcc/
* doc/invoke.texi (-fversion-loops-for-strides): Document
(loop-versioning-group-size, loop-versioning-max-inner-insns)
(loop-versioning-max-outer-insns): Document new --params.
* Makefile.in (OBJS): Add gimple-loop-versioning.o.
* common.opt (fversion-loops-for-strides): New option.
* opts.c (default_options_table): Enable fversion-loops-for-strides
at -O3.
* params.def (PARAM_LOOP_VERSIONING_GROUP_SIZE)
(PARAM_LOOP_VERSIONING_MAX_INNER_INSNS)
(PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS): New parameters.
* passes.def: Add pass_loop_versioning.
* timevar.def (TV_LOOP_VERSIONING): New time variable.
* tree-ssa-propagate.h
(substitute_and_fold_engine::substitute_and_fold): Add an optional
block parameter.
* tree-ssa-propagate.c
(substitute_and_fold_engine::substitute_and_fold): Likewise.
When passed, only walk blocks dominated by that block.
* tree-vrp.h (range_includes_p): Declare.
(range_includes_zero_p): Turn into an inline wrapper around
range_includes_p.
* tree-vrp.c (range_includes_p): New function, generalizing...
(range_includes_zero_p): ...this.
* tree-pass.h (make_pass_loop_versioning): Declare.
* gimple-loop-versioning.cc: New file.
gcc/testsuite/
* gcc.dg/loop-versioning-1.c: New test.
* gcc.dg/loop-versioning-10.c: Likewise.
* gcc.dg/loop-versioning-11.c: Likewise.
* gcc.dg/loop-versioning-2.c: Likewise.
* gcc.dg/loop-versioning-3.c: Likewise.
* gcc.dg/loop-versioning-4.c: Likewise.
* gcc.dg/loop-versioning-5.c: Likewise.
* gcc.dg/loop-versioning-6.c: Likewise.
* gcc.dg/loop-versioning-7.c: Likewise.
* gcc.dg/loop-versioning-8.c: Likewise.
* gcc.dg/loop-versioning-9.c: Likewise.
* gfortran.dg/loop_versioning_1.f90: Likewise.
* gfortran.dg/loop_versioning_2.f90: Likewise.
* gfortran.dg/loop_versioning_3.f90: Likewise.
* gfortran.dg/loop_versioning_4.f90: Likewise.
* gfortran.dg/loop_versioning_5.f90: Likewise.
* gfortran.dg/loop_versioning_6.f90: Likewise.
* gfortran.dg/loop_versioning_7.f90: Likewise.
* gfortran.dg/loop_versioning_8.f90: Likewise.
Index: gcc/doc/invoke.texi
===================================================================
--- gcc/doc/invoke.texi 2018-12-05 08:33:45.282882618 +0000
+++ gcc/doc/invoke.texi 2018-12-06 12:33:59.212098431 +0000
@@ -8256,7 +8256,8 @@ by @option{-O2} and also turns on the fo
-ftree-partial-pre @gol
-ftree-slp-vectorize @gol
-funswitch-loops @gol
--fvect-cost-model}
+-fvect-cost-model @gol
+-fversion-loops-for-strides}
@item -O0
@opindex O0
@@ -10808,6 +10809,30 @@ of the loop on both branches (modified a
Enabled by @option{-fprofile-use} and @option{-fauto-profile}.
+@item -fversion-loops-for-strides
+@opindex fversion-loops-for-strides
+If a loop iterates over an array with a variable stride, create another
+version of the loop that assumes the stride is always one. For example:
+
+@smallexample
+for (int i = 0; i < n; ++i)
+ x[i * stride] = @dots{};
+@end smallexample
+
+becomes:
+
+@smallexample
+if (stride == 1)
+ for (int i = 0; i < n; ++i)
+ x[i] = @dots{};
+else
+ for (int i = 0; i < n; ++i)
+ x[i * stride] = @dots{};
+@end smallexample
+
+This is particularly useful for assumed-shape arrays in Fortran where
+(for example) it allows better vectorization assuming contiguous accesses.
+
@item -ffunction-sections
@itemx -fdata-sections
@opindex ffunction-sections
@@ -12017,6 +12042,15 @@ Hardware autoprefetcher scheduler model
Number of lookahead cycles the model looks into; at '
' only enable instruction sorting heuristic.
+@item loop-versioning-max-inner-insns
+The maximum number of instructions that an inner loop can have
+before the loop versioning pass considers it too big to copy.
+
+@item loop-versioning-max-outer-insns
+The maximum number of instructions that an outer loop can have
+before the loop versioning pass considers it too big to copy,
+discounting any instructions in inner loops that directly benefit
+from versioning.
@end table
@end table
Index: gcc/Makefile.in
===================================================================
--- gcc/Makefile.in 2018-12-05 08:33:45.270882723 +0000
+++ gcc/Makefile.in 2018-12-06 12:33:59.208098467 +0000
@@ -1320,6 +1320,7 @@ OBJS = \
gimple-laddress.o \
gimple-loop-interchange.o \
gimple-loop-jam.o \
+ gimple-loop-versioning.o \
gimple-low.o \
gimple-pretty-print.o \
gimple-ssa-backprop.o \
Index: gcc/common.opt
===================================================================
--- gcc/common.opt 2018-12-05 08:33:45.274882688 +0000
+++ gcc/common.opt 2018-12-06 12:33:59.208098467 +0000
@@ -2775,6 +2775,10 @@ fsplit-loops
Common Report Var(flag_split_loops) Optimization
Perform loop splitting.
+fversion-loops-for-strides
+Common Report Var(flag_version_loops_for_strides) Optimization
+Version loops based on whether indices have a stride of one.
+
funwind-tables
Common Report Var(flag_unwind_tables) Optimization
Just generate unwind tables for exception handling.
Index: gcc/opts.c
===================================================================
--- gcc/opts.c 2018-12-05 08:33:45.282882618 +0000
+++ gcc/opts.c 2018-12-06 12:33:59.212098431 +0000
@@ -556,6 +556,7 @@ static const struct default_options defa
{ OPT_LEVELS_3_PLUS, OPT_ftree_slp_vectorize, NULL, 1 },
{ OPT_LEVELS_3_PLUS, OPT_funswitch_loops, NULL, 1 },
{ OPT_LEVELS_3_PLUS, OPT_fvect_cost_model_, NULL, VECT_COST_MODEL_DYNAMIC },
+ { OPT_LEVELS_3_PLUS, OPT_fversion_loops_for_strides, NULL, 1 },
/* -Ofast adds optimizations to -O3. */
{ OPT_LEVELS_FAST, OPT_ffast_math, NULL, 1 },
Index: gcc/params.def
===================================================================
--- gcc/params.def 2018-12-05 08:33:45.282882618 +0000
+++ gcc/params.def 2018-12-06 12:33:59.212098431 +0000
@@ -1365,6 +1365,19 @@ DEFPARAM(PARAM_LOGICAL_OP_NON_SHORT_CIRC
"True if a non-short-circuit operation is optimal.",
-1, -1, 1)
+DEFPARAM(PARAM_LOOP_VERSIONING_MAX_INNER_INSNS,
+ "loop-versioning-max-inner-insns",
+ "The maximum number of instructions in an inner loop that is being"
+ " considered for versioning.",
+ 200, 0, 0)
+
+DEFPARAM(PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS,
+ "loop-versioning-max-outer-insns",
+ "The maximum number of instructions in an outer loop that is being"
+ " considered for versioning, on top of the instructions in inner"
+ " loops.",
+ 100, 0, 0)
+
/*
Local variables:
Index: gcc/passes.def
===================================================================
--- gcc/passes.def 2018-12-05 08:33:45.282882618 +0000
+++ gcc/passes.def 2018-12-06 12:33:59.212098431 +0000
@@ -265,6 +265,7 @@ along with GCC; see the file COPYING3.
NEXT_PASS (pass_tree_unswitch);
NEXT_PASS (pass_scev_cprop);
NEXT_PASS (pass_loop_split);
+ NEXT_PASS (pass_loop_versioning);
NEXT_PASS (pass_loop_jam);
/* All unswitching, final value replacement and splitting can expose
empty loops. Remove them now. */
Index: gcc/timevar.def
===================================================================
--- gcc/timevar.def 2018-12-05 08:33:45.286882584 +0000
+++ gcc/timevar.def 2018-12-06 12:33:59.220098363 +0000
@@ -234,6 +234,7 @@ DEFTIMEVAR (TV_DSE1 , "
DEFTIMEVAR (TV_DSE2 , "dead store elim2")
DEFTIMEVAR (TV_LOOP , "loop analysis")
DEFTIMEVAR (TV_LOOP_INIT , "loop init")
+DEFTIMEVAR (TV_LOOP_VERSIONING , "loop versioning")
DEFTIMEVAR (TV_LOOP_MOVE_INVARIANTS , "loop invariant motion")
DEFTIMEVAR (TV_LOOP_UNROLL , "loop unrolling")
DEFTIMEVAR (TV_LOOP_DOLOOP , "loop doloop")
Index: gcc/tree-ssa-propagate.h
===================================================================
--- gcc/tree-ssa-propagate.h 2018-12-05 08:33:45.286882584 +0000
+++ gcc/tree-ssa-propagate.h 2018-12-06 12:33:59.220098363 +0000
@@ -104,7 +104,7 @@ extern void propagate_tree_value_into_st
virtual bool fold_stmt (gimple_stmt_iterator *) { return false; }
virtual tree get_value (tree) { return NULL_TREE; }
- bool substitute_and_fold (void);
+ bool substitute_and_fold (basic_block = NULL);
bool replace_uses_in (gimple *);
bool replace_phi_args_in (gphi *);
};
Index: gcc/tree-ssa-propagate.c
===================================================================
--- gcc/tree-ssa-propagate.c 2018-12-05 08:33:45.286882584 +0000
+++ gcc/tree-ssa-propagate.c 2018-12-06 12:33:59.220098363 +0000
@@ -1154,6 +1154,10 @@ substitute_and_fold_dom_walker::before_d
/* Perform final substitution and folding of propagated values.
+ Process the whole function if BLOCK is null, otherwise only
+ process the blocks that BLOCK dominates. In the latter case,
+ it is the caller's responsibility to ensure that dominator
+ information is available and up-to-date.
PROP_VALUE[I] contains the single value that should be substituted
at every use of SSA name N_I. If PROP_VALUE is NULL, no values are
@@ -1170,16 +1174,24 @@ substitute_and_fold_dom_walker::before_d
Return TRUE when something changed. */
bool
-substitute_and_fold_engine::substitute_and_fold (void)
+substitute_and_fold_engine::substitute_and_fold (basic_block block)
{
if (dump_file && (dump_flags & TDF_DETAILS))
fprintf (dump_file, "\nSubstituting values and folding statements\n\n");
memset (&prop_stats, 0, sizeof (prop_stats));
- calculate_dominance_info (CDI_DOMINATORS);
+ /* Don't call calculate_dominance_info when iterating over a subgraph.
+ Callers that are using the interface this way are likely to want to
+ iterate over several disjoint subgraphs, and it would be expensive
+ in enable-checking builds to revalidate the whole dominance tree
+ each time. */
+ if (block)
+ gcc_assert (dom_info_state (CDI_DOMINATORS));
+ else
+ calculate_dominance_info (CDI_DOMINATORS);
substitute_and_fold_dom_walker walker (CDI_DOMINATORS, this);
- walker.walk (ENTRY_BLOCK_PTR_FOR_FN (cfun));
+ walker.walk (block ? block : ENTRY_BLOCK_PTR_FOR_FN (cfun));
/* We cannot remove stmts during the BB walk, especially not release
SSA names there as that destroys the lattice of our callers.
Index: gcc/tree-vrp.h
===================================================================
--- gcc/tree-vrp.h 2018-12-05 08:33:45.290882549 +0000
+++ gcc/tree-vrp.h 2018-12-06 12:33:59.220098363 +0000
@@ -243,7 +243,7 @@ struct assert_info
extern void register_edge_assert_for (tree, edge, enum tree_code,
tree, tree, vec<assert_info> &);
extern bool stmt_interesting_for_vrp (gimple *);
-extern bool range_includes_zero_p (const value_range_base *);
+extern bool range_includes_p (const value_range_base *, HOST_WIDE_INT);
extern bool infer_value_range (gimple *, tree, tree_code *, tree *);
extern bool vrp_bitmap_equal_p (const_bitmap, const_bitmap);
@@ -285,4 +285,12 @@ extern tree get_single_symbol (tree, boo
extern void maybe_set_nonzero_bits (edge, tree);
extern value_range_kind determine_value_range (tree, wide_int *, wide_int *);
+/* Return TRUE if *VR includes the value zero. */
+
+inline bool
+range_includes_zero_p (const value_range_base *vr)
+{
+ return range_includes_p (vr, 0);
+}
+
#endif /* GCC_TREE_VRP_H */
Index: gcc/tree-vrp.c
===================================================================
--- gcc/tree-vrp.c 2018-12-05 08:33:45.290882549 +0000
+++ gcc/tree-vrp.c 2018-12-06 12:33:59.220098363 +0000
@@ -1173,15 +1173,14 @@ value_inside_range (tree val, tree min,
}
-/* Return TRUE if *VR includes the value zero. */
+/* Return TRUE if *VR includes the value X. */
bool
-range_includes_zero_p (const value_range_base *vr)
+range_includes_p (const value_range_base *vr, HOST_WIDE_INT x)
{
if (vr->varying_p () || vr->undefined_p ())
return true;
- tree zero = build_int_cst (vr->type (), 0);
- return vr->may_contain_p (zero);
+ return vr->may_contain_p (build_int_cst (vr->type (), x));
}
/* If *VR has a value range that is a single constant value return that,
Index: gcc/tree-pass.h
===================================================================
--- gcc/tree-pass.h 2018-12-05 08:33:45.286882584 +0000
+++ gcc/tree-pass.h 2018-12-06 12:33:59.220098363 +0000
@@ -362,6 +362,7 @@ extern gimple_opt_pass *make_pass_fix_lo
extern gimple_opt_pass *make_pass_tree_loop (gcc::context *ctxt);
extern gimple_opt_pass *make_pass_tree_no_loop (gcc::context *ctxt);
extern gimple_opt_pass *make_pass_tree_loop_init (gcc::context *ctxt);
+extern gimple_opt_pass *make_pass_loop_versioning (gcc::context *ctxt);
extern gimple_opt_pass *make_pass_lim (gcc::context *ctxt);
extern gimple_opt_pass *make_pass_linterchange (gcc::context *ctxt);
extern gimple_opt_pass *make_pass_tree_unswitch (gcc::context *ctxt);
Index: gcc/gimple-loop-versioning.cc
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/gimple-loop-versioning.cc 2018-12-06 12:33:59.212098431 +0000
@@ -0,0 +1,1743 @@
+/* Loop versioning pass.
+ Copyright (C) 2018 Free Software Foundation, Inc.
+
+This file is part of GCC.
+
+GCC is free software; you can redistribute it and/or modify it
+under the terms of the GNU General Public License as published by the
+Free Software Foundation; either version 3, or (at your option) any
+later version.
+
+GCC is distributed in the hope that it will be useful, but WITHOUT
+ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
+for more details.
+
+You should have received a copy of the GNU General Public License
+along with GCC; see the file COPYING3. If not see
+<http://www.gnu.org/licenses/>. */
+
+#include "config.h"
+#include "system.h"
+#include "coretypes.h"
+#include "backend.h"
+#include "tree.h"
+#include "gimple.h"
+#include "gimple-iterator.h"
+#include "tree-pass.h"
+#include "gimplify-me.h"
+#include "cfgloop.h"
+#include "tree-ssa-loop.h"
+#include "ssa.h"
+#include "tree-scalar-evolution.h"
+#include "tree-chrec.h"
+#include "tree-ssa-loop-ivopts.h"
+#include "fold-const.h"
+#include "tree-ssa-propagate.h"
+#include "tree-inline.h"
+#include "domwalk.h"
+#include "alloc-pool.h"
+#include "vr-values.h"
+#include "gimple-ssa-evrp-analyze.h"
+#include "tree-vectorizer.h"
+#include "omp-general.h"
+#include "params.h"
+
+namespace {
+
+/* This pass looks for loops that could be simplified if certain loop
+ invariant conditions were true. It is effectively a form of loop
+ splitting in which the pass produces the split conditions itself,
+ instead of using ones that are already present in the IL.
+
+ Versioning for when strides are 1
+ ---------------------------------
+
+ At the moment the only thing the pass looks for are memory references
+ like:
+
+ for (auto i : ...)
+ ...x[i * stride]...
+
+ It considers changing such loops to:
+
+ if (stride == 1)
+ for (auto i : ...) [A]
+ ...x[i]...
+ else
+ for (auto i : ...) [B]
+ ...x[i * stride]...
+
+ This can have several benefits:
+
+ (1) [A] is often easier or cheaper to vectorize than [B].
+
+ (2) The scalar code in [A] is simpler than the scalar code in [B]
+ (if the loops cannot be vectorized or need an epilogue loop).
+
+ (3) We might recognize [A] as a pattern, such as a memcpy or memset.
+
+ (4) [A] has simpler address evolutions, which can help other passes
+ like loop interchange.
+
+ The optimization is particularly useful for assumed-shape arrays in
+ Fortran, where the stride of the innermost dimension depends on the
+ array descriptor but is often equal to 1 in practice. For example:
+
+ subroutine f1(x)
+ real :: x(:)
+ x(:) = 100
+ end subroutine f1
+
+ generates the equivalent of:
+
+ raw_stride = *x.dim[0].stride;
+ stride = raw_stride != 0 ? raw_stride : 1;
+ x_base = *x.data;
+ ...
+ tmp1 = stride * S;
+ tmp2 = tmp1 - stride;
+ *x_base[tmp2] = 1.0e+2;
+
+ but in the common case that stride == 1, the last three statements
+ simplify to:
+
+ tmp3 = S + -1;
+ *x_base[tmp3] = 1.0e+2;
+
+ The optimization is in principle very simple. The difficult parts are:
+
+ (a) deciding which parts of a general address calculation correspond
+ to the inner dimension of an array, since this usually isn't explicit
+ in the IL, and for C often isn't even explicit in the source code
+
+ (b) estimating when the transformation is worthwhile
+
+ Structure
+ ---------
+
+ The pass has four phases:
+
+ (1) Walk through the statements looking for and recording potential
+ versioning opportunities. Stop if there are none.
+
+ (2) Use context-sensitive range information to see whether any versioning
+ conditions are impossible in practice. Remove them if so, and stop
+ if no opportunities remain.
+
+ (We do this only after (1) to keep compile time down when no
+ versioning opportunities exist.)
+
+ (3) Apply the cost model. Decide which versioning opportunities are
+ worthwhile and at which nesting level they should be applied.
+
+ (4) Attempt to version all the loops selected by (3), so that:
+
+ for (...)
+ ...
+
+ becomes:
+
+ if (!cond)
+ for (...) // Original loop
+ ...
+ else
+ for (...) // New loop
+ ...
+
+ Use the version condition COND to simplify the new loop. */
+
+/* Enumerates the likelihood that a particular value indexes the inner
+ dimension of an array. */
+enum inner_likelihood {
+ INNER_UNLIKELY,
+ INNER_DONT_KNOW,
+ INNER_LIKELY
+};
+
+/* Information about one term of an address_info. */
+struct address_term_info
+{
+ /* The value of the term is EXPR * MULTIPLIER. */
+ tree expr;
+ unsigned HOST_WIDE_INT multiplier;
+
+ /* The stride applied by EXPR in each iteration of some unrecorded loop,
+ or null if no stride has been identified. */
+ tree stride;
+
+ /* Enumerates the likelihood that EXPR indexes the inner dimension
+ of an array. */
+ enum inner_likelihood inner_likelihood;
+
+ /* True if STRIDE == 1 is a versioning opportunity when considered
+ in isolation. */
+ bool versioning_opportunity_p;
+};
+
+/* Information about an address calculation, and the range of constant
+ offsets applied to it. */
+struct address_info
+{
+ static const unsigned int MAX_TERMS = 8;
+
+ /* One statement that calculates the address. If multiple statements
+ share the same address, we only record the first. */
+ gimple *stmt;
+
+ /* The loop containing STMT (cached for convenience). If multiple
+ statements share the same address, they all belong to this loop. */
+ struct loop *loop;
+
+ /* A decomposition of the calculation into a sum of terms plus an
+ optional base. When BASE is provided, it is never an SSA name.
+ Once initialization is complete, all members of TERMs are SSA names. */
+ tree base;
+ auto_vec<address_term_info, MAX_TERMS> terms;
+
+ /* All bytes accessed from the address fall in the offset range
+ [MIN_OFFSET, MAX_OFFSET). */
+ HOST_WIDE_INT min_offset, max_offset;
+};
+
+/* Stores addresses based on their base and terms (ignoring the offsets). */
+struct address_info_hasher : nofree_ptr_hash <address_info>
+{
+ static hashval_t hash (const address_info *);
+ static bool equal (const address_info *, const address_info *);
+};
+
+/* Information about the versioning we'd like to apply to a loop. */
+struct loop_info
+{
+ bool worth_versioning_p () const;
+
+ /* True if we've decided not to version this loop. The remaining
+ fields are meaningless if so. */
+ bool rejected_p;
+
+ /* True if at least one subloop of this loop benefits from versioning. */
+ bool subloops_benefit_p;
+
+ /* An estimate of the total number of instructions in the loop,
+ excluding those in subloops that benefit from versioning. */
+ unsigned int num_insns;
+
+ /* The outermost loop that can handle all the version checks
+ described below. */
+ struct loop *outermost;
+
+ /* The first entry in the list of blocks that belong to this loop
+ (and not to subloops). m_next_block_in_loop provides the chain
+ pointers for the list. */
+ basic_block block_list;
+
+ /* We'd like to version the loop for the case in which these SSA names
+ (keyed off their SSA_NAME_VERSION) are all equal to 1 at runtime. */
+ bitmap_head unity_names;
+};
+
+/* The main pass structure. */
+class loop_versioning
+{
+public:
+ loop_versioning (function *);
+ ~loop_versioning ();
+ unsigned int run ();
+
+private:
+ /* Used to walk the dominator tree to find loop versioning conditions
+ that are always false. */
+ class lv_dom_walker : public dom_walker
+ {
+ public:
+ lv_dom_walker (loop_versioning &);
+
+ edge before_dom_children (basic_block) FINAL OVERRIDE;
+ void after_dom_children (basic_block) FINAL OVERRIDE;
+
+ private:
+ /* The parent pass. */
+ loop_versioning &m_lv;
+
+ /* Used to build context-dependent range information. */
+ evrp_range_analyzer m_range_analyzer;
+ };
+
+ /* Used to simplify statements based on conditions that are established
+ by the version checks. */
+ class name_prop : public substitute_and_fold_engine
+ {
+ public:
+ name_prop (loop_info &li) : m_li (li) {}
+ tree get_value (tree) FINAL OVERRIDE;
+
+ private:
+ /* Information about the versioning we've performed on the loop. */
+ loop_info &m_li;
+ };
+
+ loop_info &get_loop_info (struct loop *loop) { return m_loops[loop->num]; }
+
+ unsigned int max_insns_for_loop (struct loop *);
+ bool expensive_stmt_p (gimple *);
+
+ void version_for_unity (gimple *, tree);
+ bool acceptable_multiplier_p (tree, unsigned HOST_WIDE_INT,
+ unsigned HOST_WIDE_INT * = 0);
+ bool acceptable_type_p (tree, unsigned HOST_WIDE_INT *);
+ bool multiply_term_by (address_term_info &, tree);
+ inner_likelihood get_inner_likelihood (tree, unsigned HOST_WIDE_INT);
+ void analyze_stride (address_info &, address_term_info &,
+ tree, struct loop *);
+ bool find_per_loop_multiplication (address_info &, address_term_info &);
+ void analyze_term_using_scevs (address_info &, address_term_info &);
+ void analyze_address_fragment (address_info &);
+ void record_address_fragment (gimple *, unsigned HOST_WIDE_INT,
+ tree, unsigned HOST_WIDE_INT, HOST_WIDE_INT);
+ void analyze_expr (gimple *, tree);
+ bool analyze_block (basic_block);
+ bool analyze_blocks ();
+
+ void prune_loop_conditions (struct loop *, vr_values *);
+ bool prune_conditions ();
+
+ void merge_loop_info (struct loop *, struct loop *);
+ void add_loop_to_queue (struct loop *);
+ bool decide_whether_loop_is_versionable (struct loop *);
+ bool make_versioning_decisions ();
+
+ bool version_loop (struct loop *);
+ bool implement_versioning_decisions ();
+
+ /* The function we're optimizing. */
+ function *m_fn;
+
+ /* The obstack to use for all pass-specific bitmaps. */
+ bitmap_obstack m_bitmap_obstack;
+
+ /* An obstack to use for general allocation. */
+ obstack m_obstack;
+
+ /* The number of loops in the function. */
+ unsigned int m_nloops;
+
+ /* The total number of loop version conditions we've found. */
+ unsigned int m_num_conditions;
+
+ /* Assume that an address fragment of the form i * stride * scale
+ (for variable stride and constant scale) will not benefit from
+ versioning for stride == 1 when scale is greater than this value. */
+ unsigned HOST_WIDE_INT m_maximum_scale;
+
+ /* Information about each loop. */
+ auto_vec<loop_info> m_loops;
+
+ /* Used to form a linked list of blocks that belong to a loop,
+ started by loop_info::block_list. */
+ auto_vec<basic_block> m_next_block_in_loop;
+
+ /* The list of loops that we've decided to version. */
+ auto_vec<struct loop *> m_loops_to_version;
+
+ /* A table of addresses in the current loop, keyed off their values
+ but not their offsets. */
+ hash_table <address_info_hasher> m_address_table;
+
+ /* A list of all addresses in M_ADDRESS_TABLE, in a predictable order. */
+ auto_vec <address_info *, 32> m_address_list;
+};
+
+/* If EXPR is an SSA name and not a default definition, return the
+ defining statement, otherwise return null. */
+
+static gimple *
+maybe_get_stmt (tree expr)
+{
+ if (TREE_CODE (expr) == SSA_NAME && !SSA_NAME_IS_DEFAULT_DEF (expr))
+ return SSA_NAME_DEF_STMT (expr);
+ return NULL;
+}
+
+/* Like maybe_get_stmt, but also return null if the defining
+ statement isn't an assignment. */
+
+static gassign *
+maybe_get_assign (tree expr)
+{
+ return safe_dyn_cast <gassign *> (maybe_get_stmt (expr));
+}
+
+/* Return true if this pass should look through a cast of expression FROM
+ to type TYPE when analyzing pieces of an address. */
+
+static bool
+look_through_cast_p (tree type, tree from)
+{
+ return (INTEGRAL_TYPE_P (TREE_TYPE (from)) == INTEGRAL_TYPE_P (type)
+ && POINTER_TYPE_P (TREE_TYPE (from)) == POINTER_TYPE_P (type));
+}
+
+/* Strip all conversions of integers or pointers from EXPR, regardless
+ of whether the conversions are nops. This is useful in the context
+ of this pass because we're not trying to fold or simulate the
+ expression; we just want to see how it's structured. */
+
+static tree
+strip_casts (tree expr)
+{
+ const unsigned int MAX_NITERS = 4;
+
+ tree type = TREE_TYPE (expr);
+ while (CONVERT_EXPR_P (expr)
+ && look_through_cast_p (type, TREE_OPERAND (expr, 0)))
+ expr = TREE_OPERAND (expr, 0);
+
+ for (unsigned int niters = 0; niters < MAX_NITERS; ++niters)
+ {
+ gassign *assign = maybe_get_assign (expr);
+ if (assign
+ && CONVERT_EXPR_CODE_P (gimple_assign_rhs_code (assign))
+ && look_through_cast_p (type, gimple_assign_rhs1 (assign)))
+ expr = gimple_assign_rhs1 (assign);
+ else
+ break;
+ }
+ return expr;
+}
+
+/* Compare two address_term_infos in the same address_info. */
+
+static int
+compare_address_terms (const void *a_uncast, const void *b_uncast)
+{
+ const address_term_info *a = (const address_term_info *) a_uncast;
+ const address_term_info *b = (const address_term_info *) b_uncast;
+
+ if (a->expr != b->expr)
+ return SSA_NAME_VERSION (a->expr) < SSA_NAME_VERSION (b->expr) ? -1 : 1;
+
+ if (a->multiplier != b->multiplier)
+ return a->multiplier < b->multiplier ? -1 : 1;
+
+ return 0;
+}
+
+/* Dump ADDRESS using flags FLAGS. */
+
+static void
+dump_address_info (dump_flags_t flags, address_info &address)
+{
+ if (address.base)
+ dump_printf (flags, "%T + ", address.base);
+ for (unsigned int i = 0; i < address.terms.length (); ++i)
+ {
+ if (i != 0)
+ dump_printf (flags, " + ");
+ dump_printf (flags, "%T", address.terms[i].expr);
+ if (address.terms[i].multiplier != 1)
+ dump_printf (flags, " * %wd", address.terms[i].multiplier);
+ }
+ dump_printf (flags, " + [%wd, %wd]",
+ address.min_offset, address.max_offset - 1);
+}
+
+/* Hash an address_info based on its base and terms. */
+
+hashval_t
+address_info_hasher::hash (const address_info *info)
+{
+ inchash::hash hash;
+ hash.add_int (info->base ? TREE_CODE (info->base) : 0);
+ hash.add_int (info->terms.length ());
+ for (unsigned int i = 0; i < info->terms.length (); ++i)
+ {
+ hash.add_int (SSA_NAME_VERSION (info->terms[i].expr));
+ hash.add_hwi (info->terms[i].multiplier);
+ }
+ return hash.end ();
+}
+
+/* Return true if two address_infos have equal bases and terms. Other
+ properties might be different (such as the statement or constant
+ offset range). */
+
+bool
+address_info_hasher::equal (const address_info *a, const address_info *b)
+{
+ if (a->base != b->base
+ && (!a->base || !b->base || !operand_equal_p (a->base, b->base, 0)))
+ return false;
+
+ if (a->terms.length () != b->terms.length ())
+ return false;
+
+ for (unsigned int i = 0; i < a->terms.length (); ++i)
+ if (a->terms[i].expr != b->terms[i].expr
+ || a->terms[i].multiplier != b->terms[i].multiplier)
+ return false;
+
+ return true;
+}
+
+/* Return true if we want to version the loop, i.e. if we have a
+ specific reason for doing so and no specific reason not to. */
+
+bool
+loop_info::worth_versioning_p () const
+{
+ return (!rejected_p
+ && (!bitmap_empty_p (&unity_names) || subloops_benefit_p));
+}
+
+loop_versioning::lv_dom_walker::lv_dom_walker (loop_versioning &lv)
+ : dom_walker (CDI_DOMINATORS), m_lv (lv)
+{
+}
+
+/* Process BB before processing the blocks it dominates. */
+
+edge
+loop_versioning::lv_dom_walker::before_dom_children (basic_block bb)
+{
+ m_range_analyzer.enter (bb);
+
+ if (bb == bb->loop_father->header)
+ m_lv.prune_loop_conditions (bb->loop_father,
+ m_range_analyzer.get_vr_values ());
+
+ for (gimple_stmt_iterator si = gsi_start_bb (bb); !gsi_end_p (si);
+ gsi_next (&si))
+ m_range_analyzer.record_ranges_from_stmt (gsi_stmt (si), false);
+
+ return NULL;
+}
+
+/* Process BB after processing the blocks it dominates. */
+
+void
+loop_versioning::lv_dom_walker::after_dom_children (basic_block bb)
+{
+ m_range_analyzer.leave (bb);
+}
+
+/* Decide whether to replace VAL with a new value in a versioned loop.
+ Return the new value if so, otherwise return null. */
+
+tree
+loop_versioning::name_prop::get_value (tree val)
+{
+ if (TREE_CODE (val) == SSA_NAME
+ && bitmap_bit_p (&m_li.unity_names, SSA_NAME_VERSION (val)))
+ return build_one_cst (TREE_TYPE (val));
+ return NULL_TREE;
+}
+
+/* Initialize the structure to optimize FN. */
+
+loop_versioning::loop_versioning (function *fn)
+ : m_fn (fn),
+ m_nloops (number_of_loops (fn)),
+ m_num_conditions (0),
+ m_address_table (31)
+{
+ bitmap_obstack_initialize (&m_bitmap_obstack);
+ gcc_obstack_init (&m_obstack);
+
+ /* Initialize the loop information. */
+ m_loops.safe_grow_cleared (m_nloops);
+ for (unsigned int i = 0; i < m_nloops; ++i)
+ {
+ m_loops[i].outermost = get_loop (m_fn, 0);
+ bitmap_initialize (&m_loops[i].unity_names, &m_bitmap_obstack);
+ }
+
+ /* Initialize the list of blocks that belong to each loop. */
+ unsigned int nbbs = last_basic_block_for_fn (fn);
+ m_next_block_in_loop.safe_grow (nbbs);
+ basic_block bb;
+ FOR_EACH_BB_FN (bb, fn)
+ {
+ loop_info &li = get_loop_info (bb->loop_father);
+ m_next_block_in_loop[bb->index] = li.block_list;
+ li.block_list = bb;
+ }
+
+ /* MAX_FIXED_MODE_SIZE should be a reasonable maximum scale for
+ unvectorizable code, since it is the largest size that can be
+ handled efficiently by scalar code. omp_max_vf calculates the
+ maximum number of bytes in a vector, when such a value is relevant
+ to loop optimization. */
+ m_maximum_scale = estimated_poly_value (omp_max_vf ());
+ m_maximum_scale = MAX (m_maximum_scale, MAX_FIXED_MODE_SIZE);
+}
+
+loop_versioning::~loop_versioning ()
+{
+ bitmap_obstack_release (&m_bitmap_obstack);
+ obstack_free (&m_obstack, NULL);
+}
+
+/* Return the maximum number of instructions allowed in LOOP before
+ it becomes too big for versioning.
+
+ There are separate limits for inner and outer loops. The limit for
+ inner loops applies only to loops that benefit directly from versioning.
+ The limit for outer loops applies to all code in the outer loop and
+ its subloops that *doesn't* benefit directly from versioning; such code
+ would be "taken along for the ride". The idea is that if the cost of
+ the latter is small, it is better to version outer loops rather than
+ inner loops, both to reduce the number of repeated checks and to enable
+ more of the loop nest to be optimized as a natural nest (e.g. by loop
+ interchange or outer-loop vectorization). */
+
+unsigned int
+loop_versioning::max_insns_for_loop (struct loop *loop)
+{
+ return (loop->inner
+ ? PARAM_VALUE (PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS)
+ : PARAM_VALUE (PARAM_LOOP_VERSIONING_MAX_INNER_INSNS));
+}
+
+/* Return true if for cost reasons we should avoid versioning any loop
+ that contains STMT.
+
+ Note that we don't need to check whether versioning is invalid for
+ correctness reasons, since the versioning process does that for us.
+ The conditions involved are too rare to be worth duplicating here. */
+
+bool
+loop_versioning::expensive_stmt_p (gimple *stmt)
+{
+ if (gcall *call = dyn_cast <gcall *> (stmt))
+ /* Assume for now that the time spent in an "expensive" call would
+ overwhelm any saving from versioning. */
+ return !gimple_inexpensive_call_p (call);
+ return false;
+}
+
+/* Record that we want to version the loop that contains STMT for the
+ case in which SSA name NAME is equal to 1. We already know that NAME
+ is invariant in the loop. */
+
+void
+loop_versioning::version_for_unity (gimple *stmt, tree name)
+{
+ struct loop *loop = loop_containing_stmt (stmt);
+ loop_info &li = get_loop_info (loop);
+
+ if (bitmap_set_bit (&li.unity_names, SSA_NAME_VERSION (name)))
+ {
+ /* This is the first time we've wanted to version LOOP for NAME.
+ Keep track of the outermost loop that can handle all versioning
+ checks in LI. */
+ struct loop *outermost
+ = outermost_invariant_loop_for_expr (loop, name);
+ if (loop_depth (li.outermost) < loop_depth (outermost))
+ li.outermost = outermost;
+
+ if (dump_enabled_p ())
+ {
+ dump_printf_loc (MSG_NOTE, stmt, "want to version containing loop"
+ " for when %T == 1", name);
+ if (outermost == loop)
+ dump_printf (MSG_NOTE, "; cannot hoist check further");
+ else
+ {
+ dump_printf (MSG_NOTE, "; could implement the check at loop"
+ " depth %d", loop_depth (outermost));
+ if (loop_depth (li.outermost) > loop_depth (outermost))
+ dump_printf (MSG_NOTE, ", but other checks only allow"
+ " a depth of %d", loop_depth (li.outermost));
+ }
+ dump_printf (MSG_NOTE, "\n");
+ }
+
+ m_num_conditions += 1;
+ }
+ else
+ {
+ /* This is a duplicate request. */
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, stmt, "already asked to version containing"
+ " loop for when %T == 1\n", name);
+ }
+}
+
+/* Return true if OP1_TREE is constant and if in principle it is worth
+ versioning an address fragment of the form:
+
+ i * OP1_TREE * OP2 * stride
+
+ for the case in which stride == 1. This in practice means testing
+ whether:
+
+ OP1_TREE * OP2 <= M_MAXIMUM_SCALE.
+
+ If RESULT is nonnull, store OP1_TREE * OP2 there when returning true. */
+
+bool
+loop_versioning::acceptable_multiplier_p (tree op1_tree,
+ unsigned HOST_WIDE_INT op2,
+ unsigned HOST_WIDE_INT *result)
+{
+ if (tree_fits_uhwi_p (op1_tree))
+ {
+ unsigned HOST_WIDE_INT op1 = tree_to_uhwi (op1_tree);
+ /* The first part checks for overflow. */
+ if (op1 * op2 >= op2 && op1 * op2 <= m_maximum_scale)
+ {
+ if (result)
+ *result = op1 * op2;
+ return true;
+ }
+ }
+ return false;
+}
+
+/* Return true if it is worth using loop versioning on a memory access
+ of type TYPE. Store the size of the access in *SIZE if so. */
+
+bool
+loop_versioning::acceptable_type_p (tree type, unsigned HOST_WIDE_INT *size)
+{
+ return (TYPE_SIZE_UNIT (type)
+ && acceptable_multiplier_p (TYPE_SIZE_UNIT (type), 1, size));
+}
+
+/* See whether OP is constant and whether we can multiply TERM by that
+ constant without exceeding M_MAXIMUM_SCALE. Return true and update
+ TERM if so. */
+
+bool
+loop_versioning::multiply_term_by (address_term_info &term, tree op)
+{
+ return acceptable_multiplier_p (op, term.multiplier, &term.multiplier);
+}
+
+/* Decide whether an address fragment of the form STRIDE * MULTIPLIER
+ is likely to be indexing an innermost dimension, returning the result
+ as an INNER_* probability. */
+
+inner_likelihood
+loop_versioning::get_inner_likelihood (tree stride,
+ unsigned HOST_WIDE_INT multiplier)
+{
+ const unsigned int MAX_NITERS = 8;
+
+ /* Iterate over possible values of STRIDE. Return INNER_LIKELY if at
+ least one of those values is likely to be for the innermost dimension.
+ Record in UNLIKELY_P if at least one of those values is unlikely to be
+ for the innermost dimension.
+
+ E.g. for:
+
+ stride = cond ? a * b : 1
+
+ we should treat STRIDE as being a likely inner dimension, since
+ we know that it is 1 under at least some circumstances. (See the
+ Fortran example below.) However:
+
+ stride = a * b
+
+ on its own is unlikely to be for the innermost dimension, since
+ that would require both a and b to be 1 at runtime. */
+ bool unlikely_p = false;
+ tree worklist[MAX_NITERS];
+ unsigned int length = 0;
+ worklist[length++] = stride;
+ for (unsigned int i = 0; i < length; ++i)
+ {
+ tree expr = worklist[i];
+
+ if (CONSTANT_CLASS_P (expr))
+ {
+ /* See if EXPR * MULTIPLIER would be consistent with an individual
+ access or a small grouped access. */
+ if (acceptable_multiplier_p (expr, multiplier))
+ return INNER_LIKELY;
+ else
+ unlikely_p = true;
+ }
+ else if (gimple *stmt = maybe_get_stmt (expr))
+ {
+ /* If EXPR is set by a PHI node, queue its arguments in case
+ we find one that is consistent with an inner dimension.
+
+ An important instance of this is the Fortran handling of array
+ descriptors, which calculates the stride of the inner dimension
+ using a PHI equivalent of:
+
+ raw_stride = a.dim[0].stride;
+ stride = raw_stride != 0 ? raw_stride : 1;
+
+ (Strides for outer dimensions do not treat 0 specially.) */
+ if (gphi *phi = dyn_cast <gphi *> (stmt))
+ {
+ unsigned int nargs = gimple_phi_num_args (phi);
+ for (unsigned int j = 0; j < nargs && length < MAX_NITERS; ++j)
+ worklist[length++] = strip_casts (gimple_phi_arg_def (phi, j));
+ }
+ /* If the value is set by an assignment, expect it to be read
+ from memory (such as an array descriptor) rather than be
+ calculated. */
+ else if (gassign *assign = dyn_cast <gassign *> (stmt))
+ {
+ if (!gimple_assign_load_p (assign))
+ unlikely_p = true;
+ }
+ /* Things like calls don't really tell us anything. */
+ }
+ }
+
+ /* We didn't find any possible values of STRIDE that were likely to be
+ for the innermost dimension. If we found one that was actively
+ unlikely to be for the innermost dimension, assume that that applies
+ to STRIDE too. */
+ return unlikely_p ? INNER_UNLIKELY : INNER_DONT_KNOW;
+}
+
+/* The caller has identified that STRIDE is the stride of interest
+ in TERM, and that the stride is applied in OP_LOOP. Record this
+ information in TERM, deciding whether STRIDE is likely to be for
+ the innermost dimension of an array and whether it represents a
+ versioning opportunity. ADDRESS is the address that contains TERM. */
+
+void
+loop_versioning::analyze_stride (address_info &address,
+ address_term_info &term,
+ tree stride, struct loop *op_loop)
+{
+ term.stride = stride;
+
+ term.inner_likelihood = get_inner_likelihood (stride, term.multiplier);
+ if (dump_enabled_p ())
+ {
+ if (term.inner_likelihood == INNER_LIKELY)
+ dump_printf_loc (MSG_NOTE, address.stmt, "%T is likely to be the"
+ " innermost dimension\n", stride);
+ else if (term.inner_likelihood == INNER_UNLIKELY)
+ dump_printf_loc (MSG_NOTE, address.stmt, "%T is probably not the"
+ " innermost dimension\n", stride);
+ else
+ dump_printf_loc (MSG_NOTE, address.stmt, "cannot tell whether %T"
+ " is the innermost dimension\n", stride);
+ }
+
+ /* To be a versioning opportunity we require:
+
+ - The multiplier applied by TERM is equal to the access size,
+ so that when STRIDE is 1, the accesses in successive loop
+ iterations are consecutive.
+
+ This is deliberately conservative. We could relax it to handle
+ other cases (such as those with gaps between iterations) if we
+ find any real testcases for which it's useful.
+
+ - the stride is applied in the same loop as STMT rather than
+ in an outer loop. Although versioning for strides applied in
+ outer loops could help in some cases -- such as enabling
+ more loop interchange -- the savings are much lower than for
+ inner loops.
+
+ - the stride is an SSA name that is invariant in STMT's loop,
+ since otherwise versioning isn't possible. */
+ unsigned HOST_WIDE_INT access_size = address.max_offset - address.min_offset;
+ if (term.multiplier == access_size
+ && address.loop == op_loop
+ && TREE_CODE (stride) == SSA_NAME
+ && expr_invariant_in_loop_p (address.loop, stride))
+ {
+ term.versioning_opportunity_p = true;
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, address.stmt, "%T == 1 is a versioning"
+ " opportunity\n", stride);
+ }
+}
+
+/* See whether address term TERM (which belongs to ADDRESS) is the result
+ of multiplying a varying SSA name by a loop-invariant SSA name.
+ Return true and update TERM if so.
+
+ This handles both cases that SCEV might handle, such as:
+
+ for (int i = 0; i < n; ++i)
+ res += a[i * stride];
+
+ and ones in which the term varies arbitrarily between iterations, such as:
+
+ for (int i = 0; i < n; ++i)
+ res += a[index[i] * stride]; */
+
+bool
+loop_versioning
+::find_per_loop_multiplication (address_info &address, address_term_info &term)
+{
+ gimple *mult = maybe_get_assign (term.expr);
+ if (!mult || gimple_assign_rhs_code (mult) != MULT_EXPR)
+ return false;
+
+ struct loop *mult_loop = loop_containing_stmt (mult);
+ if (!loop_outer (mult_loop))
+ return false;
+
+ tree op1 = strip_casts (gimple_assign_rhs1 (mult));
+ tree op2 = strip_casts (gimple_assign_rhs2 (mult));
+ if (TREE_CODE (op1) != SSA_NAME || TREE_CODE (op2) != SSA_NAME)
+ return false;
+
+ bool invariant1_p = expr_invariant_in_loop_p (mult_loop, op1);
+ bool invariant2_p = expr_invariant_in_loop_p (mult_loop, op2);
+ if (invariant1_p == invariant2_p)
+ return false;
+
+ /* Make sure that the loop invariant is OP2 rather than OP1. */
+ if (invariant1_p)
+ std::swap (op1, op2);
+
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, address.stmt, "address term %T = varying %T"
+ " * loop-invariant %T\n", term.expr, op1, op2);
+ analyze_stride (address, term, op2, mult_loop);
+ return true;
+}
+
+/* Try to use scalar evolutions to find an address stride for TERM,
+ which belongs to ADDRESS.
+
+ Here we are interested in any evolution information we can find,
+ not just evolutions wrt ADDRESS->LOOP. For example, if we find that
+ an outer loop obviously iterates over the inner dimension of an array,
+ that information can help us eliminate worthless versioning opportunities
+ in inner loops. */
+
+void
+loop_versioning::analyze_term_using_scevs (address_info &address,
+ address_term_info &term)
+{
+ gimple *setter = maybe_get_stmt (term.expr);
+ if (!setter)
+ return;
+
+ struct loop *wrt_loop = loop_containing_stmt (setter);
+ if (!loop_outer (wrt_loop))
+ return;
+
+ tree chrec = strip_casts (analyze_scalar_evolution (wrt_loop, term.expr));
+ if (TREE_CODE (chrec) == POLYNOMIAL_CHREC)
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, address.stmt,
+ "address term %T = %T\n", term.expr, chrec);
+
+ /* Peel casts and accumulate constant multiplications, up to the
+ limit allowed by M_MAXIMUM_SCALE. */
+ tree stride = strip_casts (CHREC_RIGHT (chrec));
+ while (TREE_CODE (stride) == MULT_EXPR
+ && multiply_term_by (term, TREE_OPERAND (stride, 1)))
+ stride = strip_casts (TREE_OPERAND (stride, 0));
+
+ gassign *assign;
+ while ((assign = maybe_get_assign (stride))
+ && gimple_assign_rhs_code (assign) == MULT_EXPR
+ && multiply_term_by (term, gimple_assign_rhs2 (assign)))
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, address.stmt,
+ "looking through %G", assign);
+ stride = strip_casts (gimple_assign_rhs1 (assign));
+ }
+
+ analyze_stride (address, term, stride, get_chrec_loop (chrec));
+ }
+}
+
+/* Try to identify loop strides in ADDRESS and try to choose realistic
+ versioning opportunities based on these strides.
+
+ The main difficulty here isn't finding strides that could be used
+ in a version check (that's pretty easy). The problem instead is to
+ avoid versioning for some stride S that is unlikely ever to be 1 at
+ runtime. Versioning for S == 1 on its own would lead to unnecessary
+ code bloat, while adding S == 1 to more realistic version conditions
+ would lose the optimisation opportunity offered by those other conditions.
+
+ For example, versioning for a stride of 1 in the Fortran code:
+
+ integer :: a(:,:)
+ a(1,:) = 1
+
+ is not usually a good idea, since the assignment is iterating over
+ an outer dimension and is relatively unlikely to have a stride of 1.
+ (It isn't impossible, since the inner dimension might be 1, or the
+ array might be transposed.) Similarly, in:
+
+ integer :: a(:,:), b(:,:)
+ b(:,1) = a(1,:)
+
+ b(:,1) is relatively likely to have a stride of 1 while a(1,:) isn't.
+ Versioning for when both strides are 1 would lose most of the benefit
+ of versioning for b's access.
+
+ The approach we take is as follows:
+
+ - Analyze each term to see whether it has an identifiable stride,
+ regardless of which loop applies the stride.
+
+ - Evaluate the likelihood that each such stride is for the innermost
+ dimension of an array, on the scale "likely", "don't know" or "unlikely".
+
+ - If there is a single "likely" innermost stride, and that stride is
+ applied in the loop that contains STMT, version the loop for when the
+ stride is 1. This deals with the cases in which we're fairly
+ confident of doing the right thing, such as the b(:,1) reference above.
+
+ - If there are no "likely" innermost strides, and the loop that contains
+ STMT uses a stride that we rated as "don't know", version for when
+ that stride is 1. This is principally used for C code such as:
+
+ for (int i = 0; i < n; ++i)
+ a[i * x] = ...;
+
+ and:
+
+ for (int j = 0; j < n; ++j)
+ for (int i = 0; i < n; ++i)
+ a[i * x + j * y] = ...;
+
+ where nothing in the way "x" and "y" are set gives a hint as to
+ whether "i" iterates over the innermost dimension of the array.
+ In these situations it seems reasonable to assume the the
+ programmer has nested the loops appropriately (although of course
+ there are examples like GEMM in which this assumption doesn't hold
+ for all accesses in the loop).
+
+ This case is also useful for the Fortran equivalent of the
+ above C code. */
+
+void
+loop_versioning::analyze_address_fragment (address_info &address)
+{
+ if (dump_enabled_p ())
+ {
+ dump_printf_loc (MSG_NOTE, address.stmt, "analyzing address fragment ");
+ dump_address_info (MSG_NOTE, address);
+ dump_printf (MSG_NOTE, "\n");
+ }
+
+ /* Analyze each component of the sum to see whether it involves an
+ apparent stride.
+
+ There is an overlap between the addresses that
+ find_per_loop_multiplication and analyze_term_using_scevs can handle,
+ but the former is much cheaper than SCEV analysis, so try it first. */
+ for (unsigned int i = 0; i < address.terms.length (); ++i)
+ if (!find_per_loop_multiplication (address, address.terms[i]))
+ analyze_term_using_scevs (address, address.terms[i]);
+
+ /* Check for strides that are likely to be for the innermost dimension.
+
+ 1. If there is a single likely inner stride, if it is an SSA name,
+ and if it is worth versioning the loop for when the SSA name
+ equals 1, record that we want to do so.
+
+ 2. Otherwise, if there any likely inner strides, bail out. This means
+ one of:
+
+ (a) There are multiple likely inner strides. This suggests we're
+ confused and be can't be confident of doing the right thing.
+
+ (b) There is a single likely inner stride and it is a constant
+ rather than an SSA name. This can mean either that the access
+ is a natural one without any variable strides, such as:
+
+ for (int i = 0; i < n; ++i)
+ a[i] += 1;
+
+ or that a variable stride is applied to an outer dimension,
+ such as:
+
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ a[j * stride][i] += 1;
+
+ (c) There is a single likely inner stride, and it is an SSA name,
+ but it isn't a worthwhile versioning opportunity. This usually
+ means that the variable stride is applied by an outer loop,
+ such as:
+
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ a[j][i * stride] += 1;
+
+ or (using an example with a more natural loop nesting):
+
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ a[i][j] += b[i * stride];
+
+ in cases where b[i * stride] cannot (yet) be hoisted for
+ aliasing reasons.
+
+ 3. If there are no likely inner strides, fall through to the next
+ set of checks.
+
+ Pointer equality is enough to check for uniqueness in (1), since we
+ only care about SSA names. */
+ tree chosen_stride = NULL_TREE;
+ tree version_stride = NULL_TREE;
+ for (unsigned int i = 0; i < address.terms.length (); ++i)
+ if (chosen_stride != address.terms[i].stride
+ && address.terms[i].inner_likelihood == INNER_LIKELY)
+ {
+ if (chosen_stride)
+ return;
+ chosen_stride = address.terms[i].stride;
+ if (address.terms[i].versioning_opportunity_p)
+ version_stride = chosen_stride;
+ }
+
+ /* If there are no likely inner strides, see if there is a single
+ versioning opportunity for a stride that was rated as INNER_DONT_KNOW.
+ See the comment above the function for the cases that this code
+ handles. */
+ if (!chosen_stride)
+ for (unsigned int i = 0; i < address.terms.length (); ++i)
+ if (version_stride != address.terms[i].stride
+ && address.terms[i].inner_likelihood == INNER_DONT_KNOW
+ && address.terms[i].versioning_opportunity_p)
+ {
+ if (version_stride)
+ return;
+ version_stride = address.terms[i].stride;
+ }
+
+ if (version_stride)
+ version_for_unity (address.stmt, version_stride);
+}
+
+/* Treat EXPR * MULTIPLIER + OFFSET as a fragment of an address that addresses
+ TYPE_SIZE bytes and record this address fragment for later processing.
+ STMT is the statement that contains the address. */
+
+void
+loop_versioning::record_address_fragment (gimple *stmt,
+ unsigned HOST_WIDE_INT type_size,
+ tree expr,
+ unsigned HOST_WIDE_INT multiplier,
+ HOST_WIDE_INT offset)
+{
+ /* We're only interested in computed values. */
+ if (TREE_CODE (expr) != SSA_NAME)
+ return;
+
+ /* Quick exit if no part of the address is calculated in STMT's loop,
+ since such addresses have no versioning opportunities. */
+ struct loop *loop = loop_containing_stmt (stmt);
+ if (expr_invariant_in_loop_p (loop, expr))
+ return;
+
+ /* Set up an address_info for EXPR * MULTIPLIER. */
+ address_info *address = XOBNEW (&m_obstack, address_info);
+ new (address) address_info;
+ address->stmt = stmt;
+ address->loop = loop;
+ address->base = NULL_TREE;
+ address->terms.quick_grow (1);
+ address->terms[0].expr = expr;
+ address->terms[0].multiplier = multiplier;
+ address->terms[0].stride = NULL_TREE;
+ address->terms[0].inner_likelihood = INNER_UNLIKELY;
+ address->terms[0].versioning_opportunity_p = false;
+ address->min_offset = offset;
+
+ /* Peel apart the expression into a sum of address_terms, where each
+ term is multiplied by a constant. Treat a + b and a - b the same,
+ since it doesn't matter for our purposes whether an address is
+ increasing or decreasing. Distribute (a + b) * constant into
+ a * constant + b * constant.
+
+ We don't care which loop each term belongs to, since we want to
+ examine as many candidate strides as possible when determining
+ which is likely to be for the innermost dimension. We therefore
+ don't limit the search to statements in STMT's loop. */
+ for (unsigned int i = 0; i < address->terms.length (); )
+ {
+ if (gassign *assign = maybe_get_assign (address->terms[i].expr))
+ {
+ tree_code code = gimple_assign_rhs_code (assign);
+ if (code == PLUS_EXPR
+ || code == POINTER_PLUS_EXPR
+ || code == MINUS_EXPR)
+ {
+ tree op1 = gimple_assign_rhs1 (assign);
+ tree op2 = gimple_assign_rhs2 (assign);
+ if (TREE_CODE (op2) == INTEGER_CST)
+ {
+ address->terms[i].expr = strip_casts (op1);
+ /* This is heuristic only, so don't worry about truncation
+ or overflow. */
+ address->min_offset += (TREE_INT_CST_LOW (op2)
+ * address->terms[i].multiplier);
+ continue;
+ }
+ else if (address->terms.length () < address_info::MAX_TERMS)
+ {
+ unsigned int j = address->terms.length ();
+ address->terms.quick_push (address->terms[i]);
+ address->terms[i].expr = strip_casts (op1);
+ address->terms[j].expr = strip_casts (op2);
+ continue;
+ }
+ }
+ if (code == MULT_EXPR)
+ {
+ tree op1 = gimple_assign_rhs1 (assign);
+ tree op2 = gimple_assign_rhs2 (assign);
+ if (multiply_term_by (address->terms[i], op2))
+ {
+ address->terms[i].expr = strip_casts (op1);
+ continue;
+ }
+ }
+ }
+ i += 1;
+ }
+
+ /* Peel off any symbolic pointer. */
+ if (TREE_CODE (address->terms[0].expr) != SSA_NAME
+ && address->terms[0].multiplier == 1)
+ {
+ if (address->terms.length () == 1)
+ {
+ obstack_free (&m_obstack, address);
+ return;
+ }
+ address->base = address->terms[0].expr;
+ address->terms.ordered_remove (0);
+ }
+
+ /* Require all remaining terms to be SSA names. (This could be false
+ for unfolded statements, but they aren't worth dealing with.) */
+ for (unsigned int i = 0; i < address->terms.length (); ++i)
+ if (TREE_CODE (address->terms[i].expr) != SSA_NAME)
+ {
+ obstack_free (&m_obstack, address);
+ return;
+ }
+
+ /* The loop above set MIN_OFFSET based on the first byte of the
+ referenced data. Calculate the end + 1. */
+ address->max_offset = address->min_offset + type_size;
+
+ /* Put the terms into a canonical order for the hash table lookup below. */
+ address->terms.qsort (compare_address_terms);
+
+ if (dump_enabled_p ())
+ {
+ dump_printf_loc (MSG_NOTE, stmt, "recording address fragment %T", expr);
+ if (multiplier != 1)
+ dump_printf (MSG_NOTE, " * %wd", multiplier);
+ dump_printf (MSG_NOTE, " = ");
+ dump_address_info (MSG_NOTE, *address);
+ dump_printf (MSG_NOTE, "\n");
+ }
+
+ /* Pool address information with the same terms (but potentially
+ different offsets). */
+ address_info **slot = m_address_table.find_slot (address, INSERT);
+ if (address_info *old_address = *slot)
+ {
+ /* We've already seen an address with the same terms. Extend the
+ offset range to account for this access. Doing this can paper
+ over gaps, such as in:
+
+ a[i * stride * 4] + a[i * stride * 4 + 3];
+
+ where nothing references "+ 1" or "+ 2". However, the vectorizer
+ handles such gapped accesses without problems, so it's not worth
+ trying to exclude them. */
+ if (old_address->min_offset > address->min_offset)
+ old_address->min_offset = address->min_offset;
+ if (old_address->max_offset < address->max_offset)
+ old_address->max_offset = address->max_offset;
+ obstack_free (&m_obstack, address);
+ }
+ else
+ {
+ /* This is the first term we've seen an address with these terms. */
+ *slot = address;
+ m_address_list.safe_push (address);
+ }
+}
+
+/* Analyze expression EXPR, which occurs in STMT. */
+
+void
+loop_versioning::analyze_expr (gimple *stmt, tree expr)
+{
+ unsigned HOST_WIDE_INT type_size;
+
+ while (handled_component_p (expr))
+ {
+ /* See whether we can use versioning to avoid a multiplication
+ in an array index. */
+ if (TREE_CODE (expr) == ARRAY_REF
+ && acceptable_type_p (TREE_TYPE (expr), &type_size))
+ record_address_fragment (stmt, type_size,
+ TREE_OPERAND (expr, 1), type_size, 0);
+ expr = TREE_OPERAND (expr, 0);
+ }
+
+ /* See whether we can use versioning to avoid a multiplication
+ in the pointer calculation of a MEM_REF. */
+ if (TREE_CODE (expr) == MEM_REF
+ && acceptable_type_p (TREE_TYPE (expr), &type_size))
+ record_address_fragment (stmt, type_size, TREE_OPERAND (expr, 0), 1,
+ /* This is heuristic only, so don't worry
+ about truncation or overflow. */
+ TREE_INT_CST_LOW (TREE_OPERAND (expr, 1)));
+
+ /* These would be easy to handle if they existed at this stage. */
+ gcc_checking_assert (TREE_CODE (expr) != TARGET_MEM_REF);
+}
+
+/* Analyze all the statements in BB looking for useful version checks.
+ Return true on success, false if something prevents the block from
+ being versioned. */
+
+bool
+loop_versioning::analyze_block (basic_block bb)
+{
+ struct loop *loop = bb->loop_father;
+ loop_info &li = get_loop_info (loop);
+ for (gimple_stmt_iterator gsi = gsi_start_bb (bb); !gsi_end_p (gsi);
+ gsi_next (&gsi))
+ {
+ gimple *stmt = gsi_stmt (gsi);
+ if (is_gimple_debug (stmt))
+ continue;
+
+ if (expensive_stmt_p (stmt))
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, stmt, "expensive statement"
+ " prevents versioning: %G", stmt);
+ return false;
+ }
+
+ /* Only look for direct versioning opportunities in inner loops
+ since the benefit tends to be much smaller for outer loops. */
+ if (!loop->inner)
+ {
+ unsigned int nops = gimple_num_ops (stmt);
+ for (unsigned int i = 0; i < nops; ++i)
+ if (tree op = gimple_op (stmt, i))
+ analyze_expr (stmt, op);
+ }
+
+ /* The point of the instruction limit is to prevent excessive
+ code growth, so this is a size-based estimate even though
+ the optimization is aimed at speed. */
+ li.num_insns += estimate_num_insns (stmt, &eni_size_weights);
+ }
+
+ return true;
+}
+
+/* Analyze all the blocks in the function, looking for useful version checks.
+ Return true if we found one. */
+
+bool
+loop_versioning::analyze_blocks ()
+{
+ AUTO_DUMP_SCOPE ("analyze_blocks",
+ dump_user_location_t::from_function_decl (m_fn->decl));
+
+ /* For now we don't try to version the whole function, although
+ versioning at that level could be useful in some cases. */
+ get_loop_info (get_loop (m_fn, 0)).rejected_p = true;
+
+ struct loop *loop;
+ FOR_EACH_LOOP (loop, LI_FROM_INNERMOST)
+ {
+ loop_info &linfo = get_loop_info (loop);
+
+ /* See whether an inner loop prevents versioning of this loop. */
+ for (struct loop *inner = loop->inner; inner; inner = inner->next)
+ if (get_loop_info (inner).rejected_p)
+ {
+ linfo.rejected_p = true;
+ break;
+ }
+
+ /* If versioning the loop is still a possibility, examine the
+ statements in the loop to look for versioning opportunities. */
+ if (!linfo.rejected_p)
+ {
+ void *start_point = obstack_alloc (&m_obstack, 0);
+
+ for (basic_block bb = linfo.block_list; bb;
+ bb = m_next_block_in_loop[bb->index])
+ if (!analyze_block (bb))
+ {
+ linfo.rejected_p = true;
+ break;
+ }
+
+ if (!linfo.rejected_p)
+ {
+ /* Process any queued address fragments, now that we have
+ complete grouping information. */
+ address_info *address;
+ unsigned int i;
+ FOR_EACH_VEC_ELT (m_address_list, i, address)
+ analyze_address_fragment (*address);
+ }
+
+ m_address_table.empty ();
+ m_address_list.truncate (0);
+ obstack_free (&m_obstack, start_point);
+ }
+ }
+
+ return m_num_conditions != 0;
+}
+
+/* Use the ranges in VRS to remove impossible versioning conditions from
+ LOOP. */
+
+void
+loop_versioning::prune_loop_conditions (struct loop *loop, vr_values *vrs)
+{
+ loop_info &li = get_loop_info (loop);
+
+ int to_remove = -1;
+ bitmap_iterator bi;
+ unsigned int i;
+ EXECUTE_IF_SET_IN_BITMAP (&li.unity_names, 0, i, bi)
+ {
+ tree name = ssa_name (i);
+ value_range *vr = vrs->get_value_range (name);
+ if (vr && !range_includes_p (vr, 1))
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, find_loop_location (loop),
+ "%T can never be 1 in this loop\n", name);
+
+ if (to_remove >= 0)
+ bitmap_clear_bit (&li.unity_names, to_remove);
+ to_remove = i;
+ m_num_conditions -= 1;
+ }
+ }
+ if (to_remove >= 0)
+ bitmap_clear_bit (&li.unity_names, to_remove);
+}
+
+/* Remove any scheduled loop version conditions that will never be true.
+ Return true if any remain. */
+
+bool
+loop_versioning::prune_conditions ()
+{
+ AUTO_DUMP_SCOPE ("prune_loop_conditions",
+ dump_user_location_t::from_function_decl (m_fn->decl));
+
+ calculate_dominance_info (CDI_DOMINATORS);
+ lv_dom_walker dom_walker (*this);
+ dom_walker.walk (ENTRY_BLOCK_PTR_FOR_FN (m_fn));
+ return m_num_conditions != 0;
+}
+
+/* Merge the version checks for INNER into immediately-enclosing loop
+ OUTER. */
+
+void
+loop_versioning::merge_loop_info (struct loop *outer, struct loop *inner)
+{
+ loop_info &inner_li = get_loop_info (inner);
+ loop_info &outer_li = get_loop_info (outer);
+
+ if (dump_enabled_p ())
+ {
+ bitmap_iterator bi;
+ unsigned int i;
+ EXECUTE_IF_SET_IN_BITMAP (&inner_li.unity_names, 0, i, bi)
+ if (!bitmap_bit_p (&outer_li.unity_names, i))
+ dump_printf_loc (MSG_NOTE, find_loop_location (inner),
+ "hoisting check that %T == 1 to outer loop\n",
+ ssa_name (i));
+ }
+
+ bitmap_ior_into (&outer_li.unity_names, &inner_li.unity_names);
+ if (loop_depth (outer_li.outermost) < loop_depth (inner_li.outermost))
+ outer_li.outermost = inner_li.outermost;
+}
+
+/* Add LOOP to the queue of loops to version. */
+
+void
+loop_versioning::add_loop_to_queue (struct loop *loop)
+{
+ loop_info &li = get_loop_info (loop);
+
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, find_loop_location (loop),
+ "queuing this loop for versioning\n");
+ m_loops_to_version.safe_push (loop);
+
+ /* Don't try to version superloops. */
+ li.rejected_p = true;
+}
+
+/* Decide whether the cost model would allow us to version LOOP,
+ either directly or as part of a parent loop, and return true if so.
+ This does not imply that the loop is actually worth versioning in its
+ own right, just that it would be valid to version it if something
+ benefited.
+
+ We have already made this decision for all inner loops of LOOP. */
+
+bool
+loop_versioning::decide_whether_loop_is_versionable (struct loop *loop)
+{
+ loop_info &li = get_loop_info (loop);
+
+ if (li.rejected_p)
+ return false;
+
+ /* Examine the decisions made for inner loops. */
+ for (struct loop *inner = loop->inner; inner; inner = inner->next)
+ {
+ loop_info &inner_li = get_loop_info (inner);
+ if (inner_li.rejected_p)
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, find_loop_location (loop),
+ "not versioning this loop because one of its"
+ " inner loops should not be versioned\n");
+ return false;
+ }
+
+ if (inner_li.worth_versioning_p ())
+ li.subloops_benefit_p = true;
+
+ /* Accumulate the number of instructions from subloops that are not
+ the innermost, or that don't benefit from versioning. Only the
+ instructions from innermost loops that benefit from versioning
+ should be weighed against loop-versioning-max-inner-insns;
+ everything else should be weighed against
+ loop-versioning-max-outer-insns. */
+ if (!inner_li.worth_versioning_p () || inner->inner)
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, find_loop_location (loop),
+ "counting %d instructions from this loop"
+ " against its parent loop\n", inner_li.num_insns);
+ li.num_insns += inner_li.num_insns;
+ }
+ }
+
+ /* Enforce the size limits. */
+ if (li.worth_versioning_p ())
+ {
+ unsigned int max_num_insns = max_insns_for_loop (loop);
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, find_loop_location (loop),
+ "this loop has %d instructions, against"
+ " a versioning limit of %d\n",
+ li.num_insns, max_num_insns);
+ if (li.num_insns > max_num_insns)
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_MISSED_OPTIMIZATION
+ | MSG_PRIORITY_USER_FACING,
+ find_loop_location (loop),
+ "this loop is too big to version");
+ return false;
+ }
+ }
+
+ /* Hoist all version checks from subloops to this loop. */
+ for (struct loop *subloop = loop->inner; subloop; subloop = subloop->next)
+ merge_loop_info (loop, subloop);
+
+ return true;
+}
+
+/* Decide which loops to version and add them to the versioning queue.
+ Return true if there are any loops to version. */
+
+bool
+loop_versioning::make_versioning_decisions ()
+{
+ AUTO_DUMP_SCOPE ("make_versioning_decisions",
+ dump_user_location_t::from_function_decl (m_fn->decl));
+
+ struct loop *loop;
+ FOR_EACH_LOOP (loop, LI_FROM_INNERMOST)
+ {
+ loop_info &linfo = get_loop_info (loop);
+ if (decide_whether_loop_is_versionable (loop))
+ {
+ /* Commit to versioning LOOP directly if we can't hoist the
+ version checks any further. */
+ if (linfo.worth_versioning_p ()
+ && (loop_depth (loop) == 1 || linfo.outermost == loop))
+ add_loop_to_queue (loop);
+ }
+ else
+ {
+ /* We can't version this loop, so individually version any
+ subloops that would benefit and haven't been versioned yet. */
+ linfo.rejected_p = true;
+ for (struct loop *subloop = loop->inner; subloop;
+ subloop = subloop->next)
+ if (get_loop_info (subloop).worth_versioning_p ())
+ add_loop_to_queue (subloop);
+ }
+ }
+
+ return !m_loops_to_version.is_empty ();
+}
+
+/* Attempt to implement loop versioning for LOOP, using the information
+ cached in the associated loop_info. Return true on success. */
+
+bool
+loop_versioning::version_loop (struct loop *loop)
+{
+ loop_info &li = get_loop_info (loop);
+
+ /* Build up a condition that selects the original loop instead of
+ the simplified loop. */
+ tree cond = boolean_false_node;
+ bitmap_iterator bi;
+ unsigned int i;
+ EXECUTE_IF_SET_IN_BITMAP (&li.unity_names, 0, i, bi)
+ {
+ tree name = ssa_name (i);
+ tree ne_one = fold_build2 (NE_EXPR, boolean_type_node, name,
+ build_one_cst (TREE_TYPE (name)));
+ cond = fold_build2 (TRUTH_OR_EXPR, boolean_type_node, cond, ne_one);
+ }
+
+ /* Convert the condition into a suitable gcond. */
+ gimple_seq stmts = NULL;
+ cond = force_gimple_operand_1 (cond, &stmts, is_gimple_condexpr, NULL_TREE);
+
+ /* Version the loop. */
+ initialize_original_copy_tables ();
+ basic_block cond_bb;
+ struct loop *else_loop
+ = loop_version (loop, cond, &cond_bb,
+ profile_probability::unlikely (),
+ profile_probability::likely (),
+ profile_probability::unlikely (),
+ profile_probability::likely (), true);
+ free_original_copy_tables ();
+ if (!else_loop)
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_MISSED_OPTIMIZATION, find_loop_location (loop),
+ "tried but failed to version this loop for when"
+ " certain strides are 1\n");
+ return false;
+ }
+
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_OPTIMIZED_LOCATIONS, find_loop_location (loop),
+ "versioned this loop for when certain strides are 1\n");
+
+ /* Insert the statements that feed COND. */
+ if (stmts)
+ {
+ gimple_stmt_iterator gsi = gsi_last_bb (cond_bb);
+ gsi_insert_seq_before (&gsi, stmts, GSI_SAME_STMT);
+ }
+
+ /* Simplify the new loop, which is used when COND is false. */
+ name_prop (li).substitute_and_fold (else_loop->header);
+ return true;
+}
+
+/* Attempt to version all loops in the versioning queue. Return true
+ if we succeeded for at least one loop. */
+
+bool
+loop_versioning::implement_versioning_decisions ()
+{
+ /* No AUTO_DUMP_SCOPE here since all messages are top-level and
+ user-facing at this point. */
+
+ bool any_succeeded_p = false;
+
+ struct loop *loop;
+ unsigned int i;
+ FOR_EACH_VEC_ELT (m_loops_to_version, i, loop)
+ if (version_loop (loop))
+ any_succeeded_p = true;
+
+ return any_succeeded_p;
+}
+
+/* Run the pass and return a set of TODO_* flags. */
+
+unsigned int
+loop_versioning::run ()
+{
+ gcc_assert (scev_initialized_p ());
+
+ if (!analyze_blocks ()
+ || !prune_conditions ()
+ || !make_versioning_decisions ()
+ || !implement_versioning_decisions ())
+ return 0;
+
+ return TODO_update_ssa;
+}
+
+/* Loop versioning pass. */
+
+const pass_data pass_data_loop_versioning =
+{
+ GIMPLE_PASS, /* type */
+ "lversion", /* name */
+ OPTGROUP_LOOP, /* optinfo_flags */
+ TV_LOOP_VERSIONING, /* tv_id */
+ PROP_cfg, /* properties_required */
+ 0, /* properties_provided */
+ 0, /* properties_destroyed */
+ 0, /* todo_flags_start */
+ 0, /* todo_flags_finish */
+};
+
+class pass_loop_versioning : public gimple_opt_pass
+{
+public:
+ pass_loop_versioning (gcc::context *ctxt)
+ : gimple_opt_pass (pass_data_loop_versioning, ctxt)
+ {}
+
+ /* opt_pass methods: */
+ virtual bool gate (function *) { return flag_version_loops_for_strides; }
+ virtual unsigned int execute (function *);
+};
+
+unsigned int
+pass_loop_versioning::execute (function *fn)
+{
+ if (number_of_loops (fn) <= 1)
+ return 0;
+
+ return loop_versioning (fn).run ();
+}
+
+} // anon namespace
+
+gimple_opt_pass *
+make_pass_loop_versioning (gcc::context *ctxt)
+{
+ return new pass_loop_versioning (ctxt);
+}
Index: gcc/testsuite/gcc.dg/loop-versioning-1.c
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gcc.dg/loop-versioning-1.c 2018-12-06 12:33:59.216098398 +0000
@@ -0,0 +1,92 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* The simplest IV case. */
+
+void
+f1 (double *x, int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ x[stepx * i] = 100;
+}
+
+void
+f2 (double *x, int stepx, int limit)
+{
+ for (int i = 0; i < limit; i += stepx)
+ x[i] = 100;
+}
+
+void
+f3 (double *x, int stepx, int limit)
+{
+ for (double *y = x; y < x + limit; y += stepx)
+ *y = 100;
+}
+
+void
+f4 (double *x, int stepx, unsigned int n)
+{
+ for (unsigned int i = 0; i < n; ++i)
+ x[stepx * i] = 100;
+}
+
+void
+f5 (double *x, int stepx, unsigned int limit)
+{
+ for (unsigned int i = 0; i < limit; i += stepx)
+ x[i] = 100;
+}
+
+void
+f6 (double *x, int stepx, unsigned int limit)
+{
+ for (double *y = x; y < x + limit; y += stepx)
+ *y = 100;
+}
+
+double x[10000];
+
+void
+g1 (int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ x[stepx * i] = 100;
+}
+
+void
+g2 (int stepx, int limit)
+{
+ for (int i = 0; i < limit; i += stepx)
+ x[i] = 100;
+}
+
+void
+g3 (int stepx, int limit)
+{
+ for (double *y = x; y < x + limit; y += stepx)
+ *y = 100;
+}
+
+void
+g4 (int stepx, unsigned int n)
+{
+ for (unsigned int i = 0; i < n; ++i)
+ x[stepx * i] = 100;
+}
+
+void
+g5 (int stepx, unsigned int limit)
+{
+ for (unsigned int i = 0; i < limit; i += stepx)
+ x[i] = 100;
+}
+
+void
+g6 (int stepx, unsigned int limit)
+{
+ for (double *y = x; y < x + limit; y += stepx)
+ *y = 100;
+}
+
+/* { dg-final { scan-tree-dump-times {want to version containing loop} 12 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {versioned this loop} 12 "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-10.c
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gcc.dg/loop-versioning-10.c 2018-12-06 12:33:59.216098398 +0000
@@ -0,0 +1,52 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Test that we can version a gather-like operation in which a variable
+ stride is applied to the index. */
+
+int
+f1 (int *x, int *index, int step, int n)
+{
+ int res = 0;
+ for (int i = 0; i < n; ++i)
+ res += x[index[i] * step];
+ return res;
+}
+
+int
+f2 (int *x, int *index, int step, int n)
+{
+ int res = 0;
+ for (int i = 0; i < n; ++i)
+ {
+ int *ptr = x + index[i] * step;
+ res += *ptr;
+ }
+ return res;
+}
+
+int x[1000];
+
+int
+g1 (int *index, int step, int n)
+{
+ int res = 0;
+ for (int i = 0; i < n; ++i)
+ res += x[index[i] * step];
+ return res;
+}
+
+int
+g2 (int *index, int step, int n)
+{
+ int res = 0;
+ for (int i = 0; i < n; ++i)
+ {
+ int *ptr = x + index[i] * step;
+ res += *ptr;
+ }
+ return res;
+}
+
+/* { dg-final { scan-tree-dump-times {address term [^\n]* \* loop-invariant} 4 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {want to version containing loop} 4 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {versioned this loop} 4 "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-11.c
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gcc.dg/loop-versioning-11.c 2018-12-06 12:33:59.216098398 +0000
@@ -0,0 +1,29 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Test that we don't try to version for something that is never 1. */
+
+void
+f1 (double *x, int stepx, int n)
+{
+ if (stepx == 1)
+ for (int i = 0; i < n; ++i)
+ x[i] = 100;
+ else
+ for (int i = 0; i < n; ++i)
+ x[stepx * i] = 100;
+}
+
+void
+f2 (double *x, int stepx, int n)
+{
+ if (stepx <= 1)
+ for (int i = 0; i < n; ++i)
+ x[i] = 100;
+ else
+ for (int i = 0; i < n; ++i)
+ x[stepx * i] = 100;
+}
+
+/* { dg-final { scan-tree-dump-times {want to version containing loop} 2 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {can never be 1} 2 "lversion" } } */
+/* { dg-final { scan-tree-dump-not {versioned} "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-2.c
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gcc.dg/loop-versioning-2.c 2018-12-06 12:33:59.216098398 +0000
@@ -0,0 +1,73 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Versioning for step == 1 in these loops would allow loop interchange,
+ but otherwise isn't worthwhile. At the moment we decide not to version. */
+
+void
+f1 (double x[][100], int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j * step][i] = 100;
+}
+
+void
+f2 (double x[][100], int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j][i * step] = 100;
+}
+
+void
+f3 (double x[][100], int step, int limit)
+{
+ for (int i = 0; i < 100; ++i)
+ for (int j = 0; j < limit; j += step)
+ x[j][i] = 100;
+}
+
+void
+f4 (double x[][100], int step, int limit)
+{
+ for (int i = 0; i < limit; i += step)
+ for (int j = 0; j < 100; ++j)
+ x[j][i] = 100;
+}
+
+double x[100][100];
+
+void
+g1 (int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j * step][i] = 100;
+}
+
+void
+g2 (int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j][i * step] = 100;
+}
+
+void
+g3 (int step, int limit)
+{
+ for (int i = 0; i < 100; ++i)
+ for (int j = 0; j < limit; j += step)
+ x[j][i] = 100;
+}
+
+void
+g4 (int step, int limit)
+{
+ for (int i = 0; i < limit; i += step)
+ for (int j = 0; j < 100; ++j)
+ x[j][i] = 100;
+}
+
+/* { dg-final { scan-tree-dump-not {want to version} "lversion" } } */
+/* { dg-final { scan-tree-dump-not {versioned} "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-3.c
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gcc.dg/loop-versioning-3.c 2018-12-06 12:33:59.216098398 +0000
@@ -0,0 +1,24 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Versioning these loops for when both steps are 1 allows loop
+ interchange, but otherwise isn't worthwhile. At the moment we decide
+ not to version. */
+
+void
+f1 (double x[][100], int step1, int step2, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j * step1][i * step2] = 100;
+}
+
+void
+f2 (double x[][100], int step1, int step2, int limit)
+{
+ for (int i = 0; i < limit; i += step1)
+ for (int j = 0; j < limit; j += step2)
+ x[j][i] = 100;
+}
+
+/* { dg-final { scan-tree-dump-not {want to version} "lversion" } } */
+/* { dg-final { scan-tree-dump-not {versioned} "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-4.c
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gcc.dg/loop-versioning-4.c 2018-12-06 12:33:59.216098398 +0000
@@ -0,0 +1,39 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* These shouldn't be versioned; it's extremely likely that the code
+ is emulating two-dimensional arrays. */
+
+void
+f1 (double *x, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[i * step + j] = 100;
+}
+
+void
+f2 (double *x, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j * step + i] = 100;
+}
+
+void
+f3 (double *x, int *offsets, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[i * step + j + offsets[i]] = 100;
+}
+
+void
+f4 (double *x, int *offsets, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j * step + i + offsets[i]] = 100;
+}
+
+/* { dg-final { scan-tree-dump-not {want to version} "lversion" } } */
+/* { dg-final { scan-tree-dump-not {versioned} "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-5.c
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gcc.dg/loop-versioning-5.c 2018-12-06 12:33:59.216098398 +0000
@@ -0,0 +1,17 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* There's no information about whether STEP1 or STEP2 is innermost,
+ so we should assume the code is sensible and version for the inner
+ evolution, i.e. when STEP2 is 1. */
+
+void
+f1 (double *x, int step1, int step2, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[i * step1 + j * step2] = 100;
+}
+
+/* { dg-final { scan-tree-dump-times {want to version containing loop for when step2} 1 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {want to version containing loop} 1 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {versioned this loop} 1 "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-6.c
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gcc.dg/loop-versioning-6.c 2018-12-06 12:33:59.216098398 +0000
@@ -0,0 +1,31 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* The read from y in f1 will be hoisted to the outer loop. In general
+ it's not worth versioning outer loops when the inner loops don't also
+ benefit.
+
+ This test is meant to be a slight counterexample, since versioning
+ does lead to cheaper outer-loop vectorization. However, the benefit
+ isn't enough to justify the cost. */
+
+void
+f1 (double *restrict x, double *restrict y, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[i + j] = y[i * step];
+}
+
+/* A similar example in which the read can't be hoisted, but could
+ for example be handled by vectorizer alias checks. */
+
+void
+f2 (double *x, double *y, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[i + j] = y[i * step];
+}
+
+/* { dg-final { scan-tree-dump-not {want to version} "lversion" } } */
+/* { dg-final { scan-tree-dump-not {versioned} "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-7.c
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gcc.dg/loop-versioning-7.c 2018-12-06 12:33:59.216098398 +0000
@@ -0,0 +1,32 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Check that versioning can handle arrays of structures. */
+
+struct foo {
+ int a, b, c;
+};
+
+void
+f1 (struct foo *x, int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[stepx * i].a = 1;
+ x[stepx * i].b = 2;
+ x[stepx * i].c = 3;
+ }
+}
+
+void
+f2 (struct foo *x, int stepx, int limit)
+{
+ for (int i = 0; i < limit; i += stepx)
+ {
+ x[i].a = 1;
+ x[i].b = 2;
+ x[i].c = 3;
+ }
+}
+
+/* { dg-final { scan-tree-dump-times {want to version containing loop} 2 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {versioned this loop} 2 "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-8.c
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gcc.dg/loop-versioning-8.c 2018-12-06 12:33:59.216098398 +0000
@@ -0,0 +1,43 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Versioning for step == 1 in these loops would allow loop interchange,
+ but otherwise isn't worthwhile. At the moment we decide not to version. */
+
+struct foo {
+ int a[100];
+};
+
+void
+f1 (struct foo *x, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j * step].a[i] = 100;
+}
+
+void
+f2 (struct foo *x, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j].a[i * step] = 100;
+}
+
+void
+f3 (struct foo *x, int step, int limit)
+{
+ for (int i = 0; i < 100; ++i)
+ for (int j = 0; j < limit; j += step)
+ x[j].a[i] = 100;
+}
+
+void
+f4 (struct foo *x, int step, int limit)
+{
+ for (int i = 0; i < limit; i += step)
+ for (int j = 0; j < 100; ++j)
+ x[j].a[i] = 100;
+}
+
+/* { dg-final { scan-tree-dump-not {want to version} "lversion" } } */
+/* { dg-final { scan-tree-dump-not {versioned} "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-9.c
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gcc.dg/loop-versioning-9.c 2018-12-06 12:33:59.220098363 +0000
@@ -0,0 +1,48 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Check that versioning can handle small groups of accesses. */
+
+void
+f1 (int *x, int *y, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ x[i] = y[i * step * 2] + y[i * step * 2 + 1];
+}
+
+void
+f2 (int *x, int *y, __INTPTR_TYPE__ step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ x[i] = y[i * step * 2] + y[i * step * 2 + 1];
+}
+
+void
+f3 (int *x, int *y, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ x[i] = y[i * step * 3] + y[i * step * 3 + 2];
+}
+
+void
+f4 (int *x, int *y, __INTPTR_TYPE__ step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ x[i] = y[i * step * 3] + y[i * step * 3 + 2];
+}
+
+void
+f5 (int *x, int *y, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ x[i] = y[i * step * 4] + y[i * step * 4 + 3];
+}
+
+void
+f6 (int *x, int *y, __INTPTR_TYPE__ step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ x[i] = y[i * step * 4] + y[i * step * 4 + 3];
+}
+
+/* { dg-final { scan-tree-dump-times {want to version containing loop} 6 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {versioned this loop} 6 "lversion" } } */
Index: gcc/testsuite/gfortran.dg/loop_versioning_1.f90
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gfortran.dg/loop_versioning_1.f90 2018-12-06 12:33:59.220098363 +0000
@@ -0,0 +1,28 @@
+! { dg-options "-O3 -fdump-tree-lversion-details" }
+
+! The simplest IV case.
+
+subroutine f1(x)
+ real :: x(:)
+ x(:) = 100
+end subroutine f1
+
+subroutine f2(x, n, step)
+ integer :: n, step
+ real :: x(n * step)
+ do i = 1, n
+ x(i * step) = 100
+ end do
+end subroutine f2
+
+subroutine f3(x, limit, step)
+ integer :: limit, step
+ real :: x(limit)
+ do i = 1, limit, step
+ x(i) = 100
+ end do
+end subroutine f3
+
+! { dg-final { scan-tree-dump-times {likely to be the innermost dimension} 1 "lversion" } }
+! { dg-final { scan-tree-dump-times {want to version containing loop} 3 "lversion" } }
+! { dg-final { scan-tree-dump-times {versioned this loop} 3 "lversion" } }
Index: gcc/testsuite/gfortran.dg/loop_versioning_2.f90
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gfortran.dg/loop_versioning_2.f90 2018-12-06 12:33:59.220098363 +0000
@@ -0,0 +1,39 @@
+! { dg-options "-O3 -fdump-tree-lversion-details -fno-frontend-loop-interchange" }
+
+! We could version the loop for when the first dimension has a stride
+! of 1, but at present there's no real benefit. The gimple loop
+! interchange pass couldn't handle the versioned loop, and interchange
+! is instead done by the frontend (but disabled by the options above).
+
+subroutine f1(x)
+ real :: x(:, :)
+ do i = lbound(x, 1), ubound(x, 1)
+ do j = lbound(x, 2), ubound(x, 2)
+ x(i, j) = 100
+ end do
+ end do
+end subroutine f1
+
+subroutine f2(x, n, step)
+ integer :: n, step
+ real :: x(100, 100)
+ do i = 1, n
+ do j = 1, n
+ x(i * step, j) = 100
+ end do
+ end do
+end subroutine f2
+
+subroutine f3(x, n, step)
+ integer :: n, step
+ real :: x(n * step, n)
+ do i = 1, n
+ do j = 1, n
+ x(i * step, j) = 100
+ end do
+ end do
+end subroutine f3
+
+! { dg-final { scan-tree-dump-times {likely to be the innermost dimension} 1 "lversion" } }
+! { dg-final { scan-tree-dump-not {want to version} "lversion" } }
+! { dg-final { scan-tree-dump-not {versioned} "lversion" } }
Index: gcc/testsuite/gfortran.dg/loop_versioning_3.f90
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gfortran.dg/loop_versioning_3.f90 2018-12-06 12:33:59.220098363 +0000
@@ -0,0 +1,30 @@
+! { dg-options "-O3 -fdump-tree-lversion-details -fno-frontend-loop-interchange" }
+
+! Test a case in which the outer loop iterates over the inner dimension.
+! The options above prevent the frontend from interchanging the loops.
+
+subroutine f1(x, limit, step, n)
+ integer :: limit, step, n
+ real :: x(limit, n)
+ do i = 1, limit, step
+ do j = 1, n
+ x(i, j) = 100
+ end do
+ end do
+end subroutine f1
+
+subroutine f2(x, n, limit, step)
+ integer :: n, limit, step
+ real :: x(limit, n)
+ do i = 1, n
+ do j = 1, limit, step
+ x(j, i) = 100
+ end do
+ end do
+end subroutine f2
+
+! FIXME: The frontend doesn't give us enough information to tell which loop
+! is iterating over the innermost dimension, so we optimistically
+! assume the inner one is.
+! { dg-final { scan-tree-dump-not {want to version} "lversion" { xfail *-*-* } } }
+! { dg-final { scan-tree-dump-not {versioned} "lversion" { xfail *-*-* } } }
Index: gcc/testsuite/gfortran.dg/loop_versioning_4.f90
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gfortran.dg/loop_versioning_4.f90 2018-12-06 12:33:59.220098363 +0000
@@ -0,0 +1,52 @@
+! { dg-options "-O3 -fdump-tree-lversion-details -fno-frontend-loop-interchange" }
+
+! Test cases in which versioning is useful for a two-dimensional array.
+
+subroutine f1(x)
+ real :: x(:, :)
+ x(:, :) = 100
+end subroutine f1
+
+subroutine f2(x)
+ real :: x(:, :)
+ do i = lbound(x, 1), ubound(x, 1)
+ do j = lbound(x, 2), ubound(x, 2)
+ x(j, i) = 100
+ end do
+ end do
+end subroutine f2
+
+subroutine f3(x, n, step)
+ integer :: n, step
+ real :: x(100, 100)
+ do i = 1, n
+ do j = 1, n
+ x(j * step, i) = 100
+ end do
+ end do
+end subroutine f3
+
+subroutine f4(x, n, step)
+ integer :: n, step
+ real :: x(n * step, n)
+ do i = 1, n
+ do j = 1, n
+ x(j * step, i) = 100
+ end do
+ end do
+end subroutine f4
+
+subroutine f5(x, n, limit, step)
+ integer :: n, limit, step
+ real :: x(limit, n)
+ do i = 1, n
+ do j = 1, limit, step
+ x(j, i) = 100
+ end do
+ end do
+end subroutine f5
+
+! { dg-final { scan-tree-dump-times {likely to be the innermost dimension} 2 "lversion" } }
+! { dg-final { scan-tree-dump-times {want to version containing loop} 5 "lversion" } }
+! { dg-final { scan-tree-dump-times {hoisting check} 5 "lversion" } }
+! { dg-final { scan-tree-dump-times {versioned this loop} 5 "lversion" } }
Index: gcc/testsuite/gfortran.dg/loop_versioning_5.f90
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gfortran.dg/loop_versioning_5.f90 2018-12-06 12:33:59.220098363 +0000
@@ -0,0 +1,57 @@
+! { dg-options "-O3 -fdump-tree-lversion-details -fno-frontend-loop-interchange" }
+
+! Make sure that in a "badly nested" loop, we don't treat the inner loop
+! as iterating over the inner dimension with a variable stride.
+
+subroutine f1(x, n)
+ integer :: n
+ real :: x(100, 100)
+ do i = 1, n
+ do j = 1, n
+ x(i, j) = 100
+ end do
+ end do
+end subroutine f1
+
+subroutine f2(x, n, step)
+ integer :: n, step
+ real :: x(100, 100)
+ do i = 1, n
+ do j = 1, n
+ x(i, j * step) = 100
+ end do
+ end do
+end subroutine f2
+
+subroutine f3(x, n)
+ integer :: n
+ real :: x(n, n)
+ do i = 1, n
+ do j = 1, n
+ x(i, j) = 100
+ end do
+ end do
+end subroutine f3
+
+subroutine f4(x, n, step)
+ integer :: n, step
+ real :: x(n, n * step)
+ do i = 1, n
+ do j = 1, n
+ x(i, j * step) = 100
+ end do
+ end do
+end subroutine f4
+
+subroutine f5(x, n, limit, step)
+ integer :: n, limit, step
+ real :: x(n, limit)
+ do i = 1, n
+ do j = 1, limit, step
+ x(i, j) = 100
+ end do
+ end do
+end subroutine f5
+
+! { dg-final { scan-tree-dump-not {want to version} "lversion" } }
+! { dg-final { scan-tree-dump-not {versioned} "lversion" } }
Index: gcc/testsuite/gfortran.dg/loop_versioning_6.f90
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gfortran.dg/loop_versioning_6.f90 2018-12-06 12:33:59.220098363 +0000
@@ -0,0 +1,93 @@
+! { dg-options "-O3 -fdump-tree-lversion-details" }
+
+! Check that versioning can handle small groups of accesses.
+
+subroutine f1(x)
+ real :: x(:)
+ do i = lbound(x, 1), ubound(x, 1) / 2
+ x(i * 2) = 100
+ x(i * 2 + 1) = 101
+ end do
+end subroutine f1
+
+subroutine f2(x, n, step)
+ integer :: n, step
+ real :: x(n * step * 2)
+ do i = 1, n
+ x(i * step * 2) = 100
+ x(i * step * 2 + 1) = 101
+ end do
+end subroutine f2
+
+subroutine f3(x, limit, step)
+ integer :: limit, step
+ real :: x(limit * 2)
+ do i = 1, limit, step
+ x(i * 2) = 100
+ x(i * 2 + 1) = 101
+ end do
+end subroutine f3
+
+subroutine f4(x)
+ real :: x(:)
+ do i = lbound(x, 1), ubound(x, 1) / 3
+ x(i * 3) = 100
+ x(i * 3 + 1) = 101
+ x(i * 3 + 2) = 102
+ end do
+end subroutine f4
+
+subroutine f5(x, n, step)
+ integer :: n, step
+ real :: x(n * step * 3)
+ do i = 1, n
+ x(i * step * 3) = 100
+ x(i * step * 3 + 1) = 101
+ x(i * step * 3 + 2) = 102
+ end do
+end subroutine f5
+
+subroutine f6(x, limit, step)
+ integer :: limit, step
+ real :: x(limit * 3)
+ do i = 1, limit, step
+ x(i * 3) = 100
+ x(i * 3 + 1) = 101
+ x(i * 3 + 2) = 102
+ end do
+end subroutine f6
+
+subroutine f7(x)
+ real :: x(:)
+ do i = lbound(x, 1), ubound(x, 1) / 4
+ x(i * 4) = 100
+ x(i * 4 + 1) = 101
+ x(i * 4 + 2) = 102
+ x(i * 4 + 3) = 103
+ end do
+end subroutine f7
+
+subroutine f8(x, n, step)
+ integer :: n, step
+ real :: x(n * step * 4)
+ do i = 1, n
+ x(i * step * 4) = 100
+ x(i * step * 4 + 1) = 101
+ x(i * step * 4 + 2) = 102
+ x(i * step * 4 + 3) = 103
+ end do
+end subroutine f8
+
+subroutine f9(x, limit, step)
+ integer :: limit, step
+ real :: x(limit * 4)
+ do i = 1, limit, step
+ x(i * 4) = 100
+ x(i * 4 + 1) = 101
+ x(i * 4 + 2) = 102
+ x(i * 4 + 3) = 103
+ end do
+end subroutine f9
+
+! { dg-final { scan-tree-dump-times {want to version containing loop} 9 "lversion" } }
+! { dg-final { scan-tree-dump-times {versioned this loop} 9 "lversion" } }
Index: gcc/testsuite/gfortran.dg/loop_versioning_7.f90
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gfortran.dg/loop_versioning_7.f90 2018-12-06 12:33:59.220098363 +0000
@@ -0,0 +1,67 @@
+! { dg-options "-O3 -fdump-tree-lversion-details" }
+
+! Check that versioning can handle small groups of accesses, with the
+! group being a separate array dimension.
+
+subroutine f1(x, n, step)
+ integer :: n, step
+ real :: x(2, n * step)
+ do i = 1, n
+ x(1, i * step) = 100
+ x(2, i * step) = 101
+ end do
+end subroutine f1
+
+subroutine f2(x, limit, step)
+ integer :: limit, step
+ real :: x(2, limit)
+ do i = 1, limit, step
+ x(1, i) = 100
+ x(2, i) = 101
+ end do
+end subroutine f2
+
+subroutine f3(x, n, step)
+ integer :: n, step
+ real :: x(3, n * step)
+ do i = 1, n
+ x(1, i * step) = 100
+ x(2, i * step) = 101
+ x(3, i * step) = 102
+ end do
+end subroutine f3
+
+subroutine f4(x, limit, step)
+ integer :: limit, step
+ real :: x(3, limit)
+ do i = 1, limit, step
+ x(1, i) = 100
+ x(2, i) = 101
+ x(3, i) = 102
+ end do
+end subroutine f4
+
+subroutine f5(x, n, step)
+ integer :: n, step
+ real :: x(4, n * step)
+ do i = 1, n
+ x(1, i * step) = 100
+ x(2, i * step) = 101
+ x(3, i * step) = 102
+ x(4, i * step) = 103
+ end do
+end subroutine f5
+
+subroutine f6(x, limit, step)
+ integer :: limit, step
+ real :: x(4, limit)
+ do i = 1, limit, step
+ x(1, i) = 100
+ x(2, i) = 101
+ x(3, i) = 102
+ x(4, i) = 103
+ end do
+end subroutine f6
+
+! { dg-final { scan-tree-dump-times {want to version containing loop} 6 "lversion" } }
+! { dg-final { scan-tree-dump-times {versioned this loop} 6 "lversion" } }
Index: gcc/testsuite/gfortran.dg/loop_versioning_8.f90
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gfortran.dg/loop_versioning_8.f90 2018-12-06 12:33:59.220098363 +0000
@@ -0,0 +1,13 @@
+! { dg-options "-O3 -fdump-tree-lversion-details" }
+
+! Check that versioning is applied to a gather-like reduction operation.
+
+function f(x, index, n)
+ integer :: n
+ real :: x(:)
+ integer :: index(n)
+ f = sum(x(index(:)))
+end function f
+
+! { dg-final { scan-tree-dump-times {want to version containing loop} 1 "lversion" } }
+! { dg-final { scan-tree-dump-times {versioned this loop} 1 "lversion" } }
Index: gcc/testsuite/gcc.dg/loop-versioning-12.c
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gcc.dg/loop-versioning-12.c 2018-12-06 12:33:59.216098398 +0000
@@ -0,0 +1,149 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Test that we don't try to version for a step of 1 when that would
+ cause the iterations to overlap. */
+
+void
+f1 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx] = 100;
+ x[i * stepx + 1] = 99;
+ }
+}
+
+void
+f2 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx)
+ {
+ x[i] = 100;
+ x[i + 1] = 99;
+ }
+}
+
+void
+f3 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx - 16] = 100;
+ x[i * stepx - 15] = 99;
+ }
+}
+
+void
+f4 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx)
+ {
+ x[i - 16] = 100;
+ x[i - 15] = 99;
+ }
+}
+
+void
+f5 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx - 16] = 100;
+ x[i * stepx + 15] = 99;
+ }
+}
+
+void
+f6 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx)
+ {
+ x[i - 16] = 100;
+ x[i + 15] = 99;
+ }
+}
+
+void
+f7 (unsigned short *x, int stepx, int n)
+{
+ for (unsigned short *y = x; y < x + n; y += stepx)
+ {
+ y[0] = 100;
+ y[1] = 99;
+ }
+}
+
+unsigned short x[1000];
+
+void
+g1 (int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx] = 100;
+ x[i * stepx + 1] = 99;
+ }
+}
+
+void
+g2 (int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx)
+ {
+ x[i] = 100;
+ x[i + 1] = 99;
+ }
+}
+
+void
+g3 (int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx - 16] = 100;
+ x[i * stepx - 15] = 99;
+ }
+}
+
+void
+g4 (int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx)
+ {
+ x[i - 16] = 100;
+ x[i - 15] = 99;
+ }
+}
+
+void
+g5 (int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx - 16] = 100;
+ x[i * stepx + 15] = 99;
+ }
+}
+
+void
+g6 (int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx)
+ {
+ x[i - 16] = 100;
+ x[i + 15] = 99;
+ }
+}
+
+void
+g7 (int stepx, int n)
+{
+ for (unsigned short *y = x; y < x + n; y += stepx)
+ {
+ y[0] = 100;
+ y[1] = 99;
+ }
+}
+
+/* { dg-final { scan-tree-dump-not {want to version} "lversion" } } */
+/* { dg-final { scan-tree-dump-not {versioned} "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-13.c
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gcc.dg/loop-versioning-13.c 2018-12-06 12:33:59.216098398 +0000
@@ -0,0 +1,109 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Test that we do version for a step of 1 when that would lead the
+ iterations to access consecutive groups. */
+
+void
+f1 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx * 2] = 100;
+ x[i * stepx * 2 + 1] = 99;
+ }
+}
+
+void
+f2 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx * 2)
+ {
+ x[i] = 100;
+ x[i + 1] = 99;
+ }
+}
+
+void
+f3 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx * 2 - 16] = 100;
+ x[i * stepx * 2 - 15] = 99;
+ }
+}
+
+void
+f4 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx * 2)
+ {
+ x[i - 16] = 100;
+ x[i - 15] = 99;
+ }
+}
+
+void
+f5 (unsigned short *x, int stepx, int n)
+{
+ for (unsigned short *y = x; y < x + n; y += stepx * 2)
+ {
+ y[0] = 100;
+ y[1] = 99;
+ }
+}
+
+unsigned short x[1000];
+
+void
+g1 (int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx * 2] = 100;
+ x[i * stepx * 2 + 1] = 99;
+ }
+}
+
+void
+g2 (int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx * 2)
+ {
+ x[i] = 100;
+ x[i + 1] = 99;
+ }
+}
+
+void
+g3 (int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx * 2 - 16] = 100;
+ x[i * stepx * 2 - 15] = 99;
+ }
+}
+
+void
+g4 (int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx * 2)
+ {
+ x[i - 16] = 100;
+ x[i - 15] = 99;
+ }
+}
+
+void
+g5 (int stepx, int n)
+{
+ for (unsigned short *y = x; y < x + n; y += stepx * 2)
+ {
+ y[0] = 100;
+ y[1] = 99;
+ }
+}
+
+/* { dg-final { scan-tree-dump-times {want to version containing loop} 10 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {versioned this loop} 10 "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-14.c
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gcc.dg/loop-versioning-14.c 2018-12-06 12:33:59.216098398 +0000
@@ -0,0 +1,149 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Test that we don't try to version for a step of 1 when that would
+ cause the iterations to leave a gap between accesses. */
+
+void
+f1 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx * 4] = 100;
+ x[i * stepx * 4 + 1] = 99;
+ }
+}
+
+void
+f2 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx * 4)
+ {
+ x[i] = 100;
+ x[i + 1] = 99;
+ }
+}
+
+void
+f3 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx * 4 - 16] = 100;
+ x[i * stepx * 4 - 15] = 99;
+ }
+}
+
+void
+f4 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx * 4)
+ {
+ x[i - 16] = 100;
+ x[i - 15] = 99;
+ }
+}
+
+void
+f5 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx * 64 - 16] = 100;
+ x[i * stepx * 64 + 15] = 99;
+ }
+}
+
+void
+f6 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx * 64)
+ {
+ x[i - 16] = 100;
+ x[i + 15] = 99;
+ }
+}
+
+void
+f7 (unsigned short *x, int stepx, int n)
+{
+ for (unsigned short *y = x; y < x + n; y += stepx * 4)
+ {
+ y[0] = 100;
+ y[1] = 99;
+ }
+}
+
+unsigned short x[1000];
+
+void
+g1 (int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx * 4] = 100;
+ x[i * stepx * 4 + 1] = 99;
+ }
+}
+
+void
+g2 (int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx * 4)
+ {
+ x[i] = 100;
+ x[i + 1] = 99;
+ }
+}
+
+void
+g3 (int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx * 4 - 16] = 100;
+ x[i * stepx * 4 - 15] = 99;
+ }
+}
+
+void
+g4 (int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx * 4)
+ {
+ x[i - 16] = 100;
+ x[i - 15] = 99;
+ }
+}
+
+void
+g5 (int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx * 64 - 16] = 100;
+ x[i * stepx * 64 + 15] = 99;
+ }
+}
+
+void
+g6 (int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx * 64)
+ {
+ x[i - 16] = 100;
+ x[i + 15] = 99;
+ }
+}
+
+void
+g7 (int stepx, int n)
+{
+ for (unsigned short *y = x; y < x + n; y += stepx * 4)
+ {
+ y[0] = 100;
+ y[1] = 99;
+ }
+}
+
+/* { dg-final { scan-tree-dump-not {want to version} "lversion" } } */
+/* { dg-final { scan-tree-dump-not {versioned} "lversion" } } */
Index: gcc/testsuite/gcc.dg/vect/slp-43.c
===================================================================
--- gcc/testsuite/gcc.dg/vect/slp-43.c 2018-05-02 08:37:48.993604639 +0100
+++ gcc/testsuite/gcc.dg/vect/slp-43.c 2018-12-06 12:33:59.220098363 +0000
@@ -1,5 +1,5 @@
/* { dg-require-effective-target vect_int } */
-/* { dg-additional-options "-O3" } */
+/* { dg-additional-options "-O3 -fno-version-loops-for-strides" } */
#include <string.h>
#include "tree-vect.h"
Index: gcc/testsuite/gcc.dg/vect/slp-45.c
===================================================================
--- gcc/testsuite/gcc.dg/vect/slp-45.c 2018-05-02 08:37:48.997604602 +0100
+++ gcc/testsuite/gcc.dg/vect/slp-45.c 2018-12-06 12:33:59.220098363 +0000
@@ -1,5 +1,5 @@
/* { dg-require-effective-target vect_int } */
-/* { dg-additional-options "-O3" } */
+/* { dg-additional-options "-O3 -fno-version-loops-for-strides" } */
#include <string.h>
#include "tree-vect.h"
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Add a loop versioning pass
2018-12-06 13:19 ` Richard Sandiford
@ 2018-12-12 12:06 ` Richard Biener
2018-12-12 18:43 ` Richard Sandiford
0 siblings, 1 reply; 17+ messages in thread
From: Richard Biener @ 2018-12-12 12:06 UTC (permalink / raw)
To: Richard Guenther, GCC Patches, Richard Sandiford
On Thu, Dec 6, 2018 at 2:19 PM Richard Sandiford
<richard.sandiford@arm.com> wrote:
>
> Richard Biener <rguenther@suse.de> writes:
> >> > The pass contains an awful lot of heuristics :/ Like last year
> >> > with the interchange pass I would suggest to rip most of it out
> >> > and first lay infrastructure with the cases you can positively
> >> > identify without applying heuristics or "hacks" like stripping
> >> > semantically required casts. That makes it also clear which
> >> > testcases test which code-path. That said, all the analyze
> >> > multiplications/plusses/factors stuff was extremely hard to review
> >> > and I have no overall picture why this is all so complicated or
> >> > necessary.
> >>
> >> I think the tests do cover most of this -- was glad to see that
> >> when Kyrill tried taking something out, one of the tests started
> >> to fail :-). And the comments in the tests as well as the code
> >> are supposed to explain why this is desirable. But I can try to
> >> spruce up the comments in the code if they're not detailed enough.
> >>
> >> The problem is that the motivating (Fortran) case can't be identified
> >> without applying heuristics. All we see is a collection of adds and
> >> multiplies, with no semantic information to say which multiplies are for
> >> the inner dimension and which aren't. I agree it would be nicer not
> >> to have them, which is why I'd originally suggested the IFN to provide
> >> information about dimensions.
> >
> > Sure, still a new pass having _all_ the heuristic built in with
> > 10s of testcases do not make it easy to review those heuristics...
>
> Fair :-) In the patch below I've tried to cut down on the special cases
> and tried to add more comments explaining which patterns the code is
> trying to detect/reject. Probably the key part is the one beginning:
>
> + The main difficulty here isn't finding strides that could be used
> + in a version check (that's pretty easy). The problem instead is to
> + avoid versioning for some stride S that is unlikely ever to be 1 at
> + runtime. Versioning for S == 1 on its own would lead to unnecessary
> + code bloat, while adding S == 1 to more realistic version conditions
> + would lose the optimisation opportunity offered by those other conditions.
>
> >> >> +/* Return true if in principle it is worth versioning an index fragment of
> >> >> + the form:
> >> >> +
> >> >> + (i * b * SCALE) / FACTOR
> >> >> +
> >> >> + for the case in which b == 1. */
> >> >> +
> >> >> +bool
> >> >> +loop_versioning::acceptable_scale_p (tree scale, poly_uint64 factor)
> >> >> +{
> >> >> + /* See whether SCALE is a constant multiple of FACTOR, and if the
> >> >> + multiple is small enough for us to treat it as a potential grouped
> >> >> + access. For example:
> >> >> +
> >> >> + for (auto i : ...)
> >> >> + y[i] = f (x[4 * i * stride],
> >> >> + x[4 * i * stride + 1],
> >> >> + x[4 * i * stride + 2]);
> >> >> +
> >> >> + would benefit from versioning for the case in which stride == 1.
> >> >> + High multiples of i * stride are less likely to benefit, and could
> >> >> + indicate a simulated multi-dimensional array.
> >> >> +
> >> >> + This is just a heuristic, to avoid having to do expensive group
> >> >> + analysis of the data references in a loop. */
> >> >> + poly_uint64 const_scale;
> >> >> + unsigned int multiple;
> >> >> + if (poly_int_tree_p (scale, &const_scale)
> >> >> + && constant_multiple_p (const_scale, factor, &multiple))
> >> >> + {
> >> >> + unsigned int maxval = PARAM_VALUE (PARAM_LOOP_VERSIONING_GROUP_SIZE);
> >> >> + return IN_RANGE (multiple, 1, maxval);
> >> >
> >> > Hmm. So you _do_ want to version sth like
> >> >
> >> > struct X { int i; int j; } a[2048];
> >> >
> >> > for (int i = start; i < end; ++i)
> >> > a[i*s].i = 1;
> >> >
> >> > ? That is, even with s == 1 the accesses will not become contiguous?
> >> > OK, I suppose that's because you are looking at a stmt in isolation
> >> > and another stmt may access a[i*s].j here.
> >> >
> >> > That is, would it be a future improvement to run sth like the
> >> > vectorizers group access analysis on the references and perform
> >> > this check on whole groups then, possibly better being able to
> >> > constrain what is now the magic parameter PARAM_LOOP_VERSIONING_GROUP_SIZE?
> >>
> >> Yeah, possibly. The problem is that we might end up having to reproduce
> >> vectoriser heuristics. E.g. stores with gaps should be vectorisable
> >> in future with SVE, using contiguous predicated stores in which some
> >> lanes are inactive. So we don't necessarily need the .j access for
> >> this to be worthwhile.
> >>
> >> But perhaps the argument for versioning for vectorisation is stronger
> >> for grouped accesses than contiguous ones.
> >>
> >> Note that the above example doesn't rely on the grouping heuristic
> >> since we just strip the COMPONENT_REF and then analyze the ARRAY_REF
> >> with a factor of 1. The heuristic instead handles things like:
> >>
> >> a[i * stride * 2]
> >>
> >> etc., and the param is more there to decide when a constant factor is
> >> small enough to be a realistic group size instead of an outer dimension.
> >> E.g. if we see:
> >>
> >> a[i * stride * 20000]
> >>
> >> the problem is that i * stride * 20000 is likely to be applying
> >> an outer dimension index.
> >>
> >> > I guess you tried to constrain it to the stmts access size but of course
> >> > that fails short of handling later SLP vectorized cases.
> >>
> >> Right.
> >
> > So I think we want to constrain it to sth like the maximum targets
> > vector size. We want consecutive iterations to bring two memory
> > accesses to the same vector load, otherwise we can use strided
> > stores (or gathers) anyways. This means the distance between
> > accesses should be at most max_vectsize / 2? (non-power-of-two
> > interleaving is also quite limited, sth. to be considered)
>
> OK. The updated patch removes the new param and instead uses the maximum of:
>
> - omp_max_vf () (maximum vector size in bytes, or 1 if loop vectorisation
> is disabled)
>
> - MAX_FIXED_MODE_SIZE, which should be a good limit for scalar code.
>
> >> >> +{
> >> >> + const unsigned int MAX_NSPLIT = 8;
> >> >> +
> >> >> + if (dump_file && (dump_flags & TDF_DETAILS))
> >> >> + {
> >> >> + fprintf (dump_file, ";; Analyzing use of ");
> >> >> + print_generic_expr (dump_file, expr, TDF_SLIM);
> >> >> + if (maybe_ne (factor, 1U))
> >> >> + {
> >> >> + fprintf (dump_file, " (which addresses ");
> >> >> + print_dec (factor, dump_file);
> >> >> + fprintf (dump_file, " bytes)");
> >> >> + }
> >> >> + fprintf (dump_file, " in loop %d (depth %d)\n",
> >> >> + loop->num, loop_depth (loop));
> >> >> + }
> >> >> +
> >> >> + /* The main problem we have here is that we cannot assume that the
> >> >> + innermost loop iterates over the innermost dimension of an array.
> >> >> + Accidentally adding versioning checks for outer dimensions would
> >> >> + cause the version condition to be false, which as well as bloating
> >> >> + the code would defeat loop versioning benefits for other accesses.
> >> >> +
> >> >> + Unfortunately all we usually see at this stage is general address
> >> >> + arithmetic, with no positive way of identifying how many dimensions
> >> >> + an array access has and which multiplication factors in the address
> >> >> + expression correspond to which array dimensions. In C code this is
> >> >> + often not even explicit in the source, since variable-sized multi-
> >> >> + dimensional arrays are often simulated using one-dimensional arrays.
> >> >> +
> >> >> + The three main ways in which we deal with this are:
> >> >> +
> >> >> + - use heuristics that positively identify steps that are likely
> >> >> + to represent the inner dimension.
> >> >> +
> >> >> + - use heuristics that positively identify steps that are unlikely
> >> >> + to represent the inner dimension.
> >> >> +
> >> >> + - if a part of EXPR is invariant in LOOP, analyze its evolution in
> >> >> + the outer loops to see whether we can positively identify any of
> >> >> + it as iterating over the inner dimension. */
> >> >> + tree best_step = NULL_TREE;
> >> >> + auto_vec<tree, MAX_NSPLIT> worklist;
> >> >> + worklist.quick_push (expr);
> >> >> + unsigned int nsplit = 0;
> >> >> + while (!worklist.is_empty ())
> >> >> + {
> >> >> + expr = strip_casts (worklist.pop ());
> >> >> + tree_code code = TREE_CODE (expr);
> >> >> +
> >> >> + if (code == POLYNOMIAL_CHREC)
> >> >> + {
> >> >> + /* Analyze the CHREC_RIGHT as the step in an index fragment. */
> >> >> + tree step_if_innermost;
> >> >> + tree step = extract_step (CHREC_RIGHT (expr), factor,
> >> >> + &step_if_innermost);
> >> >> + if (!best_step)
> >> >> + {
> >> >> + /* This is the outermost chrec for the original expression.
> >> >> + It's not worth carrying on if the step isn't versionable,
> >> >> + or if we're pretty sure it's not for the inner dimension. */
> >> >> + if (!step_if_innermost
> >> >> + || TREE_CODE (step) != SSA_NAME
> >> >> + || !expr_invariant_in_loop_p (loop, step))
> >> >> + return;
> >> >> +
> >> >> + best_step = step;
> >> >> +
> >> >> + /* We should version for STEP == 1 if we know that that can be
> >> >> + true under some circumstances. */
> >> >> + if (integer_onep (step_if_innermost))
> >> >> + break;
> >> >> +
> >> >> + /* Bail out if this appears to be the step for the innermost
> >> >> + dimension, but isn't likely to be 1.
> >> >> +
> >> >> + ??? We could instead version for when it equals
> >> >> + STEP_IF_INNERMOST, but it's not likely to have as much
> >> >> + benefit as versioning for 1. */
> >> >> + if (step_if_innermost != step)
> >> >> + return;
> >> >> + }
> >> >> + else
> >> >> + {
> >> >> + /* This is an inner chrec. If it looks like it iterates over
> >> >> + the innermost dimension, abort any attempt to version for
> >> >> + the outermost chrec (which if we reach here wasn't itself
> >> >> + obviously iterating over the innermost dimension). */
> >> >> + if (step_if_innermost && TREE_CONSTANT (step_if_innermost))
> >> >> + return;
> >> >> + }
> >> >> + worklist.quick_push (CHREC_LEFT (expr));
> >> >> + continue;
> >> >> + }
> >> >> +
> >> >> + /* Analyze parts of a queued CHREC_LEFT. This is better than just
> >> >> + analyzing the evolution of the whole expression since the value
> >> >> + could include a mixture of analyzable and unanalyzable elements.
> >> >> + Use NSPLIT to count cases in which we add more expressions to
> >> >> + analyze, as opposed to just simplifying the existing one. */
> >> >> + if (code == PLUS_EXPR || code == POINTER_PLUS_EXPR || code == MINUS_EXPR)
> >> >> + {
> >> >> + worklist.quick_push (TREE_OPERAND (expr, 0));
> >> >> + if (nsplit++ < MAX_NSPLIT)
> >> >> + worklist.quick_push (TREE_OPERAND (expr, 1));
> >> >> + continue;
> >> >> + }
> >> >> + if (code == MULT_EXPR)
> >> >> + {
> >> >> + tree op0 = strip_casts (TREE_OPERAND (expr, 0));
> >> >> + tree op1 = TREE_OPERAND (expr, 1);
> >> >> + if (TREE_CODE (op0) == PLUS_EXPR || TREE_CODE (op0) == MINUS_EXPR)
> >> >> + {
> >> >> + tree type = TREE_TYPE (expr);
> >> >> + tree op00 = fold_convert (type, TREE_OPERAND (op0, 0));
> >> >> + worklist.quick_push (fold_build2 (MULT_EXPR, type, op00, op1));
> >> >> + if (nsplit++ < MAX_NSPLIT)
> >> >> + {
> >> >> + tree op01 = fold_convert (type, TREE_OPERAND (op0, 1));
> >> >> + worklist.quick_push (fold_build2 (MULT_EXPR, type,
> >> >> + op01, op1));
> >> >> + }
> >> >> + continue;
> >> >> + }
> >> >> + }
> >> >> +
> >> >> + /* If EXPR is invariant in LOOP, analyze it wrt the innermost loop
> >> >> + for which it could evolve (i.e. the loop containing the outermost
> >> >> + one for which EXPR is invariant). */
> >> >
> >> > This isn't how analyze_scalar_evolution works - you _always_ have to
> >> > feed the innermost loop that the expression is used in (the context).
> >> > Then you instantiate the result in the outermost loop of the nest you
> >> > are interested in. Otherwise you get garbage.
> >>
> >> I dispute this. :-) You might remember we had a similar discussion
> >> about the use in get_base_for_alignment_1.
> >
> > I don't remember ;)
> >
> >> analyze_scalar_evolution describes how the supplied value V evolves,
> >> which isn't garbage in itself. Then instantiating the SCEV tells
> >
> > Then analyze_scalar_evolution wouldn't need a 'loop' parameter.
> > It probably should get a stmt (or a use_operand) instead.
>
> Maybe :-)
>
> >> you what this means for the use in:
> >>
> >> RES = V; // A
> >> ...
> >> use of RES // B
> >>
> >> in terms of values that are available at B. But we're not interested in
> >> the second bit here. We only want to analyze *how* V evolves, not use
> >> or compute its value in a given context. Instantiating the SCEV would
> >> lose useful information.
> >
> > Instantiating only "loses" information in case there's an overall
> > effect of a loop to consider. I guess in your case it only brings
> > in not useful information.
>
> Well, any information we can get is useful here. Even if something
> is never going to be a versioning opportunity itself, it can still
> help to stop us adding pointless version checks for other parts
> of the address calculation.
>
> > That said, you can indeed pass analyze_scalar_evolution outer loops
> > of the loop the use is in (it's also in the outer loops).
> >
> >> > It looks like you are re-analyzing SSA names in the evolution - that's
> >> > odd and shouldn't be necessary (but you forget to instantiate, so...).
> >> >
> >> > I suggest to move the analyze_scalar_evolution part out of the worklist
> >> > loop where you are sure you have an SSA name.
> >>
> >> This too is because we don't care whether the whole address can
> >> be expressed as a SCEV, we just want to search for parts of the
> >> calculation that are more likely to be inner dimensions.
> >>
> >> This isn't (just) because we don't instantiate. It's because we're
> >> more aggressive than SCEV can be in looking through casts.
> >
> > :/
>
> The updated patch avoids this.
>
> >> >> + /* Bail out if OP2 can be analyzed as an evolution; we want to use
> >> >> + analyze_evolution in that case instead. There's no point trying
> >> >> + hard to avoid repeating the call to analyze_scalar_evolution since
> >> >> + that function does its own caching. */
> >> >> + if (tree_contains_chrecs (analyze_scalar_evolution (loop, op2), NULL))
> >> >
> >> > Don't you want !chrec_contains_undetermined (...) instead? I wonder what
> >> > is missing to allow all interesting expressions to be analyzed by
> >> > SCEV?
> >>
> >> This code is handling cases that vary arbitrarily between iterations,
> >> such as a histogram update. I don't think SCEV could ever provide
> >> anything useful here.
> >>
> >> analyze_scalar_evolution returns op2 if it can't analyze op2 as a SCEV.
> >
> > Hmm, right. Looks inconsistent to instantiation...
> >
> >> >> +/* Treat EXPR as a sum of products and apply analyze_product to each of the
> >> >> + products. Return true if one of the products provides a versioning
> >> >> + opportunity. FACTOR is as for analyze_product. */
> >> >> +
> >> >> +bool
> >> >> +loop_versioning::analyze_sum_of_products (struct loop *loop, tree expr,
> >> >> + poly_uint64 factor)
> >> >> +{
> >> >
> >> > This looks awfully close to what tree-affine.c does apart from more
> >> > aggressively stripping conversions? I see all the analysis parts
> >> > are limited and thus "O(1)" but still there's going to be a lot of
> >> > redundancy involved for repeated use of (derived) "IVs"? Wouldn't
> >> > it be good to have a bitmap of already handled SSA_NAMEs to stop
> >> > processing early?
> >>
> >> I did consider this but think it'll do more harm than good.
> >> Bitmap lookup is O(N) in the worst case, and the PR you were
> >> working on a month or so ago shows that that really can bite.
> >> Whereas like you say the limits make this O(1).
> >>
> >> So although Kyrill and Ramana's patch added this check, I'd prefer to
> >> leave it out. It seems like premature optimisation and I think it'll
> >> make things worse in practice.
> >
> > OK.
> >
> >> >> + const unsigned int MAX_NITERS = 8;
> >> >> +
> >> >> + tree worklist[MAX_NITERS];
> >> >> + unsigned int length = 0;
> >> >> + worklist[length++] = expr;
> >> >> + for (unsigned int i = 0; i < length; ++i)
> >> >> + {
> >> >> + expr = worklist[i];
> >> >> + gassign *assign = maybe_get_assign_strip_casts (loop, expr);
> >> >> + if (!assign)
> >> >> + continue;
> >> >> +
> >> >> + tree_code code = gimple_assign_rhs_code (assign);
> >> >> + if (code == PLUS_EXPR || code == POINTER_PLUS_EXPR || code == MINUS_EXPR)
> >> >
> >> > POINTER_MINUS_EXPR?
> >>
> >> Do you mean POINTER_DIFF_EXPR? I wouldn't expect that to be interesting
> >> since it would give information about the two pointers being subtracted.
> >>
> >> >> +/* Analyze expression EXPR, which occurs in loop LOOP. */
> >> >> +
> >> >> +void
> >> >> +loop_versioning::analyze_expr (struct loop *loop, tree expr)
> >> >> +{
> >> >> + while (handled_component_p (expr))
> >> >> + {
> >> >> + /* See whether we can use versioning to avoid a multiplication
> >> >> + in the array index. */
> >> >> + if (TREE_CODE (expr) == ARRAY_REF)
> >> >> + {
> >> >> + if (!analyze_sum_of_products (loop, TREE_OPERAND (expr, 1), 1))
> >> >> + analyze_evolution (loop, TREE_OPERAND (expr, 1), 1);
> >> >> + }
> >> >> + expr = TREE_OPERAND (expr, 0);
> >> >> + }
> >> >> +
> >> >> + if (TREE_CODE (expr) == MEM_REF)
> >> >> + {
> >> >> + tree addr = TREE_OPERAND (expr, 0);
> >> >> + /* See whether we can use versioning to avoid a multiplication
> >> >> + in the pointer calculation. This is generally only worth
> >> >> + doing if the multiplication occurs in this loop rather than
> >> >> + an outer loop. */
> >> >
> >> > Why's that so here but not above for ARRAY_REF? That is, what is
> >> > the difference between a[i] and ptr = &a[i]; *ptr?
> >>
> >> Good question. I think there was a (probably poor) reason, but clearly
> >> I didn't write it down.
> >>
> >> > > + struct loop *loop = gimple_bb (stmt)->loop_father;
> >> > > +
> >> > > + unsigned int nops = gimple_num_ops (stmt);
> >> > > + for (unsigned int i = 0; i < nops; ++i)
> >> > > + if (tree op = gimple_op (stmt, i))
> >> > > + analyze_expr (loop, op);
> >> >
> >> > I think you instead want to use gimple_walk_load_store_ops ().
> >>
> >> I agree with Kyrill's response to this.
> >
> > C++ bites you? ;) I suppose marshalling a pmf through the void *
> > data and having the callback invoke the pmf might be possible?
> > But yeah - GCC is C and most callbacks are just function pointers
> > and not pointer-to-member fns.
> >
> > You are at least also walking debug-stmts here I think
> > (unless they are expensive_stmt_p ...).
>
> Bah, yes. I remember this only about half the time :-(
>
> >> In addition, I'd originally
> >> wondered whether we should also version for ADDR_EXPRs, and that might
> >> make sense if we add more types of versioning (not just unit strides)
> >> in future.
> >
> > There's gimple_walk_load_store_addr_ops () of course.
> >
> >> >> +/* Analyze all the statements in BB looking for useful version checks. */
> >> >> +
> >> >> +void
> >> >> +loop_versioning::analyze_block (basic_block bb)
> >> >> +{
> >> >> + struct loop *loop = bb->loop_father;
> >> >> + loop_info &li = get_loop_info (loop);
> >> >> + if (li.rejected_p)
> >> >> + return;
> >> >> +
> >> >> + for (gimple_stmt_iterator gsi = gsi_start_bb (bb); !gsi_end_p (gsi);
> >> >> + gsi_next (&gsi))
> >> >> + {
> >> >> + gimple *stmt = gsi_stmt (gsi);
> >> >> + if (expensive_stmt_p (stmt))
> >> >> + {
> >> >> + if (dump_file && (dump_flags & TDF_DETAILS))
> >> >> + {
> >> >> + struct loop *loop = gimple_bb (stmt)->loop_father;
> >> >> + fprintf (dump_file, ";; Loop %d (depth %d) contains expensive"
> >> >> + " stmt: ", loop->num, loop_depth (loop));
> >> >> + print_gimple_stmt (dump_file, stmt, 0, TDF_SLIM);
> >> >> + }
> >> >> + li.rejected_p = true;
> >> >
> >> > I assume that once a loop is rejected this or another way there's
> >> > no reason to look at any outer loop of it, thus ...
> >> >
> >> >> + break;
> >> >> + }
> >> >> +
> >> >> + /* Only look for direct versioning opportunities in inner loops
> >> >> + since the benefit tends to be much smaller for outer loops. */
> >> >> + if (!loop->inner)
> >> >> + analyze_stmt (stmt);
> >> >> +
> >> >> + /* The point of the instruction limit is to prevent excessive
> >> >> + code growth, so this is a size-based estimate even though
> >> >> + the optimization is aimed at speed. */
> >> >> + li.num_insns += estimate_num_insns (stmt, &eni_size_weights);
> >> >> + }
> >> >> +}
> >> >> +
> >> >> +/* Analyze all the blocks in the function looking for useful version checks.
> >> >> + Return true if we found one. */
> >> >> +
> >> >> +bool
> >> >> +loop_versioning::analyze_blocks ()
> >> >> +{
> >> >> + /* For now we don't try to version the whole function, although
> >> >> + versioning at that level could be useful in some cases. */
> >> >> + get_loop_info (get_loop (m_fn, 0)).rejected_p = true;
> >> >> +
> >> >> + basic_block bb;
> >> >> + FOR_EACH_BB_FN (bb, m_fn)
> >> >> + if (loop_outer (bb->loop_father))
> >> >> + analyze_block (bb);
> >> >
> >> > .... I'd structure this as a
> >> >
> >> > FOR_EACH_LOOP (... LI_FROM_INNERMOST)
> >> > look at loop body, only analyze stmts belonging to loop (not subloops)
> >> >
> >> > walk eventually even open-coding this as recursion so you can quickly
> >> > finish off outer loop processing once an inner loop got disabled.
> >>
> >> OK. Thought about doing that, but it seemed relatively hard to get the
> >> list of blocks that are in a loop but not in subloops. (Let me know if
> >> there's an easy way.)
> >
> > Simply skip bbs with bb->loop_father != loop.
> >
> >> But I guess it's probably worth an extra ad-hoc
> >> walk over the blocks (not the contents) to construct a list of blocks
> >> for each loop.
> >
> > Well, constructing a list of blocks in an order that visits inner loop
> > blocks first (recursively) would be useful here and in other places
> > I guess. That can be done via get_loop_body (outermost-interesting-loop);
> > and then sorting the blocks after loop-father loop-dfs order or so.
> > But as said just skipping bb->loop_father != loop would work for me
> > (at the expense of looking at bb->loop_father N^2 times).
>
> Yeah, but I wanted to avoid even that quadraticness. The updated patch
> uses a linear walk to create a linked list of blocks for each loop.
>
> Changes from last time (to address comments that I might have snipped):
>
> - We now decompose the address fragments into a sum of the form:
>
> [base + ]var1 * const1 + var2 * const2 + ... + [CONST_MIN, CONST_MAX]
>
> then analyse each var<i> * const<i> for versioning opportunities.
> First we check whether var<i> is set by a multiplication by a
> loop invariant, and only if that fails fall back to SCEV analysis.
> There's no longer any processing of the CHREC_LEFT; only the
> CHREC_RIGHT matters. The processsing of SCEVs is generally
> much simpler than before.
>
> - We now collect grouping information for accesses in a loop by
> pooling any of the decomposed addresses fragments described above
> that differ only in [CONST_MIN, CONST_MAX]. So for example:
>
> a[i * stride] + ...;
> a[i * stride + 1] + ...;
>
> is treated as a single access of sizeof(a[0]) * 2 and:
>
> a[i * stride] + ...;
> a[i * stride + 3] + ...;
>
> is treated as a single access of sizeof(a[0]) * 4 (ignoring the gap
> in the middle).
>
> - We only version loops for conditions that would lead to consecutive groups,
> rather than cases that would leave gaps between groups.
>
> - We process loops innermost-first, and only analyse address fragments
> if we get to the end of the loop without finding something that stops
> us from versioning.
>
> - The handling of pointers and array indices is consistent.
>
> - This version drops some of the heuristics related to enabling more
> loop interchange opportunities. I've kept the tests but changed
> the expected result.
>
> - I think the only heuristic-heavy function left is get_inner_likelihood,
> which is very difficult to avoid.
>
> Tested on x86_64-linux-gnu, aarch64-linux-gnu and aarch64_be-elf.
> Also repeated the performance testing (but haven't yet tried an
> LTO variant; will do that over the weekend).
Any results? I've skimmed over the updated patch and it looks
a lot better now.
+bool
+loop_versioning
+::find_per_loop_multiplication (address_info &address, address_term_info &term)
+{
is that what coding convention allows? For grepping I'd then say we should do
bool loop_versioning::
find_per_loop_multiplication (...)
;) Anywhere else we you use
loop_versioning::foo
so please stick to that.
Otherwise OK.
I think I don't see a testcase where we could version both loops in a nest
so I'm not sure whether the transform works fine when you are only
updating SSA form at the very end of the pass. There may also be some
subtle issues with substitute_and_fold being applied to non-up-to-date SSA
form given it folds stmts looking at (single-use!) SSA edges. The single-use
guard might be what saves you here (SSA uses in the copies are not yet
updated to point to the copied DEFs).
Thanks,
Richard.
> Thanks,
> Richard
>
>
> 2018-12-06 Richard Sandiford <richard.sandiford@arm.com>
> Ramana Radhakrishnan <ramana.radhakrishnan@arm.com>
> Kyrylo Tkachov <kyrylo.tkachov@arm.com>
>
> gcc/
> * doc/invoke.texi (-fversion-loops-for-strides): Document
> (loop-versioning-group-size, loop-versioning-max-inner-insns)
> (loop-versioning-max-outer-insns): Document new --params.
> * Makefile.in (OBJS): Add gimple-loop-versioning.o.
> * common.opt (fversion-loops-for-strides): New option.
> * opts.c (default_options_table): Enable fversion-loops-for-strides
> at -O3.
> * params.def (PARAM_LOOP_VERSIONING_GROUP_SIZE)
> (PARAM_LOOP_VERSIONING_MAX_INNER_INSNS)
> (PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS): New parameters.
> * passes.def: Add pass_loop_versioning.
> * timevar.def (TV_LOOP_VERSIONING): New time variable.
> * tree-ssa-propagate.h
> (substitute_and_fold_engine::substitute_and_fold): Add an optional
> block parameter.
> * tree-ssa-propagate.c
> (substitute_and_fold_engine::substitute_and_fold): Likewise.
> When passed, only walk blocks dominated by that block.
> * tree-vrp.h (range_includes_p): Declare.
> (range_includes_zero_p): Turn into an inline wrapper around
> range_includes_p.
> * tree-vrp.c (range_includes_p): New function, generalizing...
> (range_includes_zero_p): ...this.
> * tree-pass.h (make_pass_loop_versioning): Declare.
> * gimple-loop-versioning.cc: New file.
>
> gcc/testsuite/
> * gcc.dg/loop-versioning-1.c: New test.
> * gcc.dg/loop-versioning-10.c: Likewise.
> * gcc.dg/loop-versioning-11.c: Likewise.
> * gcc.dg/loop-versioning-2.c: Likewise.
> * gcc.dg/loop-versioning-3.c: Likewise.
> * gcc.dg/loop-versioning-4.c: Likewise.
> * gcc.dg/loop-versioning-5.c: Likewise.
> * gcc.dg/loop-versioning-6.c: Likewise.
> * gcc.dg/loop-versioning-7.c: Likewise.
> * gcc.dg/loop-versioning-8.c: Likewise.
> * gcc.dg/loop-versioning-9.c: Likewise.
> * gfortran.dg/loop_versioning_1.f90: Likewise.
> * gfortran.dg/loop_versioning_2.f90: Likewise.
> * gfortran.dg/loop_versioning_3.f90: Likewise.
> * gfortran.dg/loop_versioning_4.f90: Likewise.
> * gfortran.dg/loop_versioning_5.f90: Likewise.
> * gfortran.dg/loop_versioning_6.f90: Likewise.
> * gfortran.dg/loop_versioning_7.f90: Likewise.
> * gfortran.dg/loop_versioning_8.f90: Likewise.
>
> Index: gcc/doc/invoke.texi
> ===================================================================
> --- gcc/doc/invoke.texi 2018-12-05 08:33:45.282882618 +0000
> +++ gcc/doc/invoke.texi 2018-12-06 12:33:59.212098431 +0000
> @@ -8256,7 +8256,8 @@ by @option{-O2} and also turns on the fo
> -ftree-partial-pre @gol
> -ftree-slp-vectorize @gol
> -funswitch-loops @gol
> --fvect-cost-model}
> +-fvect-cost-model @gol
> +-fversion-loops-for-strides}
>
> @item -O0
> @opindex O0
> @@ -10808,6 +10809,30 @@ of the loop on both branches (modified a
>
> Enabled by @option{-fprofile-use} and @option{-fauto-profile}.
>
> +@item -fversion-loops-for-strides
> +@opindex fversion-loops-for-strides
> +If a loop iterates over an array with a variable stride, create another
> +version of the loop that assumes the stride is always one. For example:
> +
> +@smallexample
> +for (int i = 0; i < n; ++i)
> + x[i * stride] = @dots{};
> +@end smallexample
> +
> +becomes:
> +
> +@smallexample
> +if (stride == 1)
> + for (int i = 0; i < n; ++i)
> + x[i] = @dots{};
> +else
> + for (int i = 0; i < n; ++i)
> + x[i * stride] = @dots{};
> +@end smallexample
> +
> +This is particularly useful for assumed-shape arrays in Fortran where
> +(for example) it allows better vectorization assuming contiguous accesses.
> +
> @item -ffunction-sections
> @itemx -fdata-sections
> @opindex ffunction-sections
> @@ -12017,6 +12042,15 @@ Hardware autoprefetcher scheduler model
> Number of lookahead cycles the model looks into; at '
> ' only enable instruction sorting heuristic.
>
> +@item loop-versioning-max-inner-insns
> +The maximum number of instructions that an inner loop can have
> +before the loop versioning pass considers it too big to copy.
> +
> +@item loop-versioning-max-outer-insns
> +The maximum number of instructions that an outer loop can have
> +before the loop versioning pass considers it too big to copy,
> +discounting any instructions in inner loops that directly benefit
> +from versioning.
>
> @end table
> @end table
> Index: gcc/Makefile.in
> ===================================================================
> --- gcc/Makefile.in 2018-12-05 08:33:45.270882723 +0000
> +++ gcc/Makefile.in 2018-12-06 12:33:59.208098467 +0000
> @@ -1320,6 +1320,7 @@ OBJS = \
> gimple-laddress.o \
> gimple-loop-interchange.o \
> gimple-loop-jam.o \
> + gimple-loop-versioning.o \
> gimple-low.o \
> gimple-pretty-print.o \
> gimple-ssa-backprop.o \
> Index: gcc/common.opt
> ===================================================================
> --- gcc/common.opt 2018-12-05 08:33:45.274882688 +0000
> +++ gcc/common.opt 2018-12-06 12:33:59.208098467 +0000
> @@ -2775,6 +2775,10 @@ fsplit-loops
> Common Report Var(flag_split_loops) Optimization
> Perform loop splitting.
>
> +fversion-loops-for-strides
> +Common Report Var(flag_version_loops_for_strides) Optimization
> +Version loops based on whether indices have a stride of one.
> +
> funwind-tables
> Common Report Var(flag_unwind_tables) Optimization
> Just generate unwind tables for exception handling.
> Index: gcc/opts.c
> ===================================================================
> --- gcc/opts.c 2018-12-05 08:33:45.282882618 +0000
> +++ gcc/opts.c 2018-12-06 12:33:59.212098431 +0000
> @@ -556,6 +556,7 @@ static const struct default_options defa
> { OPT_LEVELS_3_PLUS, OPT_ftree_slp_vectorize, NULL, 1 },
> { OPT_LEVELS_3_PLUS, OPT_funswitch_loops, NULL, 1 },
> { OPT_LEVELS_3_PLUS, OPT_fvect_cost_model_, NULL, VECT_COST_MODEL_DYNAMIC },
> + { OPT_LEVELS_3_PLUS, OPT_fversion_loops_for_strides, NULL, 1 },
>
> /* -Ofast adds optimizations to -O3. */
> { OPT_LEVELS_FAST, OPT_ffast_math, NULL, 1 },
> Index: gcc/params.def
> ===================================================================
> --- gcc/params.def 2018-12-05 08:33:45.282882618 +0000
> +++ gcc/params.def 2018-12-06 12:33:59.212098431 +0000
> @@ -1365,6 +1365,19 @@ DEFPARAM(PARAM_LOGICAL_OP_NON_SHORT_CIRC
> "True if a non-short-circuit operation is optimal.",
> -1, -1, 1)
>
> +DEFPARAM(PARAM_LOOP_VERSIONING_MAX_INNER_INSNS,
> + "loop-versioning-max-inner-insns",
> + "The maximum number of instructions in an inner loop that is being"
> + " considered for versioning.",
> + 200, 0, 0)
> +
> +DEFPARAM(PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS,
> + "loop-versioning-max-outer-insns",
> + "The maximum number of instructions in an outer loop that is being"
> + " considered for versioning, on top of the instructions in inner"
> + " loops.",
> + 100, 0, 0)
> +
> /*
>
> Local variables:
> Index: gcc/passes.def
> ===================================================================
> --- gcc/passes.def 2018-12-05 08:33:45.282882618 +0000
> +++ gcc/passes.def 2018-12-06 12:33:59.212098431 +0000
> @@ -265,6 +265,7 @@ along with GCC; see the file COPYING3.
> NEXT_PASS (pass_tree_unswitch);
> NEXT_PASS (pass_scev_cprop);
> NEXT_PASS (pass_loop_split);
> + NEXT_PASS (pass_loop_versioning);
> NEXT_PASS (pass_loop_jam);
> /* All unswitching, final value replacement and splitting can expose
> empty loops. Remove them now. */
> Index: gcc/timevar.def
> ===================================================================
> --- gcc/timevar.def 2018-12-05 08:33:45.286882584 +0000
> +++ gcc/timevar.def 2018-12-06 12:33:59.220098363 +0000
> @@ -234,6 +234,7 @@ DEFTIMEVAR (TV_DSE1 , "
> DEFTIMEVAR (TV_DSE2 , "dead store elim2")
> DEFTIMEVAR (TV_LOOP , "loop analysis")
> DEFTIMEVAR (TV_LOOP_INIT , "loop init")
> +DEFTIMEVAR (TV_LOOP_VERSIONING , "loop versioning")
> DEFTIMEVAR (TV_LOOP_MOVE_INVARIANTS , "loop invariant motion")
> DEFTIMEVAR (TV_LOOP_UNROLL , "loop unrolling")
> DEFTIMEVAR (TV_LOOP_DOLOOP , "loop doloop")
> Index: gcc/tree-ssa-propagate.h
> ===================================================================
> --- gcc/tree-ssa-propagate.h 2018-12-05 08:33:45.286882584 +0000
> +++ gcc/tree-ssa-propagate.h 2018-12-06 12:33:59.220098363 +0000
> @@ -104,7 +104,7 @@ extern void propagate_tree_value_into_st
> virtual bool fold_stmt (gimple_stmt_iterator *) { return false; }
> virtual tree get_value (tree) { return NULL_TREE; }
>
> - bool substitute_and_fold (void);
> + bool substitute_and_fold (basic_block = NULL);
> bool replace_uses_in (gimple *);
> bool replace_phi_args_in (gphi *);
> };
> Index: gcc/tree-ssa-propagate.c
> ===================================================================
> --- gcc/tree-ssa-propagate.c 2018-12-05 08:33:45.286882584 +0000
> +++ gcc/tree-ssa-propagate.c 2018-12-06 12:33:59.220098363 +0000
> @@ -1154,6 +1154,10 @@ substitute_and_fold_dom_walker::before_d
>
>
> /* Perform final substitution and folding of propagated values.
> + Process the whole function if BLOCK is null, otherwise only
> + process the blocks that BLOCK dominates. In the latter case,
> + it is the caller's responsibility to ensure that dominator
> + information is available and up-to-date.
>
> PROP_VALUE[I] contains the single value that should be substituted
> at every use of SSA name N_I. If PROP_VALUE is NULL, no values are
> @@ -1170,16 +1174,24 @@ substitute_and_fold_dom_walker::before_d
> Return TRUE when something changed. */
>
> bool
> -substitute_and_fold_engine::substitute_and_fold (void)
> +substitute_and_fold_engine::substitute_and_fold (basic_block block)
> {
> if (dump_file && (dump_flags & TDF_DETAILS))
> fprintf (dump_file, "\nSubstituting values and folding statements\n\n");
>
> memset (&prop_stats, 0, sizeof (prop_stats));
>
> - calculate_dominance_info (CDI_DOMINATORS);
> + /* Don't call calculate_dominance_info when iterating over a subgraph.
> + Callers that are using the interface this way are likely to want to
> + iterate over several disjoint subgraphs, and it would be expensive
> + in enable-checking builds to revalidate the whole dominance tree
> + each time. */
> + if (block)
> + gcc_assert (dom_info_state (CDI_DOMINATORS));
> + else
> + calculate_dominance_info (CDI_DOMINATORS);
> substitute_and_fold_dom_walker walker (CDI_DOMINATORS, this);
> - walker.walk (ENTRY_BLOCK_PTR_FOR_FN (cfun));
> + walker.walk (block ? block : ENTRY_BLOCK_PTR_FOR_FN (cfun));
>
> /* We cannot remove stmts during the BB walk, especially not release
> SSA names there as that destroys the lattice of our callers.
> Index: gcc/tree-vrp.h
> ===================================================================
> --- gcc/tree-vrp.h 2018-12-05 08:33:45.290882549 +0000
> +++ gcc/tree-vrp.h 2018-12-06 12:33:59.220098363 +0000
> @@ -243,7 +243,7 @@ struct assert_info
> extern void register_edge_assert_for (tree, edge, enum tree_code,
> tree, tree, vec<assert_info> &);
> extern bool stmt_interesting_for_vrp (gimple *);
> -extern bool range_includes_zero_p (const value_range_base *);
> +extern bool range_includes_p (const value_range_base *, HOST_WIDE_INT);
> extern bool infer_value_range (gimple *, tree, tree_code *, tree *);
>
> extern bool vrp_bitmap_equal_p (const_bitmap, const_bitmap);
> @@ -285,4 +285,12 @@ extern tree get_single_symbol (tree, boo
> extern void maybe_set_nonzero_bits (edge, tree);
> extern value_range_kind determine_value_range (tree, wide_int *, wide_int *);
>
> +/* Return TRUE if *VR includes the value zero. */
> +
> +inline bool
> +range_includes_zero_p (const value_range_base *vr)
> +{
> + return range_includes_p (vr, 0);
> +}
> +
> #endif /* GCC_TREE_VRP_H */
> Index: gcc/tree-vrp.c
> ===================================================================
> --- gcc/tree-vrp.c 2018-12-05 08:33:45.290882549 +0000
> +++ gcc/tree-vrp.c 2018-12-06 12:33:59.220098363 +0000
> @@ -1173,15 +1173,14 @@ value_inside_range (tree val, tree min,
> }
>
>
> -/* Return TRUE if *VR includes the value zero. */
> +/* Return TRUE if *VR includes the value X. */
>
> bool
> -range_includes_zero_p (const value_range_base *vr)
> +range_includes_p (const value_range_base *vr, HOST_WIDE_INT x)
> {
> if (vr->varying_p () || vr->undefined_p ())
> return true;
> - tree zero = build_int_cst (vr->type (), 0);
> - return vr->may_contain_p (zero);
> + return vr->may_contain_p (build_int_cst (vr->type (), x));
> }
>
> /* If *VR has a value range that is a single constant value return that,
> Index: gcc/tree-pass.h
> ===================================================================
> --- gcc/tree-pass.h 2018-12-05 08:33:45.286882584 +0000
> +++ gcc/tree-pass.h 2018-12-06 12:33:59.220098363 +0000
> @@ -362,6 +362,7 @@ extern gimple_opt_pass *make_pass_fix_lo
> extern gimple_opt_pass *make_pass_tree_loop (gcc::context *ctxt);
> extern gimple_opt_pass *make_pass_tree_no_loop (gcc::context *ctxt);
> extern gimple_opt_pass *make_pass_tree_loop_init (gcc::context *ctxt);
> +extern gimple_opt_pass *make_pass_loop_versioning (gcc::context *ctxt);
> extern gimple_opt_pass *make_pass_lim (gcc::context *ctxt);
> extern gimple_opt_pass *make_pass_linterchange (gcc::context *ctxt);
> extern gimple_opt_pass *make_pass_tree_unswitch (gcc::context *ctxt);
> Index: gcc/gimple-loop-versioning.cc
> ===================================================================
> --- /dev/null 2018-11-29 13:15:04.463550658 +0000
> +++ gcc/gimple-loop-versioning.cc 2018-12-06 12:33:59.212098431 +0000
> @@ -0,0 +1,1743 @@
> +/* Loop versioning pass.
> + Copyright (C) 2018 Free Software Foundation, Inc.
> +
> +This file is part of GCC.
> +
> +GCC is free software; you can redistribute it and/or modify it
> +under the terms of the GNU General Public License as published by the
> +Free Software Foundation; either version 3, or (at your option) any
> +later version.
> +
> +GCC is distributed in the hope that it will be useful, but WITHOUT
> +ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
> +FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
> +for more details.
> +
> +You should have received a copy of the GNU General Public License
> +along with GCC; see the file COPYING3. If not see
> +<http://www.gnu.org/licenses/>. */
> +
> +#include "config.h"
> +#include "system.h"
> +#include "coretypes.h"
> +#include "backend.h"
> +#include "tree.h"
> +#include "gimple.h"
> +#include "gimple-iterator.h"
> +#include "tree-pass.h"
> +#include "gimplify-me.h"
> +#include "cfgloop.h"
> +#include "tree-ssa-loop.h"
> +#include "ssa.h"
> +#include "tree-scalar-evolution.h"
> +#include "tree-chrec.h"
> +#include "tree-ssa-loop-ivopts.h"
> +#include "fold-const.h"
> +#include "tree-ssa-propagate.h"
> +#include "tree-inline.h"
> +#include "domwalk.h"
> +#include "alloc-pool.h"
> +#include "vr-values.h"
> +#include "gimple-ssa-evrp-analyze.h"
> +#include "tree-vectorizer.h"
> +#include "omp-general.h"
> +#include "params.h"
> +
> +namespace {
> +
> +/* This pass looks for loops that could be simplified if certain loop
> + invariant conditions were true. It is effectively a form of loop
> + splitting in which the pass produces the split conditions itself,
> + instead of using ones that are already present in the IL.
> +
> + Versioning for when strides are 1
> + ---------------------------------
> +
> + At the moment the only thing the pass looks for are memory references
> + like:
> +
> + for (auto i : ...)
> + ...x[i * stride]...
> +
> + It considers changing such loops to:
> +
> + if (stride == 1)
> + for (auto i : ...) [A]
> + ...x[i]...
> + else
> + for (auto i : ...) [B]
> + ...x[i * stride]...
> +
> + This can have several benefits:
> +
> + (1) [A] is often easier or cheaper to vectorize than [B].
> +
> + (2) The scalar code in [A] is simpler than the scalar code in [B]
> + (if the loops cannot be vectorized or need an epilogue loop).
> +
> + (3) We might recognize [A] as a pattern, such as a memcpy or memset.
> +
> + (4) [A] has simpler address evolutions, which can help other passes
> + like loop interchange.
> +
> + The optimization is particularly useful for assumed-shape arrays in
> + Fortran, where the stride of the innermost dimension depends on the
> + array descriptor but is often equal to 1 in practice. For example:
> +
> + subroutine f1(x)
> + real :: x(:)
> + x(:) = 100
> + end subroutine f1
> +
> + generates the equivalent of:
> +
> + raw_stride = *x.dim[0].stride;
> + stride = raw_stride != 0 ? raw_stride : 1;
> + x_base = *x.data;
> + ...
> + tmp1 = stride * S;
> + tmp2 = tmp1 - stride;
> + *x_base[tmp2] = 1.0e+2;
> +
> + but in the common case that stride == 1, the last three statements
> + simplify to:
> +
> + tmp3 = S + -1;
> + *x_base[tmp3] = 1.0e+2;
> +
> + The optimization is in principle very simple. The difficult parts are:
> +
> + (a) deciding which parts of a general address calculation correspond
> + to the inner dimension of an array, since this usually isn't explicit
> + in the IL, and for C often isn't even explicit in the source code
> +
> + (b) estimating when the transformation is worthwhile
> +
> + Structure
> + ---------
> +
> + The pass has four phases:
> +
> + (1) Walk through the statements looking for and recording potential
> + versioning opportunities. Stop if there are none.
> +
> + (2) Use context-sensitive range information to see whether any versioning
> + conditions are impossible in practice. Remove them if so, and stop
> + if no opportunities remain.
> +
> + (We do this only after (1) to keep compile time down when no
> + versioning opportunities exist.)
> +
> + (3) Apply the cost model. Decide which versioning opportunities are
> + worthwhile and at which nesting level they should be applied.
> +
> + (4) Attempt to version all the loops selected by (3), so that:
> +
> + for (...)
> + ...
> +
> + becomes:
> +
> + if (!cond)
> + for (...) // Original loop
> + ...
> + else
> + for (...) // New loop
> + ...
> +
> + Use the version condition COND to simplify the new loop. */
> +
> +/* Enumerates the likelihood that a particular value indexes the inner
> + dimension of an array. */
> +enum inner_likelihood {
> + INNER_UNLIKELY,
> + INNER_DONT_KNOW,
> + INNER_LIKELY
> +};
> +
> +/* Information about one term of an address_info. */
> +struct address_term_info
> +{
> + /* The value of the term is EXPR * MULTIPLIER. */
> + tree expr;
> + unsigned HOST_WIDE_INT multiplier;
> +
> + /* The stride applied by EXPR in each iteration of some unrecorded loop,
> + or null if no stride has been identified. */
> + tree stride;
> +
> + /* Enumerates the likelihood that EXPR indexes the inner dimension
> + of an array. */
> + enum inner_likelihood inner_likelihood;
> +
> + /* True if STRIDE == 1 is a versioning opportunity when considered
> + in isolation. */
> + bool versioning_opportunity_p;
> +};
> +
> +/* Information about an address calculation, and the range of constant
> + offsets applied to it. */
> +struct address_info
> +{
> + static const unsigned int MAX_TERMS = 8;
> +
> + /* One statement that calculates the address. If multiple statements
> + share the same address, we only record the first. */
> + gimple *stmt;
> +
> + /* The loop containing STMT (cached for convenience). If multiple
> + statements share the same address, they all belong to this loop. */
> + struct loop *loop;
> +
> + /* A decomposition of the calculation into a sum of terms plus an
> + optional base. When BASE is provided, it is never an SSA name.
> + Once initialization is complete, all members of TERMs are SSA names. */
> + tree base;
> + auto_vec<address_term_info, MAX_TERMS> terms;
> +
> + /* All bytes accessed from the address fall in the offset range
> + [MIN_OFFSET, MAX_OFFSET). */
> + HOST_WIDE_INT min_offset, max_offset;
> +};
> +
> +/* Stores addresses based on their base and terms (ignoring the offsets). */
> +struct address_info_hasher : nofree_ptr_hash <address_info>
> +{
> + static hashval_t hash (const address_info *);
> + static bool equal (const address_info *, const address_info *);
> +};
> +
> +/* Information about the versioning we'd like to apply to a loop. */
> +struct loop_info
> +{
> + bool worth_versioning_p () const;
> +
> + /* True if we've decided not to version this loop. The remaining
> + fields are meaningless if so. */
> + bool rejected_p;
> +
> + /* True if at least one subloop of this loop benefits from versioning. */
> + bool subloops_benefit_p;
> +
> + /* An estimate of the total number of instructions in the loop,
> + excluding those in subloops that benefit from versioning. */
> + unsigned int num_insns;
> +
> + /* The outermost loop that can handle all the version checks
> + described below. */
> + struct loop *outermost;
> +
> + /* The first entry in the list of blocks that belong to this loop
> + (and not to subloops). m_next_block_in_loop provides the chain
> + pointers for the list. */
> + basic_block block_list;
> +
> + /* We'd like to version the loop for the case in which these SSA names
> + (keyed off their SSA_NAME_VERSION) are all equal to 1 at runtime. */
> + bitmap_head unity_names;
> +};
> +
> +/* The main pass structure. */
> +class loop_versioning
> +{
> +public:
> + loop_versioning (function *);
> + ~loop_versioning ();
> + unsigned int run ();
> +
> +private:
> + /* Used to walk the dominator tree to find loop versioning conditions
> + that are always false. */
> + class lv_dom_walker : public dom_walker
> + {
> + public:
> + lv_dom_walker (loop_versioning &);
> +
> + edge before_dom_children (basic_block) FINAL OVERRIDE;
> + void after_dom_children (basic_block) FINAL OVERRIDE;
> +
> + private:
> + /* The parent pass. */
> + loop_versioning &m_lv;
> +
> + /* Used to build context-dependent range information. */
> + evrp_range_analyzer m_range_analyzer;
> + };
> +
> + /* Used to simplify statements based on conditions that are established
> + by the version checks. */
> + class name_prop : public substitute_and_fold_engine
> + {
> + public:
> + name_prop (loop_info &li) : m_li (li) {}
> + tree get_value (tree) FINAL OVERRIDE;
> +
> + private:
> + /* Information about the versioning we've performed on the loop. */
> + loop_info &m_li;
> + };
> +
> + loop_info &get_loop_info (struct loop *loop) { return m_loops[loop->num]; }
> +
> + unsigned int max_insns_for_loop (struct loop *);
> + bool expensive_stmt_p (gimple *);
> +
> + void version_for_unity (gimple *, tree);
> + bool acceptable_multiplier_p (tree, unsigned HOST_WIDE_INT,
> + unsigned HOST_WIDE_INT * = 0);
> + bool acceptable_type_p (tree, unsigned HOST_WIDE_INT *);
> + bool multiply_term_by (address_term_info &, tree);
> + inner_likelihood get_inner_likelihood (tree, unsigned HOST_WIDE_INT);
> + void analyze_stride (address_info &, address_term_info &,
> + tree, struct loop *);
> + bool find_per_loop_multiplication (address_info &, address_term_info &);
> + void analyze_term_using_scevs (address_info &, address_term_info &);
> + void analyze_address_fragment (address_info &);
> + void record_address_fragment (gimple *, unsigned HOST_WIDE_INT,
> + tree, unsigned HOST_WIDE_INT, HOST_WIDE_INT);
> + void analyze_expr (gimple *, tree);
> + bool analyze_block (basic_block);
> + bool analyze_blocks ();
> +
> + void prune_loop_conditions (struct loop *, vr_values *);
> + bool prune_conditions ();
> +
> + void merge_loop_info (struct loop *, struct loop *);
> + void add_loop_to_queue (struct loop *);
> + bool decide_whether_loop_is_versionable (struct loop *);
> + bool make_versioning_decisions ();
> +
> + bool version_loop (struct loop *);
> + bool implement_versioning_decisions ();
> +
> + /* The function we're optimizing. */
> + function *m_fn;
> +
> + /* The obstack to use for all pass-specific bitmaps. */
> + bitmap_obstack m_bitmap_obstack;
> +
> + /* An obstack to use for general allocation. */
> + obstack m_obstack;
> +
> + /* The number of loops in the function. */
> + unsigned int m_nloops;
> +
> + /* The total number of loop version conditions we've found. */
> + unsigned int m_num_conditions;
> +
> + /* Assume that an address fragment of the form i * stride * scale
> + (for variable stride and constant scale) will not benefit from
> + versioning for stride == 1 when scale is greater than this value. */
> + unsigned HOST_WIDE_INT m_maximum_scale;
> +
> + /* Information about each loop. */
> + auto_vec<loop_info> m_loops;
> +
> + /* Used to form a linked list of blocks that belong to a loop,
> + started by loop_info::block_list. */
> + auto_vec<basic_block> m_next_block_in_loop;
> +
> + /* The list of loops that we've decided to version. */
> + auto_vec<struct loop *> m_loops_to_version;
> +
> + /* A table of addresses in the current loop, keyed off their values
> + but not their offsets. */
> + hash_table <address_info_hasher> m_address_table;
> +
> + /* A list of all addresses in M_ADDRESS_TABLE, in a predictable order. */
> + auto_vec <address_info *, 32> m_address_list;
> +};
> +
> +/* If EXPR is an SSA name and not a default definition, return the
> + defining statement, otherwise return null. */
> +
> +static gimple *
> +maybe_get_stmt (tree expr)
> +{
> + if (TREE_CODE (expr) == SSA_NAME && !SSA_NAME_IS_DEFAULT_DEF (expr))
> + return SSA_NAME_DEF_STMT (expr);
> + return NULL;
> +}
> +
> +/* Like maybe_get_stmt, but also return null if the defining
> + statement isn't an assignment. */
> +
> +static gassign *
> +maybe_get_assign (tree expr)
> +{
> + return safe_dyn_cast <gassign *> (maybe_get_stmt (expr));
> +}
> +
> +/* Return true if this pass should look through a cast of expression FROM
> + to type TYPE when analyzing pieces of an address. */
> +
> +static bool
> +look_through_cast_p (tree type, tree from)
> +{
> + return (INTEGRAL_TYPE_P (TREE_TYPE (from)) == INTEGRAL_TYPE_P (type)
> + && POINTER_TYPE_P (TREE_TYPE (from)) == POINTER_TYPE_P (type));
> +}
> +
> +/* Strip all conversions of integers or pointers from EXPR, regardless
> + of whether the conversions are nops. This is useful in the context
> + of this pass because we're not trying to fold or simulate the
> + expression; we just want to see how it's structured. */
> +
> +static tree
> +strip_casts (tree expr)
> +{
> + const unsigned int MAX_NITERS = 4;
> +
> + tree type = TREE_TYPE (expr);
> + while (CONVERT_EXPR_P (expr)
> + && look_through_cast_p (type, TREE_OPERAND (expr, 0)))
> + expr = TREE_OPERAND (expr, 0);
> +
> + for (unsigned int niters = 0; niters < MAX_NITERS; ++niters)
> + {
> + gassign *assign = maybe_get_assign (expr);
> + if (assign
> + && CONVERT_EXPR_CODE_P (gimple_assign_rhs_code (assign))
> + && look_through_cast_p (type, gimple_assign_rhs1 (assign)))
> + expr = gimple_assign_rhs1 (assign);
> + else
> + break;
> + }
> + return expr;
> +}
> +
> +/* Compare two address_term_infos in the same address_info. */
> +
> +static int
> +compare_address_terms (const void *a_uncast, const void *b_uncast)
> +{
> + const address_term_info *a = (const address_term_info *) a_uncast;
> + const address_term_info *b = (const address_term_info *) b_uncast;
> +
> + if (a->expr != b->expr)
> + return SSA_NAME_VERSION (a->expr) < SSA_NAME_VERSION (b->expr) ? -1 : 1;
> +
> + if (a->multiplier != b->multiplier)
> + return a->multiplier < b->multiplier ? -1 : 1;
> +
> + return 0;
> +}
> +
> +/* Dump ADDRESS using flags FLAGS. */
> +
> +static void
> +dump_address_info (dump_flags_t flags, address_info &address)
> +{
> + if (address.base)
> + dump_printf (flags, "%T + ", address.base);
> + for (unsigned int i = 0; i < address.terms.length (); ++i)
> + {
> + if (i != 0)
> + dump_printf (flags, " + ");
> + dump_printf (flags, "%T", address.terms[i].expr);
> + if (address.terms[i].multiplier != 1)
> + dump_printf (flags, " * %wd", address.terms[i].multiplier);
> + }
> + dump_printf (flags, " + [%wd, %wd]",
> + address.min_offset, address.max_offset - 1);
> +}
> +
> +/* Hash an address_info based on its base and terms. */
> +
> +hashval_t
> +address_info_hasher::hash (const address_info *info)
> +{
> + inchash::hash hash;
> + hash.add_int (info->base ? TREE_CODE (info->base) : 0);
> + hash.add_int (info->terms.length ());
> + for (unsigned int i = 0; i < info->terms.length (); ++i)
> + {
> + hash.add_int (SSA_NAME_VERSION (info->terms[i].expr));
> + hash.add_hwi (info->terms[i].multiplier);
> + }
> + return hash.end ();
> +}
> +
> +/* Return true if two address_infos have equal bases and terms. Other
> + properties might be different (such as the statement or constant
> + offset range). */
> +
> +bool
> +address_info_hasher::equal (const address_info *a, const address_info *b)
> +{
> + if (a->base != b->base
> + && (!a->base || !b->base || !operand_equal_p (a->base, b->base, 0)))
> + return false;
> +
> + if (a->terms.length () != b->terms.length ())
> + return false;
> +
> + for (unsigned int i = 0; i < a->terms.length (); ++i)
> + if (a->terms[i].expr != b->terms[i].expr
> + || a->terms[i].multiplier != b->terms[i].multiplier)
> + return false;
> +
> + return true;
> +}
> +
> +/* Return true if we want to version the loop, i.e. if we have a
> + specific reason for doing so and no specific reason not to. */
> +
> +bool
> +loop_info::worth_versioning_p () const
> +{
> + return (!rejected_p
> + && (!bitmap_empty_p (&unity_names) || subloops_benefit_p));
> +}
> +
> +loop_versioning::lv_dom_walker::lv_dom_walker (loop_versioning &lv)
> + : dom_walker (CDI_DOMINATORS), m_lv (lv)
> +{
> +}
> +
> +/* Process BB before processing the blocks it dominates. */
> +
> +edge
> +loop_versioning::lv_dom_walker::before_dom_children (basic_block bb)
> +{
> + m_range_analyzer.enter (bb);
> +
> + if (bb == bb->loop_father->header)
> + m_lv.prune_loop_conditions (bb->loop_father,
> + m_range_analyzer.get_vr_values ());
> +
> + for (gimple_stmt_iterator si = gsi_start_bb (bb); !gsi_end_p (si);
> + gsi_next (&si))
> + m_range_analyzer.record_ranges_from_stmt (gsi_stmt (si), false);
> +
> + return NULL;
> +}
> +
> +/* Process BB after processing the blocks it dominates. */
> +
> +void
> +loop_versioning::lv_dom_walker::after_dom_children (basic_block bb)
> +{
> + m_range_analyzer.leave (bb);
> +}
> +
> +/* Decide whether to replace VAL with a new value in a versioned loop.
> + Return the new value if so, otherwise return null. */
> +
> +tree
> +loop_versioning::name_prop::get_value (tree val)
> +{
> + if (TREE_CODE (val) == SSA_NAME
> + && bitmap_bit_p (&m_li.unity_names, SSA_NAME_VERSION (val)))
> + return build_one_cst (TREE_TYPE (val));
> + return NULL_TREE;
> +}
> +
> +/* Initialize the structure to optimize FN. */
> +
> +loop_versioning::loop_versioning (function *fn)
> + : m_fn (fn),
> + m_nloops (number_of_loops (fn)),
> + m_num_conditions (0),
> + m_address_table (31)
> +{
> + bitmap_obstack_initialize (&m_bitmap_obstack);
> + gcc_obstack_init (&m_obstack);
> +
> + /* Initialize the loop information. */
> + m_loops.safe_grow_cleared (m_nloops);
> + for (unsigned int i = 0; i < m_nloops; ++i)
> + {
> + m_loops[i].outermost = get_loop (m_fn, 0);
> + bitmap_initialize (&m_loops[i].unity_names, &m_bitmap_obstack);
> + }
> +
> + /* Initialize the list of blocks that belong to each loop. */
> + unsigned int nbbs = last_basic_block_for_fn (fn);
> + m_next_block_in_loop.safe_grow (nbbs);
> + basic_block bb;
> + FOR_EACH_BB_FN (bb, fn)
> + {
> + loop_info &li = get_loop_info (bb->loop_father);
> + m_next_block_in_loop[bb->index] = li.block_list;
> + li.block_list = bb;
> + }
> +
> + /* MAX_FIXED_MODE_SIZE should be a reasonable maximum scale for
> + unvectorizable code, since it is the largest size that can be
> + handled efficiently by scalar code. omp_max_vf calculates the
> + maximum number of bytes in a vector, when such a value is relevant
> + to loop optimization. */
> + m_maximum_scale = estimated_poly_value (omp_max_vf ());
> + m_maximum_scale = MAX (m_maximum_scale, MAX_FIXED_MODE_SIZE);
> +}
> +
> +loop_versioning::~loop_versioning ()
> +{
> + bitmap_obstack_release (&m_bitmap_obstack);
> + obstack_free (&m_obstack, NULL);
> +}
> +
> +/* Return the maximum number of instructions allowed in LOOP before
> + it becomes too big for versioning.
> +
> + There are separate limits for inner and outer loops. The limit for
> + inner loops applies only to loops that benefit directly from versioning.
> + The limit for outer loops applies to all code in the outer loop and
> + its subloops that *doesn't* benefit directly from versioning; such code
> + would be "taken along for the ride". The idea is that if the cost of
> + the latter is small, it is better to version outer loops rather than
> + inner loops, both to reduce the number of repeated checks and to enable
> + more of the loop nest to be optimized as a natural nest (e.g. by loop
> + interchange or outer-loop vectorization). */
> +
> +unsigned int
> +loop_versioning::max_insns_for_loop (struct loop *loop)
> +{
> + return (loop->inner
> + ? PARAM_VALUE (PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS)
> + : PARAM_VALUE (PARAM_LOOP_VERSIONING_MAX_INNER_INSNS));
> +}
> +
> +/* Return true if for cost reasons we should avoid versioning any loop
> + that contains STMT.
> +
> + Note that we don't need to check whether versioning is invalid for
> + correctness reasons, since the versioning process does that for us.
> + The conditions involved are too rare to be worth duplicating here. */
> +
> +bool
> +loop_versioning::expensive_stmt_p (gimple *stmt)
> +{
> + if (gcall *call = dyn_cast <gcall *> (stmt))
> + /* Assume for now that the time spent in an "expensive" call would
> + overwhelm any saving from versioning. */
> + return !gimple_inexpensive_call_p (call);
> + return false;
> +}
> +
> +/* Record that we want to version the loop that contains STMT for the
> + case in which SSA name NAME is equal to 1. We already know that NAME
> + is invariant in the loop. */
> +
> +void
> +loop_versioning::version_for_unity (gimple *stmt, tree name)
> +{
> + struct loop *loop = loop_containing_stmt (stmt);
> + loop_info &li = get_loop_info (loop);
> +
> + if (bitmap_set_bit (&li.unity_names, SSA_NAME_VERSION (name)))
> + {
> + /* This is the first time we've wanted to version LOOP for NAME.
> + Keep track of the outermost loop that can handle all versioning
> + checks in LI. */
> + struct loop *outermost
> + = outermost_invariant_loop_for_expr (loop, name);
> + if (loop_depth (li.outermost) < loop_depth (outermost))
> + li.outermost = outermost;
> +
> + if (dump_enabled_p ())
> + {
> + dump_printf_loc (MSG_NOTE, stmt, "want to version containing loop"
> + " for when %T == 1", name);
> + if (outermost == loop)
> + dump_printf (MSG_NOTE, "; cannot hoist check further");
> + else
> + {
> + dump_printf (MSG_NOTE, "; could implement the check at loop"
> + " depth %d", loop_depth (outermost));
> + if (loop_depth (li.outermost) > loop_depth (outermost))
> + dump_printf (MSG_NOTE, ", but other checks only allow"
> + " a depth of %d", loop_depth (li.outermost));
> + }
> + dump_printf (MSG_NOTE, "\n");
> + }
> +
> + m_num_conditions += 1;
> + }
> + else
> + {
> + /* This is a duplicate request. */
> + if (dump_enabled_p ())
> + dump_printf_loc (MSG_NOTE, stmt, "already asked to version containing"
> + " loop for when %T == 1\n", name);
> + }
> +}
> +
> +/* Return true if OP1_TREE is constant and if in principle it is worth
> + versioning an address fragment of the form:
> +
> + i * OP1_TREE * OP2 * stride
> +
> + for the case in which stride == 1. This in practice means testing
> + whether:
> +
> + OP1_TREE * OP2 <= M_MAXIMUM_SCALE.
> +
> + If RESULT is nonnull, store OP1_TREE * OP2 there when returning true. */
> +
> +bool
> +loop_versioning::acceptable_multiplier_p (tree op1_tree,
> + unsigned HOST_WIDE_INT op2,
> + unsigned HOST_WIDE_INT *result)
> +{
> + if (tree_fits_uhwi_p (op1_tree))
> + {
> + unsigned HOST_WIDE_INT op1 = tree_to_uhwi (op1_tree);
> + /* The first part checks for overflow. */
> + if (op1 * op2 >= op2 && op1 * op2 <= m_maximum_scale)
> + {
> + if (result)
> + *result = op1 * op2;
> + return true;
> + }
> + }
> + return false;
> +}
> +
> +/* Return true if it is worth using loop versioning on a memory access
> + of type TYPE. Store the size of the access in *SIZE if so. */
> +
> +bool
> +loop_versioning::acceptable_type_p (tree type, unsigned HOST_WIDE_INT *size)
> +{
> + return (TYPE_SIZE_UNIT (type)
> + && acceptable_multiplier_p (TYPE_SIZE_UNIT (type), 1, size));
> +}
> +
> +/* See whether OP is constant and whether we can multiply TERM by that
> + constant without exceeding M_MAXIMUM_SCALE. Return true and update
> + TERM if so. */
> +
> +bool
> +loop_versioning::multiply_term_by (address_term_info &term, tree op)
> +{
> + return acceptable_multiplier_p (op, term.multiplier, &term.multiplier);
> +}
> +
> +/* Decide whether an address fragment of the form STRIDE * MULTIPLIER
> + is likely to be indexing an innermost dimension, returning the result
> + as an INNER_* probability. */
> +
> +inner_likelihood
> +loop_versioning::get_inner_likelihood (tree stride,
> + unsigned HOST_WIDE_INT multiplier)
> +{
> + const unsigned int MAX_NITERS = 8;
> +
> + /* Iterate over possible values of STRIDE. Return INNER_LIKELY if at
> + least one of those values is likely to be for the innermost dimension.
> + Record in UNLIKELY_P if at least one of those values is unlikely to be
> + for the innermost dimension.
> +
> + E.g. for:
> +
> + stride = cond ? a * b : 1
> +
> + we should treat STRIDE as being a likely inner dimension, since
> + we know that it is 1 under at least some circumstances. (See the
> + Fortran example below.) However:
> +
> + stride = a * b
> +
> + on its own is unlikely to be for the innermost dimension, since
> + that would require both a and b to be 1 at runtime. */
> + bool unlikely_p = false;
> + tree worklist[MAX_NITERS];
> + unsigned int length = 0;
> + worklist[length++] = stride;
> + for (unsigned int i = 0; i < length; ++i)
> + {
> + tree expr = worklist[i];
> +
> + if (CONSTANT_CLASS_P (expr))
> + {
> + /* See if EXPR * MULTIPLIER would be consistent with an individual
> + access or a small grouped access. */
> + if (acceptable_multiplier_p (expr, multiplier))
> + return INNER_LIKELY;
> + else
> + unlikely_p = true;
> + }
> + else if (gimple *stmt = maybe_get_stmt (expr))
> + {
> + /* If EXPR is set by a PHI node, queue its arguments in case
> + we find one that is consistent with an inner dimension.
> +
> + An important instance of this is the Fortran handling of array
> + descriptors, which calculates the stride of the inner dimension
> + using a PHI equivalent of:
> +
> + raw_stride = a.dim[0].stride;
> + stride = raw_stride != 0 ? raw_stride : 1;
> +
> + (Strides for outer dimensions do not treat 0 specially.) */
> + if (gphi *phi = dyn_cast <gphi *> (stmt))
> + {
> + unsigned int nargs = gimple_phi_num_args (phi);
> + for (unsigned int j = 0; j < nargs && length < MAX_NITERS; ++j)
> + worklist[length++] = strip_casts (gimple_phi_arg_def (phi, j));
> + }
> + /* If the value is set by an assignment, expect it to be read
> + from memory (such as an array descriptor) rather than be
> + calculated. */
> + else if (gassign *assign = dyn_cast <gassign *> (stmt))
> + {
> + if (!gimple_assign_load_p (assign))
> + unlikely_p = true;
> + }
> + /* Things like calls don't really tell us anything. */
> + }
> + }
> +
> + /* We didn't find any possible values of STRIDE that were likely to be
> + for the innermost dimension. If we found one that was actively
> + unlikely to be for the innermost dimension, assume that that applies
> + to STRIDE too. */
> + return unlikely_p ? INNER_UNLIKELY : INNER_DONT_KNOW;
> +}
> +
> +/* The caller has identified that STRIDE is the stride of interest
> + in TERM, and that the stride is applied in OP_LOOP. Record this
> + information in TERM, deciding whether STRIDE is likely to be for
> + the innermost dimension of an array and whether it represents a
> + versioning opportunity. ADDRESS is the address that contains TERM. */
> +
> +void
> +loop_versioning::analyze_stride (address_info &address,
> + address_term_info &term,
> + tree stride, struct loop *op_loop)
> +{
> + term.stride = stride;
> +
> + term.inner_likelihood = get_inner_likelihood (stride, term.multiplier);
> + if (dump_enabled_p ())
> + {
> + if (term.inner_likelihood == INNER_LIKELY)
> + dump_printf_loc (MSG_NOTE, address.stmt, "%T is likely to be the"
> + " innermost dimension\n", stride);
> + else if (term.inner_likelihood == INNER_UNLIKELY)
> + dump_printf_loc (MSG_NOTE, address.stmt, "%T is probably not the"
> + " innermost dimension\n", stride);
> + else
> + dump_printf_loc (MSG_NOTE, address.stmt, "cannot tell whether %T"
> + " is the innermost dimension\n", stride);
> + }
> +
> + /* To be a versioning opportunity we require:
> +
> + - The multiplier applied by TERM is equal to the access size,
> + so that when STRIDE is 1, the accesses in successive loop
> + iterations are consecutive.
> +
> + This is deliberately conservative. We could relax it to handle
> + other cases (such as those with gaps between iterations) if we
> + find any real testcases for which it's useful.
> +
> + - the stride is applied in the same loop as STMT rather than
> + in an outer loop. Although versioning for strides applied in
> + outer loops could help in some cases -- such as enabling
> + more loop interchange -- the savings are much lower than for
> + inner loops.
> +
> + - the stride is an SSA name that is invariant in STMT's loop,
> + since otherwise versioning isn't possible. */
> + unsigned HOST_WIDE_INT access_size = address.max_offset - address.min_offset;
> + if (term.multiplier == access_size
> + && address.loop == op_loop
> + && TREE_CODE (stride) == SSA_NAME
> + && expr_invariant_in_loop_p (address.loop, stride))
> + {
> + term.versioning_opportunity_p = true;
> + if (dump_enabled_p ())
> + dump_printf_loc (MSG_NOTE, address.stmt, "%T == 1 is a versioning"
> + " opportunity\n", stride);
> + }
> +}
> +
> +/* See whether address term TERM (which belongs to ADDRESS) is the result
> + of multiplying a varying SSA name by a loop-invariant SSA name.
> + Return true and update TERM if so.
> +
> + This handles both cases that SCEV might handle, such as:
> +
> + for (int i = 0; i < n; ++i)
> + res += a[i * stride];
> +
> + and ones in which the term varies arbitrarily between iterations, such as:
> +
> + for (int i = 0; i < n; ++i)
> + res += a[index[i] * stride]; */
> +
> +bool
> +loop_versioning
> +::find_per_loop_multiplication (address_info &address, address_term_info &term)
> +{
> + gimple *mult = maybe_get_assign (term.expr);
> + if (!mult || gimple_assign_rhs_code (mult) != MULT_EXPR)
> + return false;
> +
> + struct loop *mult_loop = loop_containing_stmt (mult);
> + if (!loop_outer (mult_loop))
> + return false;
> +
> + tree op1 = strip_casts (gimple_assign_rhs1 (mult));
> + tree op2 = strip_casts (gimple_assign_rhs2 (mult));
> + if (TREE_CODE (op1) != SSA_NAME || TREE_CODE (op2) != SSA_NAME)
> + return false;
> +
> + bool invariant1_p = expr_invariant_in_loop_p (mult_loop, op1);
> + bool invariant2_p = expr_invariant_in_loop_p (mult_loop, op2);
> + if (invariant1_p == invariant2_p)
> + return false;
> +
> + /* Make sure that the loop invariant is OP2 rather than OP1. */
> + if (invariant1_p)
> + std::swap (op1, op2);
> +
> + if (dump_enabled_p ())
> + dump_printf_loc (MSG_NOTE, address.stmt, "address term %T = varying %T"
> + " * loop-invariant %T\n", term.expr, op1, op2);
> + analyze_stride (address, term, op2, mult_loop);
> + return true;
> +}
> +
> +/* Try to use scalar evolutions to find an address stride for TERM,
> + which belongs to ADDRESS.
> +
> + Here we are interested in any evolution information we can find,
> + not just evolutions wrt ADDRESS->LOOP. For example, if we find that
> + an outer loop obviously iterates over the inner dimension of an array,
> + that information can help us eliminate worthless versioning opportunities
> + in inner loops. */
> +
> +void
> +loop_versioning::analyze_term_using_scevs (address_info &address,
> + address_term_info &term)
> +{
> + gimple *setter = maybe_get_stmt (term.expr);
> + if (!setter)
> + return;
> +
> + struct loop *wrt_loop = loop_containing_stmt (setter);
> + if (!loop_outer (wrt_loop))
> + return;
> +
> + tree chrec = strip_casts (analyze_scalar_evolution (wrt_loop, term.expr));
> + if (TREE_CODE (chrec) == POLYNOMIAL_CHREC)
> + {
> + if (dump_enabled_p ())
> + dump_printf_loc (MSG_NOTE, address.stmt,
> + "address term %T = %T\n", term.expr, chrec);
> +
> + /* Peel casts and accumulate constant multiplications, up to the
> + limit allowed by M_MAXIMUM_SCALE. */
> + tree stride = strip_casts (CHREC_RIGHT (chrec));
> + while (TREE_CODE (stride) == MULT_EXPR
> + && multiply_term_by (term, TREE_OPERAND (stride, 1)))
> + stride = strip_casts (TREE_OPERAND (stride, 0));
> +
> + gassign *assign;
> + while ((assign = maybe_get_assign (stride))
> + && gimple_assign_rhs_code (assign) == MULT_EXPR
> + && multiply_term_by (term, gimple_assign_rhs2 (assign)))
> + {
> + if (dump_enabled_p ())
> + dump_printf_loc (MSG_NOTE, address.stmt,
> + "looking through %G", assign);
> + stride = strip_casts (gimple_assign_rhs1 (assign));
> + }
> +
> + analyze_stride (address, term, stride, get_chrec_loop (chrec));
> + }
> +}
> +
> +/* Try to identify loop strides in ADDRESS and try to choose realistic
> + versioning opportunities based on these strides.
> +
> + The main difficulty here isn't finding strides that could be used
> + in a version check (that's pretty easy). The problem instead is to
> + avoid versioning for some stride S that is unlikely ever to be 1 at
> + runtime. Versioning for S == 1 on its own would lead to unnecessary
> + code bloat, while adding S == 1 to more realistic version conditions
> + would lose the optimisation opportunity offered by those other conditions.
> +
> + For example, versioning for a stride of 1 in the Fortran code:
> +
> + integer :: a(:,:)
> + a(1,:) = 1
> +
> + is not usually a good idea, since the assignment is iterating over
> + an outer dimension and is relatively unlikely to have a stride of 1.
> + (It isn't impossible, since the inner dimension might be 1, or the
> + array might be transposed.) Similarly, in:
> +
> + integer :: a(:,:), b(:,:)
> + b(:,1) = a(1,:)
> +
> + b(:,1) is relatively likely to have a stride of 1 while a(1,:) isn't.
> + Versioning for when both strides are 1 would lose most of the benefit
> + of versioning for b's access.
> +
> + The approach we take is as follows:
> +
> + - Analyze each term to see whether it has an identifiable stride,
> + regardless of which loop applies the stride.
> +
> + - Evaluate the likelihood that each such stride is for the innermost
> + dimension of an array, on the scale "likely", "don't know" or "unlikely".
> +
> + - If there is a single "likely" innermost stride, and that stride is
> + applied in the loop that contains STMT, version the loop for when the
> + stride is 1. This deals with the cases in which we're fairly
> + confident of doing the right thing, such as the b(:,1) reference above.
> +
> + - If there are no "likely" innermost strides, and the loop that contains
> + STMT uses a stride that we rated as "don't know", version for when
> + that stride is 1. This is principally used for C code such as:
> +
> + for (int i = 0; i < n; ++i)
> + a[i * x] = ...;
> +
> + and:
> +
> + for (int j = 0; j < n; ++j)
> + for (int i = 0; i < n; ++i)
> + a[i * x + j * y] = ...;
> +
> + where nothing in the way "x" and "y" are set gives a hint as to
> + whether "i" iterates over the innermost dimension of the array.
> + In these situations it seems reasonable to assume the the
> + programmer has nested the loops appropriately (although of course
> + there are examples like GEMM in which this assumption doesn't hold
> + for all accesses in the loop).
> +
> + This case is also useful for the Fortran equivalent of the
> + above C code. */
> +
> +void
> +loop_versioning::analyze_address_fragment (address_info &address)
> +{
> + if (dump_enabled_p ())
> + {
> + dump_printf_loc (MSG_NOTE, address.stmt, "analyzing address fragment ");
> + dump_address_info (MSG_NOTE, address);
> + dump_printf (MSG_NOTE, "\n");
> + }
> +
> + /* Analyze each component of the sum to see whether it involves an
> + apparent stride.
> +
> + There is an overlap between the addresses that
> + find_per_loop_multiplication and analyze_term_using_scevs can handle,
> + but the former is much cheaper than SCEV analysis, so try it first. */
> + for (unsigned int i = 0; i < address.terms.length (); ++i)
> + if (!find_per_loop_multiplication (address, address.terms[i]))
> + analyze_term_using_scevs (address, address.terms[i]);
> +
> + /* Check for strides that are likely to be for the innermost dimension.
> +
> + 1. If there is a single likely inner stride, if it is an SSA name,
> + and if it is worth versioning the loop for when the SSA name
> + equals 1, record that we want to do so.
> +
> + 2. Otherwise, if there any likely inner strides, bail out. This means
> + one of:
> +
> + (a) There are multiple likely inner strides. This suggests we're
> + confused and be can't be confident of doing the right thing.
> +
> + (b) There is a single likely inner stride and it is a constant
> + rather than an SSA name. This can mean either that the access
> + is a natural one without any variable strides, such as:
> +
> + for (int i = 0; i < n; ++i)
> + a[i] += 1;
> +
> + or that a variable stride is applied to an outer dimension,
> + such as:
> +
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + a[j * stride][i] += 1;
> +
> + (c) There is a single likely inner stride, and it is an SSA name,
> + but it isn't a worthwhile versioning opportunity. This usually
> + means that the variable stride is applied by an outer loop,
> + such as:
> +
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + a[j][i * stride] += 1;
> +
> + or (using an example with a more natural loop nesting):
> +
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + a[i][j] += b[i * stride];
> +
> + in cases where b[i * stride] cannot (yet) be hoisted for
> + aliasing reasons.
> +
> + 3. If there are no likely inner strides, fall through to the next
> + set of checks.
> +
> + Pointer equality is enough to check for uniqueness in (1), since we
> + only care about SSA names. */
> + tree chosen_stride = NULL_TREE;
> + tree version_stride = NULL_TREE;
> + for (unsigned int i = 0; i < address.terms.length (); ++i)
> + if (chosen_stride != address.terms[i].stride
> + && address.terms[i].inner_likelihood == INNER_LIKELY)
> + {
> + if (chosen_stride)
> + return;
> + chosen_stride = address.terms[i].stride;
> + if (address.terms[i].versioning_opportunity_p)
> + version_stride = chosen_stride;
> + }
> +
> + /* If there are no likely inner strides, see if there is a single
> + versioning opportunity for a stride that was rated as INNER_DONT_KNOW.
> + See the comment above the function for the cases that this code
> + handles. */
> + if (!chosen_stride)
> + for (unsigned int i = 0; i < address.terms.length (); ++i)
> + if (version_stride != address.terms[i].stride
> + && address.terms[i].inner_likelihood == INNER_DONT_KNOW
> + && address.terms[i].versioning_opportunity_p)
> + {
> + if (version_stride)
> + return;
> + version_stride = address.terms[i].stride;
> + }
> +
> + if (version_stride)
> + version_for_unity (address.stmt, version_stride);
> +}
> +
> +/* Treat EXPR * MULTIPLIER + OFFSET as a fragment of an address that addresses
> + TYPE_SIZE bytes and record this address fragment for later processing.
> + STMT is the statement that contains the address. */
> +
> +void
> +loop_versioning::record_address_fragment (gimple *stmt,
> + unsigned HOST_WIDE_INT type_size,
> + tree expr,
> + unsigned HOST_WIDE_INT multiplier,
> + HOST_WIDE_INT offset)
> +{
> + /* We're only interested in computed values. */
> + if (TREE_CODE (expr) != SSA_NAME)
> + return;
> +
> + /* Quick exit if no part of the address is calculated in STMT's loop,
> + since such addresses have no versioning opportunities. */
> + struct loop *loop = loop_containing_stmt (stmt);
> + if (expr_invariant_in_loop_p (loop, expr))
> + return;
> +
> + /* Set up an address_info for EXPR * MULTIPLIER. */
> + address_info *address = XOBNEW (&m_obstack, address_info);
> + new (address) address_info;
> + address->stmt = stmt;
> + address->loop = loop;
> + address->base = NULL_TREE;
> + address->terms.quick_grow (1);
> + address->terms[0].expr = expr;
> + address->terms[0].multiplier = multiplier;
> + address->terms[0].stride = NULL_TREE;
> + address->terms[0].inner_likelihood = INNER_UNLIKELY;
> + address->terms[0].versioning_opportunity_p = false;
> + address->min_offset = offset;
> +
> + /* Peel apart the expression into a sum of address_terms, where each
> + term is multiplied by a constant. Treat a + b and a - b the same,
> + since it doesn't matter for our purposes whether an address is
> + increasing or decreasing. Distribute (a + b) * constant into
> + a * constant + b * constant.
> +
> + We don't care which loop each term belongs to, since we want to
> + examine as many candidate strides as possible when determining
> + which is likely to be for the innermost dimension. We therefore
> + don't limit the search to statements in STMT's loop. */
> + for (unsigned int i = 0; i < address->terms.length (); )
> + {
> + if (gassign *assign = maybe_get_assign (address->terms[i].expr))
> + {
> + tree_code code = gimple_assign_rhs_code (assign);
> + if (code == PLUS_EXPR
> + || code == POINTER_PLUS_EXPR
> + || code == MINUS_EXPR)
> + {
> + tree op1 = gimple_assign_rhs1 (assign);
> + tree op2 = gimple_assign_rhs2 (assign);
> + if (TREE_CODE (op2) == INTEGER_CST)
> + {
> + address->terms[i].expr = strip_casts (op1);
> + /* This is heuristic only, so don't worry about truncation
> + or overflow. */
> + address->min_offset += (TREE_INT_CST_LOW (op2)
> + * address->terms[i].multiplier);
> + continue;
> + }
> + else if (address->terms.length () < address_info::MAX_TERMS)
> + {
> + unsigned int j = address->terms.length ();
> + address->terms.quick_push (address->terms[i]);
> + address->terms[i].expr = strip_casts (op1);
> + address->terms[j].expr = strip_casts (op2);
> + continue;
> + }
> + }
> + if (code == MULT_EXPR)
> + {
> + tree op1 = gimple_assign_rhs1 (assign);
> + tree op2 = gimple_assign_rhs2 (assign);
> + if (multiply_term_by (address->terms[i], op2))
> + {
> + address->terms[i].expr = strip_casts (op1);
> + continue;
> + }
> + }
> + }
> + i += 1;
> + }
> +
> + /* Peel off any symbolic pointer. */
> + if (TREE_CODE (address->terms[0].expr) != SSA_NAME
> + && address->terms[0].multiplier == 1)
> + {
> + if (address->terms.length () == 1)
> + {
> + obstack_free (&m_obstack, address);
> + return;
> + }
> + address->base = address->terms[0].expr;
> + address->terms.ordered_remove (0);
> + }
> +
> + /* Require all remaining terms to be SSA names. (This could be false
> + for unfolded statements, but they aren't worth dealing with.) */
> + for (unsigned int i = 0; i < address->terms.length (); ++i)
> + if (TREE_CODE (address->terms[i].expr) != SSA_NAME)
> + {
> + obstack_free (&m_obstack, address);
> + return;
> + }
> +
> + /* The loop above set MIN_OFFSET based on the first byte of the
> + referenced data. Calculate the end + 1. */
> + address->max_offset = address->min_offset + type_size;
> +
> + /* Put the terms into a canonical order for the hash table lookup below. */
> + address->terms.qsort (compare_address_terms);
> +
> + if (dump_enabled_p ())
> + {
> + dump_printf_loc (MSG_NOTE, stmt, "recording address fragment %T", expr);
> + if (multiplier != 1)
> + dump_printf (MSG_NOTE, " * %wd", multiplier);
> + dump_printf (MSG_NOTE, " = ");
> + dump_address_info (MSG_NOTE, *address);
> + dump_printf (MSG_NOTE, "\n");
> + }
> +
> + /* Pool address information with the same terms (but potentially
> + different offsets). */
> + address_info **slot = m_address_table.find_slot (address, INSERT);
> + if (address_info *old_address = *slot)
> + {
> + /* We've already seen an address with the same terms. Extend the
> + offset range to account for this access. Doing this can paper
> + over gaps, such as in:
> +
> + a[i * stride * 4] + a[i * stride * 4 + 3];
> +
> + where nothing references "+ 1" or "+ 2". However, the vectorizer
> + handles such gapped accesses without problems, so it's not worth
> + trying to exclude them. */
> + if (old_address->min_offset > address->min_offset)
> + old_address->min_offset = address->min_offset;
> + if (old_address->max_offset < address->max_offset)
> + old_address->max_offset = address->max_offset;
> + obstack_free (&m_obstack, address);
> + }
> + else
> + {
> + /* This is the first term we've seen an address with these terms. */
> + *slot = address;
> + m_address_list.safe_push (address);
> + }
> +}
> +
> +/* Analyze expression EXPR, which occurs in STMT. */
> +
> +void
> +loop_versioning::analyze_expr (gimple *stmt, tree expr)
> +{
> + unsigned HOST_WIDE_INT type_size;
> +
> + while (handled_component_p (expr))
> + {
> + /* See whether we can use versioning to avoid a multiplication
> + in an array index. */
> + if (TREE_CODE (expr) == ARRAY_REF
> + && acceptable_type_p (TREE_TYPE (expr), &type_size))
> + record_address_fragment (stmt, type_size,
> + TREE_OPERAND (expr, 1), type_size, 0);
> + expr = TREE_OPERAND (expr, 0);
> + }
> +
> + /* See whether we can use versioning to avoid a multiplication
> + in the pointer calculation of a MEM_REF. */
> + if (TREE_CODE (expr) == MEM_REF
> + && acceptable_type_p (TREE_TYPE (expr), &type_size))
> + record_address_fragment (stmt, type_size, TREE_OPERAND (expr, 0), 1,
> + /* This is heuristic only, so don't worry
> + about truncation or overflow. */
> + TREE_INT_CST_LOW (TREE_OPERAND (expr, 1)));
> +
> + /* These would be easy to handle if they existed at this stage. */
> + gcc_checking_assert (TREE_CODE (expr) != TARGET_MEM_REF);
> +}
> +
> +/* Analyze all the statements in BB looking for useful version checks.
> + Return true on success, false if something prevents the block from
> + being versioned. */
> +
> +bool
> +loop_versioning::analyze_block (basic_block bb)
> +{
> + struct loop *loop = bb->loop_father;
> + loop_info &li = get_loop_info (loop);
> + for (gimple_stmt_iterator gsi = gsi_start_bb (bb); !gsi_end_p (gsi);
> + gsi_next (&gsi))
> + {
> + gimple *stmt = gsi_stmt (gsi);
> + if (is_gimple_debug (stmt))
> + continue;
> +
> + if (expensive_stmt_p (stmt))
> + {
> + if (dump_enabled_p ())
> + dump_printf_loc (MSG_NOTE, stmt, "expensive statement"
> + " prevents versioning: %G", stmt);
> + return false;
> + }
> +
> + /* Only look for direct versioning opportunities in inner loops
> + since the benefit tends to be much smaller for outer loops. */
> + if (!loop->inner)
> + {
> + unsigned int nops = gimple_num_ops (stmt);
> + for (unsigned int i = 0; i < nops; ++i)
> + if (tree op = gimple_op (stmt, i))
> + analyze_expr (stmt, op);
> + }
> +
> + /* The point of the instruction limit is to prevent excessive
> + code growth, so this is a size-based estimate even though
> + the optimization is aimed at speed. */
> + li.num_insns += estimate_num_insns (stmt, &eni_size_weights);
> + }
> +
> + return true;
> +}
> +
> +/* Analyze all the blocks in the function, looking for useful version checks.
> + Return true if we found one. */
> +
> +bool
> +loop_versioning::analyze_blocks ()
> +{
> + AUTO_DUMP_SCOPE ("analyze_blocks",
> + dump_user_location_t::from_function_decl (m_fn->decl));
> +
> + /* For now we don't try to version the whole function, although
> + versioning at that level could be useful in some cases. */
> + get_loop_info (get_loop (m_fn, 0)).rejected_p = true;
> +
> + struct loop *loop;
> + FOR_EACH_LOOP (loop, LI_FROM_INNERMOST)
> + {
> + loop_info &linfo = get_loop_info (loop);
> +
> + /* See whether an inner loop prevents versioning of this loop. */
> + for (struct loop *inner = loop->inner; inner; inner = inner->next)
> + if (get_loop_info (inner).rejected_p)
> + {
> + linfo.rejected_p = true;
> + break;
> + }
> +
> + /* If versioning the loop is still a possibility, examine the
> + statements in the loop to look for versioning opportunities. */
> + if (!linfo.rejected_p)
> + {
> + void *start_point = obstack_alloc (&m_obstack, 0);
> +
> + for (basic_block bb = linfo.block_list; bb;
> + bb = m_next_block_in_loop[bb->index])
> + if (!analyze_block (bb))
> + {
> + linfo.rejected_p = true;
> + break;
> + }
> +
> + if (!linfo.rejected_p)
> + {
> + /* Process any queued address fragments, now that we have
> + complete grouping information. */
> + address_info *address;
> + unsigned int i;
> + FOR_EACH_VEC_ELT (m_address_list, i, address)
> + analyze_address_fragment (*address);
> + }
> +
> + m_address_table.empty ();
> + m_address_list.truncate (0);
> + obstack_free (&m_obstack, start_point);
> + }
> + }
> +
> + return m_num_conditions != 0;
> +}
> +
> +/* Use the ranges in VRS to remove impossible versioning conditions from
> + LOOP. */
> +
> +void
> +loop_versioning::prune_loop_conditions (struct loop *loop, vr_values *vrs)
> +{
> + loop_info &li = get_loop_info (loop);
> +
> + int to_remove = -1;
> + bitmap_iterator bi;
> + unsigned int i;
> + EXECUTE_IF_SET_IN_BITMAP (&li.unity_names, 0, i, bi)
> + {
> + tree name = ssa_name (i);
> + value_range *vr = vrs->get_value_range (name);
> + if (vr && !range_includes_p (vr, 1))
> + {
> + if (dump_enabled_p ())
> + dump_printf_loc (MSG_NOTE, find_loop_location (loop),
> + "%T can never be 1 in this loop\n", name);
> +
> + if (to_remove >= 0)
> + bitmap_clear_bit (&li.unity_names, to_remove);
> + to_remove = i;
> + m_num_conditions -= 1;
> + }
> + }
> + if (to_remove >= 0)
> + bitmap_clear_bit (&li.unity_names, to_remove);
> +}
> +
> +/* Remove any scheduled loop version conditions that will never be true.
> + Return true if any remain. */
> +
> +bool
> +loop_versioning::prune_conditions ()
> +{
> + AUTO_DUMP_SCOPE ("prune_loop_conditions",
> + dump_user_location_t::from_function_decl (m_fn->decl));
> +
> + calculate_dominance_info (CDI_DOMINATORS);
> + lv_dom_walker dom_walker (*this);
> + dom_walker.walk (ENTRY_BLOCK_PTR_FOR_FN (m_fn));
> + return m_num_conditions != 0;
> +}
> +
> +/* Merge the version checks for INNER into immediately-enclosing loop
> + OUTER. */
> +
> +void
> +loop_versioning::merge_loop_info (struct loop *outer, struct loop *inner)
> +{
> + loop_info &inner_li = get_loop_info (inner);
> + loop_info &outer_li = get_loop_info (outer);
> +
> + if (dump_enabled_p ())
> + {
> + bitmap_iterator bi;
> + unsigned int i;
> + EXECUTE_IF_SET_IN_BITMAP (&inner_li.unity_names, 0, i, bi)
> + if (!bitmap_bit_p (&outer_li.unity_names, i))
> + dump_printf_loc (MSG_NOTE, find_loop_location (inner),
> + "hoisting check that %T == 1 to outer loop\n",
> + ssa_name (i));
> + }
> +
> + bitmap_ior_into (&outer_li.unity_names, &inner_li.unity_names);
> + if (loop_depth (outer_li.outermost) < loop_depth (inner_li.outermost))
> + outer_li.outermost = inner_li.outermost;
> +}
> +
> +/* Add LOOP to the queue of loops to version. */
> +
> +void
> +loop_versioning::add_loop_to_queue (struct loop *loop)
> +{
> + loop_info &li = get_loop_info (loop);
> +
> + if (dump_enabled_p ())
> + dump_printf_loc (MSG_NOTE, find_loop_location (loop),
> + "queuing this loop for versioning\n");
> + m_loops_to_version.safe_push (loop);
> +
> + /* Don't try to version superloops. */
> + li.rejected_p = true;
> +}
> +
> +/* Decide whether the cost model would allow us to version LOOP,
> + either directly or as part of a parent loop, and return true if so.
> + This does not imply that the loop is actually worth versioning in its
> + own right, just that it would be valid to version it if something
> + benefited.
> +
> + We have already made this decision for all inner loops of LOOP. */
> +
> +bool
> +loop_versioning::decide_whether_loop_is_versionable (struct loop *loop)
> +{
> + loop_info &li = get_loop_info (loop);
> +
> + if (li.rejected_p)
> + return false;
> +
> + /* Examine the decisions made for inner loops. */
> + for (struct loop *inner = loop->inner; inner; inner = inner->next)
> + {
> + loop_info &inner_li = get_loop_info (inner);
> + if (inner_li.rejected_p)
> + {
> + if (dump_enabled_p ())
> + dump_printf_loc (MSG_NOTE, find_loop_location (loop),
> + "not versioning this loop because one of its"
> + " inner loops should not be versioned\n");
> + return false;
> + }
> +
> + if (inner_li.worth_versioning_p ())
> + li.subloops_benefit_p = true;
> +
> + /* Accumulate the number of instructions from subloops that are not
> + the innermost, or that don't benefit from versioning. Only the
> + instructions from innermost loops that benefit from versioning
> + should be weighed against loop-versioning-max-inner-insns;
> + everything else should be weighed against
> + loop-versioning-max-outer-insns. */
> + if (!inner_li.worth_versioning_p () || inner->inner)
> + {
> + if (dump_enabled_p ())
> + dump_printf_loc (MSG_NOTE, find_loop_location (loop),
> + "counting %d instructions from this loop"
> + " against its parent loop\n", inner_li.num_insns);
> + li.num_insns += inner_li.num_insns;
> + }
> + }
> +
> + /* Enforce the size limits. */
> + if (li.worth_versioning_p ())
> + {
> + unsigned int max_num_insns = max_insns_for_loop (loop);
> + if (dump_enabled_p ())
> + dump_printf_loc (MSG_NOTE, find_loop_location (loop),
> + "this loop has %d instructions, against"
> + " a versioning limit of %d\n",
> + li.num_insns, max_num_insns);
> + if (li.num_insns > max_num_insns)
> + {
> + if (dump_enabled_p ())
> + dump_printf_loc (MSG_MISSED_OPTIMIZATION
> + | MSG_PRIORITY_USER_FACING,
> + find_loop_location (loop),
> + "this loop is too big to version");
> + return false;
> + }
> + }
> +
> + /* Hoist all version checks from subloops to this loop. */
> + for (struct loop *subloop = loop->inner; subloop; subloop = subloop->next)
> + merge_loop_info (loop, subloop);
> +
> + return true;
> +}
> +
> +/* Decide which loops to version and add them to the versioning queue.
> + Return true if there are any loops to version. */
> +
> +bool
> +loop_versioning::make_versioning_decisions ()
> +{
> + AUTO_DUMP_SCOPE ("make_versioning_decisions",
> + dump_user_location_t::from_function_decl (m_fn->decl));
> +
> + struct loop *loop;
> + FOR_EACH_LOOP (loop, LI_FROM_INNERMOST)
> + {
> + loop_info &linfo = get_loop_info (loop);
> + if (decide_whether_loop_is_versionable (loop))
> + {
> + /* Commit to versioning LOOP directly if we can't hoist the
> + version checks any further. */
> + if (linfo.worth_versioning_p ()
> + && (loop_depth (loop) == 1 || linfo.outermost == loop))
> + add_loop_to_queue (loop);
> + }
> + else
> + {
> + /* We can't version this loop, so individually version any
> + subloops that would benefit and haven't been versioned yet. */
> + linfo.rejected_p = true;
> + for (struct loop *subloop = loop->inner; subloop;
> + subloop = subloop->next)
> + if (get_loop_info (subloop).worth_versioning_p ())
> + add_loop_to_queue (subloop);
> + }
> + }
> +
> + return !m_loops_to_version.is_empty ();
> +}
> +
> +/* Attempt to implement loop versioning for LOOP, using the information
> + cached in the associated loop_info. Return true on success. */
> +
> +bool
> +loop_versioning::version_loop (struct loop *loop)
> +{
> + loop_info &li = get_loop_info (loop);
> +
> + /* Build up a condition that selects the original loop instead of
> + the simplified loop. */
> + tree cond = boolean_false_node;
> + bitmap_iterator bi;
> + unsigned int i;
> + EXECUTE_IF_SET_IN_BITMAP (&li.unity_names, 0, i, bi)
> + {
> + tree name = ssa_name (i);
> + tree ne_one = fold_build2 (NE_EXPR, boolean_type_node, name,
> + build_one_cst (TREE_TYPE (name)));
> + cond = fold_build2 (TRUTH_OR_EXPR, boolean_type_node, cond, ne_one);
> + }
> +
> + /* Convert the condition into a suitable gcond. */
> + gimple_seq stmts = NULL;
> + cond = force_gimple_operand_1 (cond, &stmts, is_gimple_condexpr, NULL_TREE);
> +
> + /* Version the loop. */
> + initialize_original_copy_tables ();
> + basic_block cond_bb;
> + struct loop *else_loop
> + = loop_version (loop, cond, &cond_bb,
> + profile_probability::unlikely (),
> + profile_probability::likely (),
> + profile_probability::unlikely (),
> + profile_probability::likely (), true);
> + free_original_copy_tables ();
> + if (!else_loop)
> + {
> + if (dump_enabled_p ())
> + dump_printf_loc (MSG_MISSED_OPTIMIZATION, find_loop_location (loop),
> + "tried but failed to version this loop for when"
> + " certain strides are 1\n");
> + return false;
> + }
> +
> + if (dump_enabled_p ())
> + dump_printf_loc (MSG_OPTIMIZED_LOCATIONS, find_loop_location (loop),
> + "versioned this loop for when certain strides are 1\n");
> +
> + /* Insert the statements that feed COND. */
> + if (stmts)
> + {
> + gimple_stmt_iterator gsi = gsi_last_bb (cond_bb);
> + gsi_insert_seq_before (&gsi, stmts, GSI_SAME_STMT);
> + }
> +
> + /* Simplify the new loop, which is used when COND is false. */
> + name_prop (li).substitute_and_fold (else_loop->header);
> + return true;
> +}
> +
> +/* Attempt to version all loops in the versioning queue. Return true
> + if we succeeded for at least one loop. */
> +
> +bool
> +loop_versioning::implement_versioning_decisions ()
> +{
> + /* No AUTO_DUMP_SCOPE here since all messages are top-level and
> + user-facing at this point. */
> +
> + bool any_succeeded_p = false;
> +
> + struct loop *loop;
> + unsigned int i;
> + FOR_EACH_VEC_ELT (m_loops_to_version, i, loop)
> + if (version_loop (loop))
> + any_succeeded_p = true;
> +
> + return any_succeeded_p;
> +}
> +
> +/* Run the pass and return a set of TODO_* flags. */
> +
> +unsigned int
> +loop_versioning::run ()
> +{
> + gcc_assert (scev_initialized_p ());
> +
> + if (!analyze_blocks ()
> + || !prune_conditions ()
> + || !make_versioning_decisions ()
> + || !implement_versioning_decisions ())
> + return 0;
> +
> + return TODO_update_ssa;
> +}
> +
> +/* Loop versioning pass. */
> +
> +const pass_data pass_data_loop_versioning =
> +{
> + GIMPLE_PASS, /* type */
> + "lversion", /* name */
> + OPTGROUP_LOOP, /* optinfo_flags */
> + TV_LOOP_VERSIONING, /* tv_id */
> + PROP_cfg, /* properties_required */
> + 0, /* properties_provided */
> + 0, /* properties_destroyed */
> + 0, /* todo_flags_start */
> + 0, /* todo_flags_finish */
> +};
> +
> +class pass_loop_versioning : public gimple_opt_pass
> +{
> +public:
> + pass_loop_versioning (gcc::context *ctxt)
> + : gimple_opt_pass (pass_data_loop_versioning, ctxt)
> + {}
> +
> + /* opt_pass methods: */
> + virtual bool gate (function *) { return flag_version_loops_for_strides; }
> + virtual unsigned int execute (function *);
> +};
> +
> +unsigned int
> +pass_loop_versioning::execute (function *fn)
> +{
> + if (number_of_loops (fn) <= 1)
> + return 0;
> +
> + return loop_versioning (fn).run ();
> +}
> +
> +} // anon namespace
> +
> +gimple_opt_pass *
> +make_pass_loop_versioning (gcc::context *ctxt)
> +{
> + return new pass_loop_versioning (ctxt);
> +}
> Index: gcc/testsuite/gcc.dg/loop-versioning-1.c
> ===================================================================
> --- /dev/null 2018-11-29 13:15:04.463550658 +0000
> +++ gcc/testsuite/gcc.dg/loop-versioning-1.c 2018-12-06 12:33:59.216098398 +0000
> @@ -0,0 +1,92 @@
> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
> +
> +/* The simplest IV case. */
> +
> +void
> +f1 (double *x, int stepx, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + x[stepx * i] = 100;
> +}
> +
> +void
> +f2 (double *x, int stepx, int limit)
> +{
> + for (int i = 0; i < limit; i += stepx)
> + x[i] = 100;
> +}
> +
> +void
> +f3 (double *x, int stepx, int limit)
> +{
> + for (double *y = x; y < x + limit; y += stepx)
> + *y = 100;
> +}
> +
> +void
> +f4 (double *x, int stepx, unsigned int n)
> +{
> + for (unsigned int i = 0; i < n; ++i)
> + x[stepx * i] = 100;
> +}
> +
> +void
> +f5 (double *x, int stepx, unsigned int limit)
> +{
> + for (unsigned int i = 0; i < limit; i += stepx)
> + x[i] = 100;
> +}
> +
> +void
> +f6 (double *x, int stepx, unsigned int limit)
> +{
> + for (double *y = x; y < x + limit; y += stepx)
> + *y = 100;
> +}
> +
> +double x[10000];
> +
> +void
> +g1 (int stepx, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + x[stepx * i] = 100;
> +}
> +
> +void
> +g2 (int stepx, int limit)
> +{
> + for (int i = 0; i < limit; i += stepx)
> + x[i] = 100;
> +}
> +
> +void
> +g3 (int stepx, int limit)
> +{
> + for (double *y = x; y < x + limit; y += stepx)
> + *y = 100;
> +}
> +
> +void
> +g4 (int stepx, unsigned int n)
> +{
> + for (unsigned int i = 0; i < n; ++i)
> + x[stepx * i] = 100;
> +}
> +
> +void
> +g5 (int stepx, unsigned int limit)
> +{
> + for (unsigned int i = 0; i < limit; i += stepx)
> + x[i] = 100;
> +}
> +
> +void
> +g6 (int stepx, unsigned int limit)
> +{
> + for (double *y = x; y < x + limit; y += stepx)
> + *y = 100;
> +}
> +
> +/* { dg-final { scan-tree-dump-times {want to version containing loop} 12 "lversion" } } */
> +/* { dg-final { scan-tree-dump-times {versioned this loop} 12 "lversion" } } */
> Index: gcc/testsuite/gcc.dg/loop-versioning-10.c
> ===================================================================
> --- /dev/null 2018-11-29 13:15:04.463550658 +0000
> +++ gcc/testsuite/gcc.dg/loop-versioning-10.c 2018-12-06 12:33:59.216098398 +0000
> @@ -0,0 +1,52 @@
> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
> +
> +/* Test that we can version a gather-like operation in which a variable
> + stride is applied to the index. */
> +
> +int
> +f1 (int *x, int *index, int step, int n)
> +{
> + int res = 0;
> + for (int i = 0; i < n; ++i)
> + res += x[index[i] * step];
> + return res;
> +}
> +
> +int
> +f2 (int *x, int *index, int step, int n)
> +{
> + int res = 0;
> + for (int i = 0; i < n; ++i)
> + {
> + int *ptr = x + index[i] * step;
> + res += *ptr;
> + }
> + return res;
> +}
> +
> +int x[1000];
> +
> +int
> +g1 (int *index, int step, int n)
> +{
> + int res = 0;
> + for (int i = 0; i < n; ++i)
> + res += x[index[i] * step];
> + return res;
> +}
> +
> +int
> +g2 (int *index, int step, int n)
> +{
> + int res = 0;
> + for (int i = 0; i < n; ++i)
> + {
> + int *ptr = x + index[i] * step;
> + res += *ptr;
> + }
> + return res;
> +}
> +
> +/* { dg-final { scan-tree-dump-times {address term [^\n]* \* loop-invariant} 4 "lversion" } } */
> +/* { dg-final { scan-tree-dump-times {want to version containing loop} 4 "lversion" } } */
> +/* { dg-final { scan-tree-dump-times {versioned this loop} 4 "lversion" } } */
> Index: gcc/testsuite/gcc.dg/loop-versioning-11.c
> ===================================================================
> --- /dev/null 2018-11-29 13:15:04.463550658 +0000
> +++ gcc/testsuite/gcc.dg/loop-versioning-11.c 2018-12-06 12:33:59.216098398 +0000
> @@ -0,0 +1,29 @@
> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
> +
> +/* Test that we don't try to version for something that is never 1. */
> +
> +void
> +f1 (double *x, int stepx, int n)
> +{
> + if (stepx == 1)
> + for (int i = 0; i < n; ++i)
> + x[i] = 100;
> + else
> + for (int i = 0; i < n; ++i)
> + x[stepx * i] = 100;
> +}
> +
> +void
> +f2 (double *x, int stepx, int n)
> +{
> + if (stepx <= 1)
> + for (int i = 0; i < n; ++i)
> + x[i] = 100;
> + else
> + for (int i = 0; i < n; ++i)
> + x[stepx * i] = 100;
> +}
> +
> +/* { dg-final { scan-tree-dump-times {want to version containing loop} 2 "lversion" } } */
> +/* { dg-final { scan-tree-dump-times {can never be 1} 2 "lversion" } } */
> +/* { dg-final { scan-tree-dump-not {versioned} "lversion" } } */
> Index: gcc/testsuite/gcc.dg/loop-versioning-2.c
> ===================================================================
> --- /dev/null 2018-11-29 13:15:04.463550658 +0000
> +++ gcc/testsuite/gcc.dg/loop-versioning-2.c 2018-12-06 12:33:59.216098398 +0000
> @@ -0,0 +1,73 @@
> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
> +
> +/* Versioning for step == 1 in these loops would allow loop interchange,
> + but otherwise isn't worthwhile. At the moment we decide not to version. */
> +
> +void
> +f1 (double x[][100], int step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + x[j * step][i] = 100;
> +}
> +
> +void
> +f2 (double x[][100], int step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + x[j][i * step] = 100;
> +}
> +
> +void
> +f3 (double x[][100], int step, int limit)
> +{
> + for (int i = 0; i < 100; ++i)
> + for (int j = 0; j < limit; j += step)
> + x[j][i] = 100;
> +}
> +
> +void
> +f4 (double x[][100], int step, int limit)
> +{
> + for (int i = 0; i < limit; i += step)
> + for (int j = 0; j < 100; ++j)
> + x[j][i] = 100;
> +}
> +
> +double x[100][100];
> +
> +void
> +g1 (int step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + x[j * step][i] = 100;
> +}
> +
> +void
> +g2 (int step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + x[j][i * step] = 100;
> +}
> +
> +void
> +g3 (int step, int limit)
> +{
> + for (int i = 0; i < 100; ++i)
> + for (int j = 0; j < limit; j += step)
> + x[j][i] = 100;
> +}
> +
> +void
> +g4 (int step, int limit)
> +{
> + for (int i = 0; i < limit; i += step)
> + for (int j = 0; j < 100; ++j)
> + x[j][i] = 100;
> +}
> +
> +/* { dg-final { scan-tree-dump-not {want to version} "lversion" } } */
> +/* { dg-final { scan-tree-dump-not {versioned} "lversion" } } */
> Index: gcc/testsuite/gcc.dg/loop-versioning-3.c
> ===================================================================
> --- /dev/null 2018-11-29 13:15:04.463550658 +0000
> +++ gcc/testsuite/gcc.dg/loop-versioning-3.c 2018-12-06 12:33:59.216098398 +0000
> @@ -0,0 +1,24 @@
> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
> +
> +/* Versioning these loops for when both steps are 1 allows loop
> + interchange, but otherwise isn't worthwhile. At the moment we decide
> + not to version. */
> +
> +void
> +f1 (double x[][100], int step1, int step2, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + x[j * step1][i * step2] = 100;
> +}
> +
> +void
> +f2 (double x[][100], int step1, int step2, int limit)
> +{
> + for (int i = 0; i < limit; i += step1)
> + for (int j = 0; j < limit; j += step2)
> + x[j][i] = 100;
> +}
> +
> +/* { dg-final { scan-tree-dump-not {want to version} "lversion" } } */
> +/* { dg-final { scan-tree-dump-not {versioned} "lversion" } } */
> Index: gcc/testsuite/gcc.dg/loop-versioning-4.c
> ===================================================================
> --- /dev/null 2018-11-29 13:15:04.463550658 +0000
> +++ gcc/testsuite/gcc.dg/loop-versioning-4.c 2018-12-06 12:33:59.216098398 +0000
> @@ -0,0 +1,39 @@
> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
> +
> +/* These shouldn't be versioned; it's extremely likely that the code
> + is emulating two-dimensional arrays. */
> +
> +void
> +f1 (double *x, int step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + x[i * step + j] = 100;
> +}
> +
> +void
> +f2 (double *x, int step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + x[j * step + i] = 100;
> +}
> +
> +void
> +f3 (double *x, int *offsets, int step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + x[i * step + j + offsets[i]] = 100;
> +}
> +
> +void
> +f4 (double *x, int *offsets, int step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + x[j * step + i + offsets[i]] = 100;
> +}
> +
> +/* { dg-final { scan-tree-dump-not {want to version} "lversion" } } */
> +/* { dg-final { scan-tree-dump-not {versioned} "lversion" } } */
> Index: gcc/testsuite/gcc.dg/loop-versioning-5.c
> ===================================================================
> --- /dev/null 2018-11-29 13:15:04.463550658 +0000
> +++ gcc/testsuite/gcc.dg/loop-versioning-5.c 2018-12-06 12:33:59.216098398 +0000
> @@ -0,0 +1,17 @@
> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
> +
> +/* There's no information about whether STEP1 or STEP2 is innermost,
> + so we should assume the code is sensible and version for the inner
> + evolution, i.e. when STEP2 is 1. */
> +
> +void
> +f1 (double *x, int step1, int step2, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + x[i * step1 + j * step2] = 100;
> +}
> +
> +/* { dg-final { scan-tree-dump-times {want to version containing loop for when step2} 1 "lversion" } } */
> +/* { dg-final { scan-tree-dump-times {want to version containing loop} 1 "lversion" } } */
> +/* { dg-final { scan-tree-dump-times {versioned this loop} 1 "lversion" } } */
> Index: gcc/testsuite/gcc.dg/loop-versioning-6.c
> ===================================================================
> --- /dev/null 2018-11-29 13:15:04.463550658 +0000
> +++ gcc/testsuite/gcc.dg/loop-versioning-6.c 2018-12-06 12:33:59.216098398 +0000
> @@ -0,0 +1,31 @@
> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
> +
> +/* The read from y in f1 will be hoisted to the outer loop. In general
> + it's not worth versioning outer loops when the inner loops don't also
> + benefit.
> +
> + This test is meant to be a slight counterexample, since versioning
> + does lead to cheaper outer-loop vectorization. However, the benefit
> + isn't enough to justify the cost. */
> +
> +void
> +f1 (double *restrict x, double *restrict y, int step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + x[i + j] = y[i * step];
> +}
> +
> +/* A similar example in which the read can't be hoisted, but could
> + for example be handled by vectorizer alias checks. */
> +
> +void
> +f2 (double *x, double *y, int step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + x[i + j] = y[i * step];
> +}
> +
> +/* { dg-final { scan-tree-dump-not {want to version} "lversion" } } */
> +/* { dg-final { scan-tree-dump-not {versioned} "lversion" } } */
> Index: gcc/testsuite/gcc.dg/loop-versioning-7.c
> ===================================================================
> --- /dev/null 2018-11-29 13:15:04.463550658 +0000
> +++ gcc/testsuite/gcc.dg/loop-versioning-7.c 2018-12-06 12:33:59.216098398 +0000
> @@ -0,0 +1,32 @@
> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
> +
> +/* Check that versioning can handle arrays of structures. */
> +
> +struct foo {
> + int a, b, c;
> +};
> +
> +void
> +f1 (struct foo *x, int stepx, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + {
> + x[stepx * i].a = 1;
> + x[stepx * i].b = 2;
> + x[stepx * i].c = 3;
> + }
> +}
> +
> +void
> +f2 (struct foo *x, int stepx, int limit)
> +{
> + for (int i = 0; i < limit; i += stepx)
> + {
> + x[i].a = 1;
> + x[i].b = 2;
> + x[i].c = 3;
> + }
> +}
> +
> +/* { dg-final { scan-tree-dump-times {want to version containing loop} 2 "lversion" } } */
> +/* { dg-final { scan-tree-dump-times {versioned this loop} 2 "lversion" } } */
> Index: gcc/testsuite/gcc.dg/loop-versioning-8.c
> ===================================================================
> --- /dev/null 2018-11-29 13:15:04.463550658 +0000
> +++ gcc/testsuite/gcc.dg/loop-versioning-8.c 2018-12-06 12:33:59.216098398 +0000
> @@ -0,0 +1,43 @@
> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
> +
> +/* Versioning for step == 1 in these loops would allow loop interchange,
> + but otherwise isn't worthwhile. At the moment we decide not to version. */
> +
> +struct foo {
> + int a[100];
> +};
> +
> +void
> +f1 (struct foo *x, int step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + x[j * step].a[i] = 100;
> +}
> +
> +void
> +f2 (struct foo *x, int step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + x[j].a[i * step] = 100;
> +}
> +
> +void
> +f3 (struct foo *x, int step, int limit)
> +{
> + for (int i = 0; i < 100; ++i)
> + for (int j = 0; j < limit; j += step)
> + x[j].a[i] = 100;
> +}
> +
> +void
> +f4 (struct foo *x, int step, int limit)
> +{
> + for (int i = 0; i < limit; i += step)
> + for (int j = 0; j < 100; ++j)
> + x[j].a[i] = 100;
> +}
> +
> +/* { dg-final { scan-tree-dump-not {want to version} "lversion" } } */
> +/* { dg-final { scan-tree-dump-not {versioned} "lversion" } } */
> Index: gcc/testsuite/gcc.dg/loop-versioning-9.c
> ===================================================================
> --- /dev/null 2018-11-29 13:15:04.463550658 +0000
> +++ gcc/testsuite/gcc.dg/loop-versioning-9.c 2018-12-06 12:33:59.220098363 +0000
> @@ -0,0 +1,48 @@
> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
> +
> +/* Check that versioning can handle small groups of accesses. */
> +
> +void
> +f1 (int *x, int *y, int step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + x[i] = y[i * step * 2] + y[i * step * 2 + 1];
> +}
> +
> +void
> +f2 (int *x, int *y, __INTPTR_TYPE__ step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + x[i] = y[i * step * 2] + y[i * step * 2 + 1];
> +}
> +
> +void
> +f3 (int *x, int *y, int step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + x[i] = y[i * step * 3] + y[i * step * 3 + 2];
> +}
> +
> +void
> +f4 (int *x, int *y, __INTPTR_TYPE__ step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + x[i] = y[i * step * 3] + y[i * step * 3 + 2];
> +}
> +
> +void
> +f5 (int *x, int *y, int step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + x[i] = y[i * step * 4] + y[i * step * 4 + 3];
> +}
> +
> +void
> +f6 (int *x, int *y, __INTPTR_TYPE__ step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + x[i] = y[i * step * 4] + y[i * step * 4 + 3];
> +}
> +
> +/* { dg-final { scan-tree-dump-times {want to version containing loop} 6 "lversion" } } */
> +/* { dg-final { scan-tree-dump-times {versioned this loop} 6 "lversion" } } */
> Index: gcc/testsuite/gfortran.dg/loop_versioning_1.f90
> ===================================================================
> --- /dev/null 2018-11-29 13:15:04.463550658 +0000
> +++ gcc/testsuite/gfortran.dg/loop_versioning_1.f90 2018-12-06 12:33:59.220098363 +0000
> @@ -0,0 +1,28 @@
> +! { dg-options "-O3 -fdump-tree-lversion-details" }
> +
> +! The simplest IV case.
> +
> +subroutine f1(x)
> + real :: x(:)
> + x(:) = 100
> +end subroutine f1
> +
> +subroutine f2(x, n, step)
> + integer :: n, step
> + real :: x(n * step)
> + do i = 1, n
> + x(i * step) = 100
> + end do
> +end subroutine f2
> +
> +subroutine f3(x, limit, step)
> + integer :: limit, step
> + real :: x(limit)
> + do i = 1, limit, step
> + x(i) = 100
> + end do
> +end subroutine f3
> +
> +! { dg-final { scan-tree-dump-times {likely to be the innermost dimension} 1 "lversion" } }
> +! { dg-final { scan-tree-dump-times {want to version containing loop} 3 "lversion" } }
> +! { dg-final { scan-tree-dump-times {versioned this loop} 3 "lversion" } }
> Index: gcc/testsuite/gfortran.dg/loop_versioning_2.f90
> ===================================================================
> --- /dev/null 2018-11-29 13:15:04.463550658 +0000
> +++ gcc/testsuite/gfortran.dg/loop_versioning_2.f90 2018-12-06 12:33:59.220098363 +0000
> @@ -0,0 +1,39 @@
> +! { dg-options "-O3 -fdump-tree-lversion-details -fno-frontend-loop-interchange" }
> +
> +! We could version the loop for when the first dimension has a stride
> +! of 1, but at present there's no real benefit. The gimple loop
> +! interchange pass couldn't handle the versioned loop, and interchange
> +! is instead done by the frontend (but disabled by the options above).
> +
> +subroutine f1(x)
> + real :: x(:, :)
> + do i = lbound(x, 1), ubound(x, 1)
> + do j = lbound(x, 2), ubound(x, 2)
> + x(i, j) = 100
> + end do
> + end do
> +end subroutine f1
> +
> +subroutine f2(x, n, step)
> + integer :: n, step
> + real :: x(100, 100)
> + do i = 1, n
> + do j = 1, n
> + x(i * step, j) = 100
> + end do
> + end do
> +end subroutine f2
> +
> +subroutine f3(x, n, step)
> + integer :: n, step
> + real :: x(n * step, n)
> + do i = 1, n
> + do j = 1, n
> + x(i * step, j) = 100
> + end do
> + end do
> +end subroutine f3
> +
> +! { dg-final { scan-tree-dump-times {likely to be the innermost dimension} 1 "lversion" } }
> +! { dg-final { scan-tree-dump-not {want to version} "lversion" } }
> +! { dg-final { scan-tree-dump-not {versioned} "lversion" } }
> Index: gcc/testsuite/gfortran.dg/loop_versioning_3.f90
> ===================================================================
> --- /dev/null 2018-11-29 13:15:04.463550658 +0000
> +++ gcc/testsuite/gfortran.dg/loop_versioning_3.f90 2018-12-06 12:33:59.220098363 +0000
> @@ -0,0 +1,30 @@
> +! { dg-options "-O3 -fdump-tree-lversion-details -fno-frontend-loop-interchange" }
> +
> +! Test a case in which the outer loop iterates over the inner dimension.
> +! The options above prevent the frontend from interchanging the loops.
> +
> +subroutine f1(x, limit, step, n)
> + integer :: limit, step, n
> + real :: x(limit, n)
> + do i = 1, limit, step
> + do j = 1, n
> + x(i, j) = 100
> + end do
> + end do
> +end subroutine f1
> +
> +subroutine f2(x, n, limit, step)
> + integer :: n, limit, step
> + real :: x(limit, n)
> + do i = 1, n
> + do j = 1, limit, step
> + x(j, i) = 100
> + end do
> + end do
> +end subroutine f2
> +
> +! FIXME: The frontend doesn't give us enough information to tell which loop
> +! is iterating over the innermost dimension, so we optimistically
> +! assume the inner one is.
> +! { dg-final { scan-tree-dump-not {want to version} "lversion" { xfail *-*-* } } }
> +! { dg-final { scan-tree-dump-not {versioned} "lversion" { xfail *-*-* } } }
> Index: gcc/testsuite/gfortran.dg/loop_versioning_4.f90
> ===================================================================
> --- /dev/null 2018-11-29 13:15:04.463550658 +0000
> +++ gcc/testsuite/gfortran.dg/loop_versioning_4.f90 2018-12-06 12:33:59.220098363 +0000
> @@ -0,0 +1,52 @@
> +! { dg-options "-O3 -fdump-tree-lversion-details -fno-frontend-loop-interchange" }
> +
> +! Test cases in which versioning is useful for a two-dimensional array.
> +
> +subroutine f1(x)
> + real :: x(:, :)
> + x(:, :) = 100
> +end subroutine f1
> +
> +subroutine f2(x)
> + real :: x(:, :)
> + do i = lbound(x, 1), ubound(x, 1)
> + do j = lbound(x, 2), ubound(x, 2)
> + x(j, i) = 100
> + end do
> + end do
> +end subroutine f2
> +
> +subroutine f3(x, n, step)
> + integer :: n, step
> + real :: x(100, 100)
> + do i = 1, n
> + do j = 1, n
> + x(j * step, i) = 100
> + end do
> + end do
> +end subroutine f3
> +
> +subroutine f4(x, n, step)
> + integer :: n, step
> + real :: x(n * step, n)
> + do i = 1, n
> + do j = 1, n
> + x(j * step, i) = 100
> + end do
> + end do
> +end subroutine f4
> +
> +subroutine f5(x, n, limit, step)
> + integer :: n, limit, step
> + real :: x(limit, n)
> + do i = 1, n
> + do j = 1, limit, step
> + x(j, i) = 100
> + end do
> + end do
> +end subroutine f5
> +
> +! { dg-final { scan-tree-dump-times {likely to be the innermost dimension} 2 "lversion" } }
> +! { dg-final { scan-tree-dump-times {want to version containing loop} 5 "lversion" } }
> +! { dg-final { scan-tree-dump-times {hoisting check} 5 "lversion" } }
> +! { dg-final { scan-tree-dump-times {versioned this loop} 5 "lversion" } }
> Index: gcc/testsuite/gfortran.dg/loop_versioning_5.f90
> ===================================================================
> --- /dev/null 2018-11-29 13:15:04.463550658 +0000
> +++ gcc/testsuite/gfortran.dg/loop_versioning_5.f90 2018-12-06 12:33:59.220098363 +0000
> @@ -0,0 +1,57 @@
> +! { dg-options "-O3 -fdump-tree-lversion-details -fno-frontend-loop-interchange" }
> +
> +! Make sure that in a "badly nested" loop, we don't treat the inner loop
> +! as iterating over the inner dimension with a variable stride.
> +
> +subroutine f1(x, n)
> + integer :: n
> + real :: x(100, 100)
> + do i = 1, n
> + do j = 1, n
> + x(i, j) = 100
> + end do
> + end do
> +end subroutine f1
> +
> +subroutine f2(x, n, step)
> + integer :: n, step
> + real :: x(100, 100)
> + do i = 1, n
> + do j = 1, n
> + x(i, j * step) = 100
> + end do
> + end do
> +end subroutine f2
> +
> +subroutine f3(x, n)
> + integer :: n
> + real :: x(n, n)
> + do i = 1, n
> + do j = 1, n
> + x(i, j) = 100
> + end do
> + end do
> +end subroutine f3
> +
> +subroutine f4(x, n, step)
> + integer :: n, step
> + real :: x(n, n * step)
> + do i = 1, n
> + do j = 1, n
> + x(i, j * step) = 100
> + end do
> + end do
> +end subroutine f4
> +
> +subroutine f5(x, n, limit, step)
> + integer :: n, limit, step
> + real :: x(n, limit)
> + do i = 1, n
> + do j = 1, limit, step
> + x(i, j) = 100
> + end do
> + end do
> +end subroutine f5
> +
> +! { dg-final { scan-tree-dump-not {want to version} "lversion" } }
> +! { dg-final { scan-tree-dump-not {versioned} "lversion" } }
> Index: gcc/testsuite/gfortran.dg/loop_versioning_6.f90
> ===================================================================
> --- /dev/null 2018-11-29 13:15:04.463550658 +0000
> +++ gcc/testsuite/gfortran.dg/loop_versioning_6.f90 2018-12-06 12:33:59.220098363 +0000
> @@ -0,0 +1,93 @@
> +! { dg-options "-O3 -fdump-tree-lversion-details" }
> +
> +! Check that versioning can handle small groups of accesses.
> +
> +subroutine f1(x)
> + real :: x(:)
> + do i = lbound(x, 1), ubound(x, 1) / 2
> + x(i * 2) = 100
> + x(i * 2 + 1) = 101
> + end do
> +end subroutine f1
> +
> +subroutine f2(x, n, step)
> + integer :: n, step
> + real :: x(n * step * 2)
> + do i = 1, n
> + x(i * step * 2) = 100
> + x(i * step * 2 + 1) = 101
> + end do
> +end subroutine f2
> +
> +subroutine f3(x, limit, step)
> + integer :: limit, step
> + real :: x(limit * 2)
> + do i = 1, limit, step
> + x(i * 2) = 100
> + x(i * 2 + 1) = 101
> + end do
> +end subroutine f3
> +
> +subroutine f4(x)
> + real :: x(:)
> + do i = lbound(x, 1), ubound(x, 1) / 3
> + x(i * 3) = 100
> + x(i * 3 + 1) = 101
> + x(i * 3 + 2) = 102
> + end do
> +end subroutine f4
> +
> +subroutine f5(x, n, step)
> + integer :: n, step
> + real :: x(n * step * 3)
> + do i = 1, n
> + x(i * step * 3) = 100
> + x(i * step * 3 + 1) = 101
> + x(i * step * 3 + 2) = 102
> + end do
> +end subroutine f5
> +
> +subroutine f6(x, limit, step)
> + integer :: limit, step
> + real :: x(limit * 3)
> + do i = 1, limit, step
> + x(i * 3) = 100
> + x(i * 3 + 1) = 101
> + x(i * 3 + 2) = 102
> + end do
> +end subroutine f6
> +
> +subroutine f7(x)
> + real :: x(:)
> + do i = lbound(x, 1), ubound(x, 1) / 4
> + x(i * 4) = 100
> + x(i * 4 + 1) = 101
> + x(i * 4 + 2) = 102
> + x(i * 4 + 3) = 103
> + end do
> +end subroutine f7
> +
> +subroutine f8(x, n, step)
> + integer :: n, step
> + real :: x(n * step * 4)
> + do i = 1, n
> + x(i * step * 4) = 100
> + x(i * step * 4 + 1) = 101
> + x(i * step * 4 + 2) = 102
> + x(i * step * 4 + 3) = 103
> + end do
> +end subroutine f8
> +
> +subroutine f9(x, limit, step)
> + integer :: limit, step
> + real :: x(limit * 4)
> + do i = 1, limit, step
> + x(i * 4) = 100
> + x(i * 4 + 1) = 101
> + x(i * 4 + 2) = 102
> + x(i * 4 + 3) = 103
> + end do
> +end subroutine f9
> +
> +! { dg-final { scan-tree-dump-times {want to version containing loop} 9 "lversion" } }
> +! { dg-final { scan-tree-dump-times {versioned this loop} 9 "lversion" } }
> Index: gcc/testsuite/gfortran.dg/loop_versioning_7.f90
> ===================================================================
> --- /dev/null 2018-11-29 13:15:04.463550658 +0000
> +++ gcc/testsuite/gfortran.dg/loop_versioning_7.f90 2018-12-06 12:33:59.220098363 +0000
> @@ -0,0 +1,67 @@
> +! { dg-options "-O3 -fdump-tree-lversion-details" }
> +
> +! Check that versioning can handle small groups of accesses, with the
> +! group being a separate array dimension.
> +
> +subroutine f1(x, n, step)
> + integer :: n, step
> + real :: x(2, n * step)
> + do i = 1, n
> + x(1, i * step) = 100
> + x(2, i * step) = 101
> + end do
> +end subroutine f1
> +
> +subroutine f2(x, limit, step)
> + integer :: limit, step
> + real :: x(2, limit)
> + do i = 1, limit, step
> + x(1, i) = 100
> + x(2, i) = 101
> + end do
> +end subroutine f2
> +
> +subroutine f3(x, n, step)
> + integer :: n, step
> + real :: x(3, n * step)
> + do i = 1, n
> + x(1, i * step) = 100
> + x(2, i * step) = 101
> + x(3, i * step) = 102
> + end do
> +end subroutine f3
> +
> +subroutine f4(x, limit, step)
> + integer :: limit, step
> + real :: x(3, limit)
> + do i = 1, limit, step
> + x(1, i) = 100
> + x(2, i) = 101
> + x(3, i) = 102
> + end do
> +end subroutine f4
> +
> +subroutine f5(x, n, step)
> + integer :: n, step
> + real :: x(4, n * step)
> + do i = 1, n
> + x(1, i * step) = 100
> + x(2, i * step) = 101
> + x(3, i * step) = 102
> + x(4, i * step) = 103
> + end do
> +end subroutine f5
> +
> +subroutine f6(x, limit, step)
> + integer :: limit, step
> + real :: x(4, limit)
> + do i = 1, limit, step
> + x(1, i) = 100
> + x(2, i) = 101
> + x(3, i) = 102
> + x(4, i) = 103
> + end do
> +end subroutine f6
> +
> +! { dg-final { scan-tree-dump-times {want to version containing loop} 6 "lversion" } }
> +! { dg-final { scan-tree-dump-times {versioned this loop} 6 "lversion" } }
> Index: gcc/testsuite/gfortran.dg/loop_versioning_8.f90
> ===================================================================
> --- /dev/null 2018-11-29 13:15:04.463550658 +0000
> +++ gcc/testsuite/gfortran.dg/loop_versioning_8.f90 2018-12-06 12:33:59.220098363 +0000
> @@ -0,0 +1,13 @@
> +! { dg-options "-O3 -fdump-tree-lversion-details" }
> +
> +! Check that versioning is applied to a gather-like reduction operation.
> +
> +function f(x, index, n)
> + integer :: n
> + real :: x(:)
> + integer :: index(n)
> + f = sum(x(index(:)))
> +end function f
> +
> +! { dg-final { scan-tree-dump-times {want to version containing loop} 1 "lversion" } }
> +! { dg-final { scan-tree-dump-times {versioned this loop} 1 "lversion" } }
> Index: gcc/testsuite/gcc.dg/loop-versioning-12.c
> ===================================================================
> --- /dev/null 2018-11-29 13:15:04.463550658 +0000
> +++ gcc/testsuite/gcc.dg/loop-versioning-12.c 2018-12-06 12:33:59.216098398 +0000
> @@ -0,0 +1,149 @@
> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
> +
> +/* Test that we don't try to version for a step of 1 when that would
> + cause the iterations to overlap. */
> +
> +void
> +f1 (unsigned short *x, int stepx, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + {
> + x[i * stepx] = 100;
> + x[i * stepx + 1] = 99;
> + }
> +}
> +
> +void
> +f2 (unsigned short *x, int stepx, int n)
> +{
> + for (int i = 0; i < n; i += stepx)
> + {
> + x[i] = 100;
> + x[i + 1] = 99;
> + }
> +}
> +
> +void
> +f3 (unsigned short *x, int stepx, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + {
> + x[i * stepx - 16] = 100;
> + x[i * stepx - 15] = 99;
> + }
> +}
> +
> +void
> +f4 (unsigned short *x, int stepx, int n)
> +{
> + for (int i = 0; i < n; i += stepx)
> + {
> + x[i - 16] = 100;
> + x[i - 15] = 99;
> + }
> +}
> +
> +void
> +f5 (unsigned short *x, int stepx, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + {
> + x[i * stepx - 16] = 100;
> + x[i * stepx + 15] = 99;
> + }
> +}
> +
> +void
> +f6 (unsigned short *x, int stepx, int n)
> +{
> + for (int i = 0; i < n; i += stepx)
> + {
> + x[i - 16] = 100;
> + x[i + 15] = 99;
> + }
> +}
> +
> +void
> +f7 (unsigned short *x, int stepx, int n)
> +{
> + for (unsigned short *y = x; y < x + n; y += stepx)
> + {
> + y[0] = 100;
> + y[1] = 99;
> + }
> +}
> +
> +unsigned short x[1000];
> +
> +void
> +g1 (int stepx, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + {
> + x[i * stepx] = 100;
> + x[i * stepx + 1] = 99;
> + }
> +}
> +
> +void
> +g2 (int stepx, int n)
> +{
> + for (int i = 0; i < n; i += stepx)
> + {
> + x[i] = 100;
> + x[i + 1] = 99;
> + }
> +}
> +
> +void
> +g3 (int stepx, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + {
> + x[i * stepx - 16] = 100;
> + x[i * stepx - 15] = 99;
> + }
> +}
> +
> +void
> +g4 (int stepx, int n)
> +{
> + for (int i = 0; i < n; i += stepx)
> + {
> + x[i - 16] = 100;
> + x[i - 15] = 99;
> + }
> +}
> +
> +void
> +g5 (int stepx, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + {
> + x[i * stepx - 16] = 100;
> + x[i * stepx + 15] = 99;
> + }
> +}
> +
> +void
> +g6 (int stepx, int n)
> +{
> + for (int i = 0; i < n; i += stepx)
> + {
> + x[i - 16] = 100;
> + x[i + 15] = 99;
> + }
> +}
> +
> +void
> +g7 (int stepx, int n)
> +{
> + for (unsigned short *y = x; y < x + n; y += stepx)
> + {
> + y[0] = 100;
> + y[1] = 99;
> + }
> +}
> +
> +/* { dg-final { scan-tree-dump-not {want to version} "lversion" } } */
> +/* { dg-final { scan-tree-dump-not {versioned} "lversion" } } */
> Index: gcc/testsuite/gcc.dg/loop-versioning-13.c
> ===================================================================
> --- /dev/null 2018-11-29 13:15:04.463550658 +0000
> +++ gcc/testsuite/gcc.dg/loop-versioning-13.c 2018-12-06 12:33:59.216098398 +0000
> @@ -0,0 +1,109 @@
> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
> +
> +/* Test that we do version for a step of 1 when that would lead the
> + iterations to access consecutive groups. */
> +
> +void
> +f1 (unsigned short *x, int stepx, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + {
> + x[i * stepx * 2] = 100;
> + x[i * stepx * 2 + 1] = 99;
> + }
> +}
> +
> +void
> +f2 (unsigned short *x, int stepx, int n)
> +{
> + for (int i = 0; i < n; i += stepx * 2)
> + {
> + x[i] = 100;
> + x[i + 1] = 99;
> + }
> +}
> +
> +void
> +f3 (unsigned short *x, int stepx, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + {
> + x[i * stepx * 2 - 16] = 100;
> + x[i * stepx * 2 - 15] = 99;
> + }
> +}
> +
> +void
> +f4 (unsigned short *x, int stepx, int n)
> +{
> + for (int i = 0; i < n; i += stepx * 2)
> + {
> + x[i - 16] = 100;
> + x[i - 15] = 99;
> + }
> +}
> +
> +void
> +f5 (unsigned short *x, int stepx, int n)
> +{
> + for (unsigned short *y = x; y < x + n; y += stepx * 2)
> + {
> + y[0] = 100;
> + y[1] = 99;
> + }
> +}
> +
> +unsigned short x[1000];
> +
> +void
> +g1 (int stepx, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + {
> + x[i * stepx * 2] = 100;
> + x[i * stepx * 2 + 1] = 99;
> + }
> +}
> +
> +void
> +g2 (int stepx, int n)
> +{
> + for (int i = 0; i < n; i += stepx * 2)
> + {
> + x[i] = 100;
> + x[i + 1] = 99;
> + }
> +}
> +
> +void
> +g3 (int stepx, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + {
> + x[i * stepx * 2 - 16] = 100;
> + x[i * stepx * 2 - 15] = 99;
> + }
> +}
> +
> +void
> +g4 (int stepx, int n)
> +{
> + for (int i = 0; i < n; i += stepx * 2)
> + {
> + x[i - 16] = 100;
> + x[i - 15] = 99;
> + }
> +}
> +
> +void
> +g5 (int stepx, int n)
> +{
> + for (unsigned short *y = x; y < x + n; y += stepx * 2)
> + {
> + y[0] = 100;
> + y[1] = 99;
> + }
> +}
> +
> +/* { dg-final { scan-tree-dump-times {want to version containing loop} 10 "lversion" } } */
> +/* { dg-final { scan-tree-dump-times {versioned this loop} 10 "lversion" } } */
> Index: gcc/testsuite/gcc.dg/loop-versioning-14.c
> ===================================================================
> --- /dev/null 2018-11-29 13:15:04.463550658 +0000
> +++ gcc/testsuite/gcc.dg/loop-versioning-14.c 2018-12-06 12:33:59.216098398 +0000
> @@ -0,0 +1,149 @@
> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
> +
> +/* Test that we don't try to version for a step of 1 when that would
> + cause the iterations to leave a gap between accesses. */
> +
> +void
> +f1 (unsigned short *x, int stepx, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + {
> + x[i * stepx * 4] = 100;
> + x[i * stepx * 4 + 1] = 99;
> + }
> +}
> +
> +void
> +f2 (unsigned short *x, int stepx, int n)
> +{
> + for (int i = 0; i < n; i += stepx * 4)
> + {
> + x[i] = 100;
> + x[i + 1] = 99;
> + }
> +}
> +
> +void
> +f3 (unsigned short *x, int stepx, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + {
> + x[i * stepx * 4 - 16] = 100;
> + x[i * stepx * 4 - 15] = 99;
> + }
> +}
> +
> +void
> +f4 (unsigned short *x, int stepx, int n)
> +{
> + for (int i = 0; i < n; i += stepx * 4)
> + {
> + x[i - 16] = 100;
> + x[i - 15] = 99;
> + }
> +}
> +
> +void
> +f5 (unsigned short *x, int stepx, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + {
> + x[i * stepx * 64 - 16] = 100;
> + x[i * stepx * 64 + 15] = 99;
> + }
> +}
> +
> +void
> +f6 (unsigned short *x, int stepx, int n)
> +{
> + for (int i = 0; i < n; i += stepx * 64)
> + {
> + x[i - 16] = 100;
> + x[i + 15] = 99;
> + }
> +}
> +
> +void
> +f7 (unsigned short *x, int stepx, int n)
> +{
> + for (unsigned short *y = x; y < x + n; y += stepx * 4)
> + {
> + y[0] = 100;
> + y[1] = 99;
> + }
> +}
> +
> +unsigned short x[1000];
> +
> +void
> +g1 (int stepx, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + {
> + x[i * stepx * 4] = 100;
> + x[i * stepx * 4 + 1] = 99;
> + }
> +}
> +
> +void
> +g2 (int stepx, int n)
> +{
> + for (int i = 0; i < n; i += stepx * 4)
> + {
> + x[i] = 100;
> + x[i + 1] = 99;
> + }
> +}
> +
> +void
> +g3 (int stepx, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + {
> + x[i * stepx * 4 - 16] = 100;
> + x[i * stepx * 4 - 15] = 99;
> + }
> +}
> +
> +void
> +g4 (int stepx, int n)
> +{
> + for (int i = 0; i < n; i += stepx * 4)
> + {
> + x[i - 16] = 100;
> + x[i - 15] = 99;
> + }
> +}
> +
> +void
> +g5 (int stepx, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + {
> + x[i * stepx * 64 - 16] = 100;
> + x[i * stepx * 64 + 15] = 99;
> + }
> +}
> +
> +void
> +g6 (int stepx, int n)
> +{
> + for (int i = 0; i < n; i += stepx * 64)
> + {
> + x[i - 16] = 100;
> + x[i + 15] = 99;
> + }
> +}
> +
> +void
> +g7 (int stepx, int n)
> +{
> + for (unsigned short *y = x; y < x + n; y += stepx * 4)
> + {
> + y[0] = 100;
> + y[1] = 99;
> + }
> +}
> +
> +/* { dg-final { scan-tree-dump-not {want to version} "lversion" } } */
> +/* { dg-final { scan-tree-dump-not {versioned} "lversion" } } */
> Index: gcc/testsuite/gcc.dg/vect/slp-43.c
> ===================================================================
> --- gcc/testsuite/gcc.dg/vect/slp-43.c 2018-05-02 08:37:48.993604639 +0100
> +++ gcc/testsuite/gcc.dg/vect/slp-43.c 2018-12-06 12:33:59.220098363 +0000
> @@ -1,5 +1,5 @@
> /* { dg-require-effective-target vect_int } */
> -/* { dg-additional-options "-O3" } */
> +/* { dg-additional-options "-O3 -fno-version-loops-for-strides" } */
>
> #include <string.h>
> #include "tree-vect.h"
> Index: gcc/testsuite/gcc.dg/vect/slp-45.c
> ===================================================================
> --- gcc/testsuite/gcc.dg/vect/slp-45.c 2018-05-02 08:37:48.997604602 +0100
> +++ gcc/testsuite/gcc.dg/vect/slp-45.c 2018-12-06 12:33:59.220098363 +0000
> @@ -1,5 +1,5 @@
> /* { dg-require-effective-target vect_int } */
> -/* { dg-additional-options "-O3" } */
> +/* { dg-additional-options "-O3 -fno-version-loops-for-strides" } */
>
> #include <string.h>
> #include "tree-vect.h"
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Add a loop versioning pass
2018-12-12 12:06 ` Richard Biener
@ 2018-12-12 18:43 ` Richard Sandiford
2018-12-13 16:08 ` Richard Biener
0 siblings, 1 reply; 17+ messages in thread
From: Richard Sandiford @ 2018-12-12 18:43 UTC (permalink / raw)
To: Richard Biener; +Cc: Richard Guenther, GCC Patches
Richard Biener <richard.guenther@gmail.com> writes:
> On Thu, Dec 6, 2018 at 2:19 PM Richard Sandiford
>> Tested on x86_64-linux-gnu, aarch64-linux-gnu and aarch64_be-elf.
>> Also repeated the performance testing (but haven't yet tried an
>> LTO variant; will do that over the weekend).
>
> Any results?
Sorry, I should've remembered that finding time to run tests is easy,
finding time to analyse them is hard.
Speed-wise, the impact of the patch for LTO is similar to without,
with 554.roms_r being the main beneficiary for both AArch64 and x86_64.
I get a 6.8% improvement on Cortex-A57 with -Ofast -mcpu=native
-flto=jobserver.
Size-wise, there are three tests that grow by >=2% on x86_64:
549.fotonik3d_r: 5.5%
548.exchange2_r: 29.5%
554.roms_r: 39.6%
The roms increase seems fair enough given the benefit, since the
whole program uses a similar style for handling arrays.
fotonik3d is a mixed bag. Some versioning conditions are from
things like:
subroutine foo(a)
real :: a(:,:,:)
a(1,:,:) = ...
end subroutine
where we version for the middle dimension having a stride of 1.
This could be eliminated if we had more semantic information.
Other conditions are for things like:
subroutine foo(a)
real :: a(:,:,:)
a(:,1,:) = ...
end subroutine
where the pass really does help, but unfortunately those loops
aren't hot and might not even be used at all.
Some opportunities are cases in which we avoid gather loads, such as
in the "mp" loads in the hot __material_mod_MOD_mat_updatee function.
But mp feeds a gather load itself and these assignments aren't a
critical part of the loop.
(I'm testing on an AVX-only system, so these are synthesised gathers
rather than real gathers.)
The growth in 548.exchange2_r comes from reasonable changes to cold code.
The test spends almost all its time in __brute_force_MOD_digits_2, which
contains the same code before and after the patch, but which accounts
for less than 1% of .text before the patch.
> I've skimmed over the updated patch and it looks
> a lot better now.
>
> +bool
> +loop_versioning
> +::find_per_loop_multiplication (address_info &address, address_term_info &term)
> +{
>
> is that what coding convention allows?
Not sure we've settled on something, so I improvised.
> For grepping I'd then say we should do
>
> bool loop_versioning::
> find_per_loop_multiplication (...)
>
> ;)
Sounds good to me.
> Anywhere else we you use
>
> loop_versioning::foo
>
> so please stick to that.
Yeah, I used that when I could avoid an argument spilling over 80 chars,
but I think I missed that the above function now fits into that category,
Will double-check the others.
> Otherwise OK.
Thanks :-)
> I think I don't see a testcase where we could version both loops in a nest
> so I'm not sure whether the transform works fine when you are only
> updating SSA form at the very end of the pass.
You mean something like:
real :: foo(:,:), bar(:)
do i=1,n
do j=1,n
foo(i,j) = ...
end do
bar(i) = ..
end do
? I can add a test if so.
At the moment the pass only looks for direct versioning opportunities
in inner loops, so the assignment to bar wouldn't be treated as a
versioning opportunity. We should still hoist the version check
for foo outside both loops, which is tested by loop_versioning_4.f90
for cases in which the outer loop doesn't have its own array
arithmetic, but isn't tested for cases like the above.
> There may also be some subtle issues with substitute_and_fold being
> applied to non-up-to-date SSA form given it folds stmts looking at
> (single-use!) SSA edges. The single-use guard might be what saves you
> here (SSA uses in the copies are not yet updated to point to the
> copied DEFs).
OK. I was hoping that because we only apply substitute_and_fold
to new code, there would be no problem with uses elsewhere.
Would it be safer to:
- version all loops we want to version
- update SSA explicitly
- apply substitute and fold to all "new" loops
? Could we then get away with returning a 0 TODO at the end?
Thanks,
Richard
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Add a loop versioning pass
2018-12-12 18:43 ` Richard Sandiford
@ 2018-12-13 16:08 ` Richard Biener
2018-12-14 16:18 ` Richard Sandiford
0 siblings, 1 reply; 17+ messages in thread
From: Richard Biener @ 2018-12-13 16:08 UTC (permalink / raw)
To: Richard Sandiford, Richard Biener; +Cc: GCC Patches
On December 12, 2018 7:43:10 PM GMT+01:00, Richard Sandiford <richard.sandiford@arm.com> wrote:
>Richard Biener <richard.guenther@gmail.com> writes:
>> On Thu, Dec 6, 2018 at 2:19 PM Richard Sandiford
>>> Tested on x86_64-linux-gnu, aarch64-linux-gnu and aarch64_be-elf.
>>> Also repeated the performance testing (but haven't yet tried an
>>> LTO variant; will do that over the weekend).
>>
>> Any results?
>
>Sorry, I should've remembered that finding time to run tests is easy,
>finding time to analyse them is hard.
>
>Speed-wise, the impact of the patch for LTO is similar to without,
>with 554.roms_r being the main beneficiary for both AArch64 and x86_64.
>I get a 6.8% improvement on Cortex-A57 with -Ofast -mcpu=native
>-flto=jobserver.
>
>Size-wise, there are three tests that grow by >=2% on x86_64:
>
>549.fotonik3d_r: 5.5%
>548.exchange2_r: 29.5%
>554.roms_r: 39.6%
Uh. With LTO we might have a reasonable guessed profile and you do have a optimize_loop_nest_for_speed guard on the transform?
How does compile time fare with the above benchmarks?
>The roms increase seems fair enough given the benefit, since the
>whole program uses a similar style for handling arrays.
>
>fotonik3d is a mixed bag. Some versioning conditions are from
>things like:
>
> subroutine foo(a)
> real :: a(:,:,:)
> a(1,:,:) = ...
> end subroutine
>
>where we version for the middle dimension having a stride of 1.
>This could be eliminated if we had more semantic information.
>
>Other conditions are for things like:
>
> subroutine foo(a)
> real :: a(:,:,:)
> a(:,1,:) = ...
> end subroutine
>
>where the pass really does help, but unfortunately those loops
>aren't hot and might not even be used at all.
>
>Some opportunities are cases in which we avoid gather loads, such as
>in the "mp" loads in the hot __material_mod_MOD_mat_updatee function.
>But mp feeds a gather load itself and these assignments aren't a
>critical part of the loop.
>
>(I'm testing on an AVX-only system, so these are synthesised gathers
>rather than real gathers.)
>
>The growth in 548.exchange2_r comes from reasonable changes to cold
>code.
>The test spends almost all its time in __brute_force_MOD_digits_2,
>which
>contains the same code before and after the patch, but which accounts
>for less than 1% of .text before the patch.
>
>> I've skimmed over the updated patch and it looks
>> a lot better now.
>>
>> +bool
>> +loop_versioning
>> +::find_per_loop_multiplication (address_info &address,
>address_term_info &term)
>> +{
>>
>> is that what coding convention allows?
>
>Not sure we've settled on something, so I improvised.
>
>> For grepping I'd then say we should do
>>
>> bool loop_versioning::
>> find_per_loop_multiplication (...)
>>
>> ;)
>
>Sounds good to me.
>
>> Anywhere else we you use
>>
>> loop_versioning::foo
>>
>> so please stick to that.
>
>Yeah, I used that when I could avoid an argument spilling over 80
>chars,
>but I think I missed that the above function now fits into that
>category,
>Will double-check the others.
>
>> Otherwise OK.
>
>Thanks :-)
>
>> I think I don't see a testcase where we could version both loops in a
>nest
>> so I'm not sure whether the transform works fine when you are only
>> updating SSA form at the very end of the pass.
>
>You mean something like:
>
> real :: foo(:,:), bar(:)
>
> do i=1,n
> do j=1,n
> foo(i,j) = ...
> end do
> bar(i) = ..
> end do
>
>? I can add a test if so.
Please.
>At the moment the pass only looks for direct versioning opportunities
>in inner loops, so the assignment to bar wouldn't be treated as a
>versioning opportunity.
Ah, OK.
We should still hoist the version check
>for foo outside both loops, which is tested by loop_versioning_4.f90
>for cases in which the outer loop doesn't have its own array
>arithmetic, but isn't tested for cases like the above.
>
>> There may also be some subtle issues with substitute_and_fold being
>> applied to non-up-to-date SSA form given it folds stmts looking at
>> (single-use!) SSA edges. The single-use guard might be what saves
>you
>> here (SSA uses in the copies are not yet updated to point to the
>> copied DEFs).
>
>OK. I was hoping that because we only apply substitute_and_fold
>to new code, there would be no problem with uses elsewhere.
Might be, yes.
>Would it be safer to:
>
> - version all loops we want to version
> - update SSA explicitly
> - apply substitute and fold to all "new" loops
That would be definitely less fishy. But you need to get at the actual 'new' SSA names for the replacements as I guess they'll be rewritten? Or maybe those are not.
>? Could we then get away with returning a 0 TODO at the end?
Yes.
Richard.
>Thanks,
>Richard
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Add a loop versioning pass
2018-12-13 16:08 ` Richard Biener
@ 2018-12-14 16:18 ` Richard Sandiford
2018-12-15 10:27 ` Richard Biener
0 siblings, 1 reply; 17+ messages in thread
From: Richard Sandiford @ 2018-12-14 16:18 UTC (permalink / raw)
To: Richard Biener; +Cc: Richard Biener, GCC Patches
Richard Biener <rguenther@suse.de> writes:
> On December 12, 2018 7:43:10 PM GMT+01:00, Richard Sandiford <richard.sandiford@arm.com> wrote:
>>Richard Biener <richard.guenther@gmail.com> writes:
>>> On Thu, Dec 6, 2018 at 2:19 PM Richard Sandiford
>>>> Tested on x86_64-linux-gnu, aarch64-linux-gnu and aarch64_be-elf.
>>>> Also repeated the performance testing (but haven't yet tried an
>>>> LTO variant; will do that over the weekend).
>>>
>>> Any results?
>>
>>Sorry, I should've remembered that finding time to run tests is easy,
>>finding time to analyse them is hard.
>>
>>Speed-wise, the impact of the patch for LTO is similar to without,
>>with 554.roms_r being the main beneficiary for both AArch64 and x86_64.
>>I get a 6.8% improvement on Cortex-A57 with -Ofast -mcpu=native
>>-flto=jobserver.
>>
>>Size-wise, there are three tests that grow by >=2% on x86_64:
>>
>>549.fotonik3d_r: 5.5%
>>548.exchange2_r: 29.5%
>>554.roms_r: 39.6%
>
> Uh. With LTO we might have a reasonable guessed profile and you do have a optimize_loop_nest_for_speed guard on the transform?
Guard now added :-) But unfortunately it doesn't make any significant
difference. 548.exchange2_r goes from 29.5% to 27.7%, but the other
two are the same as before.
> How does compile time fare with the above benchmarks?
For 554.roms_r it's +80%(!) with -flto=1, but I think that's par for
the course given the increase in function sizes.
For 549.fotonik3d_r it's +5% with -flto=1.
For 503.bwaves_r (as an example of a benchmark whose size doesn't change),
the difference is in the noise.
[...]
>>You mean something like:
>>
>> real :: foo(:,:), bar(:)
>>
>> do i=1,n
>> do j=1,n
>> foo(i,j) = ...
>> end do
>> bar(i) = ..
>> end do
>>
>>? I can add a test if so.
>
> Please.
OK, I've added them to loop_versioning_4.f90.
>>> There may also be some subtle issues with substitute_and_fold being
>>> applied to non-up-to-date SSA form given it folds stmts looking at
>>> (single-use!) SSA edges. The single-use guard might be what saves
>>you
>>> here (SSA uses in the copies are not yet updated to point to the
>>> copied DEFs).
>>
>>OK. I was hoping that because we only apply substitute_and_fold
>>to new code, there would be no problem with uses elsewhere.
>
> Might be, yes.
>
>>Would it be safer to:
>>
>> - version all loops we want to version
>> - update SSA explicitly
>> - apply substitute and fold to all "new" loops
>
> That would be definitely less fishy. But you need to get at the actual
> 'new' SSA names for the replacements as I guess they'll be rewritten?
> Or maybe those are not.
>
>>? Could we then get away with returning a 0 TODO at the end?
>
> Yes.
OK, the updated patch does it this way.
Tested as before.
Thanks,
Richard
2018-12-14 Richard Sandiford <richard.sandiford@arm.com>
Ramana Radhakrishnan <ramana.radhakrishnan@arm.com>
Kyrylo Tkachov <kyrylo.tkachov@arm.com>
gcc/
* doc/invoke.texi (-fversion-loops-for-strides): Document
(loop-versioning-group-size, loop-versioning-max-inner-insns)
(loop-versioning-max-outer-insns): Document new --params.
* Makefile.in (OBJS): Add gimple-loop-versioning.o.
* common.opt (fversion-loops-for-strides): New option.
* opts.c (default_options_table): Enable fversion-loops-for-strides
at -O3.
* params.def (PARAM_LOOP_VERSIONING_GROUP_SIZE)
(PARAM_LOOP_VERSIONING_MAX_INNER_INSNS)
(PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS): New parameters.
* passes.def: Add pass_loop_versioning.
* timevar.def (TV_LOOP_VERSIONING): New time variable.
* tree-ssa-propagate.h
(substitute_and_fold_engine::substitute_and_fold): Add an optional
block parameter.
* tree-ssa-propagate.c
(substitute_and_fold_engine::substitute_and_fold): Likewise.
When passed, only walk blocks dominated by that block.
* tree-vrp.h (range_includes_p): Declare.
(range_includes_zero_p): Turn into an inline wrapper around
range_includes_p.
* tree-vrp.c (range_includes_p): New function, generalizing...
(range_includes_zero_p): ...this.
* tree-pass.h (make_pass_loop_versioning): Declare.
* gimple-loop-versioning.cc: New file.
gcc/testsuite/
* gcc.dg/loop-versioning-1.c: New test.
* gcc.dg/loop-versioning-10.c: Likewise.
* gcc.dg/loop-versioning-11.c: Likewise.
* gcc.dg/loop-versioning-2.c: Likewise.
* gcc.dg/loop-versioning-3.c: Likewise.
* gcc.dg/loop-versioning-4.c: Likewise.
* gcc.dg/loop-versioning-5.c: Likewise.
* gcc.dg/loop-versioning-6.c: Likewise.
* gcc.dg/loop-versioning-7.c: Likewise.
* gcc.dg/loop-versioning-8.c: Likewise.
* gcc.dg/loop-versioning-9.c: Likewise.
* gfortran.dg/loop_versioning_1.f90: Likewise.
* gfortran.dg/loop_versioning_2.f90: Likewise.
* gfortran.dg/loop_versioning_3.f90: Likewise.
* gfortran.dg/loop_versioning_4.f90: Likewise.
* gfortran.dg/loop_versioning_5.f90: Likewise.
* gfortran.dg/loop_versioning_6.f90: Likewise.
* gfortran.dg/loop_versioning_7.f90: Likewise.
* gfortran.dg/loop_versioning_8.f90: Likewise.
Index: gcc/doc/invoke.texi
===================================================================
--- gcc/doc/invoke.texi 2018-12-11 15:49:19.061544092 +0000
+++ gcc/doc/invoke.texi 2018-12-14 15:02:53.813883464 +0000
@@ -8220,7 +8220,8 @@ by @option{-O2} and also turns on the fo
-ftree-partial-pre @gol
-ftree-slp-vectorize @gol
-funswitch-loops @gol
--fvect-cost-model}
+-fvect-cost-model @gol
+-fversion-loops-for-strides}
@item -O0
@opindex O0
@@ -10772,6 +10773,30 @@ of the loop on both branches (modified a
Enabled by @option{-fprofile-use} and @option{-fauto-profile}.
+@item -fversion-loops-for-strides
+@opindex fversion-loops-for-strides
+If a loop iterates over an array with a variable stride, create another
+version of the loop that assumes the stride is always one. For example:
+
+@smallexample
+for (int i = 0; i < n; ++i)
+ x[i * stride] = @dots{};
+@end smallexample
+
+becomes:
+
+@smallexample
+if (stride == 1)
+ for (int i = 0; i < n; ++i)
+ x[i] = @dots{};
+else
+ for (int i = 0; i < n; ++i)
+ x[i * stride] = @dots{};
+@end smallexample
+
+This is particularly useful for assumed-shape arrays in Fortran where
+(for example) it allows better vectorization assuming contiguous accesses.
+
@item -ffunction-sections
@itemx -fdata-sections
@opindex ffunction-sections
@@ -11981,6 +12006,15 @@ Hardware autoprefetcher scheduler model
Number of lookahead cycles the model looks into; at '
' only enable instruction sorting heuristic.
+@item loop-versioning-max-inner-insns
+The maximum number of instructions that an inner loop can have
+before the loop versioning pass considers it too big to copy.
+
+@item loop-versioning-max-outer-insns
+The maximum number of instructions that an outer loop can have
+before the loop versioning pass considers it too big to copy,
+discounting any instructions in inner loops that directly benefit
+from versioning.
@end table
@end table
Index: gcc/Makefile.in
===================================================================
--- gcc/Makefile.in 2018-12-06 18:00:29.000000000 +0000
+++ gcc/Makefile.in 2018-12-14 15:02:53.809883498 +0000
@@ -1320,6 +1320,7 @@ OBJS = \
gimple-laddress.o \
gimple-loop-interchange.o \
gimple-loop-jam.o \
+ gimple-loop-versioning.o \
gimple-low.o \
gimple-pretty-print.o \
gimple-ssa-backprop.o \
Index: gcc/common.opt
===================================================================
--- gcc/common.opt 2018-12-06 18:00:29.000000000 +0000
+++ gcc/common.opt 2018-12-14 15:02:53.809883498 +0000
@@ -2775,6 +2775,10 @@ fsplit-loops
Common Report Var(flag_split_loops) Optimization
Perform loop splitting.
+fversion-loops-for-strides
+Common Report Var(flag_version_loops_for_strides) Optimization
+Version loops based on whether indices have a stride of one.
+
funwind-tables
Common Report Var(flag_unwind_tables) Optimization
Just generate unwind tables for exception handling.
Index: gcc/opts.c
===================================================================
--- gcc/opts.c 2018-12-06 18:00:29.000000000 +0000
+++ gcc/opts.c 2018-12-14 15:02:53.813883464 +0000
@@ -556,6 +556,7 @@ static const struct default_options defa
{ OPT_LEVELS_3_PLUS, OPT_ftree_slp_vectorize, NULL, 1 },
{ OPT_LEVELS_3_PLUS, OPT_funswitch_loops, NULL, 1 },
{ OPT_LEVELS_3_PLUS, OPT_fvect_cost_model_, NULL, VECT_COST_MODEL_DYNAMIC },
+ { OPT_LEVELS_3_PLUS, OPT_fversion_loops_for_strides, NULL, 1 },
/* -Ofast adds optimizations to -O3. */
{ OPT_LEVELS_FAST, OPT_ffast_math, NULL, 1 },
Index: gcc/params.def
===================================================================
--- gcc/params.def 2018-12-06 18:00:29.000000000 +0000
+++ gcc/params.def 2018-12-14 15:02:53.817883430 +0000
@@ -1365,6 +1365,19 @@ DEFPARAM(PARAM_LOGICAL_OP_NON_SHORT_CIRC
"True if a non-short-circuit operation is optimal.",
-1, -1, 1)
+DEFPARAM(PARAM_LOOP_VERSIONING_MAX_INNER_INSNS,
+ "loop-versioning-max-inner-insns",
+ "The maximum number of instructions in an inner loop that is being"
+ " considered for versioning.",
+ 200, 0, 0)
+
+DEFPARAM(PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS,
+ "loop-versioning-max-outer-insns",
+ "The maximum number of instructions in an outer loop that is being"
+ " considered for versioning, on top of the instructions in inner"
+ " loops.",
+ 100, 0, 0)
+
/*
Local variables:
Index: gcc/passes.def
===================================================================
--- gcc/passes.def 2018-12-06 18:00:29.000000000 +0000
+++ gcc/passes.def 2018-12-14 15:02:53.817883430 +0000
@@ -265,6 +265,7 @@ along with GCC; see the file COPYING3.
NEXT_PASS (pass_tree_unswitch);
NEXT_PASS (pass_scev_cprop);
NEXT_PASS (pass_loop_split);
+ NEXT_PASS (pass_loop_versioning);
NEXT_PASS (pass_loop_jam);
/* All unswitching, final value replacement and splitting can expose
empty loops. Remove them now. */
Index: gcc/timevar.def
===================================================================
--- gcc/timevar.def 2018-12-06 18:00:29.000000000 +0000
+++ gcc/timevar.def 2018-12-14 15:02:53.821883396 +0000
@@ -234,6 +234,7 @@ DEFTIMEVAR (TV_DSE1 , "
DEFTIMEVAR (TV_DSE2 , "dead store elim2")
DEFTIMEVAR (TV_LOOP , "loop analysis")
DEFTIMEVAR (TV_LOOP_INIT , "loop init")
+DEFTIMEVAR (TV_LOOP_VERSIONING , "loop versioning")
DEFTIMEVAR (TV_LOOP_MOVE_INVARIANTS , "loop invariant motion")
DEFTIMEVAR (TV_LOOP_UNROLL , "loop unrolling")
DEFTIMEVAR (TV_LOOP_DOLOOP , "loop doloop")
Index: gcc/tree-ssa-propagate.h
===================================================================
--- gcc/tree-ssa-propagate.h 2018-12-06 18:00:29.000000000 +0000
+++ gcc/tree-ssa-propagate.h 2018-12-14 15:02:53.821883396 +0000
@@ -104,7 +104,7 @@ extern void propagate_tree_value_into_st
virtual bool fold_stmt (gimple_stmt_iterator *) { return false; }
virtual tree get_value (tree) { return NULL_TREE; }
- bool substitute_and_fold (void);
+ bool substitute_and_fold (basic_block = NULL);
bool replace_uses_in (gimple *);
bool replace_phi_args_in (gphi *);
};
Index: gcc/tree-ssa-propagate.c
===================================================================
--- gcc/tree-ssa-propagate.c 2018-12-06 18:00:29.000000000 +0000
+++ gcc/tree-ssa-propagate.c 2018-12-14 15:02:53.821883396 +0000
@@ -1154,6 +1154,10 @@ substitute_and_fold_dom_walker::before_d
/* Perform final substitution and folding of propagated values.
+ Process the whole function if BLOCK is null, otherwise only
+ process the blocks that BLOCK dominates. In the latter case,
+ it is the caller's responsibility to ensure that dominator
+ information is available and up-to-date.
PROP_VALUE[I] contains the single value that should be substituted
at every use of SSA name N_I. If PROP_VALUE is NULL, no values are
@@ -1170,16 +1174,24 @@ substitute_and_fold_dom_walker::before_d
Return TRUE when something changed. */
bool
-substitute_and_fold_engine::substitute_and_fold (void)
+substitute_and_fold_engine::substitute_and_fold (basic_block block)
{
if (dump_file && (dump_flags & TDF_DETAILS))
fprintf (dump_file, "\nSubstituting values and folding statements\n\n");
memset (&prop_stats, 0, sizeof (prop_stats));
- calculate_dominance_info (CDI_DOMINATORS);
+ /* Don't call calculate_dominance_info when iterating over a subgraph.
+ Callers that are using the interface this way are likely to want to
+ iterate over several disjoint subgraphs, and it would be expensive
+ in enable-checking builds to revalidate the whole dominance tree
+ each time. */
+ if (block)
+ gcc_assert (dom_info_state (CDI_DOMINATORS));
+ else
+ calculate_dominance_info (CDI_DOMINATORS);
substitute_and_fold_dom_walker walker (CDI_DOMINATORS, this);
- walker.walk (ENTRY_BLOCK_PTR_FOR_FN (cfun));
+ walker.walk (block ? block : ENTRY_BLOCK_PTR_FOR_FN (cfun));
/* We cannot remove stmts during the BB walk, especially not release
SSA names there as that destroys the lattice of our callers.
Index: gcc/tree-vrp.h
===================================================================
--- gcc/tree-vrp.h 2018-12-06 18:00:29.000000000 +0000
+++ gcc/tree-vrp.h 2018-12-14 15:02:53.821883396 +0000
@@ -243,7 +243,7 @@ struct assert_info
extern void register_edge_assert_for (tree, edge, enum tree_code,
tree, tree, vec<assert_info> &);
extern bool stmt_interesting_for_vrp (gimple *);
-extern bool range_includes_zero_p (const value_range_base *);
+extern bool range_includes_p (const value_range_base *, HOST_WIDE_INT);
extern bool infer_value_range (gimple *, tree, tree_code *, tree *);
extern bool vrp_bitmap_equal_p (const_bitmap, const_bitmap);
@@ -285,4 +285,12 @@ extern tree get_single_symbol (tree, boo
extern void maybe_set_nonzero_bits (edge, tree);
extern value_range_kind determine_value_range (tree, wide_int *, wide_int *);
+/* Return TRUE if *VR includes the value zero. */
+
+inline bool
+range_includes_zero_p (const value_range_base *vr)
+{
+ return range_includes_p (vr, 0);
+}
+
#endif /* GCC_TREE_VRP_H */
Index: gcc/tree-vrp.c
===================================================================
--- gcc/tree-vrp.c 2018-12-07 14:59:04.255508495 +0000
+++ gcc/tree-vrp.c 2018-12-14 15:02:53.821883396 +0000
@@ -1173,15 +1173,14 @@ value_inside_range (tree val, tree min,
}
-/* Return TRUE if *VR includes the value zero. */
+/* Return TRUE if *VR includes the value X. */
bool
-range_includes_zero_p (const value_range_base *vr)
+range_includes_p (const value_range_base *vr, HOST_WIDE_INT x)
{
if (vr->varying_p () || vr->undefined_p ())
return true;
- tree zero = build_int_cst (vr->type (), 0);
- return vr->may_contain_p (zero);
+ return vr->may_contain_p (build_int_cst (vr->type (), x));
}
/* If *VR has a value range that is a single constant value return that,
Index: gcc/tree-pass.h
===================================================================
--- gcc/tree-pass.h 2018-12-06 18:00:29.000000000 +0000
+++ gcc/tree-pass.h 2018-12-14 15:02:53.821883396 +0000
@@ -362,6 +362,7 @@ extern gimple_opt_pass *make_pass_fix_lo
extern gimple_opt_pass *make_pass_tree_loop (gcc::context *ctxt);
extern gimple_opt_pass *make_pass_tree_no_loop (gcc::context *ctxt);
extern gimple_opt_pass *make_pass_tree_loop_init (gcc::context *ctxt);
+extern gimple_opt_pass *make_pass_loop_versioning (gcc::context *ctxt);
extern gimple_opt_pass *make_pass_lim (gcc::context *ctxt);
extern gimple_opt_pass *make_pass_linterchange (gcc::context *ctxt);
extern gimple_opt_pass *make_pass_tree_unswitch (gcc::context *ctxt);
Index: gcc/gimple-loop-versioning.cc
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/gimple-loop-versioning.cc 2018-12-14 15:02:53.813883464 +0000
@@ -0,0 +1,1758 @@
+/* Loop versioning pass.
+ Copyright (C) 2018 Free Software Foundation, Inc.
+
+This file is part of GCC.
+
+GCC is free software; you can redistribute it and/or modify it
+under the terms of the GNU General Public License as published by the
+Free Software Foundation; either version 3, or (at your option) any
+later version.
+
+GCC is distributed in the hope that it will be useful, but WITHOUT
+ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
+for more details.
+
+You should have received a copy of the GNU General Public License
+along with GCC; see the file COPYING3. If not see
+<http://www.gnu.org/licenses/>. */
+
+#include "config.h"
+#include "system.h"
+#include "coretypes.h"
+#include "backend.h"
+#include "tree.h"
+#include "gimple.h"
+#include "gimple-iterator.h"
+#include "tree-pass.h"
+#include "gimplify-me.h"
+#include "cfgloop.h"
+#include "tree-ssa-loop.h"
+#include "ssa.h"
+#include "tree-scalar-evolution.h"
+#include "tree-chrec.h"
+#include "tree-ssa-loop-ivopts.h"
+#include "fold-const.h"
+#include "tree-ssa-propagate.h"
+#include "tree-inline.h"
+#include "domwalk.h"
+#include "alloc-pool.h"
+#include "vr-values.h"
+#include "gimple-ssa-evrp-analyze.h"
+#include "tree-vectorizer.h"
+#include "omp-general.h"
+#include "predict.h"
+#include "tree-into-ssa.h"
+#include "params.h"
+
+namespace {
+
+/* This pass looks for loops that could be simplified if certain loop
+ invariant conditions were true. It is effectively a form of loop
+ splitting in which the pass produces the split conditions itself,
+ instead of using ones that are already present in the IL.
+
+ Versioning for when strides are 1
+ ---------------------------------
+
+ At the moment the only thing the pass looks for are memory references
+ like:
+
+ for (auto i : ...)
+ ...x[i * stride]...
+
+ It considers changing such loops to:
+
+ if (stride == 1)
+ for (auto i : ...) [A]
+ ...x[i]...
+ else
+ for (auto i : ...) [B]
+ ...x[i * stride]...
+
+ This can have several benefits:
+
+ (1) [A] is often easier or cheaper to vectorize than [B].
+
+ (2) The scalar code in [A] is simpler than the scalar code in [B]
+ (if the loops cannot be vectorized or need an epilogue loop).
+
+ (3) We might recognize [A] as a pattern, such as a memcpy or memset.
+
+ (4) [A] has simpler address evolutions, which can help other passes
+ like loop interchange.
+
+ The optimization is particularly useful for assumed-shape arrays in
+ Fortran, where the stride of the innermost dimension depends on the
+ array descriptor but is often equal to 1 in practice. For example:
+
+ subroutine f1(x)
+ real :: x(:)
+ x(:) = 100
+ end subroutine f1
+
+ generates the equivalent of:
+
+ raw_stride = *x.dim[0].stride;
+ stride = raw_stride != 0 ? raw_stride : 1;
+ x_base = *x.data;
+ ...
+ tmp1 = stride * S;
+ tmp2 = tmp1 - stride;
+ *x_base[tmp2] = 1.0e+2;
+
+ but in the common case that stride == 1, the last three statements
+ simplify to:
+
+ tmp3 = S + -1;
+ *x_base[tmp3] = 1.0e+2;
+
+ The optimization is in principle very simple. The difficult parts are:
+
+ (a) deciding which parts of a general address calculation correspond
+ to the inner dimension of an array, since this usually isn't explicit
+ in the IL, and for C often isn't even explicit in the source code
+
+ (b) estimating when the transformation is worthwhile
+
+ Structure
+ ---------
+
+ The pass has four phases:
+
+ (1) Walk through the statements looking for and recording potential
+ versioning opportunities. Stop if there are none.
+
+ (2) Use context-sensitive range information to see whether any versioning
+ conditions are impossible in practice. Remove them if so, and stop
+ if no opportunities remain.
+
+ (We do this only after (1) to keep compile time down when no
+ versioning opportunities exist.)
+
+ (3) Apply the cost model. Decide which versioning opportunities are
+ worthwhile and at which nesting level they should be applied.
+
+ (4) Attempt to version all the loops selected by (3), so that:
+
+ for (...)
+ ...
+
+ becomes:
+
+ if (!cond)
+ for (...) // Original loop
+ ...
+ else
+ for (...) // New loop
+ ...
+
+ Use the version condition COND to simplify the new loop. */
+
+/* Enumerates the likelihood that a particular value indexes the inner
+ dimension of an array. */
+enum inner_likelihood {
+ INNER_UNLIKELY,
+ INNER_DONT_KNOW,
+ INNER_LIKELY
+};
+
+/* Information about one term of an address_info. */
+struct address_term_info
+{
+ /* The value of the term is EXPR * MULTIPLIER. */
+ tree expr;
+ unsigned HOST_WIDE_INT multiplier;
+
+ /* The stride applied by EXPR in each iteration of some unrecorded loop,
+ or null if no stride has been identified. */
+ tree stride;
+
+ /* Enumerates the likelihood that EXPR indexes the inner dimension
+ of an array. */
+ enum inner_likelihood inner_likelihood;
+
+ /* True if STRIDE == 1 is a versioning opportunity when considered
+ in isolation. */
+ bool versioning_opportunity_p;
+};
+
+/* Information about an address calculation, and the range of constant
+ offsets applied to it. */
+struct address_info
+{
+ static const unsigned int MAX_TERMS = 8;
+
+ /* One statement that calculates the address. If multiple statements
+ share the same address, we only record the first. */
+ gimple *stmt;
+
+ /* The loop containing STMT (cached for convenience). If multiple
+ statements share the same address, they all belong to this loop. */
+ struct loop *loop;
+
+ /* A decomposition of the calculation into a sum of terms plus an
+ optional base. When BASE is provided, it is never an SSA name.
+ Once initialization is complete, all members of TERMs are SSA names. */
+ tree base;
+ auto_vec<address_term_info, MAX_TERMS> terms;
+
+ /* All bytes accessed from the address fall in the offset range
+ [MIN_OFFSET, MAX_OFFSET). */
+ HOST_WIDE_INT min_offset, max_offset;
+};
+
+/* Stores addresses based on their base and terms (ignoring the offsets). */
+struct address_info_hasher : nofree_ptr_hash <address_info>
+{
+ static hashval_t hash (const address_info *);
+ static bool equal (const address_info *, const address_info *);
+};
+
+/* Information about the versioning we'd like to apply to a loop. */
+struct loop_info
+{
+ bool worth_versioning_p () const;
+
+ /* True if we've decided not to version this loop. The remaining
+ fields are meaningless if so. */
+ bool rejected_p;
+
+ /* True if at least one subloop of this loop benefits from versioning. */
+ bool subloops_benefit_p;
+
+ /* An estimate of the total number of instructions in the loop,
+ excluding those in subloops that benefit from versioning. */
+ unsigned int num_insns;
+
+ /* The outermost loop that can handle all the version checks
+ described below. */
+ struct loop *outermost;
+
+ /* The first entry in the list of blocks that belong to this loop
+ (and not to subloops). m_next_block_in_loop provides the chain
+ pointers for the list. */
+ basic_block block_list;
+
+ /* We'd like to version the loop for the case in which these SSA names
+ (keyed off their SSA_NAME_VERSION) are all equal to 1 at runtime. */
+ bitmap_head unity_names;
+
+ /* If versioning succeeds, this points the version of the loop that
+ assumes the version conditions holds. */
+ struct loop *optimized_loop;
+};
+
+/* The main pass structure. */
+class loop_versioning
+{
+public:
+ loop_versioning (function *);
+ ~loop_versioning ();
+ unsigned int run ();
+
+private:
+ /* Used to walk the dominator tree to find loop versioning conditions
+ that are always false. */
+ class lv_dom_walker : public dom_walker
+ {
+ public:
+ lv_dom_walker (loop_versioning &);
+
+ edge before_dom_children (basic_block) FINAL OVERRIDE;
+ void after_dom_children (basic_block) FINAL OVERRIDE;
+
+ private:
+ /* The parent pass. */
+ loop_versioning &m_lv;
+
+ /* Used to build context-dependent range information. */
+ evrp_range_analyzer m_range_analyzer;
+ };
+
+ /* Used to simplify statements based on conditions that are established
+ by the version checks. */
+ class name_prop : public substitute_and_fold_engine
+ {
+ public:
+ name_prop (loop_info &li) : m_li (li) {}
+ tree get_value (tree) FINAL OVERRIDE;
+
+ private:
+ /* Information about the versioning we've performed on the loop. */
+ loop_info &m_li;
+ };
+
+ loop_info &get_loop_info (struct loop *loop) { return m_loops[loop->num]; }
+
+ unsigned int max_insns_for_loop (struct loop *);
+ bool expensive_stmt_p (gimple *);
+
+ void version_for_unity (gimple *, tree);
+ bool acceptable_multiplier_p (tree, unsigned HOST_WIDE_INT,
+ unsigned HOST_WIDE_INT * = 0);
+ bool acceptable_type_p (tree, unsigned HOST_WIDE_INT *);
+ bool multiply_term_by (address_term_info &, tree);
+ inner_likelihood get_inner_likelihood (tree, unsigned HOST_WIDE_INT);
+ void analyze_stride (address_info &, address_term_info &,
+ tree, struct loop *);
+ bool find_per_loop_multiplication (address_info &, address_term_info &);
+ void analyze_term_using_scevs (address_info &, address_term_info &);
+ void analyze_address_fragment (address_info &);
+ void record_address_fragment (gimple *, unsigned HOST_WIDE_INT,
+ tree, unsigned HOST_WIDE_INT, HOST_WIDE_INT);
+ void analyze_expr (gimple *, tree);
+ bool analyze_block (basic_block);
+ bool analyze_blocks ();
+
+ void prune_loop_conditions (struct loop *, vr_values *);
+ bool prune_conditions ();
+
+ void merge_loop_info (struct loop *, struct loop *);
+ void add_loop_to_queue (struct loop *);
+ bool decide_whether_loop_is_versionable (struct loop *);
+ bool make_versioning_decisions ();
+
+ bool version_loop (struct loop *);
+ void implement_versioning_decisions ();
+
+ /* The function we're optimizing. */
+ function *m_fn;
+
+ /* The obstack to use for all pass-specific bitmaps. */
+ bitmap_obstack m_bitmap_obstack;
+
+ /* An obstack to use for general allocation. */
+ obstack m_obstack;
+
+ /* The number of loops in the function. */
+ unsigned int m_nloops;
+
+ /* The total number of loop version conditions we've found. */
+ unsigned int m_num_conditions;
+
+ /* Assume that an address fragment of the form i * stride * scale
+ (for variable stride and constant scale) will not benefit from
+ versioning for stride == 1 when scale is greater than this value. */
+ unsigned HOST_WIDE_INT m_maximum_scale;
+
+ /* Information about each loop. */
+ auto_vec<loop_info> m_loops;
+
+ /* Used to form a linked list of blocks that belong to a loop,
+ started by loop_info::block_list. */
+ auto_vec<basic_block> m_next_block_in_loop;
+
+ /* The list of loops that we've decided to version. */
+ auto_vec<struct loop *> m_loops_to_version;
+
+ /* A table of addresses in the current loop, keyed off their values
+ but not their offsets. */
+ hash_table <address_info_hasher> m_address_table;
+
+ /* A list of all addresses in M_ADDRESS_TABLE, in a predictable order. */
+ auto_vec <address_info *, 32> m_address_list;
+};
+
+/* If EXPR is an SSA name and not a default definition, return the
+ defining statement, otherwise return null. */
+
+static gimple *
+maybe_get_stmt (tree expr)
+{
+ if (TREE_CODE (expr) == SSA_NAME && !SSA_NAME_IS_DEFAULT_DEF (expr))
+ return SSA_NAME_DEF_STMT (expr);
+ return NULL;
+}
+
+/* Like maybe_get_stmt, but also return null if the defining
+ statement isn't an assignment. */
+
+static gassign *
+maybe_get_assign (tree expr)
+{
+ return safe_dyn_cast <gassign *> (maybe_get_stmt (expr));
+}
+
+/* Return true if this pass should look through a cast of expression FROM
+ to type TYPE when analyzing pieces of an address. */
+
+static bool
+look_through_cast_p (tree type, tree from)
+{
+ return (INTEGRAL_TYPE_P (TREE_TYPE (from)) == INTEGRAL_TYPE_P (type)
+ && POINTER_TYPE_P (TREE_TYPE (from)) == POINTER_TYPE_P (type));
+}
+
+/* Strip all conversions of integers or pointers from EXPR, regardless
+ of whether the conversions are nops. This is useful in the context
+ of this pass because we're not trying to fold or simulate the
+ expression; we just want to see how it's structured. */
+
+static tree
+strip_casts (tree expr)
+{
+ const unsigned int MAX_NITERS = 4;
+
+ tree type = TREE_TYPE (expr);
+ while (CONVERT_EXPR_P (expr)
+ && look_through_cast_p (type, TREE_OPERAND (expr, 0)))
+ expr = TREE_OPERAND (expr, 0);
+
+ for (unsigned int niters = 0; niters < MAX_NITERS; ++niters)
+ {
+ gassign *assign = maybe_get_assign (expr);
+ if (assign
+ && CONVERT_EXPR_CODE_P (gimple_assign_rhs_code (assign))
+ && look_through_cast_p (type, gimple_assign_rhs1 (assign)))
+ expr = gimple_assign_rhs1 (assign);
+ else
+ break;
+ }
+ return expr;
+}
+
+/* Compare two address_term_infos in the same address_info. */
+
+static int
+compare_address_terms (const void *a_uncast, const void *b_uncast)
+{
+ const address_term_info *a = (const address_term_info *) a_uncast;
+ const address_term_info *b = (const address_term_info *) b_uncast;
+
+ if (a->expr != b->expr)
+ return SSA_NAME_VERSION (a->expr) < SSA_NAME_VERSION (b->expr) ? -1 : 1;
+
+ if (a->multiplier != b->multiplier)
+ return a->multiplier < b->multiplier ? -1 : 1;
+
+ return 0;
+}
+
+/* Dump ADDRESS using flags FLAGS. */
+
+static void
+dump_address_info (dump_flags_t flags, address_info &address)
+{
+ if (address.base)
+ dump_printf (flags, "%T + ", address.base);
+ for (unsigned int i = 0; i < address.terms.length (); ++i)
+ {
+ if (i != 0)
+ dump_printf (flags, " + ");
+ dump_printf (flags, "%T", address.terms[i].expr);
+ if (address.terms[i].multiplier != 1)
+ dump_printf (flags, " * %wd", address.terms[i].multiplier);
+ }
+ dump_printf (flags, " + [%wd, %wd]",
+ address.min_offset, address.max_offset - 1);
+}
+
+/* Hash an address_info based on its base and terms. */
+
+hashval_t
+address_info_hasher::hash (const address_info *info)
+{
+ inchash::hash hash;
+ hash.add_int (info->base ? TREE_CODE (info->base) : 0);
+ hash.add_int (info->terms.length ());
+ for (unsigned int i = 0; i < info->terms.length (); ++i)
+ {
+ hash.add_int (SSA_NAME_VERSION (info->terms[i].expr));
+ hash.add_hwi (info->terms[i].multiplier);
+ }
+ return hash.end ();
+}
+
+/* Return true if two address_infos have equal bases and terms. Other
+ properties might be different (such as the statement or constant
+ offset range). */
+
+bool
+address_info_hasher::equal (const address_info *a, const address_info *b)
+{
+ if (a->base != b->base
+ && (!a->base || !b->base || !operand_equal_p (a->base, b->base, 0)))
+ return false;
+
+ if (a->terms.length () != b->terms.length ())
+ return false;
+
+ for (unsigned int i = 0; i < a->terms.length (); ++i)
+ if (a->terms[i].expr != b->terms[i].expr
+ || a->terms[i].multiplier != b->terms[i].multiplier)
+ return false;
+
+ return true;
+}
+
+/* Return true if we want to version the loop, i.e. if we have a
+ specific reason for doing so and no specific reason not to. */
+
+bool
+loop_info::worth_versioning_p () const
+{
+ return (!rejected_p
+ && (!bitmap_empty_p (&unity_names) || subloops_benefit_p));
+}
+
+loop_versioning::lv_dom_walker::lv_dom_walker (loop_versioning &lv)
+ : dom_walker (CDI_DOMINATORS), m_lv (lv), m_range_analyzer (false)
+{
+}
+
+/* Process BB before processing the blocks it dominates. */
+
+edge
+loop_versioning::lv_dom_walker::before_dom_children (basic_block bb)
+{
+ m_range_analyzer.enter (bb);
+
+ if (bb == bb->loop_father->header)
+ m_lv.prune_loop_conditions (bb->loop_father,
+ m_range_analyzer.get_vr_values ());
+
+ for (gimple_stmt_iterator si = gsi_start_bb (bb); !gsi_end_p (si);
+ gsi_next (&si))
+ m_range_analyzer.record_ranges_from_stmt (gsi_stmt (si), false);
+
+ return NULL;
+}
+
+/* Process BB after processing the blocks it dominates. */
+
+void
+loop_versioning::lv_dom_walker::after_dom_children (basic_block bb)
+{
+ m_range_analyzer.leave (bb);
+}
+
+/* Decide whether to replace VAL with a new value in a versioned loop.
+ Return the new value if so, otherwise return null. */
+
+tree
+loop_versioning::name_prop::get_value (tree val)
+{
+ if (TREE_CODE (val) == SSA_NAME
+ && bitmap_bit_p (&m_li.unity_names, SSA_NAME_VERSION (val)))
+ return build_one_cst (TREE_TYPE (val));
+ return NULL_TREE;
+}
+
+/* Initialize the structure to optimize FN. */
+
+loop_versioning::loop_versioning (function *fn)
+ : m_fn (fn),
+ m_nloops (number_of_loops (fn)),
+ m_num_conditions (0),
+ m_address_table (31)
+{
+ bitmap_obstack_initialize (&m_bitmap_obstack);
+ gcc_obstack_init (&m_obstack);
+
+ /* Initialize the loop information. */
+ m_loops.safe_grow_cleared (m_nloops);
+ for (unsigned int i = 0; i < m_nloops; ++i)
+ {
+ m_loops[i].outermost = get_loop (m_fn, 0);
+ bitmap_initialize (&m_loops[i].unity_names, &m_bitmap_obstack);
+ }
+
+ /* Initialize the list of blocks that belong to each loop. */
+ unsigned int nbbs = last_basic_block_for_fn (fn);
+ m_next_block_in_loop.safe_grow (nbbs);
+ basic_block bb;
+ FOR_EACH_BB_FN (bb, fn)
+ {
+ loop_info &li = get_loop_info (bb->loop_father);
+ m_next_block_in_loop[bb->index] = li.block_list;
+ li.block_list = bb;
+ }
+
+ /* MAX_FIXED_MODE_SIZE should be a reasonable maximum scale for
+ unvectorizable code, since it is the largest size that can be
+ handled efficiently by scalar code. omp_max_vf calculates the
+ maximum number of bytes in a vector, when such a value is relevant
+ to loop optimization. */
+ m_maximum_scale = estimated_poly_value (omp_max_vf ());
+ m_maximum_scale = MAX (m_maximum_scale, MAX_FIXED_MODE_SIZE);
+}
+
+loop_versioning::~loop_versioning ()
+{
+ bitmap_obstack_release (&m_bitmap_obstack);
+ obstack_free (&m_obstack, NULL);
+}
+
+/* Return the maximum number of instructions allowed in LOOP before
+ it becomes too big for versioning.
+
+ There are separate limits for inner and outer loops. The limit for
+ inner loops applies only to loops that benefit directly from versioning.
+ The limit for outer loops applies to all code in the outer loop and
+ its subloops that *doesn't* benefit directly from versioning; such code
+ would be "taken along for the ride". The idea is that if the cost of
+ the latter is small, it is better to version outer loops rather than
+ inner loops, both to reduce the number of repeated checks and to enable
+ more of the loop nest to be optimized as a natural nest (e.g. by loop
+ interchange or outer-loop vectorization). */
+
+unsigned int
+loop_versioning::max_insns_for_loop (struct loop *loop)
+{
+ return (loop->inner
+ ? PARAM_VALUE (PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS)
+ : PARAM_VALUE (PARAM_LOOP_VERSIONING_MAX_INNER_INSNS));
+}
+
+/* Return true if for cost reasons we should avoid versioning any loop
+ that contains STMT.
+
+ Note that we don't need to check whether versioning is invalid for
+ correctness reasons, since the versioning process does that for us.
+ The conditions involved are too rare to be worth duplicating here. */
+
+bool
+loop_versioning::expensive_stmt_p (gimple *stmt)
+{
+ if (gcall *call = dyn_cast <gcall *> (stmt))
+ /* Assume for now that the time spent in an "expensive" call would
+ overwhelm any saving from versioning. */
+ return !gimple_inexpensive_call_p (call);
+ return false;
+}
+
+/* Record that we want to version the loop that contains STMT for the
+ case in which SSA name NAME is equal to 1. We already know that NAME
+ is invariant in the loop. */
+
+void
+loop_versioning::version_for_unity (gimple *stmt, tree name)
+{
+ struct loop *loop = loop_containing_stmt (stmt);
+ loop_info &li = get_loop_info (loop);
+
+ if (bitmap_set_bit (&li.unity_names, SSA_NAME_VERSION (name)))
+ {
+ /* This is the first time we've wanted to version LOOP for NAME.
+ Keep track of the outermost loop that can handle all versioning
+ checks in LI. */
+ struct loop *outermost
+ = outermost_invariant_loop_for_expr (loop, name);
+ if (loop_depth (li.outermost) < loop_depth (outermost))
+ li.outermost = outermost;
+
+ if (dump_enabled_p ())
+ {
+ dump_printf_loc (MSG_NOTE, stmt, "want to version containing loop"
+ " for when %T == 1", name);
+ if (outermost == loop)
+ dump_printf (MSG_NOTE, "; cannot hoist check further");
+ else
+ {
+ dump_printf (MSG_NOTE, "; could implement the check at loop"
+ " depth %d", loop_depth (outermost));
+ if (loop_depth (li.outermost) > loop_depth (outermost))
+ dump_printf (MSG_NOTE, ", but other checks only allow"
+ " a depth of %d", loop_depth (li.outermost));
+ }
+ dump_printf (MSG_NOTE, "\n");
+ }
+
+ m_num_conditions += 1;
+ }
+ else
+ {
+ /* This is a duplicate request. */
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, stmt, "already asked to version containing"
+ " loop for when %T == 1\n", name);
+ }
+}
+
+/* Return true if OP1_TREE is constant and if in principle it is worth
+ versioning an address fragment of the form:
+
+ i * OP1_TREE * OP2 * stride
+
+ for the case in which stride == 1. This in practice means testing
+ whether:
+
+ OP1_TREE * OP2 <= M_MAXIMUM_SCALE.
+
+ If RESULT is nonnull, store OP1_TREE * OP2 there when returning true. */
+
+bool
+loop_versioning::acceptable_multiplier_p (tree op1_tree,
+ unsigned HOST_WIDE_INT op2,
+ unsigned HOST_WIDE_INT *result)
+{
+ if (tree_fits_uhwi_p (op1_tree))
+ {
+ unsigned HOST_WIDE_INT op1 = tree_to_uhwi (op1_tree);
+ /* The first part checks for overflow. */
+ if (op1 * op2 >= op2 && op1 * op2 <= m_maximum_scale)
+ {
+ if (result)
+ *result = op1 * op2;
+ return true;
+ }
+ }
+ return false;
+}
+
+/* Return true if it is worth using loop versioning on a memory access
+ of type TYPE. Store the size of the access in *SIZE if so. */
+
+bool
+loop_versioning::acceptable_type_p (tree type, unsigned HOST_WIDE_INT *size)
+{
+ return (TYPE_SIZE_UNIT (type)
+ && acceptable_multiplier_p (TYPE_SIZE_UNIT (type), 1, size));
+}
+
+/* See whether OP is constant and whether we can multiply TERM by that
+ constant without exceeding M_MAXIMUM_SCALE. Return true and update
+ TERM if so. */
+
+bool
+loop_versioning::multiply_term_by (address_term_info &term, tree op)
+{
+ return acceptable_multiplier_p (op, term.multiplier, &term.multiplier);
+}
+
+/* Decide whether an address fragment of the form STRIDE * MULTIPLIER
+ is likely to be indexing an innermost dimension, returning the result
+ as an INNER_* probability. */
+
+inner_likelihood
+loop_versioning::get_inner_likelihood (tree stride,
+ unsigned HOST_WIDE_INT multiplier)
+{
+ const unsigned int MAX_NITERS = 8;
+
+ /* Iterate over possible values of STRIDE. Return INNER_LIKELY if at
+ least one of those values is likely to be for the innermost dimension.
+ Record in UNLIKELY_P if at least one of those values is unlikely to be
+ for the innermost dimension.
+
+ E.g. for:
+
+ stride = cond ? a * b : 1
+
+ we should treat STRIDE as being a likely inner dimension, since
+ we know that it is 1 under at least some circumstances. (See the
+ Fortran example below.) However:
+
+ stride = a * b
+
+ on its own is unlikely to be for the innermost dimension, since
+ that would require both a and b to be 1 at runtime. */
+ bool unlikely_p = false;
+ tree worklist[MAX_NITERS];
+ unsigned int length = 0;
+ worklist[length++] = stride;
+ for (unsigned int i = 0; i < length; ++i)
+ {
+ tree expr = worklist[i];
+
+ if (CONSTANT_CLASS_P (expr))
+ {
+ /* See if EXPR * MULTIPLIER would be consistent with an individual
+ access or a small grouped access. */
+ if (acceptable_multiplier_p (expr, multiplier))
+ return INNER_LIKELY;
+ else
+ unlikely_p = true;
+ }
+ else if (gimple *stmt = maybe_get_stmt (expr))
+ {
+ /* If EXPR is set by a PHI node, queue its arguments in case
+ we find one that is consistent with an inner dimension.
+
+ An important instance of this is the Fortran handling of array
+ descriptors, which calculates the stride of the inner dimension
+ using a PHI equivalent of:
+
+ raw_stride = a.dim[0].stride;
+ stride = raw_stride != 0 ? raw_stride : 1;
+
+ (Strides for outer dimensions do not treat 0 specially.) */
+ if (gphi *phi = dyn_cast <gphi *> (stmt))
+ {
+ unsigned int nargs = gimple_phi_num_args (phi);
+ for (unsigned int j = 0; j < nargs && length < MAX_NITERS; ++j)
+ worklist[length++] = strip_casts (gimple_phi_arg_def (phi, j));
+ }
+ /* If the value is set by an assignment, expect it to be read
+ from memory (such as an array descriptor) rather than be
+ calculated. */
+ else if (gassign *assign = dyn_cast <gassign *> (stmt))
+ {
+ if (!gimple_assign_load_p (assign))
+ unlikely_p = true;
+ }
+ /* Things like calls don't really tell us anything. */
+ }
+ }
+
+ /* We didn't find any possible values of STRIDE that were likely to be
+ for the innermost dimension. If we found one that was actively
+ unlikely to be for the innermost dimension, assume that that applies
+ to STRIDE too. */
+ return unlikely_p ? INNER_UNLIKELY : INNER_DONT_KNOW;
+}
+
+/* The caller has identified that STRIDE is the stride of interest
+ in TERM, and that the stride is applied in OP_LOOP. Record this
+ information in TERM, deciding whether STRIDE is likely to be for
+ the innermost dimension of an array and whether it represents a
+ versioning opportunity. ADDRESS is the address that contains TERM. */
+
+void
+loop_versioning::analyze_stride (address_info &address,
+ address_term_info &term,
+ tree stride, struct loop *op_loop)
+{
+ term.stride = stride;
+
+ term.inner_likelihood = get_inner_likelihood (stride, term.multiplier);
+ if (dump_enabled_p ())
+ {
+ if (term.inner_likelihood == INNER_LIKELY)
+ dump_printf_loc (MSG_NOTE, address.stmt, "%T is likely to be the"
+ " innermost dimension\n", stride);
+ else if (term.inner_likelihood == INNER_UNLIKELY)
+ dump_printf_loc (MSG_NOTE, address.stmt, "%T is probably not the"
+ " innermost dimension\n", stride);
+ else
+ dump_printf_loc (MSG_NOTE, address.stmt, "cannot tell whether %T"
+ " is the innermost dimension\n", stride);
+ }
+
+ /* To be a versioning opportunity we require:
+
+ - The multiplier applied by TERM is equal to the access size,
+ so that when STRIDE is 1, the accesses in successive loop
+ iterations are consecutive.
+
+ This is deliberately conservative. We could relax it to handle
+ other cases (such as those with gaps between iterations) if we
+ find any real testcases for which it's useful.
+
+ - the stride is applied in the same loop as STMT rather than
+ in an outer loop. Although versioning for strides applied in
+ outer loops could help in some cases -- such as enabling
+ more loop interchange -- the savings are much lower than for
+ inner loops.
+
+ - the stride is an SSA name that is invariant in STMT's loop,
+ since otherwise versioning isn't possible. */
+ unsigned HOST_WIDE_INT access_size = address.max_offset - address.min_offset;
+ if (term.multiplier == access_size
+ && address.loop == op_loop
+ && TREE_CODE (stride) == SSA_NAME
+ && expr_invariant_in_loop_p (address.loop, stride))
+ {
+ term.versioning_opportunity_p = true;
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, address.stmt, "%T == 1 is a versioning"
+ " opportunity\n", stride);
+ }
+}
+
+/* See whether address term TERM (which belongs to ADDRESS) is the result
+ of multiplying a varying SSA name by a loop-invariant SSA name.
+ Return true and update TERM if so.
+
+ This handles both cases that SCEV might handle, such as:
+
+ for (int i = 0; i < n; ++i)
+ res += a[i * stride];
+
+ and ones in which the term varies arbitrarily between iterations, such as:
+
+ for (int i = 0; i < n; ++i)
+ res += a[index[i] * stride]; */
+
+bool
+loop_versioning::find_per_loop_multiplication (address_info &address,
+ address_term_info &term)
+{
+ gimple *mult = maybe_get_assign (term.expr);
+ if (!mult || gimple_assign_rhs_code (mult) != MULT_EXPR)
+ return false;
+
+ struct loop *mult_loop = loop_containing_stmt (mult);
+ if (!loop_outer (mult_loop))
+ return false;
+
+ tree op1 = strip_casts (gimple_assign_rhs1 (mult));
+ tree op2 = strip_casts (gimple_assign_rhs2 (mult));
+ if (TREE_CODE (op1) != SSA_NAME || TREE_CODE (op2) != SSA_NAME)
+ return false;
+
+ bool invariant1_p = expr_invariant_in_loop_p (mult_loop, op1);
+ bool invariant2_p = expr_invariant_in_loop_p (mult_loop, op2);
+ if (invariant1_p == invariant2_p)
+ return false;
+
+ /* Make sure that the loop invariant is OP2 rather than OP1. */
+ if (invariant1_p)
+ std::swap (op1, op2);
+
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, address.stmt, "address term %T = varying %T"
+ " * loop-invariant %T\n", term.expr, op1, op2);
+ analyze_stride (address, term, op2, mult_loop);
+ return true;
+}
+
+/* Try to use scalar evolutions to find an address stride for TERM,
+ which belongs to ADDRESS.
+
+ Here we are interested in any evolution information we can find,
+ not just evolutions wrt ADDRESS->LOOP. For example, if we find that
+ an outer loop obviously iterates over the inner dimension of an array,
+ that information can help us eliminate worthless versioning opportunities
+ in inner loops. */
+
+void
+loop_versioning::analyze_term_using_scevs (address_info &address,
+ address_term_info &term)
+{
+ gimple *setter = maybe_get_stmt (term.expr);
+ if (!setter)
+ return;
+
+ struct loop *wrt_loop = loop_containing_stmt (setter);
+ if (!loop_outer (wrt_loop))
+ return;
+
+ tree chrec = strip_casts (analyze_scalar_evolution (wrt_loop, term.expr));
+ if (TREE_CODE (chrec) == POLYNOMIAL_CHREC)
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, address.stmt,
+ "address term %T = %T\n", term.expr, chrec);
+
+ /* Peel casts and accumulate constant multiplications, up to the
+ limit allowed by M_MAXIMUM_SCALE. */
+ tree stride = strip_casts (CHREC_RIGHT (chrec));
+ while (TREE_CODE (stride) == MULT_EXPR
+ && multiply_term_by (term, TREE_OPERAND (stride, 1)))
+ stride = strip_casts (TREE_OPERAND (stride, 0));
+
+ gassign *assign;
+ while ((assign = maybe_get_assign (stride))
+ && gimple_assign_rhs_code (assign) == MULT_EXPR
+ && multiply_term_by (term, gimple_assign_rhs2 (assign)))
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, address.stmt,
+ "looking through %G", assign);
+ stride = strip_casts (gimple_assign_rhs1 (assign));
+ }
+
+ analyze_stride (address, term, stride, get_chrec_loop (chrec));
+ }
+}
+
+/* Try to identify loop strides in ADDRESS and try to choose realistic
+ versioning opportunities based on these strides.
+
+ The main difficulty here isn't finding strides that could be used
+ in a version check (that's pretty easy). The problem instead is to
+ avoid versioning for some stride S that is unlikely ever to be 1 at
+ runtime. Versioning for S == 1 on its own would lead to unnecessary
+ code bloat, while adding S == 1 to more realistic version conditions
+ would lose the optimisation opportunity offered by those other conditions.
+
+ For example, versioning for a stride of 1 in the Fortran code:
+
+ integer :: a(:,:)
+ a(1,:) = 1
+
+ is not usually a good idea, since the assignment is iterating over
+ an outer dimension and is relatively unlikely to have a stride of 1.
+ (It isn't impossible, since the inner dimension might be 1, or the
+ array might be transposed.) Similarly, in:
+
+ integer :: a(:,:), b(:,:)
+ b(:,1) = a(1,:)
+
+ b(:,1) is relatively likely to have a stride of 1 while a(1,:) isn't.
+ Versioning for when both strides are 1 would lose most of the benefit
+ of versioning for b's access.
+
+ The approach we take is as follows:
+
+ - Analyze each term to see whether it has an identifiable stride,
+ regardless of which loop applies the stride.
+
+ - Evaluate the likelihood that each such stride is for the innermost
+ dimension of an array, on the scale "likely", "don't know" or "unlikely".
+
+ - If there is a single "likely" innermost stride, and that stride is
+ applied in the loop that contains STMT, version the loop for when the
+ stride is 1. This deals with the cases in which we're fairly
+ confident of doing the right thing, such as the b(:,1) reference above.
+
+ - If there are no "likely" innermost strides, and the loop that contains
+ STMT uses a stride that we rated as "don't know", version for when
+ that stride is 1. This is principally used for C code such as:
+
+ for (int i = 0; i < n; ++i)
+ a[i * x] = ...;
+
+ and:
+
+ for (int j = 0; j < n; ++j)
+ for (int i = 0; i < n; ++i)
+ a[i * x + j * y] = ...;
+
+ where nothing in the way "x" and "y" are set gives a hint as to
+ whether "i" iterates over the innermost dimension of the array.
+ In these situations it seems reasonable to assume the the
+ programmer has nested the loops appropriately (although of course
+ there are examples like GEMM in which this assumption doesn't hold
+ for all accesses in the loop).
+
+ This case is also useful for the Fortran equivalent of the
+ above C code. */
+
+void
+loop_versioning::analyze_address_fragment (address_info &address)
+{
+ if (dump_enabled_p ())
+ {
+ dump_printf_loc (MSG_NOTE, address.stmt, "analyzing address fragment ");
+ dump_address_info (MSG_NOTE, address);
+ dump_printf (MSG_NOTE, "\n");
+ }
+
+ /* Analyze each component of the sum to see whether it involves an
+ apparent stride.
+
+ There is an overlap between the addresses that
+ find_per_loop_multiplication and analyze_term_using_scevs can handle,
+ but the former is much cheaper than SCEV analysis, so try it first. */
+ for (unsigned int i = 0; i < address.terms.length (); ++i)
+ if (!find_per_loop_multiplication (address, address.terms[i]))
+ analyze_term_using_scevs (address, address.terms[i]);
+
+ /* Check for strides that are likely to be for the innermost dimension.
+
+ 1. If there is a single likely inner stride, if it is an SSA name,
+ and if it is worth versioning the loop for when the SSA name
+ equals 1, record that we want to do so.
+
+ 2. Otherwise, if there any likely inner strides, bail out. This means
+ one of:
+
+ (a) There are multiple likely inner strides. This suggests we're
+ confused and be can't be confident of doing the right thing.
+
+ (b) There is a single likely inner stride and it is a constant
+ rather than an SSA name. This can mean either that the access
+ is a natural one without any variable strides, such as:
+
+ for (int i = 0; i < n; ++i)
+ a[i] += 1;
+
+ or that a variable stride is applied to an outer dimension,
+ such as:
+
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ a[j * stride][i] += 1;
+
+ (c) There is a single likely inner stride, and it is an SSA name,
+ but it isn't a worthwhile versioning opportunity. This usually
+ means that the variable stride is applied by an outer loop,
+ such as:
+
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ a[j][i * stride] += 1;
+
+ or (using an example with a more natural loop nesting):
+
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ a[i][j] += b[i * stride];
+
+ in cases where b[i * stride] cannot (yet) be hoisted for
+ aliasing reasons.
+
+ 3. If there are no likely inner strides, fall through to the next
+ set of checks.
+
+ Pointer equality is enough to check for uniqueness in (1), since we
+ only care about SSA names. */
+ tree chosen_stride = NULL_TREE;
+ tree version_stride = NULL_TREE;
+ for (unsigned int i = 0; i < address.terms.length (); ++i)
+ if (chosen_stride != address.terms[i].stride
+ && address.terms[i].inner_likelihood == INNER_LIKELY)
+ {
+ if (chosen_stride)
+ return;
+ chosen_stride = address.terms[i].stride;
+ if (address.terms[i].versioning_opportunity_p)
+ version_stride = chosen_stride;
+ }
+
+ /* If there are no likely inner strides, see if there is a single
+ versioning opportunity for a stride that was rated as INNER_DONT_KNOW.
+ See the comment above the function for the cases that this code
+ handles. */
+ if (!chosen_stride)
+ for (unsigned int i = 0; i < address.terms.length (); ++i)
+ if (version_stride != address.terms[i].stride
+ && address.terms[i].inner_likelihood == INNER_DONT_KNOW
+ && address.terms[i].versioning_opportunity_p)
+ {
+ if (version_stride)
+ return;
+ version_stride = address.terms[i].stride;
+ }
+
+ if (version_stride)
+ version_for_unity (address.stmt, version_stride);
+}
+
+/* Treat EXPR * MULTIPLIER + OFFSET as a fragment of an address that addresses
+ TYPE_SIZE bytes and record this address fragment for later processing.
+ STMT is the statement that contains the address. */
+
+void
+loop_versioning::record_address_fragment (gimple *stmt,
+ unsigned HOST_WIDE_INT type_size,
+ tree expr,
+ unsigned HOST_WIDE_INT multiplier,
+ HOST_WIDE_INT offset)
+{
+ /* We're only interested in computed values. */
+ if (TREE_CODE (expr) != SSA_NAME)
+ return;
+
+ /* Quick exit if no part of the address is calculated in STMT's loop,
+ since such addresses have no versioning opportunities. */
+ struct loop *loop = loop_containing_stmt (stmt);
+ if (expr_invariant_in_loop_p (loop, expr))
+ return;
+
+ /* Set up an address_info for EXPR * MULTIPLIER. */
+ address_info *address = XOBNEW (&m_obstack, address_info);
+ new (address) address_info;
+ address->stmt = stmt;
+ address->loop = loop;
+ address->base = NULL_TREE;
+ address->terms.quick_grow (1);
+ address->terms[0].expr = expr;
+ address->terms[0].multiplier = multiplier;
+ address->terms[0].stride = NULL_TREE;
+ address->terms[0].inner_likelihood = INNER_UNLIKELY;
+ address->terms[0].versioning_opportunity_p = false;
+ address->min_offset = offset;
+
+ /* Peel apart the expression into a sum of address_terms, where each
+ term is multiplied by a constant. Treat a + b and a - b the same,
+ since it doesn't matter for our purposes whether an address is
+ increasing or decreasing. Distribute (a + b) * constant into
+ a * constant + b * constant.
+
+ We don't care which loop each term belongs to, since we want to
+ examine as many candidate strides as possible when determining
+ which is likely to be for the innermost dimension. We therefore
+ don't limit the search to statements in STMT's loop. */
+ for (unsigned int i = 0; i < address->terms.length (); )
+ {
+ if (gassign *assign = maybe_get_assign (address->terms[i].expr))
+ {
+ tree_code code = gimple_assign_rhs_code (assign);
+ if (code == PLUS_EXPR
+ || code == POINTER_PLUS_EXPR
+ || code == MINUS_EXPR)
+ {
+ tree op1 = gimple_assign_rhs1 (assign);
+ tree op2 = gimple_assign_rhs2 (assign);
+ if (TREE_CODE (op2) == INTEGER_CST)
+ {
+ address->terms[i].expr = strip_casts (op1);
+ /* This is heuristic only, so don't worry about truncation
+ or overflow. */
+ address->min_offset += (TREE_INT_CST_LOW (op2)
+ * address->terms[i].multiplier);
+ continue;
+ }
+ else if (address->terms.length () < address_info::MAX_TERMS)
+ {
+ unsigned int j = address->terms.length ();
+ address->terms.quick_push (address->terms[i]);
+ address->terms[i].expr = strip_casts (op1);
+ address->terms[j].expr = strip_casts (op2);
+ continue;
+ }
+ }
+ if (code == MULT_EXPR)
+ {
+ tree op1 = gimple_assign_rhs1 (assign);
+ tree op2 = gimple_assign_rhs2 (assign);
+ if (multiply_term_by (address->terms[i], op2))
+ {
+ address->terms[i].expr = strip_casts (op1);
+ continue;
+ }
+ }
+ }
+ i += 1;
+ }
+
+ /* Peel off any symbolic pointer. */
+ if (TREE_CODE (address->terms[0].expr) != SSA_NAME
+ && address->terms[0].multiplier == 1)
+ {
+ if (address->terms.length () == 1)
+ {
+ obstack_free (&m_obstack, address);
+ return;
+ }
+ address->base = address->terms[0].expr;
+ address->terms.ordered_remove (0);
+ }
+
+ /* Require all remaining terms to be SSA names. (This could be false
+ for unfolded statements, but they aren't worth dealing with.) */
+ for (unsigned int i = 0; i < address->terms.length (); ++i)
+ if (TREE_CODE (address->terms[i].expr) != SSA_NAME)
+ {
+ obstack_free (&m_obstack, address);
+ return;
+ }
+
+ /* The loop above set MIN_OFFSET based on the first byte of the
+ referenced data. Calculate the end + 1. */
+ address->max_offset = address->min_offset + type_size;
+
+ /* Put the terms into a canonical order for the hash table lookup below. */
+ address->terms.qsort (compare_address_terms);
+
+ if (dump_enabled_p ())
+ {
+ dump_printf_loc (MSG_NOTE, stmt, "recording address fragment %T", expr);
+ if (multiplier != 1)
+ dump_printf (MSG_NOTE, " * %wd", multiplier);
+ dump_printf (MSG_NOTE, " = ");
+ dump_address_info (MSG_NOTE, *address);
+ dump_printf (MSG_NOTE, "\n");
+ }
+
+ /* Pool address information with the same terms (but potentially
+ different offsets). */
+ address_info **slot = m_address_table.find_slot (address, INSERT);
+ if (address_info *old_address = *slot)
+ {
+ /* We've already seen an address with the same terms. Extend the
+ offset range to account for this access. Doing this can paper
+ over gaps, such as in:
+
+ a[i * stride * 4] + a[i * stride * 4 + 3];
+
+ where nothing references "+ 1" or "+ 2". However, the vectorizer
+ handles such gapped accesses without problems, so it's not worth
+ trying to exclude them. */
+ if (old_address->min_offset > address->min_offset)
+ old_address->min_offset = address->min_offset;
+ if (old_address->max_offset < address->max_offset)
+ old_address->max_offset = address->max_offset;
+ obstack_free (&m_obstack, address);
+ }
+ else
+ {
+ /* This is the first time we've seen an address with these terms. */
+ *slot = address;
+ m_address_list.safe_push (address);
+ }
+}
+
+/* Analyze expression EXPR, which occurs in STMT. */
+
+void
+loop_versioning::analyze_expr (gimple *stmt, tree expr)
+{
+ unsigned HOST_WIDE_INT type_size;
+
+ while (handled_component_p (expr))
+ {
+ /* See whether we can use versioning to avoid a multiplication
+ in an array index. */
+ if (TREE_CODE (expr) == ARRAY_REF
+ && acceptable_type_p (TREE_TYPE (expr), &type_size))
+ record_address_fragment (stmt, type_size,
+ TREE_OPERAND (expr, 1), type_size, 0);
+ expr = TREE_OPERAND (expr, 0);
+ }
+
+ /* See whether we can use versioning to avoid a multiplication
+ in the pointer calculation of a MEM_REF. */
+ if (TREE_CODE (expr) == MEM_REF
+ && acceptable_type_p (TREE_TYPE (expr), &type_size))
+ record_address_fragment (stmt, type_size, TREE_OPERAND (expr, 0), 1,
+ /* This is heuristic only, so don't worry
+ about truncation or overflow. */
+ TREE_INT_CST_LOW (TREE_OPERAND (expr, 1)));
+
+ /* These would be easy to handle if they existed at this stage. */
+ gcc_checking_assert (TREE_CODE (expr) != TARGET_MEM_REF);
+}
+
+/* Analyze all the statements in BB looking for useful version checks.
+ Return true on success, false if something prevents the block from
+ being versioned. */
+
+bool
+loop_versioning::analyze_block (basic_block bb)
+{
+ struct loop *loop = bb->loop_father;
+ loop_info &li = get_loop_info (loop);
+ for (gimple_stmt_iterator gsi = gsi_start_bb (bb); !gsi_end_p (gsi);
+ gsi_next (&gsi))
+ {
+ gimple *stmt = gsi_stmt (gsi);
+ if (is_gimple_debug (stmt))
+ continue;
+
+ if (expensive_stmt_p (stmt))
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, stmt, "expensive statement"
+ " prevents versioning: %G", stmt);
+ return false;
+ }
+
+ /* Only look for direct versioning opportunities in inner loops
+ since the benefit tends to be much smaller for outer loops. */
+ if (!loop->inner)
+ {
+ unsigned int nops = gimple_num_ops (stmt);
+ for (unsigned int i = 0; i < nops; ++i)
+ if (tree op = gimple_op (stmt, i))
+ analyze_expr (stmt, op);
+ }
+
+ /* The point of the instruction limit is to prevent excessive
+ code growth, so this is a size-based estimate even though
+ the optimization is aimed at speed. */
+ li.num_insns += estimate_num_insns (stmt, &eni_size_weights);
+ }
+
+ return true;
+}
+
+/* Analyze all the blocks in the function, looking for useful version checks.
+ Return true if we found one. */
+
+bool
+loop_versioning::analyze_blocks ()
+{
+ AUTO_DUMP_SCOPE ("analyze_blocks",
+ dump_user_location_t::from_function_decl (m_fn->decl));
+
+ /* For now we don't try to version the whole function, although
+ versioning at that level could be useful in some cases. */
+ get_loop_info (get_loop (m_fn, 0)).rejected_p = true;
+
+ struct loop *loop;
+ FOR_EACH_LOOP (loop, LI_FROM_INNERMOST)
+ {
+ loop_info &linfo = get_loop_info (loop);
+
+ /* Ignore cold loops. */
+ if (!optimize_loop_for_speed_p (loop))
+ linfo.rejected_p = true;
+
+ /* See whether an inner loop prevents versioning of this loop. */
+ if (!linfo.rejected_p)
+ for (struct loop *inner = loop->inner; inner; inner = inner->next)
+ if (get_loop_info (inner).rejected_p)
+ {
+ linfo.rejected_p = true;
+ break;
+ }
+
+ /* If versioning the loop is still a possibility, examine the
+ statements in the loop to look for versioning opportunities. */
+ if (!linfo.rejected_p)
+ {
+ void *start_point = obstack_alloc (&m_obstack, 0);
+
+ for (basic_block bb = linfo.block_list; bb;
+ bb = m_next_block_in_loop[bb->index])
+ if (!analyze_block (bb))
+ {
+ linfo.rejected_p = true;
+ break;
+ }
+
+ if (!linfo.rejected_p)
+ {
+ /* Process any queued address fragments, now that we have
+ complete grouping information. */
+ address_info *address;
+ unsigned int i;
+ FOR_EACH_VEC_ELT (m_address_list, i, address)
+ analyze_address_fragment (*address);
+ }
+
+ m_address_table.empty ();
+ m_address_list.truncate (0);
+ obstack_free (&m_obstack, start_point);
+ }
+ }
+
+ return m_num_conditions != 0;
+}
+
+/* Use the ranges in VRS to remove impossible versioning conditions from
+ LOOP. */
+
+void
+loop_versioning::prune_loop_conditions (struct loop *loop, vr_values *vrs)
+{
+ loop_info &li = get_loop_info (loop);
+
+ int to_remove = -1;
+ bitmap_iterator bi;
+ unsigned int i;
+ EXECUTE_IF_SET_IN_BITMAP (&li.unity_names, 0, i, bi)
+ {
+ tree name = ssa_name (i);
+ value_range *vr = vrs->get_value_range (name);
+ if (vr && !range_includes_p (vr, 1))
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, find_loop_location (loop),
+ "%T can never be 1 in this loop\n", name);
+
+ if (to_remove >= 0)
+ bitmap_clear_bit (&li.unity_names, to_remove);
+ to_remove = i;
+ m_num_conditions -= 1;
+ }
+ }
+ if (to_remove >= 0)
+ bitmap_clear_bit (&li.unity_names, to_remove);
+}
+
+/* Remove any scheduled loop version conditions that will never be true.
+ Return true if any remain. */
+
+bool
+loop_versioning::prune_conditions ()
+{
+ AUTO_DUMP_SCOPE ("prune_loop_conditions",
+ dump_user_location_t::from_function_decl (m_fn->decl));
+
+ calculate_dominance_info (CDI_DOMINATORS);
+ lv_dom_walker dom_walker (*this);
+ dom_walker.walk (ENTRY_BLOCK_PTR_FOR_FN (m_fn));
+ return m_num_conditions != 0;
+}
+
+/* Merge the version checks for INNER into immediately-enclosing loop
+ OUTER. */
+
+void
+loop_versioning::merge_loop_info (struct loop *outer, struct loop *inner)
+{
+ loop_info &inner_li = get_loop_info (inner);
+ loop_info &outer_li = get_loop_info (outer);
+
+ if (dump_enabled_p ())
+ {
+ bitmap_iterator bi;
+ unsigned int i;
+ EXECUTE_IF_SET_IN_BITMAP (&inner_li.unity_names, 0, i, bi)
+ if (!bitmap_bit_p (&outer_li.unity_names, i))
+ dump_printf_loc (MSG_NOTE, find_loop_location (inner),
+ "hoisting check that %T == 1 to outer loop\n",
+ ssa_name (i));
+ }
+
+ bitmap_ior_into (&outer_li.unity_names, &inner_li.unity_names);
+ if (loop_depth (outer_li.outermost) < loop_depth (inner_li.outermost))
+ outer_li.outermost = inner_li.outermost;
+}
+
+/* Add LOOP to the queue of loops to version. */
+
+void
+loop_versioning::add_loop_to_queue (struct loop *loop)
+{
+ loop_info &li = get_loop_info (loop);
+
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, find_loop_location (loop),
+ "queuing this loop for versioning\n");
+ m_loops_to_version.safe_push (loop);
+
+ /* Don't try to version superloops. */
+ li.rejected_p = true;
+}
+
+/* Decide whether the cost model would allow us to version LOOP,
+ either directly or as part of a parent loop, and return true if so.
+ This does not imply that the loop is actually worth versioning in its
+ own right, just that it would be valid to version it if something
+ benefited.
+
+ We have already made this decision for all inner loops of LOOP. */
+
+bool
+loop_versioning::decide_whether_loop_is_versionable (struct loop *loop)
+{
+ loop_info &li = get_loop_info (loop);
+
+ if (li.rejected_p)
+ return false;
+
+ /* Examine the decisions made for inner loops. */
+ for (struct loop *inner = loop->inner; inner; inner = inner->next)
+ {
+ loop_info &inner_li = get_loop_info (inner);
+ if (inner_li.rejected_p)
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, find_loop_location (loop),
+ "not versioning this loop because one of its"
+ " inner loops should not be versioned\n");
+ return false;
+ }
+
+ if (inner_li.worth_versioning_p ())
+ li.subloops_benefit_p = true;
+
+ /* Accumulate the number of instructions from subloops that are not
+ the innermost, or that don't benefit from versioning. Only the
+ instructions from innermost loops that benefit from versioning
+ should be weighed against loop-versioning-max-inner-insns;
+ everything else should be weighed against
+ loop-versioning-max-outer-insns. */
+ if (!inner_li.worth_versioning_p () || inner->inner)
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, find_loop_location (loop),
+ "counting %d instructions from this loop"
+ " against its parent loop\n", inner_li.num_insns);
+ li.num_insns += inner_li.num_insns;
+ }
+ }
+
+ /* Enforce the size limits. */
+ if (li.worth_versioning_p ())
+ {
+ unsigned int max_num_insns = max_insns_for_loop (loop);
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, find_loop_location (loop),
+ "this loop has %d instructions, against"
+ " a versioning limit of %d\n",
+ li.num_insns, max_num_insns);
+ if (li.num_insns > max_num_insns)
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_MISSED_OPTIMIZATION
+ | MSG_PRIORITY_USER_FACING,
+ find_loop_location (loop),
+ "this loop is too big to version");
+ return false;
+ }
+ }
+
+ /* Hoist all version checks from subloops to this loop. */
+ for (struct loop *subloop = loop->inner; subloop; subloop = subloop->next)
+ merge_loop_info (loop, subloop);
+
+ return true;
+}
+
+/* Decide which loops to version and add them to the versioning queue.
+ Return true if there are any loops to version. */
+
+bool
+loop_versioning::make_versioning_decisions ()
+{
+ AUTO_DUMP_SCOPE ("make_versioning_decisions",
+ dump_user_location_t::from_function_decl (m_fn->decl));
+
+ struct loop *loop;
+ FOR_EACH_LOOP (loop, LI_FROM_INNERMOST)
+ {
+ loop_info &linfo = get_loop_info (loop);
+ if (decide_whether_loop_is_versionable (loop))
+ {
+ /* Commit to versioning LOOP directly if we can't hoist the
+ version checks any further. */
+ if (linfo.worth_versioning_p ()
+ && (loop_depth (loop) == 1 || linfo.outermost == loop))
+ add_loop_to_queue (loop);
+ }
+ else
+ {
+ /* We can't version this loop, so individually version any
+ subloops that would benefit and haven't been versioned yet. */
+ linfo.rejected_p = true;
+ for (struct loop *subloop = loop->inner; subloop;
+ subloop = subloop->next)
+ if (get_loop_info (subloop).worth_versioning_p ())
+ add_loop_to_queue (subloop);
+ }
+ }
+
+ return !m_loops_to_version.is_empty ();
+}
+
+/* Attempt to implement loop versioning for LOOP, using the information
+ cached in the associated loop_info. Return true on success. */
+
+bool
+loop_versioning::version_loop (struct loop *loop)
+{
+ loop_info &li = get_loop_info (loop);
+
+ /* Build up a condition that selects the original loop instead of
+ the simplified loop. */
+ tree cond = boolean_false_node;
+ bitmap_iterator bi;
+ unsigned int i;
+ EXECUTE_IF_SET_IN_BITMAP (&li.unity_names, 0, i, bi)
+ {
+ tree name = ssa_name (i);
+ tree ne_one = fold_build2 (NE_EXPR, boolean_type_node, name,
+ build_one_cst (TREE_TYPE (name)));
+ cond = fold_build2 (TRUTH_OR_EXPR, boolean_type_node, cond, ne_one);
+ }
+
+ /* Convert the condition into a suitable gcond. */
+ gimple_seq stmts = NULL;
+ cond = force_gimple_operand_1 (cond, &stmts, is_gimple_condexpr, NULL_TREE);
+
+ /* Version the loop. */
+ initialize_original_copy_tables ();
+ basic_block cond_bb;
+ li.optimized_loop = loop_version (loop, cond, &cond_bb,
+ profile_probability::unlikely (),
+ profile_probability::likely (),
+ profile_probability::unlikely (),
+ profile_probability::likely (), true);
+ free_original_copy_tables ();
+ if (!li.optimized_loop)
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_MISSED_OPTIMIZATION, find_loop_location (loop),
+ "tried but failed to version this loop for when"
+ " certain strides are 1\n");
+ return false;
+ }
+
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_OPTIMIZED_LOCATIONS, find_loop_location (loop),
+ "versioned this loop for when certain strides are 1\n");
+
+ /* Insert the statements that feed COND. */
+ if (stmts)
+ {
+ gimple_stmt_iterator gsi = gsi_last_bb (cond_bb);
+ gsi_insert_seq_before (&gsi, stmts, GSI_SAME_STMT);
+ }
+
+ return true;
+}
+
+/* Attempt to version all loops in the versioning queue. */
+
+void
+loop_versioning::implement_versioning_decisions ()
+{
+ /* No AUTO_DUMP_SCOPE here since all messages are top-level and
+ user-facing at this point. */
+
+ bool any_succeeded_p = false;
+ struct loop *loop;
+ unsigned int i;
+ FOR_EACH_VEC_ELT (m_loops_to_version, i, loop)
+ if (version_loop (loop))
+ any_succeeded_p = true;
+ if (!any_succeeded_p)
+ return;
+
+ update_ssa (TODO_update_ssa);
+
+ /* Simplify the new loop, which is used when COND is false. */
+ FOR_EACH_VEC_ELT (m_loops_to_version, i, loop)
+ {
+ loop_info &linfo = get_loop_info (loop);
+ if (linfo.optimized_loop)
+ name_prop (linfo).substitute_and_fold (linfo.optimized_loop->header);
+ }
+}
+
+/* Run the pass and return a set of TODO_* flags. */
+
+unsigned int
+loop_versioning::run ()
+{
+ gcc_assert (scev_initialized_p ());
+
+ if (analyze_blocks ()
+ && prune_conditions ()
+ && make_versioning_decisions ())
+ implement_versioning_decisions ();
+
+ return 0;
+}
+
+/* Loop versioning pass. */
+
+const pass_data pass_data_loop_versioning =
+{
+ GIMPLE_PASS, /* type */
+ "lversion", /* name */
+ OPTGROUP_LOOP, /* optinfo_flags */
+ TV_LOOP_VERSIONING, /* tv_id */
+ PROP_cfg, /* properties_required */
+ 0, /* properties_provided */
+ 0, /* properties_destroyed */
+ 0, /* todo_flags_start */
+ 0, /* todo_flags_finish */
+};
+
+class pass_loop_versioning : public gimple_opt_pass
+{
+public:
+ pass_loop_versioning (gcc::context *ctxt)
+ : gimple_opt_pass (pass_data_loop_versioning, ctxt)
+ {}
+
+ /* opt_pass methods: */
+ virtual bool gate (function *) { return flag_version_loops_for_strides; }
+ virtual unsigned int execute (function *);
+};
+
+unsigned int
+pass_loop_versioning::execute (function *fn)
+{
+ if (number_of_loops (fn) <= 1)
+ return 0;
+
+ return loop_versioning (fn).run ();
+}
+
+} // anon namespace
+
+gimple_opt_pass *
+make_pass_loop_versioning (gcc::context *ctxt)
+{
+ return new pass_loop_versioning (ctxt);
+}
Index: gcc/testsuite/gcc.dg/loop-versioning-1.c
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gcc.dg/loop-versioning-1.c 2018-12-14 15:02:53.817883430 +0000
@@ -0,0 +1,92 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* The simplest IV case. */
+
+void
+f1 (double *x, int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ x[stepx * i] = 100;
+}
+
+void
+f2 (double *x, int stepx, int limit)
+{
+ for (int i = 0; i < limit; i += stepx)
+ x[i] = 100;
+}
+
+void
+f3 (double *x, int stepx, int limit)
+{
+ for (double *y = x; y < x + limit; y += stepx)
+ *y = 100;
+}
+
+void
+f4 (double *x, int stepx, unsigned int n)
+{
+ for (unsigned int i = 0; i < n; ++i)
+ x[stepx * i] = 100;
+}
+
+void
+f5 (double *x, int stepx, unsigned int limit)
+{
+ for (unsigned int i = 0; i < limit; i += stepx)
+ x[i] = 100;
+}
+
+void
+f6 (double *x, int stepx, unsigned int limit)
+{
+ for (double *y = x; y < x + limit; y += stepx)
+ *y = 100;
+}
+
+double x[10000];
+
+void
+g1 (int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ x[stepx * i] = 100;
+}
+
+void
+g2 (int stepx, int limit)
+{
+ for (int i = 0; i < limit; i += stepx)
+ x[i] = 100;
+}
+
+void
+g3 (int stepx, int limit)
+{
+ for (double *y = x; y < x + limit; y += stepx)
+ *y = 100;
+}
+
+void
+g4 (int stepx, unsigned int n)
+{
+ for (unsigned int i = 0; i < n; ++i)
+ x[stepx * i] = 100;
+}
+
+void
+g5 (int stepx, unsigned int limit)
+{
+ for (unsigned int i = 0; i < limit; i += stepx)
+ x[i] = 100;
+}
+
+void
+g6 (int stepx, unsigned int limit)
+{
+ for (double *y = x; y < x + limit; y += stepx)
+ *y = 100;
+}
+
+/* { dg-final { scan-tree-dump-times {want to version containing loop} 12 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {versioned this loop} 12 "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-10.c
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gcc.dg/loop-versioning-10.c 2018-12-14 15:02:53.817883430 +0000
@@ -0,0 +1,52 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Test that we can version a gather-like operation in which a variable
+ stride is applied to the index. */
+
+int
+f1 (int *x, int *index, int step, int n)
+{
+ int res = 0;
+ for (int i = 0; i < n; ++i)
+ res += x[index[i] * step];
+ return res;
+}
+
+int
+f2 (int *x, int *index, int step, int n)
+{
+ int res = 0;
+ for (int i = 0; i < n; ++i)
+ {
+ int *ptr = x + index[i] * step;
+ res += *ptr;
+ }
+ return res;
+}
+
+int x[1000];
+
+int
+g1 (int *index, int step, int n)
+{
+ int res = 0;
+ for (int i = 0; i < n; ++i)
+ res += x[index[i] * step];
+ return res;
+}
+
+int
+g2 (int *index, int step, int n)
+{
+ int res = 0;
+ for (int i = 0; i < n; ++i)
+ {
+ int *ptr = x + index[i] * step;
+ res += *ptr;
+ }
+ return res;
+}
+
+/* { dg-final { scan-tree-dump-times {address term [^\n]* \* loop-invariant} 4 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {want to version containing loop} 4 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {versioned this loop} 4 "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-11.c
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gcc.dg/loop-versioning-11.c 2018-12-14 15:02:53.817883430 +0000
@@ -0,0 +1,29 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Test that we don't try to version for something that is never 1. */
+
+void
+f1 (double *x, int stepx, int n)
+{
+ if (stepx == 1)
+ for (int i = 0; i < n; ++i)
+ x[i] = 100;
+ else
+ for (int i = 0; i < n; ++i)
+ x[stepx * i] = 100;
+}
+
+void
+f2 (double *x, int stepx, int n)
+{
+ if (stepx <= 1)
+ for (int i = 0; i < n; ++i)
+ x[i] = 100;
+ else
+ for (int i = 0; i < n; ++i)
+ x[stepx * i] = 100;
+}
+
+/* { dg-final { scan-tree-dump-times {want to version containing loop} 2 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {can never be 1} 2 "lversion" } } */
+/* { dg-final { scan-tree-dump-not {versioned} "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-2.c
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gcc.dg/loop-versioning-2.c 2018-12-14 15:02:53.817883430 +0000
@@ -0,0 +1,73 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Versioning for step == 1 in these loops would allow loop interchange,
+ but otherwise isn't worthwhile. At the moment we decide not to version. */
+
+void
+f1 (double x[][100], int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j * step][i] = 100;
+}
+
+void
+f2 (double x[][100], int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j][i * step] = 100;
+}
+
+void
+f3 (double x[][100], int step, int limit)
+{
+ for (int i = 0; i < 100; ++i)
+ for (int j = 0; j < limit; j += step)
+ x[j][i] = 100;
+}
+
+void
+f4 (double x[][100], int step, int limit)
+{
+ for (int i = 0; i < limit; i += step)
+ for (int j = 0; j < 100; ++j)
+ x[j][i] = 100;
+}
+
+double x[100][100];
+
+void
+g1 (int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j * step][i] = 100;
+}
+
+void
+g2 (int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j][i * step] = 100;
+}
+
+void
+g3 (int step, int limit)
+{
+ for (int i = 0; i < 100; ++i)
+ for (int j = 0; j < limit; j += step)
+ x[j][i] = 100;
+}
+
+void
+g4 (int step, int limit)
+{
+ for (int i = 0; i < limit; i += step)
+ for (int j = 0; j < 100; ++j)
+ x[j][i] = 100;
+}
+
+/* { dg-final { scan-tree-dump-not {want to version} "lversion" } } */
+/* { dg-final { scan-tree-dump-not {versioned} "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-3.c
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gcc.dg/loop-versioning-3.c 2018-12-14 15:02:53.817883430 +0000
@@ -0,0 +1,24 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Versioning these loops for when both steps are 1 allows loop
+ interchange, but otherwise isn't worthwhile. At the moment we decide
+ not to version. */
+
+void
+f1 (double x[][100], int step1, int step2, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j * step1][i * step2] = 100;
+}
+
+void
+f2 (double x[][100], int step1, int step2, int limit)
+{
+ for (int i = 0; i < limit; i += step1)
+ for (int j = 0; j < limit; j += step2)
+ x[j][i] = 100;
+}
+
+/* { dg-final { scan-tree-dump-not {want to version} "lversion" } } */
+/* { dg-final { scan-tree-dump-not {versioned} "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-4.c
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gcc.dg/loop-versioning-4.c 2018-12-14 15:02:53.817883430 +0000
@@ -0,0 +1,39 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* These shouldn't be versioned; it's extremely likely that the code
+ is emulating two-dimensional arrays. */
+
+void
+f1 (double *x, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[i * step + j] = 100;
+}
+
+void
+f2 (double *x, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j * step + i] = 100;
+}
+
+void
+f3 (double *x, int *offsets, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[i * step + j + offsets[i]] = 100;
+}
+
+void
+f4 (double *x, int *offsets, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j * step + i + offsets[i]] = 100;
+}
+
+/* { dg-final { scan-tree-dump-not {want to version} "lversion" } } */
+/* { dg-final { scan-tree-dump-not {versioned} "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-5.c
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gcc.dg/loop-versioning-5.c 2018-12-14 15:02:53.817883430 +0000
@@ -0,0 +1,17 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* There's no information about whether STEP1 or STEP2 is innermost,
+ so we should assume the code is sensible and version for the inner
+ evolution, i.e. when STEP2 is 1. */
+
+void
+f1 (double *x, int step1, int step2, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[i * step1 + j * step2] = 100;
+}
+
+/* { dg-final { scan-tree-dump-times {want to version containing loop for when step2} 1 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {want to version containing loop} 1 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {versioned this loop} 1 "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-6.c
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gcc.dg/loop-versioning-6.c 2018-12-14 15:02:53.817883430 +0000
@@ -0,0 +1,31 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* The read from y in f1 will be hoisted to the outer loop. In general
+ it's not worth versioning outer loops when the inner loops don't also
+ benefit.
+
+ This test is meant to be a slight counterexample, since versioning
+ does lead to cheaper outer-loop vectorization. However, the benefit
+ isn't enough to justify the cost. */
+
+void
+f1 (double *restrict x, double *restrict y, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[i + j] = y[i * step];
+}
+
+/* A similar example in which the read can't be hoisted, but could
+ for example be handled by vectorizer alias checks. */
+
+void
+f2 (double *x, double *y, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[i + j] = y[i * step];
+}
+
+/* { dg-final { scan-tree-dump-not {want to version} "lversion" } } */
+/* { dg-final { scan-tree-dump-not {versioned} "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-7.c
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gcc.dg/loop-versioning-7.c 2018-12-14 15:02:53.817883430 +0000
@@ -0,0 +1,32 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Check that versioning can handle arrays of structures. */
+
+struct foo {
+ int a, b, c;
+};
+
+void
+f1 (struct foo *x, int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[stepx * i].a = 1;
+ x[stepx * i].b = 2;
+ x[stepx * i].c = 3;
+ }
+}
+
+void
+f2 (struct foo *x, int stepx, int limit)
+{
+ for (int i = 0; i < limit; i += stepx)
+ {
+ x[i].a = 1;
+ x[i].b = 2;
+ x[i].c = 3;
+ }
+}
+
+/* { dg-final { scan-tree-dump-times {want to version containing loop} 2 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {versioned this loop} 2 "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-8.c
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gcc.dg/loop-versioning-8.c 2018-12-14 15:02:53.817883430 +0000
@@ -0,0 +1,43 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Versioning for step == 1 in these loops would allow loop interchange,
+ but otherwise isn't worthwhile. At the moment we decide not to version. */
+
+struct foo {
+ int a[100];
+};
+
+void
+f1 (struct foo *x, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j * step].a[i] = 100;
+}
+
+void
+f2 (struct foo *x, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j].a[i * step] = 100;
+}
+
+void
+f3 (struct foo *x, int step, int limit)
+{
+ for (int i = 0; i < 100; ++i)
+ for (int j = 0; j < limit; j += step)
+ x[j].a[i] = 100;
+}
+
+void
+f4 (struct foo *x, int step, int limit)
+{
+ for (int i = 0; i < limit; i += step)
+ for (int j = 0; j < 100; ++j)
+ x[j].a[i] = 100;
+}
+
+/* { dg-final { scan-tree-dump-not {want to version} "lversion" } } */
+/* { dg-final { scan-tree-dump-not {versioned} "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-9.c
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gcc.dg/loop-versioning-9.c 2018-12-14 15:02:53.817883430 +0000
@@ -0,0 +1,48 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Check that versioning can handle small groups of accesses. */
+
+void
+f1 (int *x, int *y, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ x[i] = y[i * step * 2] + y[i * step * 2 + 1];
+}
+
+void
+f2 (int *x, int *y, __INTPTR_TYPE__ step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ x[i] = y[i * step * 2] + y[i * step * 2 + 1];
+}
+
+void
+f3 (int *x, int *y, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ x[i] = y[i * step * 3] + y[i * step * 3 + 2];
+}
+
+void
+f4 (int *x, int *y, __INTPTR_TYPE__ step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ x[i] = y[i * step * 3] + y[i * step * 3 + 2];
+}
+
+void
+f5 (int *x, int *y, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ x[i] = y[i * step * 4] + y[i * step * 4 + 3];
+}
+
+void
+f6 (int *x, int *y, __INTPTR_TYPE__ step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ x[i] = y[i * step * 4] + y[i * step * 4 + 3];
+}
+
+/* { dg-final { scan-tree-dump-times {want to version containing loop} 6 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {versioned this loop} 6 "lversion" } } */
Index: gcc/testsuite/gfortran.dg/loop_versioning_1.f90
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gfortran.dg/loop_versioning_1.f90 2018-12-14 15:02:53.817883430 +0000
@@ -0,0 +1,28 @@
+! { dg-options "-O3 -fdump-tree-lversion-details" }
+
+! The simplest IV case.
+
+subroutine f1(x)
+ real :: x(:)
+ x(:) = 100
+end subroutine f1
+
+subroutine f2(x, n, step)
+ integer :: n, step
+ real :: x(n * step)
+ do i = 1, n
+ x(i * step) = 100
+ end do
+end subroutine f2
+
+subroutine f3(x, limit, step)
+ integer :: limit, step
+ real :: x(limit)
+ do i = 1, limit, step
+ x(i) = 100
+ end do
+end subroutine f3
+
+! { dg-final { scan-tree-dump-times {likely to be the innermost dimension} 1 "lversion" } }
+! { dg-final { scan-tree-dump-times {want to version containing loop} 3 "lversion" } }
+! { dg-final { scan-tree-dump-times {versioned this loop} 3 "lversion" } }
Index: gcc/testsuite/gfortran.dg/loop_versioning_2.f90
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gfortran.dg/loop_versioning_2.f90 2018-12-14 15:02:53.817883430 +0000
@@ -0,0 +1,39 @@
+! { dg-options "-O3 -fdump-tree-lversion-details -fno-frontend-loop-interchange" }
+
+! We could version the loop for when the first dimension has a stride
+! of 1, but at present there's no real benefit. The gimple loop
+! interchange pass couldn't handle the versioned loop, and interchange
+! is instead done by the frontend (but disabled by the options above).
+
+subroutine f1(x)
+ real :: x(:, :)
+ do i = lbound(x, 1), ubound(x, 1)
+ do j = lbound(x, 2), ubound(x, 2)
+ x(i, j) = 100
+ end do
+ end do
+end subroutine f1
+
+subroutine f2(x, n, step)
+ integer :: n, step
+ real :: x(100, 100)
+ do i = 1, n
+ do j = 1, n
+ x(i * step, j) = 100
+ end do
+ end do
+end subroutine f2
+
+subroutine f3(x, n, step)
+ integer :: n, step
+ real :: x(n * step, n)
+ do i = 1, n
+ do j = 1, n
+ x(i * step, j) = 100
+ end do
+ end do
+end subroutine f3
+
+! { dg-final { scan-tree-dump-times {likely to be the innermost dimension} 1 "lversion" } }
+! { dg-final { scan-tree-dump-not {want to version} "lversion" } }
+! { dg-final { scan-tree-dump-not {versioned} "lversion" } }
Index: gcc/testsuite/gfortran.dg/loop_versioning_3.f90
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gfortran.dg/loop_versioning_3.f90 2018-12-14 15:02:53.817883430 +0000
@@ -0,0 +1,30 @@
+! { dg-options "-O3 -fdump-tree-lversion-details -fno-frontend-loop-interchange" }
+
+! Test a case in which the outer loop iterates over the inner dimension.
+! The options above prevent the frontend from interchanging the loops.
+
+subroutine f1(x, limit, step, n)
+ integer :: limit, step, n
+ real :: x(limit, n)
+ do i = 1, limit, step
+ do j = 1, n
+ x(i, j) = 100
+ end do
+ end do
+end subroutine f1
+
+subroutine f2(x, n, limit, step)
+ integer :: n, limit, step
+ real :: x(limit, n)
+ do i = 1, n
+ do j = 1, limit, step
+ x(j, i) = 100
+ end do
+ end do
+end subroutine f2
+
+! FIXME: The frontend doesn't give us enough information to tell which loop
+! is iterating over the innermost dimension, so we optimistically
+! assume the inner one is.
+! { dg-final { scan-tree-dump-not {want to version} "lversion" { xfail *-*-* } } }
+! { dg-final { scan-tree-dump-not {versioned} "lversion" { xfail *-*-* } } }
Index: gcc/testsuite/gfortran.dg/loop_versioning_4.f90
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gfortran.dg/loop_versioning_4.f90 2018-12-14 15:02:53.817883430 +0000
@@ -0,0 +1,95 @@
+! { dg-options "-O3 -fdump-tree-lversion-details -fno-frontend-loop-interchange" }
+
+! Test cases in which versioning is useful for a two-dimensional array.
+
+subroutine f1(x)
+ real :: x(:, :)
+ x(:, :) = 100
+end subroutine f1
+
+subroutine f2(x)
+ real :: x(:, :)
+ do i = lbound(x, 1), ubound(x, 1)
+ do j = lbound(x, 2), ubound(x, 2)
+ x(j, i) = 100
+ end do
+ end do
+end subroutine f2
+
+subroutine f3(x, n, step)
+ integer :: n, step
+ real :: x(100, 100)
+ do i = 1, n
+ do j = 1, n
+ x(j * step, i) = 100
+ end do
+ end do
+end subroutine f3
+
+subroutine f4(x, n, step)
+ integer :: n, step
+ real :: x(n * step, n)
+ do i = 1, n
+ do j = 1, n
+ x(j * step, i) = 100
+ end do
+ end do
+end subroutine f4
+
+subroutine f5(x, n, limit, step)
+ integer :: n, limit, step
+ real :: x(limit, n)
+ do i = 1, n
+ do j = 1, limit, step
+ x(j, i) = 100
+ end do
+ end do
+end subroutine f5
+
+subroutine f6(x, y)
+ real :: x(:, :), y(:)
+ do i = lbound(x, 1), ubound(x, 1)
+ do j = lbound(x, 2), ubound(x, 2)
+ x(j, i) = 100
+ end do
+ y(i) = 200
+ end do
+end subroutine f6
+
+subroutine f7(x, y, n, step)
+ integer :: n, step
+ real :: x(100, 100), y(100)
+ do i = 1, n
+ do j = 1, n
+ x(j * step, i) = 100
+ end do
+ y(i * step) = 200
+ end do
+end subroutine f7
+
+subroutine f8(x, y, n, step)
+ integer :: n, step
+ real :: x(n * step, n), y(n * step)
+ do i = 1, n
+ do j = 1, n
+ x(j * step, i) = 100
+ end do
+ y(i * step) = 200
+ end do
+end subroutine f8
+
+subroutine f9(x, n, limit, step)
+ integer :: n, limit, step
+ real :: x(limit, n), y(limit)
+ do i = 1, n
+ do j = 1, limit, step
+ x(j, i) = 100
+ end do
+ y(i) = 200
+ end do
+end subroutine f9
+
+! { dg-final { scan-tree-dump-times {likely to be the innermost dimension} 3 "lversion" } }
+! { dg-final { scan-tree-dump-times {want to version containing loop} 9 "lversion" } }
+! { dg-final { scan-tree-dump-times {hoisting check} 9 "lversion" } }
+! { dg-final { scan-tree-dump-times {versioned this loop} 9 "lversion" } }
Index: gcc/testsuite/gfortran.dg/loop_versioning_5.f90
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gfortran.dg/loop_versioning_5.f90 2018-12-14 15:02:53.817883430 +0000
@@ -0,0 +1,57 @@
+! { dg-options "-O3 -fdump-tree-lversion-details -fno-frontend-loop-interchange" }
+
+! Make sure that in a "badly nested" loop, we don't treat the inner loop
+! as iterating over the inner dimension with a variable stride.
+
+subroutine f1(x, n)
+ integer :: n
+ real :: x(100, 100)
+ do i = 1, n
+ do j = 1, n
+ x(i, j) = 100
+ end do
+ end do
+end subroutine f1
+
+subroutine f2(x, n, step)
+ integer :: n, step
+ real :: x(100, 100)
+ do i = 1, n
+ do j = 1, n
+ x(i, j * step) = 100
+ end do
+ end do
+end subroutine f2
+
+subroutine f3(x, n)
+ integer :: n
+ real :: x(n, n)
+ do i = 1, n
+ do j = 1, n
+ x(i, j) = 100
+ end do
+ end do
+end subroutine f3
+
+subroutine f4(x, n, step)
+ integer :: n, step
+ real :: x(n, n * step)
+ do i = 1, n
+ do j = 1, n
+ x(i, j * step) = 100
+ end do
+ end do
+end subroutine f4
+
+subroutine f5(x, n, limit, step)
+ integer :: n, limit, step
+ real :: x(n, limit)
+ do i = 1, n
+ do j = 1, limit, step
+ x(i, j) = 100
+ end do
+ end do
+end subroutine f5
+
+! { dg-final { scan-tree-dump-not {want to version} "lversion" } }
+! { dg-final { scan-tree-dump-not {versioned} "lversion" } }
Index: gcc/testsuite/gfortran.dg/loop_versioning_6.f90
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gfortran.dg/loop_versioning_6.f90 2018-12-14 15:02:53.817883430 +0000
@@ -0,0 +1,93 @@
+! { dg-options "-O3 -fdump-tree-lversion-details" }
+
+! Check that versioning can handle small groups of accesses.
+
+subroutine f1(x)
+ real :: x(:)
+ do i = lbound(x, 1), ubound(x, 1) / 2
+ x(i * 2) = 100
+ x(i * 2 + 1) = 101
+ end do
+end subroutine f1
+
+subroutine f2(x, n, step)
+ integer :: n, step
+ real :: x(n * step * 2)
+ do i = 1, n
+ x(i * step * 2) = 100
+ x(i * step * 2 + 1) = 101
+ end do
+end subroutine f2
+
+subroutine f3(x, limit, step)
+ integer :: limit, step
+ real :: x(limit * 2)
+ do i = 1, limit, step
+ x(i * 2) = 100
+ x(i * 2 + 1) = 101
+ end do
+end subroutine f3
+
+subroutine f4(x)
+ real :: x(:)
+ do i = lbound(x, 1), ubound(x, 1) / 3
+ x(i * 3) = 100
+ x(i * 3 + 1) = 101
+ x(i * 3 + 2) = 102
+ end do
+end subroutine f4
+
+subroutine f5(x, n, step)
+ integer :: n, step
+ real :: x(n * step * 3)
+ do i = 1, n
+ x(i * step * 3) = 100
+ x(i * step * 3 + 1) = 101
+ x(i * step * 3 + 2) = 102
+ end do
+end subroutine f5
+
+subroutine f6(x, limit, step)
+ integer :: limit, step
+ real :: x(limit * 3)
+ do i = 1, limit, step
+ x(i * 3) = 100
+ x(i * 3 + 1) = 101
+ x(i * 3 + 2) = 102
+ end do
+end subroutine f6
+
+subroutine f7(x)
+ real :: x(:)
+ do i = lbound(x, 1), ubound(x, 1) / 4
+ x(i * 4) = 100
+ x(i * 4 + 1) = 101
+ x(i * 4 + 2) = 102
+ x(i * 4 + 3) = 103
+ end do
+end subroutine f7
+
+subroutine f8(x, n, step)
+ integer :: n, step
+ real :: x(n * step * 4)
+ do i = 1, n
+ x(i * step * 4) = 100
+ x(i * step * 4 + 1) = 101
+ x(i * step * 4 + 2) = 102
+ x(i * step * 4 + 3) = 103
+ end do
+end subroutine f8
+
+subroutine f9(x, limit, step)
+ integer :: limit, step
+ real :: x(limit * 4)
+ do i = 1, limit, step
+ x(i * 4) = 100
+ x(i * 4 + 1) = 101
+ x(i * 4 + 2) = 102
+ x(i * 4 + 3) = 103
+ end do
+end subroutine f9
+
+! { dg-final { scan-tree-dump-times {want to version containing loop} 9 "lversion" } }
+! { dg-final { scan-tree-dump-times {versioned this loop} 9 "lversion" } }
Index: gcc/testsuite/gfortran.dg/loop_versioning_7.f90
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gfortran.dg/loop_versioning_7.f90 2018-12-14 15:02:53.821883396 +0000
@@ -0,0 +1,67 @@
+! { dg-options "-O3 -fdump-tree-lversion-details" }
+
+! Check that versioning can handle small groups of accesses, with the
+! group being a separate array dimension.
+
+subroutine f1(x, n, step)
+ integer :: n, step
+ real :: x(2, n * step)
+ do i = 1, n
+ x(1, i * step) = 100
+ x(2, i * step) = 101
+ end do
+end subroutine f1
+
+subroutine f2(x, limit, step)
+ integer :: limit, step
+ real :: x(2, limit)
+ do i = 1, limit, step
+ x(1, i) = 100
+ x(2, i) = 101
+ end do
+end subroutine f2
+
+subroutine f3(x, n, step)
+ integer :: n, step
+ real :: x(3, n * step)
+ do i = 1, n
+ x(1, i * step) = 100
+ x(2, i * step) = 101
+ x(3, i * step) = 102
+ end do
+end subroutine f3
+
+subroutine f4(x, limit, step)
+ integer :: limit, step
+ real :: x(3, limit)
+ do i = 1, limit, step
+ x(1, i) = 100
+ x(2, i) = 101
+ x(3, i) = 102
+ end do
+end subroutine f4
+
+subroutine f5(x, n, step)
+ integer :: n, step
+ real :: x(4, n * step)
+ do i = 1, n
+ x(1, i * step) = 100
+ x(2, i * step) = 101
+ x(3, i * step) = 102
+ x(4, i * step) = 103
+ end do
+end subroutine f5
+
+subroutine f6(x, limit, step)
+ integer :: limit, step
+ real :: x(4, limit)
+ do i = 1, limit, step
+ x(1, i) = 100
+ x(2, i) = 101
+ x(3, i) = 102
+ x(4, i) = 103
+ end do
+end subroutine f6
+
+! { dg-final { scan-tree-dump-times {want to version containing loop} 6 "lversion" } }
+! { dg-final { scan-tree-dump-times {versioned this loop} 6 "lversion" } }
Index: gcc/testsuite/gfortran.dg/loop_versioning_8.f90
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gfortran.dg/loop_versioning_8.f90 2018-12-14 15:02:53.821883396 +0000
@@ -0,0 +1,13 @@
+! { dg-options "-O3 -fdump-tree-lversion-details" }
+
+! Check that versioning is applied to a gather-like reduction operation.
+
+function f(x, index, n)
+ integer :: n
+ real :: x(:)
+ integer :: index(n)
+ f = sum(x(index(:)))
+end function f
+
+! { dg-final { scan-tree-dump-times {want to version containing loop} 1 "lversion" } }
+! { dg-final { scan-tree-dump-times {versioned this loop} 1 "lversion" } }
Index: gcc/testsuite/gcc.dg/loop-versioning-12.c
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gcc.dg/loop-versioning-12.c 2018-12-14 15:02:53.817883430 +0000
@@ -0,0 +1,149 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Test that we don't try to version for a step of 1 when that would
+ cause the iterations to overlap. */
+
+void
+f1 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx] = 100;
+ x[i * stepx + 1] = 99;
+ }
+}
+
+void
+f2 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx)
+ {
+ x[i] = 100;
+ x[i + 1] = 99;
+ }
+}
+
+void
+f3 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx - 16] = 100;
+ x[i * stepx - 15] = 99;
+ }
+}
+
+void
+f4 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx)
+ {
+ x[i - 16] = 100;
+ x[i - 15] = 99;
+ }
+}
+
+void
+f5 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx - 16] = 100;
+ x[i * stepx + 15] = 99;
+ }
+}
+
+void
+f6 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx)
+ {
+ x[i - 16] = 100;
+ x[i + 15] = 99;
+ }
+}
+
+void
+f7 (unsigned short *x, int stepx, int n)
+{
+ for (unsigned short *y = x; y < x + n; y += stepx)
+ {
+ y[0] = 100;
+ y[1] = 99;
+ }
+}
+
+unsigned short x[1000];
+
+void
+g1 (int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx] = 100;
+ x[i * stepx + 1] = 99;
+ }
+}
+
+void
+g2 (int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx)
+ {
+ x[i] = 100;
+ x[i + 1] = 99;
+ }
+}
+
+void
+g3 (int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx - 16] = 100;
+ x[i * stepx - 15] = 99;
+ }
+}
+
+void
+g4 (int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx)
+ {
+ x[i - 16] = 100;
+ x[i - 15] = 99;
+ }
+}
+
+void
+g5 (int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx - 16] = 100;
+ x[i * stepx + 15] = 99;
+ }
+}
+
+void
+g6 (int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx)
+ {
+ x[i - 16] = 100;
+ x[i + 15] = 99;
+ }
+}
+
+void
+g7 (int stepx, int n)
+{
+ for (unsigned short *y = x; y < x + n; y += stepx)
+ {
+ y[0] = 100;
+ y[1] = 99;
+ }
+}
+
+/* { dg-final { scan-tree-dump-not {want to version} "lversion" } } */
+/* { dg-final { scan-tree-dump-not {versioned} "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-13.c
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gcc.dg/loop-versioning-13.c 2018-12-14 15:02:53.817883430 +0000
@@ -0,0 +1,109 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Test that we do version for a step of 1 when that would lead the
+ iterations to access consecutive groups. */
+
+void
+f1 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx * 2] = 100;
+ x[i * stepx * 2 + 1] = 99;
+ }
+}
+
+void
+f2 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx * 2)
+ {
+ x[i] = 100;
+ x[i + 1] = 99;
+ }
+}
+
+void
+f3 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx * 2 - 16] = 100;
+ x[i * stepx * 2 - 15] = 99;
+ }
+}
+
+void
+f4 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx * 2)
+ {
+ x[i - 16] = 100;
+ x[i - 15] = 99;
+ }
+}
+
+void
+f5 (unsigned short *x, int stepx, int n)
+{
+ for (unsigned short *y = x; y < x + n; y += stepx * 2)
+ {
+ y[0] = 100;
+ y[1] = 99;
+ }
+}
+
+unsigned short x[1000];
+
+void
+g1 (int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx * 2] = 100;
+ x[i * stepx * 2 + 1] = 99;
+ }
+}
+
+void
+g2 (int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx * 2)
+ {
+ x[i] = 100;
+ x[i + 1] = 99;
+ }
+}
+
+void
+g3 (int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx * 2 - 16] = 100;
+ x[i * stepx * 2 - 15] = 99;
+ }
+}
+
+void
+g4 (int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx * 2)
+ {
+ x[i - 16] = 100;
+ x[i - 15] = 99;
+ }
+}
+
+void
+g5 (int stepx, int n)
+{
+ for (unsigned short *y = x; y < x + n; y += stepx * 2)
+ {
+ y[0] = 100;
+ y[1] = 99;
+ }
+}
+
+/* { dg-final { scan-tree-dump-times {want to version containing loop} 10 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {versioned this loop} 10 "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-14.c
===================================================================
--- /dev/null 2018-11-29 13:15:04.463550658 +0000
+++ gcc/testsuite/gcc.dg/loop-versioning-14.c 2018-12-14 15:02:53.817883430 +0000
@@ -0,0 +1,149 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Test that we don't try to version for a step of 1 when that would
+ cause the iterations to leave a gap between accesses. */
+
+void
+f1 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx * 4] = 100;
+ x[i * stepx * 4 + 1] = 99;
+ }
+}
+
+void
+f2 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx * 4)
+ {
+ x[i] = 100;
+ x[i + 1] = 99;
+ }
+}
+
+void
+f3 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx * 4 - 16] = 100;
+ x[i * stepx * 4 - 15] = 99;
+ }
+}
+
+void
+f4 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx * 4)
+ {
+ x[i - 16] = 100;
+ x[i - 15] = 99;
+ }
+}
+
+void
+f5 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx * 64 - 16] = 100;
+ x[i * stepx * 64 + 15] = 99;
+ }
+}
+
+void
+f6 (unsigned short *x, int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx * 64)
+ {
+ x[i - 16] = 100;
+ x[i + 15] = 99;
+ }
+}
+
+void
+f7 (unsigned short *x, int stepx, int n)
+{
+ for (unsigned short *y = x; y < x + n; y += stepx * 4)
+ {
+ y[0] = 100;
+ y[1] = 99;
+ }
+}
+
+unsigned short x[1000];
+
+void
+g1 (int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx * 4] = 100;
+ x[i * stepx * 4 + 1] = 99;
+ }
+}
+
+void
+g2 (int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx * 4)
+ {
+ x[i] = 100;
+ x[i + 1] = 99;
+ }
+}
+
+void
+g3 (int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx * 4 - 16] = 100;
+ x[i * stepx * 4 - 15] = 99;
+ }
+}
+
+void
+g4 (int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx * 4)
+ {
+ x[i - 16] = 100;
+ x[i - 15] = 99;
+ }
+}
+
+void
+g5 (int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[i * stepx * 64 - 16] = 100;
+ x[i * stepx * 64 + 15] = 99;
+ }
+}
+
+void
+g6 (int stepx, int n)
+{
+ for (int i = 0; i < n; i += stepx * 64)
+ {
+ x[i - 16] = 100;
+ x[i + 15] = 99;
+ }
+}
+
+void
+g7 (int stepx, int n)
+{
+ for (unsigned short *y = x; y < x + n; y += stepx * 4)
+ {
+ y[0] = 100;
+ y[1] = 99;
+ }
+}
+
+/* { dg-final { scan-tree-dump-not {want to version} "lversion" } } */
+/* { dg-final { scan-tree-dump-not {versioned} "lversion" } } */
Index: gcc/testsuite/gcc.dg/vect/slp-43.c
===================================================================
--- gcc/testsuite/gcc.dg/vect/slp-43.c 2018-12-06 18:00:29.000000000 +0000
+++ gcc/testsuite/gcc.dg/vect/slp-43.c 2018-12-14 15:02:53.817883430 +0000
@@ -1,5 +1,5 @@
/* { dg-require-effective-target vect_int } */
-/* { dg-additional-options "-O3" } */
+/* { dg-additional-options "-O3 -fno-version-loops-for-strides" } */
#include <string.h>
#include "tree-vect.h"
Index: gcc/testsuite/gcc.dg/vect/slp-45.c
===================================================================
--- gcc/testsuite/gcc.dg/vect/slp-45.c 2018-12-06 18:00:29.000000000 +0000
+++ gcc/testsuite/gcc.dg/vect/slp-45.c 2018-12-14 15:02:53.817883430 +0000
@@ -1,5 +1,5 @@
/* { dg-require-effective-target vect_int } */
-/* { dg-additional-options "-O3" } */
+/* { dg-additional-options "-O3 -fno-version-loops-for-strides" } */
#include <string.h>
#include "tree-vect.h"
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Add a loop versioning pass
2018-12-14 16:18 ` Richard Sandiford
@ 2018-12-15 10:27 ` Richard Biener
0 siblings, 0 replies; 17+ messages in thread
From: Richard Biener @ 2018-12-15 10:27 UTC (permalink / raw)
To: Richard Sandiford; +Cc: Richard Biener, GCC Patches
On December 14, 2018 5:18:38 PM GMT+01:00, Richard Sandiford <richard.sandiford@arm.com> wrote:
>Richard Biener <rguenther@suse.de> writes:
>> On December 12, 2018 7:43:10 PM GMT+01:00, Richard Sandiford
><richard.sandiford@arm.com> wrote:
>>>Richard Biener <richard.guenther@gmail.com> writes:
>>>> On Thu, Dec 6, 2018 at 2:19 PM Richard Sandiford
>>>>> Tested on x86_64-linux-gnu, aarch64-linux-gnu and aarch64_be-elf.
>>>>> Also repeated the performance testing (but haven't yet tried an
>>>>> LTO variant; will do that over the weekend).
>>>>
>>>> Any results?
>>>
>>>Sorry, I should've remembered that finding time to run tests is easy,
>>>finding time to analyse them is hard.
>>>
>>>Speed-wise, the impact of the patch for LTO is similar to without,
>>>with 554.roms_r being the main beneficiary for both AArch64 and
>x86_64.
>>>I get a 6.8% improvement on Cortex-A57 with -Ofast -mcpu=native
>>>-flto=jobserver.
>>>
>>>Size-wise, there are three tests that grow by >=2% on x86_64:
>>>
>>>549.fotonik3d_r: 5.5%
>>>548.exchange2_r: 29.5%
>>>554.roms_r: 39.6%
>>
>> Uh. With LTO we might have a reasonable guessed profile and you do
>have a optimize_loop_nest_for_speed guard on the transform?
>
>Guard now added :-) But unfortunately it doesn't make any significant
>difference. 548.exchange2_r goes from 29.5% to 27.7%, but the other
>two are the same as before.
>
>> How does compile time fare with the above benchmarks?
>
>For 554.roms_r it's +80%(!) with -flto=1, but I think that's par for
>the course given the increase in function sizes.
:(
>For 549.fotonik3d_r it's +5% with -flto=1.
>
>For 503.bwaves_r (as an example of a benchmark whose size doesn't
>change),
>the difference is in the noise.
>
>[...]
>
>>>You mean something like:
>>>
>>> real :: foo(:,:), bar(:)
>>>
>>> do i=1,n
>>> do j=1,n
>>> foo(i,j) = ...
>>> end do
>>> bar(i) = ..
>>> end do
>>>
>>>? I can add a test if so.
>>
>> Please.
>
>OK, I've added them to loop_versioning_4.f90.
>
>>>> There may also be some subtle issues with substitute_and_fold being
>>>> applied to non-up-to-date SSA form given it folds stmts looking at
>>>> (single-use!) SSA edges. The single-use guard might be what saves
>>>you
>>>> here (SSA uses in the copies are not yet updated to point to the
>>>> copied DEFs).
>>>
>>>OK. I was hoping that because we only apply substitute_and_fold
>>>to new code, there would be no problem with uses elsewhere.
>>
>> Might be, yes.
>>
>>>Would it be safer to:
>>>
>>> - version all loops we want to version
>>> - update SSA explicitly
>>> - apply substitute and fold to all "new" loops
>>
>> That would be definitely less fishy. But you need to get at the
>actual
>> 'new' SSA names for the replacements as I guess they'll be rewritten?
>> Or maybe those are not.
>>
>>>? Could we then get away with returning a 0 TODO at the end?
>>
>> Yes.
>
>OK, the updated patch does it this way.
>
>Tested as before.
OK.
Thanks,
Richard.
>Thanks,
>Richard
>
>
>2018-12-14 Richard Sandiford <richard.sandiford@arm.com>
> Ramana Radhakrishnan <ramana.radhakrishnan@arm.com>
> Kyrylo Tkachov <kyrylo.tkachov@arm.com>
>
>gcc/
> * doc/invoke.texi (-fversion-loops-for-strides): Document
> (loop-versioning-group-size, loop-versioning-max-inner-insns)
> (loop-versioning-max-outer-insns): Document new --params.
> * Makefile.in (OBJS): Add gimple-loop-versioning.o.
> * common.opt (fversion-loops-for-strides): New option.
> * opts.c (default_options_table): Enable fversion-loops-for-strides
> at -O3.
> * params.def (PARAM_LOOP_VERSIONING_GROUP_SIZE)
> (PARAM_LOOP_VERSIONING_MAX_INNER_INSNS)
> (PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS): New parameters.
> * passes.def: Add pass_loop_versioning.
> * timevar.def (TV_LOOP_VERSIONING): New time variable.
> * tree-ssa-propagate.h
> (substitute_and_fold_engine::substitute_and_fold): Add an optional
> block parameter.
> * tree-ssa-propagate.c
> (substitute_and_fold_engine::substitute_and_fold): Likewise.
> When passed, only walk blocks dominated by that block.
> * tree-vrp.h (range_includes_p): Declare.
> (range_includes_zero_p): Turn into an inline wrapper around
> range_includes_p.
> * tree-vrp.c (range_includes_p): New function, generalizing...
> (range_includes_zero_p): ...this.
> * tree-pass.h (make_pass_loop_versioning): Declare.
> * gimple-loop-versioning.cc: New file.
>
>gcc/testsuite/
> * gcc.dg/loop-versioning-1.c: New test.
> * gcc.dg/loop-versioning-10.c: Likewise.
> * gcc.dg/loop-versioning-11.c: Likewise.
> * gcc.dg/loop-versioning-2.c: Likewise.
> * gcc.dg/loop-versioning-3.c: Likewise.
> * gcc.dg/loop-versioning-4.c: Likewise.
> * gcc.dg/loop-versioning-5.c: Likewise.
> * gcc.dg/loop-versioning-6.c: Likewise.
> * gcc.dg/loop-versioning-7.c: Likewise.
> * gcc.dg/loop-versioning-8.c: Likewise.
> * gcc.dg/loop-versioning-9.c: Likewise.
> * gfortran.dg/loop_versioning_1.f90: Likewise.
> * gfortran.dg/loop_versioning_2.f90: Likewise.
> * gfortran.dg/loop_versioning_3.f90: Likewise.
> * gfortran.dg/loop_versioning_4.f90: Likewise.
> * gfortran.dg/loop_versioning_5.f90: Likewise.
> * gfortran.dg/loop_versioning_6.f90: Likewise.
> * gfortran.dg/loop_versioning_7.f90: Likewise.
> * gfortran.dg/loop_versioning_8.f90: Likewise.
>
>Index: gcc/doc/invoke.texi
>===================================================================
>--- gcc/doc/invoke.texi 2018-12-11 15:49:19.061544092 +0000
>+++ gcc/doc/invoke.texi 2018-12-14 15:02:53.813883464 +0000
>@@ -8220,7 +8220,8 @@ by @option{-O2} and also turns on the fo
> -ftree-partial-pre @gol
> -ftree-slp-vectorize @gol
> -funswitch-loops @gol
>--fvect-cost-model}
>+-fvect-cost-model @gol
>+-fversion-loops-for-strides}
>
> @item -O0
> @opindex O0
>@@ -10772,6 +10773,30 @@ of the loop on both branches (modified a
>
> Enabled by @option{-fprofile-use} and @option{-fauto-profile}.
>
>+@item -fversion-loops-for-strides
>+@opindex fversion-loops-for-strides
>+If a loop iterates over an array with a variable stride, create
>another
>+version of the loop that assumes the stride is always one. For
>example:
>+
>+@smallexample
>+for (int i = 0; i < n; ++i)
>+ x[i * stride] = @dots{};
>+@end smallexample
>+
>+becomes:
>+
>+@smallexample
>+if (stride == 1)
>+ for (int i = 0; i < n; ++i)
>+ x[i] = @dots{};
>+else
>+ for (int i = 0; i < n; ++i)
>+ x[i * stride] = @dots{};
>+@end smallexample
>+
>+This is particularly useful for assumed-shape arrays in Fortran where
>+(for example) it allows better vectorization assuming contiguous
>accesses.
>+
> @item -ffunction-sections
> @itemx -fdata-sections
> @opindex ffunction-sections
>@@ -11981,6 +12006,15 @@ Hardware autoprefetcher scheduler model
> Number of lookahead cycles the model looks into; at '
> ' only enable instruction sorting heuristic.
>
>+@item loop-versioning-max-inner-insns
>+The maximum number of instructions that an inner loop can have
>+before the loop versioning pass considers it too big to copy.
>+
>+@item loop-versioning-max-outer-insns
>+The maximum number of instructions that an outer loop can have
>+before the loop versioning pass considers it too big to copy,
>+discounting any instructions in inner loops that directly benefit
>+from versioning.
>
> @end table
> @end table
>Index: gcc/Makefile.in
>===================================================================
>--- gcc/Makefile.in 2018-12-06 18:00:29.000000000 +0000
>+++ gcc/Makefile.in 2018-12-14 15:02:53.809883498 +0000
>@@ -1320,6 +1320,7 @@ OBJS = \
> gimple-laddress.o \
> gimple-loop-interchange.o \
> gimple-loop-jam.o \
>+ gimple-loop-versioning.o \
> gimple-low.o \
> gimple-pretty-print.o \
> gimple-ssa-backprop.o \
>Index: gcc/common.opt
>===================================================================
>--- gcc/common.opt 2018-12-06 18:00:29.000000000 +0000
>+++ gcc/common.opt 2018-12-14 15:02:53.809883498 +0000
>@@ -2775,6 +2775,10 @@ fsplit-loops
> Common Report Var(flag_split_loops) Optimization
> Perform loop splitting.
>
>+fversion-loops-for-strides
>+Common Report Var(flag_version_loops_for_strides) Optimization
>+Version loops based on whether indices have a stride of one.
>+
> funwind-tables
> Common Report Var(flag_unwind_tables) Optimization
> Just generate unwind tables for exception handling.
>Index: gcc/opts.c
>===================================================================
>--- gcc/opts.c 2018-12-06 18:00:29.000000000 +0000
>+++ gcc/opts.c 2018-12-14 15:02:53.813883464 +0000
>@@ -556,6 +556,7 @@ static const struct default_options defa
> { OPT_LEVELS_3_PLUS, OPT_ftree_slp_vectorize, NULL, 1 },
> { OPT_LEVELS_3_PLUS, OPT_funswitch_loops, NULL, 1 },
>{ OPT_LEVELS_3_PLUS, OPT_fvect_cost_model_, NULL,
>VECT_COST_MODEL_DYNAMIC },
>+ { OPT_LEVELS_3_PLUS, OPT_fversion_loops_for_strides, NULL, 1 },
>
> /* -Ofast adds optimizations to -O3. */
> { OPT_LEVELS_FAST, OPT_ffast_math, NULL, 1 },
>Index: gcc/params.def
>===================================================================
>--- gcc/params.def 2018-12-06 18:00:29.000000000 +0000
>+++ gcc/params.def 2018-12-14 15:02:53.817883430 +0000
>@@ -1365,6 +1365,19 @@ DEFPARAM(PARAM_LOGICAL_OP_NON_SHORT_CIRC
> "True if a non-short-circuit operation is optimal.",
> -1, -1, 1)
>
>+DEFPARAM(PARAM_LOOP_VERSIONING_MAX_INNER_INSNS,
>+ "loop-versioning-max-inner-insns",
>+ "The maximum number of instructions in an inner loop that is being"
>+ " considered for versioning.",
>+ 200, 0, 0)
>+
>+DEFPARAM(PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS,
>+ "loop-versioning-max-outer-insns",
>+ "The maximum number of instructions in an outer loop that is being"
>+ " considered for versioning, on top of the instructions in inner"
>+ " loops.",
>+ 100, 0, 0)
>+
> /*
>
> Local variables:
>Index: gcc/passes.def
>===================================================================
>--- gcc/passes.def 2018-12-06 18:00:29.000000000 +0000
>+++ gcc/passes.def 2018-12-14 15:02:53.817883430 +0000
>@@ -265,6 +265,7 @@ along with GCC; see the file COPYING3.
> NEXT_PASS (pass_tree_unswitch);
> NEXT_PASS (pass_scev_cprop);
> NEXT_PASS (pass_loop_split);
>+ NEXT_PASS (pass_loop_versioning);
> NEXT_PASS (pass_loop_jam);
> /* All unswitching, final value replacement and splitting can expose
> empty loops. Remove them now. */
>Index: gcc/timevar.def
>===================================================================
>--- gcc/timevar.def 2018-12-06 18:00:29.000000000 +0000
>+++ gcc/timevar.def 2018-12-14 15:02:53.821883396 +0000
>@@ -234,6 +234,7 @@ DEFTIMEVAR (TV_DSE1 , "
> DEFTIMEVAR (TV_DSE2 , "dead store elim2")
> DEFTIMEVAR (TV_LOOP , "loop analysis")
> DEFTIMEVAR (TV_LOOP_INIT , "loop init")
>+DEFTIMEVAR (TV_LOOP_VERSIONING , "loop versioning")
> DEFTIMEVAR (TV_LOOP_MOVE_INVARIANTS , "loop invariant motion")
> DEFTIMEVAR (TV_LOOP_UNROLL , "loop unrolling")
> DEFTIMEVAR (TV_LOOP_DOLOOP , "loop doloop")
>Index: gcc/tree-ssa-propagate.h
>===================================================================
>--- gcc/tree-ssa-propagate.h 2018-12-06 18:00:29.000000000 +0000
>+++ gcc/tree-ssa-propagate.h 2018-12-14 15:02:53.821883396 +0000
>@@ -104,7 +104,7 @@ extern void propagate_tree_value_into_st
> virtual bool fold_stmt (gimple_stmt_iterator *) { return false; }
> virtual tree get_value (tree) { return NULL_TREE; }
>
>- bool substitute_and_fold (void);
>+ bool substitute_and_fold (basic_block = NULL);
> bool replace_uses_in (gimple *);
> bool replace_phi_args_in (gphi *);
> };
>Index: gcc/tree-ssa-propagate.c
>===================================================================
>--- gcc/tree-ssa-propagate.c 2018-12-06 18:00:29.000000000 +0000
>+++ gcc/tree-ssa-propagate.c 2018-12-14 15:02:53.821883396 +0000
>@@ -1154,6 +1154,10 @@ substitute_and_fold_dom_walker::before_d
>
>
> /* Perform final substitution and folding of propagated values.
>+ Process the whole function if BLOCK is null, otherwise only
>+ process the blocks that BLOCK dominates. In the latter case,
>+ it is the caller's responsibility to ensure that dominator
>+ information is available and up-to-date.
>
> PROP_VALUE[I] contains the single value that should be substituted
> at every use of SSA name N_I. If PROP_VALUE is NULL, no values are
>@@ -1170,16 +1174,24 @@ substitute_and_fold_dom_walker::before_d
> Return TRUE when something changed. */
>
> bool
>-substitute_and_fold_engine::substitute_and_fold (void)
>+substitute_and_fold_engine::substitute_and_fold (basic_block block)
> {
> if (dump_file && (dump_flags & TDF_DETAILS))
>fprintf (dump_file, "\nSubstituting values and folding
>statements\n\n");
>
> memset (&prop_stats, 0, sizeof (prop_stats));
>
>- calculate_dominance_info (CDI_DOMINATORS);
>+ /* Don't call calculate_dominance_info when iterating over a
>subgraph.
>+ Callers that are using the interface this way are likely to want
>to
>+ iterate over several disjoint subgraphs, and it would be
>expensive
>+ in enable-checking builds to revalidate the whole dominance tree
>+ each time. */
>+ if (block)
>+ gcc_assert (dom_info_state (CDI_DOMINATORS));
>+ else
>+ calculate_dominance_info (CDI_DOMINATORS);
> substitute_and_fold_dom_walker walker (CDI_DOMINATORS, this);
>- walker.walk (ENTRY_BLOCK_PTR_FOR_FN (cfun));
>+ walker.walk (block ? block : ENTRY_BLOCK_PTR_FOR_FN (cfun));
>
> /* We cannot remove stmts during the BB walk, especially not release
> SSA names there as that destroys the lattice of our callers.
>Index: gcc/tree-vrp.h
>===================================================================
>--- gcc/tree-vrp.h 2018-12-06 18:00:29.000000000 +0000
>+++ gcc/tree-vrp.h 2018-12-14 15:02:53.821883396 +0000
>@@ -243,7 +243,7 @@ struct assert_info
> extern void register_edge_assert_for (tree, edge, enum tree_code,
> tree, tree, vec<assert_info> &);
> extern bool stmt_interesting_for_vrp (gimple *);
>-extern bool range_includes_zero_p (const value_range_base *);
>+extern bool range_includes_p (const value_range_base *,
>HOST_WIDE_INT);
> extern bool infer_value_range (gimple *, tree, tree_code *, tree *);
>
> extern bool vrp_bitmap_equal_p (const_bitmap, const_bitmap);
>@@ -285,4 +285,12 @@ extern tree get_single_symbol (tree, boo
> extern void maybe_set_nonzero_bits (edge, tree);
>extern value_range_kind determine_value_range (tree, wide_int *,
>wide_int *);
>
>+/* Return TRUE if *VR includes the value zero. */
>+
>+inline bool
>+range_includes_zero_p (const value_range_base *vr)
>+{
>+ return range_includes_p (vr, 0);
>+}
>+
> #endif /* GCC_TREE_VRP_H */
>Index: gcc/tree-vrp.c
>===================================================================
>--- gcc/tree-vrp.c 2018-12-07 14:59:04.255508495 +0000
>+++ gcc/tree-vrp.c 2018-12-14 15:02:53.821883396 +0000
>@@ -1173,15 +1173,14 @@ value_inside_range (tree val, tree min,
> }
>
>
>-/* Return TRUE if *VR includes the value zero. */
>+/* Return TRUE if *VR includes the value X. */
>
> bool
>-range_includes_zero_p (const value_range_base *vr)
>+range_includes_p (const value_range_base *vr, HOST_WIDE_INT x)
> {
> if (vr->varying_p () || vr->undefined_p ())
> return true;
>- tree zero = build_int_cst (vr->type (), 0);
>- return vr->may_contain_p (zero);
>+ return vr->may_contain_p (build_int_cst (vr->type (), x));
> }
>
>/* If *VR has a value range that is a single constant value return
>that,
>Index: gcc/tree-pass.h
>===================================================================
>--- gcc/tree-pass.h 2018-12-06 18:00:29.000000000 +0000
>+++ gcc/tree-pass.h 2018-12-14 15:02:53.821883396 +0000
>@@ -362,6 +362,7 @@ extern gimple_opt_pass *make_pass_fix_lo
> extern gimple_opt_pass *make_pass_tree_loop (gcc::context *ctxt);
> extern gimple_opt_pass *make_pass_tree_no_loop (gcc::context *ctxt);
> extern gimple_opt_pass *make_pass_tree_loop_init (gcc::context *ctxt);
>+extern gimple_opt_pass *make_pass_loop_versioning (gcc::context
>*ctxt);
> extern gimple_opt_pass *make_pass_lim (gcc::context *ctxt);
> extern gimple_opt_pass *make_pass_linterchange (gcc::context *ctxt);
> extern gimple_opt_pass *make_pass_tree_unswitch (gcc::context *ctxt);
>Index: gcc/gimple-loop-versioning.cc
>===================================================================
>--- /dev/null 2018-11-29 13:15:04.463550658 +0000
>+++ gcc/gimple-loop-versioning.cc 2018-12-14 15:02:53.813883464 +0000
>@@ -0,0 +1,1758 @@
>+/* Loop versioning pass.
>+ Copyright (C) 2018 Free Software Foundation, Inc.
>+
>+This file is part of GCC.
>+
>+GCC is free software; you can redistribute it and/or modify it
>+under the terms of the GNU General Public License as published by the
>+Free Software Foundation; either version 3, or (at your option) any
>+later version.
>+
>+GCC is distributed in the hope that it will be useful, but WITHOUT
>+ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
>+FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
>+for more details.
>+
>+You should have received a copy of the GNU General Public License
>+along with GCC; see the file COPYING3. If not see
>+<http://www.gnu.org/licenses/>. */
>+
>+#include "config.h"
>+#include "system.h"
>+#include "coretypes.h"
>+#include "backend.h"
>+#include "tree.h"
>+#include "gimple.h"
>+#include "gimple-iterator.h"
>+#include "tree-pass.h"
>+#include "gimplify-me.h"
>+#include "cfgloop.h"
>+#include "tree-ssa-loop.h"
>+#include "ssa.h"
>+#include "tree-scalar-evolution.h"
>+#include "tree-chrec.h"
>+#include "tree-ssa-loop-ivopts.h"
>+#include "fold-const.h"
>+#include "tree-ssa-propagate.h"
>+#include "tree-inline.h"
>+#include "domwalk.h"
>+#include "alloc-pool.h"
>+#include "vr-values.h"
>+#include "gimple-ssa-evrp-analyze.h"
>+#include "tree-vectorizer.h"
>+#include "omp-general.h"
>+#include "predict.h"
>+#include "tree-into-ssa.h"
>+#include "params.h"
>+
>+namespace {
>+
>+/* This pass looks for loops that could be simplified if certain loop
>+ invariant conditions were true. It is effectively a form of loop
>+ splitting in which the pass produces the split conditions itself,
>+ instead of using ones that are already present in the IL.
>+
>+ Versioning for when strides are 1
>+ ---------------------------------
>+
>+ At the moment the only thing the pass looks for are memory
>references
>+ like:
>+
>+ for (auto i : ...)
>+ ...x[i * stride]...
>+
>+ It considers changing such loops to:
>+
>+ if (stride == 1)
>+ for (auto i : ...) [A]
>+ ...x[i]...
>+ else
>+ for (auto i : ...) [B]
>+ ...x[i * stride]...
>+
>+ This can have several benefits:
>+
>+ (1) [A] is often easier or cheaper to vectorize than [B].
>+
>+ (2) The scalar code in [A] is simpler than the scalar code in [B]
>+ (if the loops cannot be vectorized or need an epilogue loop).
>+
>+ (3) We might recognize [A] as a pattern, such as a memcpy or
>memset.
>+
>+ (4) [A] has simpler address evolutions, which can help other passes
>+ like loop interchange.
>+
>+ The optimization is particularly useful for assumed-shape arrays in
>+ Fortran, where the stride of the innermost dimension depends on the
>+ array descriptor but is often equal to 1 in practice. For example:
>+
>+ subroutine f1(x)
>+ real :: x(:)
>+ x(:) = 100
>+ end subroutine f1
>+
>+ generates the equivalent of:
>+
>+ raw_stride = *x.dim[0].stride;
>+ stride = raw_stride != 0 ? raw_stride : 1;
>+ x_base = *x.data;
>+ ...
>+ tmp1 = stride * S;
>+ tmp2 = tmp1 - stride;
>+ *x_base[tmp2] = 1.0e+2;
>+
>+ but in the common case that stride == 1, the last three statements
>+ simplify to:
>+
>+ tmp3 = S + -1;
>+ *x_base[tmp3] = 1.0e+2;
>+
>+ The optimization is in principle very simple. The difficult parts
>are:
>+
>+ (a) deciding which parts of a general address calculation
>correspond
>+ to the inner dimension of an array, since this usually isn't
>explicit
>+ in the IL, and for C often isn't even explicit in the source
>code
>+
>+ (b) estimating when the transformation is worthwhile
>+
>+ Structure
>+ ---------
>+
>+ The pass has four phases:
>+
>+ (1) Walk through the statements looking for and recording potential
>+ versioning opportunities. Stop if there are none.
>+
>+ (2) Use context-sensitive range information to see whether any
>versioning
>+ conditions are impossible in practice. Remove them if so, and
>stop
>+ if no opportunities remain.
>+
>+ (We do this only after (1) to keep compile time down when no
>+ versioning opportunities exist.)
>+
>+ (3) Apply the cost model. Decide which versioning opportunities
>are
>+ worthwhile and at which nesting level they should be applied.
>+
>+ (4) Attempt to version all the loops selected by (3), so that:
>+
>+ for (...)
>+ ...
>+
>+ becomes:
>+
>+ if (!cond)
>+ for (...) // Original loop
>+ ...
>+ else
>+ for (...) // New loop
>+ ...
>+
>+ Use the version condition COND to simplify the new loop. */
>+
>+/* Enumerates the likelihood that a particular value indexes the inner
>+ dimension of an array. */
>+enum inner_likelihood {
>+ INNER_UNLIKELY,
>+ INNER_DONT_KNOW,
>+ INNER_LIKELY
>+};
>+
>+/* Information about one term of an address_info. */
>+struct address_term_info
>+{
>+ /* The value of the term is EXPR * MULTIPLIER. */
>+ tree expr;
>+ unsigned HOST_WIDE_INT multiplier;
>+
>+ /* The stride applied by EXPR in each iteration of some unrecorded
>loop,
>+ or null if no stride has been identified. */
>+ tree stride;
>+
>+ /* Enumerates the likelihood that EXPR indexes the inner dimension
>+ of an array. */
>+ enum inner_likelihood inner_likelihood;
>+
>+ /* True if STRIDE == 1 is a versioning opportunity when considered
>+ in isolation. */
>+ bool versioning_opportunity_p;
>+};
>+
>+/* Information about an address calculation, and the range of constant
>+ offsets applied to it. */
>+struct address_info
>+{
>+ static const unsigned int MAX_TERMS = 8;
>+
>+ /* One statement that calculates the address. If multiple
>statements
>+ share the same address, we only record the first. */
>+ gimple *stmt;
>+
>+ /* The loop containing STMT (cached for convenience). If multiple
>+ statements share the same address, they all belong to this loop.
>*/
>+ struct loop *loop;
>+
>+ /* A decomposition of the calculation into a sum of terms plus an
>+ optional base. When BASE is provided, it is never an SSA name.
>+ Once initialization is complete, all members of TERMs are SSA
>names. */
>+ tree base;
>+ auto_vec<address_term_info, MAX_TERMS> terms;
>+
>+ /* All bytes accessed from the address fall in the offset range
>+ [MIN_OFFSET, MAX_OFFSET). */
>+ HOST_WIDE_INT min_offset, max_offset;
>+};
>+
>+/* Stores addresses based on their base and terms (ignoring the
>offsets). */
>+struct address_info_hasher : nofree_ptr_hash <address_info>
>+{
>+ static hashval_t hash (const address_info *);
>+ static bool equal (const address_info *, const address_info *);
>+};
>+
>+/* Information about the versioning we'd like to apply to a loop. */
>+struct loop_info
>+{
>+ bool worth_versioning_p () const;
>+
>+ /* True if we've decided not to version this loop. The remaining
>+ fields are meaningless if so. */
>+ bool rejected_p;
>+
>+ /* True if at least one subloop of this loop benefits from
>versioning. */
>+ bool subloops_benefit_p;
>+
>+ /* An estimate of the total number of instructions in the loop,
>+ excluding those in subloops that benefit from versioning. */
>+ unsigned int num_insns;
>+
>+ /* The outermost loop that can handle all the version checks
>+ described below. */
>+ struct loop *outermost;
>+
>+ /* The first entry in the list of blocks that belong to this loop
>+ (and not to subloops). m_next_block_in_loop provides the chain
>+ pointers for the list. */
>+ basic_block block_list;
>+
>+ /* We'd like to version the loop for the case in which these SSA
>names
>+ (keyed off their SSA_NAME_VERSION) are all equal to 1 at runtime.
> */
>+ bitmap_head unity_names;
>+
>+ /* If versioning succeeds, this points the version of the loop that
>+ assumes the version conditions holds. */
>+ struct loop *optimized_loop;
>+};
>+
>+/* The main pass structure. */
>+class loop_versioning
>+{
>+public:
>+ loop_versioning (function *);
>+ ~loop_versioning ();
>+ unsigned int run ();
>+
>+private:
>+ /* Used to walk the dominator tree to find loop versioning
>conditions
>+ that are always false. */
>+ class lv_dom_walker : public dom_walker
>+ {
>+ public:
>+ lv_dom_walker (loop_versioning &);
>+
>+ edge before_dom_children (basic_block) FINAL OVERRIDE;
>+ void after_dom_children (basic_block) FINAL OVERRIDE;
>+
>+ private:
>+ /* The parent pass. */
>+ loop_versioning &m_lv;
>+
>+ /* Used to build context-dependent range information. */
>+ evrp_range_analyzer m_range_analyzer;
>+ };
>+
>+ /* Used to simplify statements based on conditions that are
>established
>+ by the version checks. */
>+ class name_prop : public substitute_and_fold_engine
>+ {
>+ public:
>+ name_prop (loop_info &li) : m_li (li) {}
>+ tree get_value (tree) FINAL OVERRIDE;
>+
>+ private:
>+ /* Information about the versioning we've performed on the loop.
>*/
>+ loop_info &m_li;
>+ };
>+
>+ loop_info &get_loop_info (struct loop *loop) { return
>m_loops[loop->num]; }
>+
>+ unsigned int max_insns_for_loop (struct loop *);
>+ bool expensive_stmt_p (gimple *);
>+
>+ void version_for_unity (gimple *, tree);
>+ bool acceptable_multiplier_p (tree, unsigned HOST_WIDE_INT,
>+ unsigned HOST_WIDE_INT * = 0);
>+ bool acceptable_type_p (tree, unsigned HOST_WIDE_INT *);
>+ bool multiply_term_by (address_term_info &, tree);
>+ inner_likelihood get_inner_likelihood (tree, unsigned
>HOST_WIDE_INT);
>+ void analyze_stride (address_info &, address_term_info &,
>+ tree, struct loop *);
>+ bool find_per_loop_multiplication (address_info &, address_term_info
>&);
>+ void analyze_term_using_scevs (address_info &, address_term_info &);
>+ void analyze_address_fragment (address_info &);
>+ void record_address_fragment (gimple *, unsigned HOST_WIDE_INT,
>+ tree, unsigned HOST_WIDE_INT, HOST_WIDE_INT);
>+ void analyze_expr (gimple *, tree);
>+ bool analyze_block (basic_block);
>+ bool analyze_blocks ();
>+
>+ void prune_loop_conditions (struct loop *, vr_values *);
>+ bool prune_conditions ();
>+
>+ void merge_loop_info (struct loop *, struct loop *);
>+ void add_loop_to_queue (struct loop *);
>+ bool decide_whether_loop_is_versionable (struct loop *);
>+ bool make_versioning_decisions ();
>+
>+ bool version_loop (struct loop *);
>+ void implement_versioning_decisions ();
>+
>+ /* The function we're optimizing. */
>+ function *m_fn;
>+
>+ /* The obstack to use for all pass-specific bitmaps. */
>+ bitmap_obstack m_bitmap_obstack;
>+
>+ /* An obstack to use for general allocation. */
>+ obstack m_obstack;
>+
>+ /* The number of loops in the function. */
>+ unsigned int m_nloops;
>+
>+ /* The total number of loop version conditions we've found. */
>+ unsigned int m_num_conditions;
>+
>+ /* Assume that an address fragment of the form i * stride * scale
>+ (for variable stride and constant scale) will not benefit from
>+ versioning for stride == 1 when scale is greater than this value.
> */
>+ unsigned HOST_WIDE_INT m_maximum_scale;
>+
>+ /* Information about each loop. */
>+ auto_vec<loop_info> m_loops;
>+
>+ /* Used to form a linked list of blocks that belong to a loop,
>+ started by loop_info::block_list. */
>+ auto_vec<basic_block> m_next_block_in_loop;
>+
>+ /* The list of loops that we've decided to version. */
>+ auto_vec<struct loop *> m_loops_to_version;
>+
>+ /* A table of addresses in the current loop, keyed off their values
>+ but not their offsets. */
>+ hash_table <address_info_hasher> m_address_table;
>+
>+ /* A list of all addresses in M_ADDRESS_TABLE, in a predictable
>order. */
>+ auto_vec <address_info *, 32> m_address_list;
>+};
>+
>+/* If EXPR is an SSA name and not a default definition, return the
>+ defining statement, otherwise return null. */
>+
>+static gimple *
>+maybe_get_stmt (tree expr)
>+{
>+ if (TREE_CODE (expr) == SSA_NAME && !SSA_NAME_IS_DEFAULT_DEF (expr))
>+ return SSA_NAME_DEF_STMT (expr);
>+ return NULL;
>+}
>+
>+/* Like maybe_get_stmt, but also return null if the defining
>+ statement isn't an assignment. */
>+
>+static gassign *
>+maybe_get_assign (tree expr)
>+{
>+ return safe_dyn_cast <gassign *> (maybe_get_stmt (expr));
>+}
>+
>+/* Return true if this pass should look through a cast of expression
>FROM
>+ to type TYPE when analyzing pieces of an address. */
>+
>+static bool
>+look_through_cast_p (tree type, tree from)
>+{
>+ return (INTEGRAL_TYPE_P (TREE_TYPE (from)) == INTEGRAL_TYPE_P (type)
>+ && POINTER_TYPE_P (TREE_TYPE (from)) == POINTER_TYPE_P (type));
>+}
>+
>+/* Strip all conversions of integers or pointers from EXPR, regardless
>+ of whether the conversions are nops. This is useful in the context
>+ of this pass because we're not trying to fold or simulate the
>+ expression; we just want to see how it's structured. */
>+
>+static tree
>+strip_casts (tree expr)
>+{
>+ const unsigned int MAX_NITERS = 4;
>+
>+ tree type = TREE_TYPE (expr);
>+ while (CONVERT_EXPR_P (expr)
>+ && look_through_cast_p (type, TREE_OPERAND (expr, 0)))
>+ expr = TREE_OPERAND (expr, 0);
>+
>+ for (unsigned int niters = 0; niters < MAX_NITERS; ++niters)
>+ {
>+ gassign *assign = maybe_get_assign (expr);
>+ if (assign
>+ && CONVERT_EXPR_CODE_P (gimple_assign_rhs_code (assign))
>+ && look_through_cast_p (type, gimple_assign_rhs1 (assign)))
>+ expr = gimple_assign_rhs1 (assign);
>+ else
>+ break;
>+ }
>+ return expr;
>+}
>+
>+/* Compare two address_term_infos in the same address_info. */
>+
>+static int
>+compare_address_terms (const void *a_uncast, const void *b_uncast)
>+{
>+ const address_term_info *a = (const address_term_info *) a_uncast;
>+ const address_term_info *b = (const address_term_info *) b_uncast;
>+
>+ if (a->expr != b->expr)
>+ return SSA_NAME_VERSION (a->expr) < SSA_NAME_VERSION (b->expr) ?
>-1 : 1;
>+
>+ if (a->multiplier != b->multiplier)
>+ return a->multiplier < b->multiplier ? -1 : 1;
>+
>+ return 0;
>+}
>+
>+/* Dump ADDRESS using flags FLAGS. */
>+
>+static void
>+dump_address_info (dump_flags_t flags, address_info &address)
>+{
>+ if (address.base)
>+ dump_printf (flags, "%T + ", address.base);
>+ for (unsigned int i = 0; i < address.terms.length (); ++i)
>+ {
>+ if (i != 0)
>+ dump_printf (flags, " + ");
>+ dump_printf (flags, "%T", address.terms[i].expr);
>+ if (address.terms[i].multiplier != 1)
>+ dump_printf (flags, " * %wd", address.terms[i].multiplier);
>+ }
>+ dump_printf (flags, " + [%wd, %wd]",
>+ address.min_offset, address.max_offset - 1);
>+}
>+
>+/* Hash an address_info based on its base and terms. */
>+
>+hashval_t
>+address_info_hasher::hash (const address_info *info)
>+{
>+ inchash::hash hash;
>+ hash.add_int (info->base ? TREE_CODE (info->base) : 0);
>+ hash.add_int (info->terms.length ());
>+ for (unsigned int i = 0; i < info->terms.length (); ++i)
>+ {
>+ hash.add_int (SSA_NAME_VERSION (info->terms[i].expr));
>+ hash.add_hwi (info->terms[i].multiplier);
>+ }
>+ return hash.end ();
>+}
>+
>+/* Return true if two address_infos have equal bases and terms. Other
>+ properties might be different (such as the statement or constant
>+ offset range). */
>+
>+bool
>+address_info_hasher::equal (const address_info *a, const address_info
>*b)
>+{
>+ if (a->base != b->base
>+ && (!a->base || !b->base || !operand_equal_p (a->base, b->base,
>0)))
>+ return false;
>+
>+ if (a->terms.length () != b->terms.length ())
>+ return false;
>+
>+ for (unsigned int i = 0; i < a->terms.length (); ++i)
>+ if (a->terms[i].expr != b->terms[i].expr
>+ || a->terms[i].multiplier != b->terms[i].multiplier)
>+ return false;
>+
>+ return true;
>+}
>+
>+/* Return true if we want to version the loop, i.e. if we have a
>+ specific reason for doing so and no specific reason not to. */
>+
>+bool
>+loop_info::worth_versioning_p () const
>+{
>+ return (!rejected_p
>+ && (!bitmap_empty_p (&unity_names) || subloops_benefit_p));
>+}
>+
>+loop_versioning::lv_dom_walker::lv_dom_walker (loop_versioning &lv)
>+ : dom_walker (CDI_DOMINATORS), m_lv (lv), m_range_analyzer (false)
>+{
>+}
>+
>+/* Process BB before processing the blocks it dominates. */
>+
>+edge
>+loop_versioning::lv_dom_walker::before_dom
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Add a loop versioning pass
2018-10-25 16:16 ` David Malcolm
@ 2018-11-28 14:16 ` Richard Sandiford
0 siblings, 0 replies; 17+ messages in thread
From: Richard Sandiford @ 2018-11-28 14:16 UTC (permalink / raw)
To: David Malcolm; +Cc: gcc-patches
David Malcolm <dmalcolm@redhat.com> writes:
> On Wed, 2018-10-24 at 14:05 +0100, Richard Sandiford wrote:
>> This patch adds a pass that versions loops with variable index
>> strides
>> for the case in which the stride is 1. E.g.:
>>
>> for (int i = 0; i < n; ++i)
>> x[i * stride] = ...;
>>
>> becomes:
>>
>> if (stepx == 1)
>> for (int i = 0; i < n; ++i)
>> x[i] = ...;
>> else
>> for (int i = 0; i < n; ++i)
>> x[i * stride] = ...;
>>
>> This is useful for both vector code and scalar code, and in some
>> cases
>> can enable further optimisations like loop interchange or pattern
>> recognition.
>>
>> The pass gives a 7.6% improvement on Cortex-A72 for 554.roms_r at -O3
>> and a 2.4% improvement for 465.tonto. I haven't found any SPEC tests
>> that regress.
>>
>> Sizewise, there's a 10% increase in .text for both 554.roms_r and
>> 465.tonto. That's obviously a lot, but in tonto's case it's because
>> the whole program is written using assumed-shape arrays and pointers,
>> so a large number of functions really do benefit from versioning.
>> roms likewise makes heavy use of assumed-shape arrays, and that
>> improvement in performance IMO justifies the code growth.
>>
>> The next biggest .text increase is 4.5% for 548.exchange2_r. I did
>> see
>> a small (0.4%) speed improvement there, but although both 3-iteration
>> runs
>> produced stable results, that might still be noise. There was a
>> slightly
>> larger (non-noise) improvement for a 256-bit SVE model.
>>
>> 481.wrf and 521.wrf_r .text grew by 2.8% and 2.5% respectively, but
>> without any noticeable improvement in performance. No other test
>> grew
>> by more than 2%.
>>
>> Although the main SPEC beneficiaries are all Fortran tests, the
>> benchmarks we use for SVE also include some C and C++ tests that
>> benefit.
>>
>> Using -frepack-arrays gives the same benefits in many Fortran cases.
>> The problem is that using that option inappropriately can force a
>> full
>> array copy for arguments that the function only reads once, and so it
>> isn't really something we can turn on by default. The new pass is
>> supposed to give most of the benefits of -frepack-arrays without
>> the risk of unnecessary repacking.
>>
>> The patch therefore enables the pass by default at -O3.
>>
>> Tested on aarch64-linux-gnu and x86_64-linux-gnu. OK to install?
>>
>> Richard
>>
>
> [...snip...]
>
>> +/* Run the pass and return a set of TODO_* flags. */
>> +
>> +unsigned int
>> +loop_versioning::run ()
>> +{
>> + gcc_assert (scev_initialized_p ());
>> +
>> + if (!analyze_blocks ()
>> + || !prune_conditions ()
>> + || !make_versioning_decisions ()
>> + || !implement_versioning_decisions ())
>> + return 0;
>> +
>> + return TODO_update_ssa;
>> +}
>> +
>> +/* Loop versioningting pass. */
>
> (typo)
Huh, no idea how I even got there.
>> +
>> +namespace {
>
> Could the whole file be within this anonymous namespace, rather than
> just the opt_pass subclass? (hiding class loop_versioning, so that the
> optimizer knows that the only thing visible outside the TU is
> make_pass_loop_versioning). This can be a pain to debug, but you can
> always comment out the anon namespace locally when debugging.
Yeah, I prefer that style too, so that the TU only exports public
interfaces. I'd got the impression from earlier threads that this
was frowned on for GCC and so I was deliberately avoiding it, but if
it's OK then great.
Thanks,
Richard
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Add a loop versioning pass
2018-10-25 14:16 ` Richard Biener
2018-10-25 16:03 ` Richard Sandiford
@ 2018-10-26 14:49 ` Richard Biener
1 sibling, 0 replies; 17+ messages in thread
From: Richard Biener @ 2018-10-26 14:49 UTC (permalink / raw)
To: GCC Patches, Richard Sandiford
On Thu, Oct 25, 2018 at 3:52 PM Richard Biener
<richard.guenther@gmail.com> wrote:
>
> On Wed, Oct 24, 2018 at 3:05 PM Richard Sandiford
> <richard.sandiford@arm.com> wrote:
> >
> > This patch adds a pass that versions loops with variable index strides
> > for the case in which the stride is 1. E.g.:
> >
> > for (int i = 0; i < n; ++i)
> > x[i * stride] = ...;
> >
> > becomes:
> >
> > if (stepx == 1)
> > for (int i = 0; i < n; ++i)
> > x[i] = ...;
> > else
> > for (int i = 0; i < n; ++i)
> > x[i * stride] = ...;
> >
> > This is useful for both vector code and scalar code, and in some cases
> > can enable further optimisations like loop interchange or pattern
> > recognition.
> >
> > The pass gives a 7.6% improvement on Cortex-A72 for 554.roms_r at -O3
> > and a 2.4% improvement for 465.tonto. I haven't found any SPEC tests
> > that regress.
> >
> > Sizewise, there's a 10% increase in .text for both 554.roms_r and
> > 465.tonto. That's obviously a lot, but in tonto's case it's because
> > the whole program is written using assumed-shape arrays and pointers,
> > so a large number of functions really do benefit from versioning.
> > roms likewise makes heavy use of assumed-shape arrays, and that
> > improvement in performance IMO justifies the code growth.
> >
> > The next biggest .text increase is 4.5% for 548.exchange2_r. I did see
> > a small (0.4%) speed improvement there, but although both 3-iteration runs
> > produced stable results, that might still be noise. There was a slightly
> > larger (non-noise) improvement for a 256-bit SVE model.
> >
> > 481.wrf and 521.wrf_r .text grew by 2.8% and 2.5% respectively, but
> > without any noticeable improvement in performance. No other test grew
> > by more than 2%.
> >
> > Although the main SPEC beneficiaries are all Fortran tests, the
> > benchmarks we use for SVE also include some C and C++ tests that
> > benefit.
> >
> > Using -frepack-arrays gives the same benefits in many Fortran cases.
> > The problem is that using that option inappropriately can force a full
> > array copy for arguments that the function only reads once, and so it
> > isn't really something we can turn on by default. The new pass is
> > supposed to give most of the benefits of -frepack-arrays without
> > the risk of unnecessary repacking.
> >
> > The patch therefore enables the pass by default at -O3.
> >
> > Tested on aarch64-linux-gnu and x86_64-linux-gnu. OK to install?
>
> I will give this a thorough review tomorror (sorry for the delay),
So I didn't really finish but one thing I noticed is
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + {
> + fprintf (dump_file, ";; Want to version loop %d (depth %d)"
> + " for when ", loop->num, loop_depth (loop));
> + print_generic_expr (dump_file, name, TDF_SLIM);
> + fprintf (dump_file, " == 1");
Since you are writing a new pass you want to use the new dump interface.
if (dump_enabled_p ())
dump_printf (MSG_NOTE, ";; Want to version loop %d (depth %d)"
" for when %E == 1", loop->num, loop_depth (loop), name);
...
it's much nicer to be able to use %E/%G than separate calls for the
tree parts.
I'm also cut&pasting my overall comment part but not the incomplete
set of individual comments/questions yet (hope to finish that on Monday):
> Sizewise, there's a 10% increase in .text for both 554.roms_r and
> 465.tonto. That's obviously a lot, but in tonto's case it's because
> the whole program is written using assumed-shape arrays and pointers,
> so a large number of functions really do benefit from versioning.
> roms likewise makes heavy use of assumed-shape arrays, and that
> improvement in performance IMO justifies the code growth.
Ouch. I know that at least with LTO IPA-CP can do "quite" some
propagation of constant strides. Not sure if we're aggressive
enough in actually doing the cloning for all cases we figure out
strides though. But my question is how we can avoid doing the
versioning for loops in the copy that did not have the IPA-CPed
stride of one? Ideally we'd be able to mark individual references
as {definitely,likely,unlikely,not}-unit-stride?
> The next biggest .text increase is 4.5% for 548.exchange2_r. I did see
> a small (0.4%) speed improvement there, but although both 3-iteration runs
> produced stable results, that might still be noise. There was a slightly
> larger (non-noise) improvement for a 256-bit SVE model.
>
> 481.wrf and 521.wrf_r .text grew by 2.8% and 2.5% respectively, but
> without any noticeable improvement in performance. No other test grew
> by more than 2%.
>
> Although the main SPEC beneficiaries are all Fortran tests, the
> benchmarks we use for SVE also include some C and C++ tests that
> benefit.
Did you see any slowdown, for example because versioning was forced
to be on an innermost loop? I'm thinking of the testcase in
PR87561 where we do have strided accesses in the innermost loop.
Since you cite performance numbers how did you measure them?
I assume -Ofast -march=native but did you check with -flto?
> Using -frepack-arrays gives the same benefits in many Fortran cases.
> The problem is that using that option inappropriately can force a full
> array copy for arguments that the function only reads once, and so it
> isn't really something we can turn on by default. The new pass is
> supposed to give most of the benefits of -frepack-arrays without
> the risk of unnecessary repacking.
>
> The patch therefore enables the pass by default at -O3.
I think that's reasonable.
One possible enhancement would be to add a value-profile for the
strides so we can guide this optimization better.
Thanks,
Richard.
> but
> one question that comes up is how we avoid excessive versioning.
> Consider this pass versioning for stride1, then loop distribution
> versioning for aliasing and then vectorization versioning for alignment.
>
> Do we have any idea on how to track these versionings so we can
> eventually commonize paths that didn't get any followup optimization
> applied?
>
> Also consider the case where loop distribution applies transforms
> with versioning for aliasing to _both_ loop nests from your versioning.
>
> Maybe we can at least add some loop->number_of_times_versioned
> counter and some dumping/statistics to get an idea how big an
> issue this might be?
>
> Thanks,
> Richard.
>
> > Richard
> >
> >
> > 2018-10-24 Richard Sandiford <richard.sandiford@arm.com>
> >
> > gcc/
> > * doc/invoke.texi (-fversion-loops-for-strides): Document
> > (loop-versioning-group-size, loop-versioning-max-inner-insns)
> > (loop-versioning-max-outer-insns): Document new --params.
> > * Makefile.in (OBJS): Add gimple-loop-versioning.o.
> > * common.opt (fversion-loops-for-strides): New option.
> > * opts.c (default_options_table): Enable fversion-loops-for-strides
> > at -O3.
> > * params.def (PARAM_LOOP_VERSIONING_GROUP_SIZE)
> > (PARAM_LOOP_VERSIONING_MAX_INNER_INSNS)
> > (PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS): New parameters.
> > * passes.def: Add pass_loop_versioning.
> > * timevar.def (TV_LOOP_VERSIONING): New time variable.
> > * tree-ssa-propagate.h
> > (substitute_and_fold_engine::substitute_and_fold): Add an optional
> > block parameter.
> > * tree-ssa-propagate.c
> > (substitute_and_fold_engine::substitute_and_fold): Likewise.
> > When passed, only walk blocks dominated by that block.
> > * tree-vrp.h (range_includes_p): Declare.
> > (range_includes_zero_p): Turn into an inline wrapper around
> > range_includes_p.
> > * tree-vrp.c (range_includes_p): New function, generalizing...
> > (range_includes_zero_p): ...this.
> > * tree-pass.h (make_pass_loop_versioning): Declare.
> > * gimple-loop-versioning.cc: New file.
> >
> > gcc/testsuite/
> > * gcc.dg/loop-versioning-1.c: New test.
> > * gcc.dg/loop-versioning-10.c: Likewise.
> > * gcc.dg/loop-versioning-11.c: Likewise.
> > * gcc.dg/loop-versioning-2.c: Likewise.
> > * gcc.dg/loop-versioning-3.c: Likewise.
> > * gcc.dg/loop-versioning-4.c: Likewise.
> > * gcc.dg/loop-versioning-5.c: Likewise.
> > * gcc.dg/loop-versioning-6.c: Likewise.
> > * gcc.dg/loop-versioning-7.c: Likewise.
> > * gcc.dg/loop-versioning-8.c: Likewise.
> > * gcc.dg/loop-versioning-9.c: Likewise.
> > * gfortran.dg/loop_versioning_1.f90: Likewise.
> > * gfortran.dg/loop_versioning_2.f90: Likewise.
> > * gfortran.dg/loop_versioning_3.f90: Likewise.
> > * gfortran.dg/loop_versioning_4.f90: Likewise.
> > * gfortran.dg/loop_versioning_5.f90: Likewise.
> > * gfortran.dg/loop_versioning_6.f90: Likewise.
> > * gfortran.dg/loop_versioning_7.f90: Likewise.
> > * gfortran.dg/loop_versioning_8.f90: Likewise.
> >
> > Index: gcc/doc/invoke.texi
> > ===================================================================
> > --- gcc/doc/invoke.texi 2018-10-24 14:02:14.000000000 +0100
> > +++ gcc/doc/invoke.texi 2018-10-24 14:02:15.184152693 +0100
> > @@ -7934,7 +7934,8 @@ by @option{-O2} and also turns on the fo
> > -fvect-cost-model @gol
> > -ftree-partial-pre @gol
> > -fpeel-loops @gol
> > --fipa-cp-clone}
> > +-fipa-cp-clone @gol
> > +-fversion-loops-for-strides}
> >
> > @item -O0
> > @opindex O0
> > @@ -10358,6 +10359,29 @@ for one side of the iteration space and
> > Move branches with loop invariant conditions out of the loop, with duplicates
> > of the loop on both branches (modified according to result of the condition).
> >
> > +@item -fversion-loops-for-strides
> > +@opindex fversion-loops-for-strides
> > +If a loop iterates over an array with a variable stride, create another
> > +version of the loop that assumes the stride is always 1. For example:
> > +
> > +@smallexample
> > +for (int i = 0; i < n; ++i)
> > + x[i * stride] = @dots{};
> > +@end smallexample
> > +
> > +becomes:
> > +
> > +@smallexample
> > +if (stride == 1)
> > + for (int i = 0; i < n; ++i)
> > + x[i] = @dots{};
> > +else
> > + for (int i = 0; i < n; ++i)
> > + x[i * stride] = @dots{};
> > +@end smallexample
> > +
> > +This is particularly useful for assumed-shape arrays in Fortran.
> > +
> > @item -ffunction-sections
> > @itemx -fdata-sections
> > @opindex ffunction-sections
> > @@ -11567,6 +11591,20 @@ Hardware autoprefetcher scheduler model
> > Number of lookahead cycles the model looks into; at '
> > ' only enable instruction sorting heuristic.
> >
> > +@item loop-versioning-group-size
> > +Make the loop versioning pass optimize @samp{a[i * index * @var{N}]}
> > +in the same way as it would optimize @samp{a[i * index]} when @var{N}
> > +is less than or equal to this value.
> > +
> > +@item loop-versioning-max-inner-insns
> > +The maximum number of instructions that an inner loop can have
> > +before the loop versioning pass considers it too big to copy.
> > +
> > +@item loop-versioning-max-outer-insns
> > +The maximum number of instructions that an outer loop can have
> > +before the loop versioning pass considers it too big to copy,
> > +discounting any instructions in inner loops that directly benefit
> > +from versioning.
> >
> > @end table
> > @end table
> > Index: gcc/Makefile.in
> > ===================================================================
> > --- gcc/Makefile.in 2018-10-24 14:02:14.000000000 +0100
> > +++ gcc/Makefile.in 2018-10-24 14:02:15.180152727 +0100
> > @@ -1312,6 +1312,7 @@ OBJS = \
> > gimple-laddress.o \
> > gimple-loop-interchange.o \
> > gimple-loop-jam.o \
> > + gimple-loop-versioning.o \
> > gimple-low.o \
> > gimple-pretty-print.o \
> > gimple-ssa-backprop.o \
> > Index: gcc/common.opt
> > ===================================================================
> > --- gcc/common.opt 2018-10-24 14:02:14.000000000 +0100
> > +++ gcc/common.opt 2018-10-24 14:02:15.180152727 +0100
> > @@ -2712,6 +2712,10 @@ fsplit-loops
> > Common Report Var(flag_split_loops) Optimization
> > Perform loop splitting.
> >
> > +fversion-loops-for-strides
> > +Common Report Var(flag_version_loops_for_strides) Optimization
> > +Version loops based on whether indices have a stride of 1.
> > +
> > funwind-tables
> > Common Report Var(flag_unwind_tables) Optimization
> > Just generate unwind tables for exception handling.
> > Index: gcc/opts.c
> > ===================================================================
> > --- gcc/opts.c 2018-10-24 14:02:14.000000000 +0100
> > +++ gcc/opts.c 2018-10-24 14:02:15.184152693 +0100
> > @@ -544,6 +544,7 @@ static const struct default_options defa
> > { OPT_LEVELS_3_PLUS, OPT_fipa_cp_clone, NULL, 1 },
> > { OPT_LEVELS_3_PLUS, OPT_ftree_partial_pre, NULL, 1 },
> > { OPT_LEVELS_3_PLUS, OPT_fpeel_loops, NULL, 1 },
> > + { OPT_LEVELS_3_PLUS, OPT_fversion_loops_for_strides, NULL, 1 },
> >
> > /* -Ofast adds optimizations to -O3. */
> > { OPT_LEVELS_FAST, OPT_ffast_math, NULL, 1 },
> > Index: gcc/params.def
> > ===================================================================
> > --- gcc/params.def 2018-10-24 14:02:14.000000000 +0100
> > +++ gcc/params.def 2018-10-24 14:02:15.184152693 +0100
> > @@ -1360,6 +1360,25 @@ DEFPARAM(PARAM_AVOID_FMA_MAX_BITS,
> > "Maximum number of bits for which we avoid creating FMAs.",
> > 0, 0, 512)
> >
> > +DEFPARAM(PARAM_LOOP_VERSIONING_GROUP_SIZE,
> > + "loop-versioning-group-size",
> > + "The maximum constant N for which accesses of the form x[N * step]"
> > + " are worth versioning for the case in which step is 1",
> > + 4, 1, 0)
> > +
> > +DEFPARAM(PARAM_LOOP_VERSIONING_MAX_INNER_INSNS,
> > + "loop-versioning-max-inner-insns",
> > + "The maximum number of instructions in an inner loop that is being"
> > + " considered for versioning",
> > + 200, 0, 0)
> > +
> > +DEFPARAM(PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS,
> > + "loop-versioning-max-outer-insns",
> > + "The maximum number of instructions in an outer loop that is being"
> > + " considered for versioning, on top of the instructions in inner"
> > + " loops",
> > + 100, 0, 0)
> > +
> > /*
> >
> > Local variables:
> > Index: gcc/passes.def
> > ===================================================================
> > --- gcc/passes.def 2018-10-24 14:02:14.000000000 +0100
> > +++ gcc/passes.def 2018-10-24 14:02:15.184152693 +0100
> > @@ -260,6 +260,7 @@ along with GCC; see the file COPYING3.
> > NEXT_PASS (pass_tree_loop);
> > PUSH_INSERT_PASSES_WITHIN (pass_tree_loop)
> > NEXT_PASS (pass_tree_loop_init);
> > + NEXT_PASS (pass_loop_versioning);
> > NEXT_PASS (pass_tree_unswitch);
> > NEXT_PASS (pass_scev_cprop);
> > NEXT_PASS (pass_loop_split);
> > Index: gcc/timevar.def
> > ===================================================================
> > --- gcc/timevar.def 2018-10-24 14:02:14.000000000 +0100
> > +++ gcc/timevar.def 2018-10-24 14:02:15.188152659 +0100
> > @@ -234,6 +234,7 @@ DEFTIMEVAR (TV_DSE1 , "
> > DEFTIMEVAR (TV_DSE2 , "dead store elim2")
> > DEFTIMEVAR (TV_LOOP , "loop analysis")
> > DEFTIMEVAR (TV_LOOP_INIT , "loop init")
> > +DEFTIMEVAR (TV_LOOP_VERSIONING , "loop versioning")
> > DEFTIMEVAR (TV_LOOP_MOVE_INVARIANTS , "loop invariant motion")
> > DEFTIMEVAR (TV_LOOP_UNROLL , "loop unrolling")
> > DEFTIMEVAR (TV_LOOP_DOLOOP , "loop doloop")
> > Index: gcc/tree-ssa-propagate.h
> > ===================================================================
> > --- gcc/tree-ssa-propagate.h 2018-10-24 14:02:14.000000000 +0100
> > +++ gcc/tree-ssa-propagate.h 2018-10-24 14:02:15.188152659 +0100
> > @@ -104,7 +104,7 @@ extern void propagate_tree_value_into_st
> > virtual bool fold_stmt (gimple_stmt_iterator *) { return false; }
> > virtual tree get_value (tree) { return NULL_TREE; }
> >
> > - bool substitute_and_fold (void);
> > + bool substitute_and_fold (basic_block = NULL);
> > bool replace_uses_in (gimple *);
> > bool replace_phi_args_in (gphi *);
> > };
> > Index: gcc/tree-ssa-propagate.c
> > ===================================================================
> > --- gcc/tree-ssa-propagate.c 2018-10-24 14:02:14.000000000 +0100
> > +++ gcc/tree-ssa-propagate.c 2018-10-24 14:02:15.188152659 +0100
> > @@ -1152,6 +1152,10 @@ substitute_and_fold_dom_walker::before_d
> >
> >
> > /* Perform final substitution and folding of propagated values.
> > + Process the whole function if BLOCK is null, otherwise only
> > + process the blocks that BLOCK dominates. In the latter case,
> > + it is the caller's responsibility to ensure that dominator
> > + information is available and up-to-date.
> >
> > PROP_VALUE[I] contains the single value that should be substituted
> > at every use of SSA name N_I. If PROP_VALUE is NULL, no values are
> > @@ -1168,16 +1172,24 @@ substitute_and_fold_dom_walker::before_d
> > Return TRUE when something changed. */
> >
> > bool
> > -substitute_and_fold_engine::substitute_and_fold (void)
> > +substitute_and_fold_engine::substitute_and_fold (basic_block block)
> > {
> > if (dump_file && (dump_flags & TDF_DETAILS))
> > fprintf (dump_file, "\nSubstituting values and folding statements\n\n");
> >
> > memset (&prop_stats, 0, sizeof (prop_stats));
> >
> > - calculate_dominance_info (CDI_DOMINATORS);
> > + /* Don't call calculate_dominance_info when iterating over a subgraph.
> > + Callers that are using the interface this way are likely to want to
> > + iterate over several disjoint subgraphs, and it would be expensive
> > + in enable-checking builds to revalidate the whole dominance tree
> > + each time. */
> > + if (block)
> > + gcc_assert (dom_info_state (CDI_DOMINATORS));
> > + else
> > + calculate_dominance_info (CDI_DOMINATORS);
> > substitute_and_fold_dom_walker walker (CDI_DOMINATORS, this);
> > - walker.walk (ENTRY_BLOCK_PTR_FOR_FN (cfun));
> > + walker.walk (block ? block : ENTRY_BLOCK_PTR_FOR_FN (cfun));
> >
> > /* We cannot remove stmts during the BB walk, especially not release
> > SSA names there as that destroys the lattice of our callers.
> > Index: gcc/tree-vrp.h
> > ===================================================================
> > --- gcc/tree-vrp.h 2018-10-24 14:02:14.000000000 +0100
> > +++ gcc/tree-vrp.h 2018-10-24 14:02:15.188152659 +0100
> > @@ -86,7 +86,7 @@ extern void register_edge_assert_for (tr
> > tree, tree, vec<assert_info> &);
> > extern bool stmt_interesting_for_vrp (gimple *);
> > extern void set_value_range_to_varying (value_range *);
> > -extern bool range_includes_zero_p (const value_range *);
> > +extern bool range_includes_p (const value_range *, HOST_WIDE_INT);
> > extern bool infer_value_range (gimple *, tree, tree_code *, tree *);
> >
> > extern void set_value_range_to_nonnull (value_range *, tree);
> > @@ -122,4 +122,13 @@ extern int value_inside_range (tree, tre
> > extern tree get_single_symbol (tree, bool *, tree *);
> > extern void maybe_set_nonzero_bits (edge, tree);
> > extern value_range_type determine_value_range (tree, wide_int *, wide_int *);
> > +
> > +/* Return TRUE if *VR includes the value zero. */
> > +
> > +inline bool
> > +range_includes_zero_p (const value_range *vr)
> > +{
> > + return range_includes_p (vr, 0);
> > +}
> > +
> > #endif /* GCC_TREE_VRP_H */
> > Index: gcc/tree-vrp.c
> > ===================================================================
> > --- gcc/tree-vrp.c 2018-10-24 14:02:14.000000000 +0100
> > +++ gcc/tree-vrp.c 2018-10-24 14:02:15.188152659 +0100
> > @@ -844,10 +844,10 @@ value_inside_range (tree val, tree min,
> > }
> >
> >
> > -/* Return TRUE if *VR includes the value zero. */
> > +/* Return TRUE if *VR includes the value X. */
> >
> > bool
> > -range_includes_zero_p (const value_range *vr)
> > +range_includes_p (const value_range *vr, HOST_WIDE_INT x)
> > {
> > if (vr->type == VR_VARYING)
> > return true;
> > @@ -856,13 +856,13 @@ range_includes_zero_p (const value_range
> > if (vr->type == VR_UNDEFINED)
> > return true;
> >
> > - tree zero = build_int_cst (TREE_TYPE (vr->min), 0);
> > + tree x_cst = build_int_cst (TREE_TYPE (vr->min), x);
> > if (vr->type == VR_ANTI_RANGE)
> > {
> > - int res = value_inside_range (zero, vr->min, vr->max);
> > + int res = value_inside_range (x_cst, vr->min, vr->max);
> > return res == 0 || res == -2;
> > }
> > - return value_inside_range (zero, vr->min, vr->max) != 0;
> > + return value_inside_range (x_cst, vr->min, vr->max) != 0;
> > }
> >
> > /* If *VR has a value rante that is a single constant value return that,
> > Index: gcc/tree-pass.h
> > ===================================================================
> > --- gcc/tree-pass.h 2018-10-24 14:02:14.000000000 +0100
> > +++ gcc/tree-pass.h 2018-10-24 14:02:15.188152659 +0100
> > @@ -362,6 +362,7 @@ extern gimple_opt_pass *make_pass_fix_lo
> > extern gimple_opt_pass *make_pass_tree_loop (gcc::context *ctxt);
> > extern gimple_opt_pass *make_pass_tree_no_loop (gcc::context *ctxt);
> > extern gimple_opt_pass *make_pass_tree_loop_init (gcc::context *ctxt);
> > +extern gimple_opt_pass *make_pass_loop_versioning (gcc::context *ctxt);
> > extern gimple_opt_pass *make_pass_lim (gcc::context *ctxt);
> > extern gimple_opt_pass *make_pass_linterchange (gcc::context *ctxt);
> > extern gimple_opt_pass *make_pass_tree_unswitch (gcc::context *ctxt);
> > Index: gcc/gimple-loop-versioning.cc
> > ===================================================================
> > --- /dev/null 2018-09-14 11:16:31.122530289 +0100
> > +++ gcc/gimple-loop-versioning.cc 2018-10-24 14:02:15.184152693 +0100
> > @@ -0,0 +1,1417 @@
> > +/* Loop versioning pass.
> > + Copyright (C) 2018 Free Software Foundation, Inc.
> > +
> > +This file is part of GCC.
> > +
> > +GCC is free software; you can redistribute it and/or modify it
> > +under the terms of the GNU General Public License as published by the
> > +Free Software Foundation; either version 3, or (at your option) any
> > +later version.
> > +
> > +GCC is distributed in the hope that it will be useful, but WITHOUT
> > +ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
> > +FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
> > +for more details.
> > +
> > +You should have received a copy of the GNU General Public License
> > +along with GCC; see the file COPYING3. If not see
> > +<http://www.gnu.org/licenses/>. */
> > +
> > +#include "config.h"
> > +#include "system.h"
> > +#include "coretypes.h"
> > +#include "backend.h"
> > +#include "tree.h"
> > +#include "gimple.h"
> > +#include "gimple-iterator.h"
> > +#include "tree-pass.h"
> > +#include "gimplify-me.h"
> > +#include "cfgloop.h"
> > +#include "tree-ssa-loop.h"
> > +#include "ssa.h"
> > +#include "tree-scalar-evolution.h"
> > +#include "tree-chrec.h"
> > +#include "tree-ssa-loop-ivopts.h"
> > +#include "fold-const.h"
> > +#include "tree-ssa-propagate.h"
> > +#include "tree-inline.h"
> > +#include "domwalk.h"
> > +#include "alloc-pool.h"
> > +#include "vr-values.h"
> > +#include "gimple-ssa-evrp-analyze.h"
> > +#include "gimple-pretty-print.h"
> > +#include "params.h"
> > +
> > +/* This pass looks for loops that could be simplified if certain loop
> > + invariant conditions were true. It is effectively a form of loop
> > + splitting in which the pass produces the split conditions itself,
> > + instead of using ones that are already present in the IL.
> > +
> > + Versioning for when strides are 1
> > + ---------------------------------
> > +
> > + At the moment the only thing the pass looks for are memory references
> > + like:
> > +
> > + for (auto i : ...)
> > + ...x[i * stride]...
> > +
> > + It considers changing such loops to:
> > +
> > + if (stride == 1)
> > + for (auto i : ...) [A]
> > + ...x[i]...
> > + else
> > + for (auto i : ...) [B]
> > + ...x[i * stride]...
> > +
> > + This can have several benefits:
> > +
> > + (1) [A] is often easier or cheaper to vectorize than [B].
> > +
> > + (2) The scalar code in [A] is simpler than the scalar code in [B]
> > + (if the loops cannot be vectorized or need an epilogue loop).
> > +
> > + (3) We might recognize [A] as a pattern, such as a memcpy or memset.
> > +
> > + (4) [A] has simpler address evolutions, which can help other passes
> > + like loop interchange.
> > +
> > + The optimization is particularly useful for assumed-shape arrays in
> > + Fortran, where the stride of the innermost dimension depends on the
> > + array descriptor but is often equal to 1 in practice. For example:
> > +
> > + subroutine f1(x)
> > + real :: x(:)
> > + x(:) = 100
> > + end subroutine f1
> > +
> > + generates the equivalent of:
> > +
> > + raw_stride = *x.dim[0].stride;
> > + stride = raw_stride != 0 ? raw_stride : 1;
> > + x_base = *x.data;
> > + ...
> > + tmp1 = stride * S;
> > + tmp2 = tmp1 - stride;
> > + *x_base[tmp2] = 1.0e+2;
> > +
> > + but in the common case that stride == 1, the last three statements
> > + simplify to:
> > +
> > + tmp3 = S + -1;
> > + *x_base[tmp3] = 1.0e+2;
> > +
> > + The optimization is in principle very simple. The difficult parts are:
> > +
> > + (a) deciding which parts of a general address calculation correspond
> > + to the inner dimension of an array, since this usually isn't explicit
> > + in the IL, and for C often isn't even explicit in the source code
> > +
> > + (b) estimating when the transformation is worthwhile
> > +
> > + Structure
> > + ---------
> > +
> > + The pass has four phases:
> > +
> > + (1) Walk through the statements looking for and recording potential
> > + versioning opportunities. Stop if there are none.
> > +
> > + (2) Use context-sensitive range information to see whether any versioning
> > + conditions are impossible in practice. Remove them if so, and stop
> > + if no opportunities remain.
> > +
> > + (We do this only after (1) to keep compile time down when no
> > + versioning opportunities exist.)
> > +
> > + (3) Apply the cost model. Decide which versioning opportunities are
> > + worthwhile and at which nesting level they should be applied.
> > +
> > + (4) Attempt to version all the loops selected by (3), so that:
> > +
> > + for (...)
> > + ...
> > +
> > + becomes:
> > +
> > + if (!cond)
> > + for (...) // Original loop
> > + ...
> > + else
> > + for (...) // New loop
> > + ...
> > +
> > + Use the version condition COND to simplify the new loop. */
> > +class loop_versioning {
> > +public:
> > + loop_versioning (function *);
> > + ~loop_versioning ();
> > + unsigned int run ();
> > +
> > +private:
> > + /* Information about the versioning we'd like to apply to a loop. */
> > + struct loop_info {
> > + bool worth_versioning_p () const;
> > +
> > + /* True if we've decided not to version this loop. The remaining
> > + fields are meaningless if so. */
> > + bool rejected_p;
> > +
> > + /* True if at least one subloop of this loop benefits from versioning. */
> > + bool subloops_benefit_p;
> > +
> > + /* An estimate of the total number of instructions in the loop,
> > + excluding those in subloops that benefit from versioning. */
> > + unsigned int num_insns;
> > +
> > + /* The outermost loop that can handle all the version checks
> > + described below. */
> > + struct loop *outermost;
> > +
> > + /* We'd like to version the loop for the case in which these
> > + SSA_NAMEs are all equal to 1 at runtime. */
> > + vec<tree> unity_names;
> > +
> > + /* The set of SSA_NAMEs in UNITY_NAMES, keyed off the SSA_NAME_VERSION. */
> > + bitmap_head unity_name_ids;
> > + };
> > +
> > + /* Used to walk the dominator tree to find loop versioning conditions
> > + that are always false. */
> > + class lv_dom_walker : public dom_walker
> > + {
> > + public:
> > + lv_dom_walker (loop_versioning &);
> > +
> > + edge before_dom_children (basic_block) FINAL OVERRIDE;
> > + void after_dom_children (basic_block) FINAL OVERRIDE;
> > +
> > + private:
> > + /* The parent pass. */
> > + loop_versioning &m_lv;
> > +
> > + /* Used to build context-dependent range information. */
> > + evrp_range_analyzer m_range_analyzer;
> > + };
> > +
> > + /* Used to simplify statements based on conditions that are established
> > + by the version checks. */
> > + class name_prop : public substitute_and_fold_engine
> > + {
> > + public:
> > + name_prop (loop_info &li) : m_li (li) {}
> > + tree get_value (tree) FINAL OVERRIDE;
> > +
> > + private:
> > + /* Information about the versioning we've performed on the loop. */
> > + loop_info &m_li;
> > + };
> > +
> > + loop_info &get_loop_info (struct loop *loop) { return m_loops[loop->num]; }
> > +
> > + unsigned int max_insns_for_loop (struct loop *);
> > + bool expensive_stmt_p (gimple *);
> > +
> > + void version_for_unity (struct loop *, tree);
> > + bool acceptable_scale_p (tree, poly_uint64);
> > + tree get_step_if_innermost (tree, poly_uint64);
> > + tree extract_step (tree, poly_uint64, tree *);
> > + void analyze_evolution (struct loop *, tree, poly_uint64);
> > + bool analyze_product (struct loop *, gassign *, poly_uint64);
> > + bool analyze_sum_of_products (struct loop *, tree, poly_uint64);
> > + void analyze_pointer (struct loop *, tree, tree);
> > + void analyze_expr (struct loop *, tree);
> > + void analyze_stmt (gimple *);
> > + void analyze_block (basic_block);
> > + bool analyze_blocks ();
> > +
> > + void prune_loop_conditions (struct loop *, vr_values *);
> > + bool prune_conditions ();
> > +
> > + void merge_loop_info (struct loop *, struct loop *);
> > + void add_loop_to_queue (struct loop *);
> > + bool decide_whether_loop_is_versionable (struct loop *);
> > + bool make_versioning_decisions ();
> > +
> > + bool version_loop (struct loop *);
> > + bool implement_versioning_decisions ();
> > +
> > + /* The function we're optimizing. */
> > + function *m_fn;
> > +
> > + /* The obstack to use for all pass-specific bitmaps. */
> > + bitmap_obstack m_obstack;
> > +
> > + /* The number of loops in the function. */
> > + unsigned int m_nloops;
> > +
> > + /* The total number of loop version conditions we've found. */
> > + unsigned int m_num_conditions;
> > +
> > + /* Information about each loop. */
> > + auto_vec<loop_info> m_loops;
> > +
> > + /* The list of loops that we've decided to version. */
> > + auto_vec<struct loop *> m_loops_to_version;
> > +};
> > +
> > +/* If EXPR is an SSA name and not a default definition, return the
> > + defining statement, otherwise return null. */
> > +
> > +static gimple *
> > +maybe_get_stmt (tree expr)
> > +{
> > + if (TREE_CODE (expr) == SSA_NAME && !SSA_NAME_IS_DEFAULT_DEF (expr))
> > + return SSA_NAME_DEF_STMT (expr);
> > + return NULL;
> > +}
> > +
> > +/* Like maybe_get_stmt, but also return null if the defining
> > + statement isn't an assignment. */
> > +
> > +static gassign *
> > +maybe_get_assign (tree expr)
> > +{
> > + return safe_dyn_cast <gassign *> (maybe_get_stmt (expr));
> > +}
> > +
> > +/* If EXPR is an SSA name, look through any casts to see whether the
> > + unconverted value is defined in LOOP by a gassign. Return the
> > + gassign if so, otherwise return null. */
> > +
> > +gassign *
> > +maybe_get_assign_strip_casts (struct loop *loop, tree expr)
> > +{
> > + const unsigned int MAX_NITERS = 4;
> > +
> > + tree type = TREE_TYPE (expr);
> > + for (unsigned int niters = 0; niters < MAX_NITERS; ++niters)
> > + {
> > + gassign *assign = maybe_get_assign (expr);
> > + if (!assign || gimple_bb (assign)->loop_father != loop)
> > + return NULL;
> > + expr = gimple_assign_rhs1 (assign);
> > + if (CONVERT_EXPR_CODE_P (gimple_assign_rhs_code (assign))
> > + && INTEGRAL_TYPE_P (TREE_TYPE (expr)) == INTEGRAL_TYPE_P (type)
> > + && POINTER_TYPE_P (TREE_TYPE (expr)) == POINTER_TYPE_P (type))
> > + ;
> > + else
> > + return assign;
> > + }
> > + return NULL;
> > +}
> > +
> > +/* Strip all conversions of integers from EXPR, regardless of whether
> > + the conversions are nops. This is useful in the context of this pass
> > + because we're not trying to fold or simulate the expression; we just
> > + want to see how it's structured. */
> > +
> > +static tree
> > +strip_casts (tree expr)
> > +{
> > + while (CONVERT_EXPR_P (expr)
> > + && INTEGRAL_TYPE_P (TREE_TYPE (TREE_OPERAND (expr, 0))))
> > + expr = TREE_OPERAND (expr, 0);
> > + return expr;
> > +}
> > +
> > +/* Return true if we want to version the loop, i.e. if we have a
> > + specific reason for doing so and no specific reason not to. */
> > +
> > +bool
> > +loop_versioning::loop_info::worth_versioning_p () const
> > +{
> > + return !rejected_p && (!unity_names.is_empty () || subloops_benefit_p);
> > +}
> > +
> > +loop_versioning::lv_dom_walker::lv_dom_walker (loop_versioning &lv)
> > + : dom_walker (CDI_DOMINATORS), m_lv (lv)
> > +{
> > +}
> > +
> > +/* Process BB before processing the blocks it dominates. */
> > +
> > +edge
> > +loop_versioning::lv_dom_walker::before_dom_children (basic_block bb)
> > +{
> > + m_range_analyzer.enter (bb);
> > +
> > + if (bb == bb->loop_father->header)
> > + m_lv.prune_loop_conditions (bb->loop_father,
> > + m_range_analyzer.get_vr_values ());
> > +
> > + for (gimple_stmt_iterator si = gsi_start_bb (bb); !gsi_end_p (si);
> > + gsi_next (&si))
> > + m_range_analyzer.record_ranges_from_stmt (gsi_stmt (si), false);
> > +
> > + return NULL;
> > +}
> > +
> > +/* Process BB after processing the blocks it dominates. */
> > +
> > +void
> > +loop_versioning::lv_dom_walker::after_dom_children (basic_block bb)
> > +{
> > + m_range_analyzer.leave (bb);
> > +}
> > +
> > +/* Decide whether to replace VAL with a new value in a versioned loop.
> > + Return the new value if so, otherwise return null. */
> > +
> > +tree
> > +loop_versioning::name_prop::get_value (tree val)
> > +{
> > + if (TREE_CODE (val) == SSA_NAME
> > + && bitmap_bit_p (&m_li.unity_name_ids, SSA_NAME_VERSION (val)))
> > + return build_one_cst (TREE_TYPE (val));
> > + return NULL_TREE;
> > +}
> > +
> > +/* Initialize the structure to optimize FN. */
> > +
> > +loop_versioning::loop_versioning (function *fn)
> > + : m_fn (fn),
> > + m_nloops (number_of_loops (fn)),
> > + m_num_conditions (0)
> > +{
> > + bitmap_obstack_initialize (&m_obstack);
> > +
> > + m_loops.safe_grow_cleared (m_nloops);
> > + for (unsigned int i = 0; i < m_nloops; ++i)
> > + {
> > + m_loops[i].outermost = get_loop (m_fn, 0);
> > + bitmap_initialize (&m_loops[i].unity_name_ids, &m_obstack);
> > + }
> > +}
> > +
> > +loop_versioning::~loop_versioning ()
> > +{
> > + for (unsigned int i = 0; i < m_nloops; ++i)
> > + m_loops[i].unity_names.release ();
> > + bitmap_obstack_release (&m_obstack);
> > +}
> > +
> > +/* Return the maximum number of instructions allowed in LOOP before
> > + it becomes too big for versioning.
> > +
> > + There are separate limits for inner and outer loops. The limit for
> > + inner loops applies only to loops that benefit directly from versioning.
> > + The limit for outer loops applies to all code in the outer loop and
> > + its subloops that *doesn't* benefit directly from versioning; such code
> > + would be "taken along for the ride". The idea is that if the cost of
> > + the latter is small, it is better to version outer loops rather than
> > + inner loops, both to reduce the number of repeated checks and to enable
> > + more of the loop nest to be optimized as a natural nest (e.g. by loop
> > + interchange or outer-loop vectorization). */
> > +
> > +unsigned int
> > +loop_versioning::max_insns_for_loop (struct loop *loop)
> > +{
> > + return (loop->inner
> > + ? PARAM_VALUE (PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS)
> > + : PARAM_VALUE (PARAM_LOOP_VERSIONING_MAX_INNER_INSNS));
> > +}
> > +
> > +/* Return true if for cost reasons we should avoid versioning any loop
> > + that contains STMT.
> > +
> > + Note that we don't need to check whether versioning is invalid for
> > + correctness reasons, since the versioning process does that for us.
> > + The conditions involved are too rare to be worth duplicating here. */
> > +
> > +bool
> > +loop_versioning::expensive_stmt_p (gimple *stmt)
> > +{
> > + if (gcall *call = dyn_cast <gcall *> (stmt))
> > + /* Assume for now that the time spent in an "expensive" call would
> > + overwhelm any saving from versioning. */
> > + return !gimple_inexpensive_call_p (call);
> > + return false;
> > +}
> > +
> > +/* Record that we want to version LOOP for the case in which SSA name NAME
> > + is equal to 1. We already know that NAME is invariant in LOOP. */
> > +
> > +void
> > +loop_versioning::version_for_unity (struct loop *loop, tree name)
> > +{
> > + loop_info &li = get_loop_info (loop);
> > +
> > + if (bitmap_set_bit (&li.unity_name_ids, SSA_NAME_VERSION (name)))
> > + {
> > + /* This is the first time we've wanted to version LOOP for NAME. */
> > + li.unity_names.safe_push (name);
> > +
> > + /* Keep track of the outermost loop that can handle all versioning
> > + checks in LI. */
> > + struct loop *outermost
> > + = outermost_invariant_loop_for_expr (loop, name);
> > + if (loop_depth (li.outermost) < loop_depth (outermost))
> > + li.outermost = outermost;
> > +
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + {
> > + fprintf (dump_file, ";; Want to version loop %d (depth %d)"
> > + " for when ", loop->num, loop_depth (loop));
> > + print_generic_expr (dump_file, name, TDF_SLIM);
> > + fprintf (dump_file, " == 1");
> > + if (outermost == loop)
> > + fprintf (dump_file, "; cannot hoist check further");
> > + else
> > + {
> > + fprintf (dump_file, "; could hoist check to loop %d (depth %d)",
> > + outermost->num, loop_depth (outermost));
> > + if (loop_depth (li.outermost) > loop_depth (outermost))
> > + fprintf (dump_file, ", but that's further than"
> > + " other checks allow");
> > + }
> > + fprintf (dump_file, "\n");
> > + }
> > +
> > + m_num_conditions += 1;
> > + }
> > + else
> > + {
> > + /* This is a duplicate request. */
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + {
> > + fprintf (dump_file, ";; Already want to version loop for when ");
> > + print_generic_expr (dump_file, name, TDF_SLIM);
> > + fprintf (dump_file, " == 1\n");
> > + }
> > + }
> > +}
> > +
> > +/* Return true if in principle it is worth versioning an index fragment of
> > + the form:
> > +
> > + (i * b * SCALE) / FACTOR
> > +
> > + for the case in which b == 1. */
> > +
> > +bool
> > +loop_versioning::acceptable_scale_p (tree scale, poly_uint64 factor)
> > +{
> > + /* See whether SCALE is a constant multiple of FACTOR, and if the
> > + multiple is small enough for us to treat it as a potential grouped
> > + access. For example:
> > +
> > + for (auto i : ...)
> > + y[i] = f (x[4 * i * stride],
> > + x[4 * i * stride + 1],
> > + x[4 * i * stride + 2]);
> > +
> > + would benefit from versioning for the case in which stride == 1.
> > + High multiples of i * stride are less likely to benefit, and could
> > + indicate a simulated multi-dimensional array.
> > +
> > + This is just a heuristic, to avoid having to do expensive group
> > + analysis of the data references in a loop. */
> > + poly_uint64 const_scale;
> > + unsigned int multiple;
> > + if (poly_int_tree_p (scale, &const_scale)
> > + && constant_multiple_p (const_scale, factor, &multiple))
> > + {
> > + unsigned int maxval = PARAM_VALUE (PARAM_LOOP_VERSIONING_GROUP_SIZE);
> > + return IN_RANGE (multiple, 1, maxval);
> > + }
> > +
> > + return false;
> > +}
> > +
> > +/* Decide whether an index fragment of the form:
> > +
> > + (i * STEP) / FACTOR
> > +
> > + is likely to be for an innermost dimension. If we think it is,
> > + return one of the constant values that it could have (returning 1 if
> > + that's a possibility). If think it isn't, return null. Otherwise
> > + return STEP, to indicate that it might or might not be an inner
> > + dimension. */
> > +
> > +tree
> > +loop_versioning::get_step_if_innermost (tree step, poly_uint64 factor)
> > +{
> > + const unsigned int MAX_NITERS = 8;
> > +
> > + tree likely = NULL_TREE;
> > + tree unlikely = NULL_TREE;
> > + tree worklist[MAX_NITERS];
> > + unsigned int length = 0;
> > + worklist[length++] = step;
> > + for (unsigned int i = 0; i < length; ++i)
> > + {
> > + tree expr = worklist[i];
> > +
> > + if (TREE_CONSTANT (expr))
> > + {
> > + /* See if multiplying by EXPR applies a scale that would be
> > + consistent with an individual access or a small grouped
> > + access. */
> > + if (acceptable_scale_p (expr, factor))
> > + {
> > + likely = expr;
> > + if (integer_onep (expr))
> > + break;
> > + }
> > + else
> > + unlikely = expr;
> > + continue;
> > + }
> > +
> > + /* Otherwise we can only handle SSA names. */
> > + gimple *stmt = maybe_get_stmt (expr);
> > + if (!stmt)
> > + continue;
> > +
> > + /* If EXPR is set by a PHI node, queue its arguments in case
> > + we find one that is consistent with an inner dimension.
> > +
> > + An important instance of this is the Fortran handling of array
> > + descriptors, which calculates the stride of the inner dimension
> > + using a PHI equivalent of:
> > +
> > + raw_stride = a.dim[0].stride;
> > + stride = raw_stride != 0 ? raw_stride : 1;
> > +
> > + (Strides for outer dimensions do not treat 0 specially.) */
> > + if (gphi *phi = dyn_cast <gphi *> (stmt))
> > + {
> > + unsigned int nargs = gimple_phi_num_args (phi);
> > + for (unsigned int j = 0; j < nargs && length < MAX_NITERS; ++j)
> > + worklist[length++] = gimple_phi_arg_def (phi, j);
> > + continue;
> > + }
> > +
> > + /* If the value is set by an assignment, expect it to be read from
> > + memory (such as an array descriptor) rather than be calculated. */
> > + if (gassign *assign = dyn_cast <gassign *> (stmt))
> > + {
> > + if (gimple_assign_single_p (assign)
> > + && is_gimple_lvalue (gimple_assign_rhs1 (assign)))
> > + continue;
> > +
> > + unlikely = expr;
> > + }
> > +
> > + /* Things like calls don't really tell us anything. */
> > + }
> > +
> > + if (likely)
> > + {
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + fprintf (dump_file, "likely to be innermost dimension\n");
> > + return likely;
> > + }
> > +
> > + if (unlikely)
> > + {
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + fprintf (dump_file, "probably not innermost dimension\n");
> > + return NULL_TREE;
> > + }
> > +
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + {
> > + if (maybe_ne (factor, 1U))
> > + fprintf (dump_file, "didn't find expected scaling factor\n");
> > + else
> > + fprintf (dump_file, "no information about value\n");
> > + }
> > + return step;
> > +}
> > +
> > +/* STEP appears in an index fragment of the form:
> > +
> > + {..., +, STEP}_n / FACTOR
> > +
> > + Remove any conversions and grouping or scaling factors from STEP and
> > + return the underlying index step. Set *STEP_IF_INNERMOST to the value
> > + returned by get_step_if_innermost. */
> > +
> > +tree
> > +loop_versioning::extract_step (tree step, poly_uint64 factor,
> > + tree *step_if_innermost)
> > +{
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + {
> > + fprintf (dump_file, ";; step ");
> > + print_generic_expr (dump_file, step, TDF_SLIM);
> > + fprintf (dump_file, ": ");
> > + }
> > +
> > + step = strip_casts (step);
> > +
> > + /* Peel any scaling, which generally happens after conversion to
> > + pointer width. For example, on LP64 systems:
> > +
> > + int *x, i, stride;
> > + ... x[4 * i * stride] ...;
> > +
> > + multiplies i * stride by 4 using ints, then widens the result
> > + to pointer width before multiplying by sizeof (int). */
> > + poly_uint64 scale;
> > + if (TREE_CODE (step) == MULT_EXPR
> > + && poly_int_tree_p (TREE_OPERAND (step, 1), &scale)
> > + && known_eq (scale, factor))
> > + {
> > + step = strip_casts (TREE_OPERAND (step, 0));
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + fprintf (dump_file, "scaled, ");
> > + factor = 1;
> > + }
> > +
> > + /* Peel any grouping multiple; see acceptable_scale_p for details. */
> > + if (TREE_CODE (step) == MULT_EXPR
> > + && acceptable_scale_p (TREE_OPERAND (step, 1), factor))
> > + {
> > + step = strip_casts (TREE_OPERAND (step, 0));
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + fprintf (dump_file, "%sgrouped, ",
> > + maybe_ne (factor, 1U) ? "scaled, " : "");
> > + factor = 1;
> > + }
> > +
> > + *step_if_innermost = get_step_if_innermost (step, factor);
> > + return step;
> > +}
> > +
> > +/* Analyze the evolution of index fragment EXPR / FACTOR in LOOP and its
> > + containing loops to see whether any part of it could be simplified
> > + by versioning. Register the versioning opportunities if so. */
> > +
> > +void
> > +loop_versioning::analyze_evolution (struct loop *loop, tree expr,
> > + poly_uint64 factor)
> > +{
> > + const unsigned int MAX_NSPLIT = 8;
> > +
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + {
> > + fprintf (dump_file, ";; Analyzing use of ");
> > + print_generic_expr (dump_file, expr, TDF_SLIM);
> > + if (maybe_ne (factor, 1U))
> > + {
> > + fprintf (dump_file, " (which addresses ");
> > + print_dec (factor, dump_file);
> > + fprintf (dump_file, " bytes)");
> > + }
> > + fprintf (dump_file, " in loop %d (depth %d)\n",
> > + loop->num, loop_depth (loop));
> > + }
> > +
> > + /* The main problem we have here is that we cannot assume that the
> > + innermost loop iterates over the innermost dimension of an array.
> > + Accidentally adding versioning checks for outer dimensions would
> > + cause the version condition to be false, which as well as bloating
> > + the code would defeat loop versioning benefits for other accesses.
> > +
> > + Unfortunately all we usually see at this stage is general address
> > + arithmetic, with no positive way of identifying how many dimensions
> > + an array access has and which multiplication factors in the address
> > + expression correspond to which array dimensions. In C code this is
> > + often not even explicit in the source, since variable-sized multi-
> > + dimensional arrays are often simulated using one-dimensional arrays.
> > +
> > + The three main ways in which we deal with this are:
> > +
> > + - use heuristics that positively identify steps that are likely
> > + to represent the inner dimension.
> > +
> > + - use heuristics that positively identify steps that are unlikely
> > + to represent the inner dimension.
> > +
> > + - if a part of EXPR is invariant in LOOP, analyze its evolution in
> > + the outer loops to see whether we can positively identify any of
> > + it as iterating over the inner dimension. */
> > + tree best_step = NULL_TREE;
> > + auto_vec<tree, MAX_NSPLIT> worklist;
> > + worklist.quick_push (expr);
> > + unsigned int nsplit = 0;
> > + while (!worklist.is_empty ())
> > + {
> > + expr = strip_casts (worklist.pop ());
> > + tree_code code = TREE_CODE (expr);
> > +
> > + if (code == POLYNOMIAL_CHREC)
> > + {
> > + /* Analyze the CHREC_RIGHT as the step in an index fragment. */
> > + tree step_if_innermost;
> > + tree step = extract_step (CHREC_RIGHT (expr), factor,
> > + &step_if_innermost);
> > + if (!best_step)
> > + {
> > + /* This is the outermost chrec for the original expression.
> > + It's not worth carrying on if the step isn't versionable,
> > + or if we're pretty sure it's not for the inner dimension. */
> > + if (!step_if_innermost
> > + || TREE_CODE (step) != SSA_NAME
> > + || !expr_invariant_in_loop_p (loop, step))
> > + return;
> > +
> > + best_step = step;
> > +
> > + /* We should version for STEP == 1 if we know that that can be
> > + true under some circumstances. */
> > + if (integer_onep (step_if_innermost))
> > + break;
> > +
> > + /* Bail out if this appears to be the step for the innermost
> > + dimension, but isn't likely to be 1.
> > +
> > + ??? We could instead version for when it equals
> > + STEP_IF_INNERMOST, but it's not likely to have as much
> > + benefit as versioning for 1. */
> > + if (step_if_innermost != step)
> > + return;
> > + }
> > + else
> > + {
> > + /* This is an inner chrec. If it looks like it iterates over
> > + the innermost dimension, abort any attempt to version for
> > + the outermost chrec (which if we reach here wasn't itself
> > + obviously iterating over the innermost dimension). */
> > + if (step_if_innermost && TREE_CONSTANT (step_if_innermost))
> > + return;
> > + }
> > + worklist.quick_push (CHREC_LEFT (expr));
> > + continue;
> > + }
> > +
> > + /* Analyze parts of a queued CHREC_LEFT. This is better than just
> > + analyzing the evolution of the whole expression since the value
> > + could include a mixture of analyzable and unanalyzable elements.
> > + Use NSPLIT to count cases in which we add more expressions to
> > + analyze, as opposed to just simplifying the existing one. */
> > + if (code == PLUS_EXPR || code == POINTER_PLUS_EXPR || code == MINUS_EXPR)
> > + {
> > + worklist.quick_push (TREE_OPERAND (expr, 0));
> > + if (nsplit++ < MAX_NSPLIT)
> > + worklist.quick_push (TREE_OPERAND (expr, 1));
> > + continue;
> > + }
> > + if (code == MULT_EXPR)
> > + {
> > + tree op0 = strip_casts (TREE_OPERAND (expr, 0));
> > + tree op1 = TREE_OPERAND (expr, 1);
> > + if (TREE_CODE (op0) == PLUS_EXPR || TREE_CODE (op0) == MINUS_EXPR)
> > + {
> > + tree type = TREE_TYPE (expr);
> > + tree op00 = fold_convert (type, TREE_OPERAND (op0, 0));
> > + worklist.quick_push (fold_build2 (MULT_EXPR, type, op00, op1));
> > + if (nsplit++ < MAX_NSPLIT)
> > + {
> > + tree op01 = fold_convert (type, TREE_OPERAND (op0, 1));
> > + worklist.quick_push (fold_build2 (MULT_EXPR, type,
> > + op01, op1));
> > + }
> > + continue;
> > + }
> > + }
> > +
> > + /* If EXPR is invariant in LOOP, analyze it wrt the innermost loop
> > + for which it could evolve (i.e. the loop containing the outermost
> > + one for which EXPR is invariant). */
> > + struct loop *wrt_loop = outermost_invariant_loop_for_expr (loop, expr);
> > + if (wrt_loop)
> > + {
> > + wrt_loop = loop_outer (wrt_loop);
> > + if (!wrt_loop)
> > + continue;
> > + }
> > + else
> > + wrt_loop = loop;
> > + tree evolution = analyze_scalar_evolution (wrt_loop, expr);
> > + tree chrec = strip_casts (evolution);
> > + if (TREE_CODE (chrec) == POLYNOMIAL_CHREC)
> > + {
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + {
> > + fprintf (dump_file, ";; evolution of ");
> > + print_generic_expr (dump_file, expr, TDF_SLIM);
> > + fprintf (dump_file, " in loop %d (depth %d): ",
> > + wrt_loop->num, loop_depth (wrt_loop));
> > + print_generic_expr (dump_file, evolution, TDF_SLIM);
> > + fprintf (dump_file, "\n");
> > + }
> > + worklist.quick_push (chrec);
> > + }
> > + else
> > + {
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + {
> > + fprintf (dump_file, ";; cannot analyze ");
> > + print_generic_expr (dump_file, expr, TDF_SLIM);
> > + fprintf (dump_file, " any further\n");
> > + }
> > + }
> > + }
> > + if (best_step)
> > + version_for_unity (loop, best_step);
> > +}
> > +
> > +/* Analyze multiplication MULT to see whether we can identify "gather-like"
> > + versioning opportunities such as:
> > +
> > + for (int i = 0; i < n; ++i)
> > + res += a[index[i] * stride];
> > +
> > + Return true if there was a versioning opportunity.
> > +
> > + LOOP is the loop that contains MULT. Dividing the value of MULT
> > + by FACTOR converts it into an element index. */
> > +
> > +bool
> > +loop_versioning::analyze_product (struct loop *loop, gassign *mult,
> > + poly_uint64 factor)
> > +{
> > + /* Record the original LHS for the dump message below. */
> > + tree lhs = gimple_assign_lhs (mult);
> > +
> > + /* Peel any scaling, which generally happens after conversion to
> > + pointer width. For example, on LP64 systems:
> > +
> > + int *x, i, stride;
> > + ... x[4 * i * stride] ...;
> > +
> > + multiplies i * stride by 4 using ints, then widens the result
> > + to pointer width before multiplying by sizeof (int). */
> > + poly_uint64 scale;
> > + if (poly_int_tree_p (gimple_assign_rhs2 (mult), &scale)
> > + && known_eq (scale, factor))
> > + {
> > + mult = maybe_get_assign_strip_casts (loop, gimple_assign_rhs1 (mult));
> > + if (!mult || gimple_assign_rhs_code (mult) != MULT_EXPR)
> > + return false;
> > + factor = 1;
> > + }
> > +
> > + /* Peel any grouping multiple; see acceptable_scale_p for details. */
> > + if (acceptable_scale_p (gimple_assign_rhs2 (mult), factor))
> > + {
> > + mult = maybe_get_assign_strip_casts (loop, gimple_assign_rhs1 (mult));
> > + if (!mult || gimple_assign_rhs_code (mult) != MULT_EXPR)
> > + return false;
> > + }
> > + else if (maybe_ne (factor, 1U))
> > + /* We expect to see a scaling multiplication when FACTOR is > 1. */
> > + return false;
> > +
> > + /* See whether this multiplication involves a loop-invariant SSA name
> > + and a non-invariant SSA name. */
> > + tree op1 = gimple_assign_rhs1 (mult);
> > + tree op2 = gimple_assign_rhs2 (mult);
> > + if (TREE_CODE (op1) != SSA_NAME || TREE_CODE (op2) != SSA_NAME)
> > + return false;
> > +
> > + bool invariant1_p = expr_invariant_in_loop_p (loop, op1);
> > + bool invariant2_p = expr_invariant_in_loop_p (loop, op2);
> > + if (invariant1_p == invariant2_p)
> > + return false;
> > +
> > + /* Make sure that the invariant is OP1 and the other operand is OP2. */
> > + if (invariant2_p)
> > + std::swap (op1, op2);
> > +
> > + /* Bail out if OP2 can be analyzed as an evolution; we want to use
> > + analyze_evolution in that case instead. There's no point trying
> > + hard to avoid repeating the call to analyze_scalar_evolution since
> > + that function does its own caching. */
> > + if (tree_contains_chrecs (analyze_scalar_evolution (loop, op2), NULL))
> > + return false;
> > +
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + {
> > + fprintf (dump_file, ";; Address fragment ");
> > + print_generic_expr (dump_file, lhs, TDF_SLIM);
> > + fprintf (dump_file, " multiplies invariant ");
> > + print_generic_expr (dump_file, op1, TDF_SLIM);
> > + fprintf (dump_file, " by ");
> > + print_generic_expr (dump_file, op2, TDF_SLIM);
> > + fprintf (dump_file, ", which isn't a scalar evolution\n");
> > + }
> > +
> > + version_for_unity (loop, op1);
> > + return true;
> > +}
> > +
> > +/* Treat EXPR as a sum of products and apply analyze_product to each of the
> > + products. Return true if one of the products provides a versioning
> > + opportunity. FACTOR is as for analyze_product. */
> > +
> > +bool
> > +loop_versioning::analyze_sum_of_products (struct loop *loop, tree expr,
> > + poly_uint64 factor)
> > +{
> > + const unsigned int MAX_NITERS = 8;
> > +
> > + tree worklist[MAX_NITERS];
> > + unsigned int length = 0;
> > + worklist[length++] = expr;
> > + for (unsigned int i = 0; i < length; ++i)
> > + {
> > + expr = worklist[i];
> > + gassign *assign = maybe_get_assign_strip_casts (loop, expr);
> > + if (!assign)
> > + continue;
> > +
> > + tree_code code = gimple_assign_rhs_code (assign);
> > + if (code == PLUS_EXPR || code == POINTER_PLUS_EXPR || code == MINUS_EXPR)
> > + {
> > + if (length < MAX_NITERS)
> > + worklist[length++] = gimple_assign_rhs1 (assign);
> > + if (length < MAX_NITERS)
> > + worklist[length++] = gimple_assign_rhs2 (assign);
> > + }
> > + else if (code == MULT_EXPR && analyze_product (loop, assign, factor))
> > + return true;
> > + }
> > + return false;
> > +}
> > +
> > +/* Analyze pointer expression EXPR, which occurs in loop LOOP and which
> > + is used to address a value of type TYPE. */
> > +
> > +void
> > +loop_versioning::analyze_pointer (struct loop *loop, tree expr, tree type)
> > +{
> > + poly_uint64 factor;
> > + if (poly_int_tree_p (TYPE_SIZE_UNIT (type), &factor))
> > + {
> > + if (!analyze_sum_of_products (loop, expr, factor))
> > + analyze_evolution (loop, expr, factor);
> > + }
> > +}
> > +
> > +/* Analyze expression EXPR, which occurs in loop LOOP. */
> > +
> > +void
> > +loop_versioning::analyze_expr (struct loop *loop, tree expr)
> > +{
> > + while (handled_component_p (expr))
> > + {
> > + /* See whether we can use versioning to avoid a multiplication
> > + in the array index. */
> > + if (TREE_CODE (expr) == ARRAY_REF)
> > + {
> > + if (!analyze_sum_of_products (loop, TREE_OPERAND (expr, 1), 1))
> > + analyze_evolution (loop, TREE_OPERAND (expr, 1), 1);
> > + }
> > + expr = TREE_OPERAND (expr, 0);
> > + }
> > +
> > + if (TREE_CODE (expr) == MEM_REF)
> > + {
> > + tree addr = TREE_OPERAND (expr, 0);
> > + /* See whether we can use versioning to avoid a multiplication
> > + in the pointer calculation. This is generally only worth
> > + doing if the multiplication occurs in this loop rather than
> > + an outer loop. */
> > + if (!expr_invariant_in_loop_p (loop, addr))
> > + analyze_pointer (loop, addr, TREE_TYPE (expr));
> > + }
> > +
> > + /* These would be easy to handle if they existed at this stage. */
> > + gcc_checking_assert (TREE_CODE (expr) != TARGET_MEM_REF);
> > +}
> > +
> > +/* Analyze STMT looking for useful version checks. */
> > +
> > +void
> > +loop_versioning::analyze_stmt (gimple *stmt)
> > +{
> > + struct loop *loop = gimple_bb (stmt)->loop_father;
> > +
> > + unsigned int nops = gimple_num_ops (stmt);
> > + for (unsigned int i = 0; i < nops; ++i)
> > + if (tree op = gimple_op (stmt, i))
> > + analyze_expr (loop, op);
> > +}
> > +
> > +/* Analyze all the statements in BB looking for useful version checks. */
> > +
> > +void
> > +loop_versioning::analyze_block (basic_block bb)
> > +{
> > + struct loop *loop = bb->loop_father;
> > + loop_info &li = get_loop_info (loop);
> > + if (li.rejected_p)
> > + return;
> > +
> > + for (gimple_stmt_iterator gsi = gsi_start_bb (bb); !gsi_end_p (gsi);
> > + gsi_next (&gsi))
> > + {
> > + gimple *stmt = gsi_stmt (gsi);
> > + if (expensive_stmt_p (stmt))
> > + {
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + {
> > + struct loop *loop = gimple_bb (stmt)->loop_father;
> > + fprintf (dump_file, ";; Loop %d (depth %d) contains expensive"
> > + " stmt: ", loop->num, loop_depth (loop));
> > + print_gimple_stmt (dump_file, stmt, 0, TDF_SLIM);
> > + }
> > + li.rejected_p = true;
> > + break;
> > + }
> > +
> > + /* Only look for direct versioning opportunities in inner loops
> > + since the benefit tends to be much smaller for outer loops. */
> > + if (!loop->inner)
> > + analyze_stmt (stmt);
> > +
> > + /* The point of the instruction limit is to prevent excessive
> > + code growth, so this is a size-based estimate even though
> > + the optimization is aimed at speed. */
> > + li.num_insns += estimate_num_insns (stmt, &eni_size_weights);
> > + }
> > +}
> > +
> > +/* Analyze all the blocks in the function looking for useful version checks.
> > + Return true if we found one. */
> > +
> > +bool
> > +loop_versioning::analyze_blocks ()
> > +{
> > + /* For now we don't try to version the whole function, although
> > + versioning at that level could be useful in some cases. */
> > + get_loop_info (get_loop (m_fn, 0)).rejected_p = true;
> > +
> > + basic_block bb;
> > + FOR_EACH_BB_FN (bb, m_fn)
> > + if (loop_outer (bb->loop_father))
> > + analyze_block (bb);
> > +
> > + return m_num_conditions != 0;
> > +}
> > +
> > +/* Use the ranges in VRS to remove impossible versioning conditions from
> > + LOOP. */
> > +
> > +void
> > +loop_versioning::prune_loop_conditions (struct loop *loop, vr_values *vrs)
> > +{
> > + loop_info &li = get_loop_info (loop);
> > +
> > + unsigned int i = li.unity_names.length ();
> > + while (i > 0)
> > + {
> > + i -= 1;
> > + tree name = li.unity_names[i];
> > + value_range *vr = vrs->get_value_range (name);
> > + if (vr && !range_includes_p (vr, 1))
> > + {
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + {
> > + fprintf (dump_file, ";; ");
> > + print_generic_expr (dump_file, name, TDF_SLIM);
> > + fprintf (dump_file, " can never be 1 in loop %d\n", loop->num);
> > + }
> > +
> > + li.unity_names.unordered_remove (i);
> > + bitmap_clear_bit (&li.unity_name_ids, SSA_NAME_VERSION (name));
> > + m_num_conditions -= 1;
> > + }
> > + }
> > +}
> > +
> > +/* Remove any scheduled loop version conditions that will never be true.
> > + Return true if any remain. */
> > +
> > +bool
> > +loop_versioning::prune_conditions ()
> > +{
> > + calculate_dominance_info (CDI_DOMINATORS);
> > + lv_dom_walker dom_walker (*this);
> > + dom_walker.walk (ENTRY_BLOCK_PTR_FOR_FN (m_fn));
> > + return m_num_conditions != 0;
> > +}
> > +
> > +/* Merge the version checks for INNER into immediately-enclosing loop
> > + OUTER. */
> > +
> > +void
> > +loop_versioning::merge_loop_info (struct loop *outer, struct loop *inner)
> > +{
> > + loop_info &inner_li = get_loop_info (inner);
> > + loop_info &outer_li = get_loop_info (outer);
> > +
> > + tree name;
> > + unsigned int i;
> > + FOR_EACH_VEC_ELT (inner_li.unity_names, i, name)
> > + if (bitmap_set_bit (&outer_li.unity_name_ids, SSA_NAME_VERSION (name)))
> > + {
> > + outer_li.unity_names.safe_push (name);
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + {
> > + fprintf (dump_file, ";; Hoisting check that ");
> > + print_generic_expr (dump_file, name, TDF_SLIM);
> > + fprintf (dump_file, " == 1 from loop %d (depth %d) to loop %d\n",
> > + inner->num, loop_depth (inner), outer->num);
> > + }
> > + }
> > +
> > + if (loop_depth (outer_li.outermost) < loop_depth (inner_li.outermost))
> > + outer_li.outermost = inner_li.outermost;
> > +}
> > +
> > +/* Add LOOP to the queue of loops to version. */
> > +
> > +void
> > +loop_versioning::add_loop_to_queue (struct loop *loop)
> > +{
> > + loop_info &li = get_loop_info (loop);
> > +
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + fprintf (dump_file, ";; Queuing loop %d (depth %d) for versioning\n",
> > + loop->num, loop_depth (loop));
> > + m_loops_to_version.safe_push (loop);
> > +
> > + /* Don't try to version superloops. */
> > + li.rejected_p = true;
> > +}
> > +
> > +/* Decide whether the cost model would allow us to version LOOP,
> > + either directly or as part of a parent loop, and return true if so.
> > + This does not imply that the loop is actually worth versioning in its
> > + own right, just that it would be valid to version it if something
> > + benefited.
> > +
> > + We have already made this decision for all inner loops of LOOP. */
> > +
> > +bool
> > +loop_versioning::decide_whether_loop_is_versionable (struct loop *loop)
> > +{
> > + loop_info &li = get_loop_info (loop);
> > +
> > + if (li.rejected_p)
> > + return false;
> > +
> > + /* Examine the decisions made for inner loops. */
> > + for (struct loop *inner = loop->inner; inner; inner = inner->next)
> > + {
> > + loop_info &inner_li = get_loop_info (inner);
> > + if (inner_li.rejected_p)
> > + {
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + fprintf (dump_file, ";; Not versioning loop %d (depth %d)"
> > + " because inner loop %d should not be versioned\n",
> > + loop->num, loop_depth (loop), inner->num);
> > + return false;
> > + }
> > +
> > + if (inner_li.worth_versioning_p ())
> > + li.subloops_benefit_p = true;
> > +
> > + /* Accumulate the number of instructions from subloops that are not
> > + the innermost, or that don't benefit from versioning. Only the
> > + instructions from innermost loops that benefit from versioning
> > + should be weighed against loop-versioning-max-inner-insns;
> > + everything else should be weighed against
> > + loop-versioning-max-outer-insns. */
> > + if (!inner_li.worth_versioning_p () || inner->inner)
> > + {
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + fprintf (dump_file, ";; Counting %d instructions from"
> > + " loop %d (depth %d) against parent loop %d\n",
> > + inner_li.num_insns, inner->num, loop_depth (inner),
> > + loop->num);
> > + li.num_insns += inner_li.num_insns;
> > + }
> > + }
> > +
> > + /* Enforce the size limits. */
> > + if (li.worth_versioning_p ())
> > + {
> > + unsigned int max_num_insns = max_insns_for_loop (loop);
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + fprintf (dump_file, ";; Loop %d (depth %d) has %d instructions,"
> > + " against a limit of %d\n", loop->num, loop_depth (loop),
> > + li.num_insns, max_num_insns);
> > + if (li.num_insns > max_num_insns)
> > + {
> > + if (dump_file && (dump_flags & TDF_DETAILS))
> > + fprintf (dump_file, ";; Not versioning loop %d (depth %d)"
> > + " because it is too big\n", loop->num, loop_depth (loop));
> > + return false;
> > + }
> > + }
> > +
> > + /* Hoist all version checks for subloops to this loop. */
> > + for (struct loop *subloop = loop->inner; subloop; subloop = subloop->next)
> > + merge_loop_info (loop, subloop);
> > +
> > + return true;
> > +}
> > +
> > +/* Decide which loops to version and add them to the versioning queue.
> > + Return true if there are any loops to version. */
> > +
> > +bool
> > +loop_versioning::make_versioning_decisions ()
> > +{
> > + struct loop *loop;
> > + FOR_EACH_LOOP (loop, LI_FROM_INNERMOST)
> > + {
> > + loop_info &linfo = get_loop_info (loop);
> > + if (decide_whether_loop_is_versionable (loop))
> > + {
> > + /* Commit to versioning LOOP directly if we can't hoist the
> > + version checks any further. */
> > + if (linfo.worth_versioning_p ()
> > + && (loop_depth (loop) == 1 || linfo.outermost == loop))
> > + add_loop_to_queue (loop);
> > + }
> > + else
> > + {
> > + /* We can't version this loop, so individually version any
> > + subloops that would benefit and haven't been versioned yet. */
> > + linfo.rejected_p = true;
> > + for (struct loop *subloop = loop->inner; subloop;
> > + subloop = subloop->next)
> > + if (get_loop_info (subloop).worth_versioning_p ())
> > + add_loop_to_queue (subloop);
> > + }
> > + }
> > +
> > + return !m_loops_to_version.is_empty ();
> > +}
> > +
> > +/* Attempt to implement loop versioning for LOOP, using the information
> > + cached in the associated loop_info. Return true on success. */
> > +
> > +bool
> > +loop_versioning::version_loop (struct loop *loop)
> > +{
> > + loop_info &li = get_loop_info (loop);
> > +
> > + /* Not protected by TDF_DETAILS since this is the main piece of
> > + information. */
> > + if (dump_file)
> > + fprintf (dump_file, ";; Versioning loop %d (depth %d)\n",
> > + loop->num, loop_depth (loop));
> > +
> > + /* Build up a condition that selects the original loop instead of
> > + the simplified loop. */
> > + tree cond = boolean_false_node;
> > + tree name;
> > + unsigned int i;
> > + FOR_EACH_VEC_ELT (li.unity_names, i, name)
> > + {
> > + tree ne_one = fold_build2 (NE_EXPR, boolean_type_node, name,
> > + build_one_cst (TREE_TYPE (name)));
> > +
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Add a loop versioning pass
2018-10-24 13:41 Richard Sandiford
2018-10-25 14:16 ` Richard Biener
@ 2018-10-25 16:16 ` David Malcolm
2018-11-28 14:16 ` Richard Sandiford
1 sibling, 1 reply; 17+ messages in thread
From: David Malcolm @ 2018-10-25 16:16 UTC (permalink / raw)
To: Richard Sandiford, gcc-patches
On Wed, 2018-10-24 at 14:05 +0100, Richard Sandiford wrote:
> This patch adds a pass that versions loops with variable index
> strides
> for the case in which the stride is 1. E.g.:
>
> for (int i = 0; i < n; ++i)
> x[i * stride] = ...;
>
> becomes:
>
> if (stepx == 1)
> for (int i = 0; i < n; ++i)
> x[i] = ...;
> else
> for (int i = 0; i < n; ++i)
> x[i * stride] = ...;
>
> This is useful for both vector code and scalar code, and in some
> cases
> can enable further optimisations like loop interchange or pattern
> recognition.
>
> The pass gives a 7.6% improvement on Cortex-A72 for 554.roms_r at -O3
> and a 2.4% improvement for 465.tonto. I haven't found any SPEC tests
> that regress.
>
> Sizewise, there's a 10% increase in .text for both 554.roms_r and
> 465.tonto. That's obviously a lot, but in tonto's case it's because
> the whole program is written using assumed-shape arrays and pointers,
> so a large number of functions really do benefit from versioning.
> roms likewise makes heavy use of assumed-shape arrays, and that
> improvement in performance IMO justifies the code growth.
>
> The next biggest .text increase is 4.5% for 548.exchange2_r. I did
> see
> a small (0.4%) speed improvement there, but although both 3-iteration
> runs
> produced stable results, that might still be noise. There was a
> slightly
> larger (non-noise) improvement for a 256-bit SVE model.
>
> 481.wrf and 521.wrf_r .text grew by 2.8% and 2.5% respectively, but
> without any noticeable improvement in performance. No other test
> grew
> by more than 2%.
>
> Although the main SPEC beneficiaries are all Fortran tests, the
> benchmarks we use for SVE also include some C and C++ tests that
> benefit.
>
> Using -frepack-arrays gives the same benefits in many Fortran cases.
> The problem is that using that option inappropriately can force a
> full
> array copy for arguments that the function only reads once, and so it
> isn't really something we can turn on by default. The new pass is
> supposed to give most of the benefits of -frepack-arrays without
> the risk of unnecessary repacking.
>
> The patch therefore enables the pass by default at -O3.
>
> Tested on aarch64-linux-gnu and x86_64-linux-gnu. OK to install?
>
> Richard
>
[...snip...]
> +/* Run the pass and return a set of TODO_* flags. */
> +
> +unsigned int
> +loop_versioning::run ()
> +{
> + gcc_assert (scev_initialized_p ());
> +
> + if (!analyze_blocks ()
> + || !prune_conditions ()
> + || !make_versioning_decisions ()
> + || !implement_versioning_decisions ())
> + return 0;
> +
> + return TODO_update_ssa;
> +}
> +
> +/* Loop versioningting pass. */
(typo)
> +
> +namespace {
Could the whole file be within this anonymous namespace, rather than
just the opt_pass subclass? (hiding class loop_versioning, so that the
optimizer knows that the only thing visible outside the TU is
make_pass_loop_versioning). This can be a pain to debug, but you can
always comment out the anon namespace locally when debugging.
> +
> +const pass_data pass_data_loop_versioning =
> +{
> + GIMPLE_PASS, /* type */
> + "lversion", /* name */
> + OPTGROUP_LOOP, /* optinfo_flags */
> + TV_LOOP_VERSIONING, /* tv_id */
> + PROP_cfg, /* properties_required */
> + 0, /* properties_provided */
> + 0, /* properties_destroyed */
> + 0, /* todo_flags_start */
> + 0, /* todo_flags_finish */
> +};
[...snip...]
> +
> +} // anon namespace
> +
> +gimple_opt_pass *
> +make_pass_loop_versioning (gcc::context *ctxt)
> +{
> + return new pass_loop_versioning (ctxt);
> +}
[...snip...]
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Add a loop versioning pass
2018-10-25 14:16 ` Richard Biener
@ 2018-10-25 16:03 ` Richard Sandiford
2018-10-26 14:49 ` Richard Biener
1 sibling, 0 replies; 17+ messages in thread
From: Richard Sandiford @ 2018-10-25 16:03 UTC (permalink / raw)
To: Richard Biener; +Cc: GCC Patches
Richard Biener <richard.guenther@gmail.com> writes:
> On Wed, Oct 24, 2018 at 3:05 PM Richard Sandiford
> <richard.sandiford@arm.com> wrote:
>>
>> This patch adds a pass that versions loops with variable index strides
>> for the case in which the stride is 1. E.g.:
>>
>> for (int i = 0; i < n; ++i)
>> x[i * stride] = ...;
>>
>> becomes:
>>
>> if (stepx == 1)
>> for (int i = 0; i < n; ++i)
>> x[i] = ...;
>> else
>> for (int i = 0; i < n; ++i)
>> x[i * stride] = ...;
>>
>> This is useful for both vector code and scalar code, and in some cases
>> can enable further optimisations like loop interchange or pattern
>> recognition.
>>
>> The pass gives a 7.6% improvement on Cortex-A72 for 554.roms_r at -O3
>> and a 2.4% improvement for 465.tonto. I haven't found any SPEC tests
>> that regress.
>>
>> Sizewise, there's a 10% increase in .text for both 554.roms_r and
>> 465.tonto. That's obviously a lot, but in tonto's case it's because
>> the whole program is written using assumed-shape arrays and pointers,
>> so a large number of functions really do benefit from versioning.
>> roms likewise makes heavy use of assumed-shape arrays, and that
>> improvement in performance IMO justifies the code growth.
>>
>> The next biggest .text increase is 4.5% for 548.exchange2_r. I did see
>> a small (0.4%) speed improvement there, but although both 3-iteration runs
>> produced stable results, that might still be noise. There was a slightly
>> larger (non-noise) improvement for a 256-bit SVE model.
>>
>> 481.wrf and 521.wrf_r .text grew by 2.8% and 2.5% respectively, but
>> without any noticeable improvement in performance. No other test grew
>> by more than 2%.
>>
>> Although the main SPEC beneficiaries are all Fortran tests, the
>> benchmarks we use for SVE also include some C and C++ tests that
>> benefit.
>>
>> Using -frepack-arrays gives the same benefits in many Fortran cases.
>> The problem is that using that option inappropriately can force a full
>> array copy for arguments that the function only reads once, and so it
>> isn't really something we can turn on by default. The new pass is
>> supposed to give most of the benefits of -frepack-arrays without
>> the risk of unnecessary repacking.
>>
>> The patch therefore enables the pass by default at -O3.
>>
>> Tested on aarch64-linux-gnu and x86_64-linux-gnu. OK to install?
>
> I will give this a thorough review tomorror (sorry for the delay), but
> one question that comes up is how we avoid excessive versioning.
> Consider this pass versioning for stride1, then loop distribution
> versioning for aliasing and then vectorization versioning for alignment.
Yeah, this could happen. :-)
> Do we have any idea on how to track these versionings so we can
> eventually commonize paths that didn't get any followup optimization
> applied?
The idea was that this pass would help even for scalar code (due to
cheaper addressing), so it's not something we should necessarily undo
if there is no follow-on optimisation.
> Also consider the case where loop distribution applies transforms
> with versioning for aliasing to _both_ loop nests from your versioning.
I think that's a good thing though. The aliasing checks are going to be
cheaper for a stride of 1 compared to a variable stride.
> Maybe we can at least add some loop->number_of_times_versioned
> counter and some dumping/statistics to get an idea how big an
> issue this might be?
That might help. If we could be more confident that the "faster"
loop versions are very likely to be used, we could throttle the
optimisation of the unlikely versions.
But so far I haven't seen an example where this really hurts.
Thanks,
Richard
>
> Thanks,
> Richard.
>
>> Richard
>>
>>
>> 2018-10-24 Richard Sandiford <richard.sandiford@arm.com>
>>
>> gcc/
>> * doc/invoke.texi (-fversion-loops-for-strides): Document
>> (loop-versioning-group-size, loop-versioning-max-inner-insns)
>> (loop-versioning-max-outer-insns): Document new --params.
>> * Makefile.in (OBJS): Add gimple-loop-versioning.o.
>> * common.opt (fversion-loops-for-strides): New option.
>> * opts.c (default_options_table): Enable fversion-loops-for-strides
>> at -O3.
>> * params.def (PARAM_LOOP_VERSIONING_GROUP_SIZE)
>> (PARAM_LOOP_VERSIONING_MAX_INNER_INSNS)
>> (PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS): New parameters.
>> * passes.def: Add pass_loop_versioning.
>> * timevar.def (TV_LOOP_VERSIONING): New time variable.
>> * tree-ssa-propagate.h
>> (substitute_and_fold_engine::substitute_and_fold): Add an optional
>> block parameter.
>> * tree-ssa-propagate.c
>> (substitute_and_fold_engine::substitute_and_fold): Likewise.
>> When passed, only walk blocks dominated by that block.
>> * tree-vrp.h (range_includes_p): Declare.
>> (range_includes_zero_p): Turn into an inline wrapper around
>> range_includes_p.
>> * tree-vrp.c (range_includes_p): New function, generalizing...
>> (range_includes_zero_p): ...this.
>> * tree-pass.h (make_pass_loop_versioning): Declare.
>> * gimple-loop-versioning.cc: New file.
>>
>> gcc/testsuite/
>> * gcc.dg/loop-versioning-1.c: New test.
>> * gcc.dg/loop-versioning-10.c: Likewise.
>> * gcc.dg/loop-versioning-11.c: Likewise.
>> * gcc.dg/loop-versioning-2.c: Likewise.
>> * gcc.dg/loop-versioning-3.c: Likewise.
>> * gcc.dg/loop-versioning-4.c: Likewise.
>> * gcc.dg/loop-versioning-5.c: Likewise.
>> * gcc.dg/loop-versioning-6.c: Likewise.
>> * gcc.dg/loop-versioning-7.c: Likewise.
>> * gcc.dg/loop-versioning-8.c: Likewise.
>> * gcc.dg/loop-versioning-9.c: Likewise.
>> * gfortran.dg/loop_versioning_1.f90: Likewise.
>> * gfortran.dg/loop_versioning_2.f90: Likewise.
>> * gfortran.dg/loop_versioning_3.f90: Likewise.
>> * gfortran.dg/loop_versioning_4.f90: Likewise.
>> * gfortran.dg/loop_versioning_5.f90: Likewise.
>> * gfortran.dg/loop_versioning_6.f90: Likewise.
>> * gfortran.dg/loop_versioning_7.f90: Likewise.
>> * gfortran.dg/loop_versioning_8.f90: Likewise.
>>
>> Index: gcc/doc/invoke.texi
>> ===================================================================
>> --- gcc/doc/invoke.texi 2018-10-24 14:02:14.000000000 +0100
>> +++ gcc/doc/invoke.texi 2018-10-24 14:02:15.184152693 +0100
>> @@ -7934,7 +7934,8 @@ by @option{-O2} and also turns on the fo
>> -fvect-cost-model @gol
>> -ftree-partial-pre @gol
>> -fpeel-loops @gol
>> --fipa-cp-clone}
>> +-fipa-cp-clone @gol
>> +-fversion-loops-for-strides}
>>
>> @item -O0
>> @opindex O0
>> @@ -10358,6 +10359,29 @@ for one side of the iteration space and
>> Move branches with loop invariant conditions out of the loop, with duplicates
>> of the loop on both branches (modified according to result of the condition).
>>
>> +@item -fversion-loops-for-strides
>> +@opindex fversion-loops-for-strides
>> +If a loop iterates over an array with a variable stride, create another
>> +version of the loop that assumes the stride is always 1. For example:
>> +
>> +@smallexample
>> +for (int i = 0; i < n; ++i)
>> + x[i * stride] = @dots{};
>> +@end smallexample
>> +
>> +becomes:
>> +
>> +@smallexample
>> +if (stride == 1)
>> + for (int i = 0; i < n; ++i)
>> + x[i] = @dots{};
>> +else
>> + for (int i = 0; i < n; ++i)
>> + x[i * stride] = @dots{};
>> +@end smallexample
>> +
>> +This is particularly useful for assumed-shape arrays in Fortran.
>> +
>> @item -ffunction-sections
>> @itemx -fdata-sections
>> @opindex ffunction-sections
>> @@ -11567,6 +11591,20 @@ Hardware autoprefetcher scheduler model
>> Number of lookahead cycles the model looks into; at '
>> ' only enable instruction sorting heuristic.
>>
>> +@item loop-versioning-group-size
>> +Make the loop versioning pass optimize @samp{a[i * index * @var{N}]}
>> +in the same way as it would optimize @samp{a[i * index]} when @var{N}
>> +is less than or equal to this value.
>> +
>> +@item loop-versioning-max-inner-insns
>> +The maximum number of instructions that an inner loop can have
>> +before the loop versioning pass considers it too big to copy.
>> +
>> +@item loop-versioning-max-outer-insns
>> +The maximum number of instructions that an outer loop can have
>> +before the loop versioning pass considers it too big to copy,
>> +discounting any instructions in inner loops that directly benefit
>> +from versioning.
>>
>> @end table
>> @end table
>> Index: gcc/Makefile.in
>> ===================================================================
>> --- gcc/Makefile.in 2018-10-24 14:02:14.000000000 +0100
>> +++ gcc/Makefile.in 2018-10-24 14:02:15.180152727 +0100
>> @@ -1312,6 +1312,7 @@ OBJS = \
>> gimple-laddress.o \
>> gimple-loop-interchange.o \
>> gimple-loop-jam.o \
>> + gimple-loop-versioning.o \
>> gimple-low.o \
>> gimple-pretty-print.o \
>> gimple-ssa-backprop.o \
>> Index: gcc/common.opt
>> ===================================================================
>> --- gcc/common.opt 2018-10-24 14:02:14.000000000 +0100
>> +++ gcc/common.opt 2018-10-24 14:02:15.180152727 +0100
>> @@ -2712,6 +2712,10 @@ fsplit-loops
>> Common Report Var(flag_split_loops) Optimization
>> Perform loop splitting.
>>
>> +fversion-loops-for-strides
>> +Common Report Var(flag_version_loops_for_strides) Optimization
>> +Version loops based on whether indices have a stride of 1.
>> +
>> funwind-tables
>> Common Report Var(flag_unwind_tables) Optimization
>> Just generate unwind tables for exception handling.
>> Index: gcc/opts.c
>> ===================================================================
>> --- gcc/opts.c 2018-10-24 14:02:14.000000000 +0100
>> +++ gcc/opts.c 2018-10-24 14:02:15.184152693 +0100
>> @@ -544,6 +544,7 @@ static const struct default_options defa
>> { OPT_LEVELS_3_PLUS, OPT_fipa_cp_clone, NULL, 1 },
>> { OPT_LEVELS_3_PLUS, OPT_ftree_partial_pre, NULL, 1 },
>> { OPT_LEVELS_3_PLUS, OPT_fpeel_loops, NULL, 1 },
>> + { OPT_LEVELS_3_PLUS, OPT_fversion_loops_for_strides, NULL, 1 },
>>
>> /* -Ofast adds optimizations to -O3. */
>> { OPT_LEVELS_FAST, OPT_ffast_math, NULL, 1 },
>> Index: gcc/params.def
>> ===================================================================
>> --- gcc/params.def 2018-10-24 14:02:14.000000000 +0100
>> +++ gcc/params.def 2018-10-24 14:02:15.184152693 +0100
>> @@ -1360,6 +1360,25 @@ DEFPARAM(PARAM_AVOID_FMA_MAX_BITS,
>> "Maximum number of bits for which we avoid creating FMAs.",
>> 0, 0, 512)
>>
>> +DEFPARAM(PARAM_LOOP_VERSIONING_GROUP_SIZE,
>> + "loop-versioning-group-size",
>> + "The maximum constant N for which accesses of the form x[N * step]"
>> + " are worth versioning for the case in which step is 1",
>> + 4, 1, 0)
>> +
>> +DEFPARAM(PARAM_LOOP_VERSIONING_MAX_INNER_INSNS,
>> + "loop-versioning-max-inner-insns",
>> + "The maximum number of instructions in an inner loop that is being"
>> + " considered for versioning",
>> + 200, 0, 0)
>> +
>> +DEFPARAM(PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS,
>> + "loop-versioning-max-outer-insns",
>> + "The maximum number of instructions in an outer loop that is being"
>> + " considered for versioning, on top of the instructions in inner"
>> + " loops",
>> + 100, 0, 0)
>> +
>> /*
>>
>> Local variables:
>> Index: gcc/passes.def
>> ===================================================================
>> --- gcc/passes.def 2018-10-24 14:02:14.000000000 +0100
>> +++ gcc/passes.def 2018-10-24 14:02:15.184152693 +0100
>> @@ -260,6 +260,7 @@ along with GCC; see the file COPYING3.
>> NEXT_PASS (pass_tree_loop);
>> PUSH_INSERT_PASSES_WITHIN (pass_tree_loop)
>> NEXT_PASS (pass_tree_loop_init);
>> + NEXT_PASS (pass_loop_versioning);
>> NEXT_PASS (pass_tree_unswitch);
>> NEXT_PASS (pass_scev_cprop);
>> NEXT_PASS (pass_loop_split);
>> Index: gcc/timevar.def
>> ===================================================================
>> --- gcc/timevar.def 2018-10-24 14:02:14.000000000 +0100
>> +++ gcc/timevar.def 2018-10-24 14:02:15.188152659 +0100
>> @@ -234,6 +234,7 @@ DEFTIMEVAR (TV_DSE1 , "
>> DEFTIMEVAR (TV_DSE2 , "dead store elim2")
>> DEFTIMEVAR (TV_LOOP , "loop analysis")
>> DEFTIMEVAR (TV_LOOP_INIT , "loop init")
>> +DEFTIMEVAR (TV_LOOP_VERSIONING , "loop versioning")
>> DEFTIMEVAR (TV_LOOP_MOVE_INVARIANTS , "loop invariant motion")
>> DEFTIMEVAR (TV_LOOP_UNROLL , "loop unrolling")
>> DEFTIMEVAR (TV_LOOP_DOLOOP , "loop doloop")
>> Index: gcc/tree-ssa-propagate.h
>> ===================================================================
>> --- gcc/tree-ssa-propagate.h 2018-10-24 14:02:14.000000000 +0100
>> +++ gcc/tree-ssa-propagate.h 2018-10-24 14:02:15.188152659 +0100
>> @@ -104,7 +104,7 @@ extern void propagate_tree_value_into_st
>> virtual bool fold_stmt (gimple_stmt_iterator *) { return false; }
>> virtual tree get_value (tree) { return NULL_TREE; }
>>
>> - bool substitute_and_fold (void);
>> + bool substitute_and_fold (basic_block = NULL);
>> bool replace_uses_in (gimple *);
>> bool replace_phi_args_in (gphi *);
>> };
>> Index: gcc/tree-ssa-propagate.c
>> ===================================================================
>> --- gcc/tree-ssa-propagate.c 2018-10-24 14:02:14.000000000 +0100
>> +++ gcc/tree-ssa-propagate.c 2018-10-24 14:02:15.188152659 +0100
>> @@ -1152,6 +1152,10 @@ substitute_and_fold_dom_walker::before_d
>>
>>
>> /* Perform final substitution and folding of propagated values.
>> + Process the whole function if BLOCK is null, otherwise only
>> + process the blocks that BLOCK dominates. In the latter case,
>> + it is the caller's responsibility to ensure that dominator
>> + information is available and up-to-date.
>>
>> PROP_VALUE[I] contains the single value that should be substituted
>> at every use of SSA name N_I. If PROP_VALUE is NULL, no values are
>> @@ -1168,16 +1172,24 @@ substitute_and_fold_dom_walker::before_d
>> Return TRUE when something changed. */
>>
>> bool
>> -substitute_and_fold_engine::substitute_and_fold (void)
>> +substitute_and_fold_engine::substitute_and_fold (basic_block block)
>> {
>> if (dump_file && (dump_flags & TDF_DETAILS))
>> fprintf (dump_file, "\nSubstituting values and folding statements\n\n");
>>
>> memset (&prop_stats, 0, sizeof (prop_stats));
>>
>> - calculate_dominance_info (CDI_DOMINATORS);
>> + /* Don't call calculate_dominance_info when iterating over a subgraph.
>> + Callers that are using the interface this way are likely to want to
>> + iterate over several disjoint subgraphs, and it would be expensive
>> + in enable-checking builds to revalidate the whole dominance tree
>> + each time. */
>> + if (block)
>> + gcc_assert (dom_info_state (CDI_DOMINATORS));
>> + else
>> + calculate_dominance_info (CDI_DOMINATORS);
>> substitute_and_fold_dom_walker walker (CDI_DOMINATORS, this);
>> - walker.walk (ENTRY_BLOCK_PTR_FOR_FN (cfun));
>> + walker.walk (block ? block : ENTRY_BLOCK_PTR_FOR_FN (cfun));
>>
>> /* We cannot remove stmts during the BB walk, especially not release
>> SSA names there as that destroys the lattice of our callers.
>> Index: gcc/tree-vrp.h
>> ===================================================================
>> --- gcc/tree-vrp.h 2018-10-24 14:02:14.000000000 +0100
>> +++ gcc/tree-vrp.h 2018-10-24 14:02:15.188152659 +0100
>> @@ -86,7 +86,7 @@ extern void register_edge_assert_for (tr
>> tree, tree, vec<assert_info> &);
>> extern bool stmt_interesting_for_vrp (gimple *);
>> extern void set_value_range_to_varying (value_range *);
>> -extern bool range_includes_zero_p (const value_range *);
>> +extern bool range_includes_p (const value_range *, HOST_WIDE_INT);
>> extern bool infer_value_range (gimple *, tree, tree_code *, tree *);
>>
>> extern void set_value_range_to_nonnull (value_range *, tree);
>> @@ -122,4 +122,13 @@ extern int value_inside_range (tree, tre
>> extern tree get_single_symbol (tree, bool *, tree *);
>> extern void maybe_set_nonzero_bits (edge, tree);
>> extern value_range_type determine_value_range (tree, wide_int *, wide_int *);
>> +
>> +/* Return TRUE if *VR includes the value zero. */
>> +
>> +inline bool
>> +range_includes_zero_p (const value_range *vr)
>> +{
>> + return range_includes_p (vr, 0);
>> +}
>> +
>> #endif /* GCC_TREE_VRP_H */
>> Index: gcc/tree-vrp.c
>> ===================================================================
>> --- gcc/tree-vrp.c 2018-10-24 14:02:14.000000000 +0100
>> +++ gcc/tree-vrp.c 2018-10-24 14:02:15.188152659 +0100
>> @@ -844,10 +844,10 @@ value_inside_range (tree val, tree min,
>> }
>>
>>
>> -/* Return TRUE if *VR includes the value zero. */
>> +/* Return TRUE if *VR includes the value X. */
>>
>> bool
>> -range_includes_zero_p (const value_range *vr)
>> +range_includes_p (const value_range *vr, HOST_WIDE_INT x)
>> {
>> if (vr->type == VR_VARYING)
>> return true;
>> @@ -856,13 +856,13 @@ range_includes_zero_p (const value_range
>> if (vr->type == VR_UNDEFINED)
>> return true;
>>
>> - tree zero = build_int_cst (TREE_TYPE (vr->min), 0);
>> + tree x_cst = build_int_cst (TREE_TYPE (vr->min), x);
>> if (vr->type == VR_ANTI_RANGE)
>> {
>> - int res = value_inside_range (zero, vr->min, vr->max);
>> + int res = value_inside_range (x_cst, vr->min, vr->max);
>> return res == 0 || res == -2;
>> }
>> - return value_inside_range (zero, vr->min, vr->max) != 0;
>> + return value_inside_range (x_cst, vr->min, vr->max) != 0;
>> }
>>
>> /* If *VR has a value rante that is a single constant value return that,
>> Index: gcc/tree-pass.h
>> ===================================================================
>> --- gcc/tree-pass.h 2018-10-24 14:02:14.000000000 +0100
>> +++ gcc/tree-pass.h 2018-10-24 14:02:15.188152659 +0100
>> @@ -362,6 +362,7 @@ extern gimple_opt_pass *make_pass_fix_lo
>> extern gimple_opt_pass *make_pass_tree_loop (gcc::context *ctxt);
>> extern gimple_opt_pass *make_pass_tree_no_loop (gcc::context *ctxt);
>> extern gimple_opt_pass *make_pass_tree_loop_init (gcc::context *ctxt);
>> +extern gimple_opt_pass *make_pass_loop_versioning (gcc::context *ctxt);
>> extern gimple_opt_pass *make_pass_lim (gcc::context *ctxt);
>> extern gimple_opt_pass *make_pass_linterchange (gcc::context *ctxt);
>> extern gimple_opt_pass *make_pass_tree_unswitch (gcc::context *ctxt);
>> Index: gcc/gimple-loop-versioning.cc
>> ===================================================================
>> --- /dev/null 2018-09-14 11:16:31.122530289 +0100
>> +++ gcc/gimple-loop-versioning.cc 2018-10-24 14:02:15.184152693 +0100
>> @@ -0,0 +1,1417 @@
>> +/* Loop versioning pass.
>> + Copyright (C) 2018 Free Software Foundation, Inc.
>> +
>> +This file is part of GCC.
>> +
>> +GCC is free software; you can redistribute it and/or modify it
>> +under the terms of the GNU General Public License as published by the
>> +Free Software Foundation; either version 3, or (at your option) any
>> +later version.
>> +
>> +GCC is distributed in the hope that it will be useful, but WITHOUT
>> +ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
>> +FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
>> +for more details.
>> +
>> +You should have received a copy of the GNU General Public License
>> +along with GCC; see the file COPYING3. If not see
>> +<http://www.gnu.org/licenses/>. */
>> +
>> +#include "config.h"
>> +#include "system.h"
>> +#include "coretypes.h"
>> +#include "backend.h"
>> +#include "tree.h"
>> +#include "gimple.h"
>> +#include "gimple-iterator.h"
>> +#include "tree-pass.h"
>> +#include "gimplify-me.h"
>> +#include "cfgloop.h"
>> +#include "tree-ssa-loop.h"
>> +#include "ssa.h"
>> +#include "tree-scalar-evolution.h"
>> +#include "tree-chrec.h"
>> +#include "tree-ssa-loop-ivopts.h"
>> +#include "fold-const.h"
>> +#include "tree-ssa-propagate.h"
>> +#include "tree-inline.h"
>> +#include "domwalk.h"
>> +#include "alloc-pool.h"
>> +#include "vr-values.h"
>> +#include "gimple-ssa-evrp-analyze.h"
>> +#include "gimple-pretty-print.h"
>> +#include "params.h"
>> +
>> +/* This pass looks for loops that could be simplified if certain loop
>> + invariant conditions were true. It is effectively a form of loop
>> + splitting in which the pass produces the split conditions itself,
>> + instead of using ones that are already present in the IL.
>> +
>> + Versioning for when strides are 1
>> + ---------------------------------
>> +
>> + At the moment the only thing the pass looks for are memory references
>> + like:
>> +
>> + for (auto i : ...)
>> + ...x[i * stride]...
>> +
>> + It considers changing such loops to:
>> +
>> + if (stride == 1)
>> + for (auto i : ...) [A]
>> + ...x[i]...
>> + else
>> + for (auto i : ...) [B]
>> + ...x[i * stride]...
>> +
>> + This can have several benefits:
>> +
>> + (1) [A] is often easier or cheaper to vectorize than [B].
>> +
>> + (2) The scalar code in [A] is simpler than the scalar code in [B]
>> + (if the loops cannot be vectorized or need an epilogue loop).
>> +
>> + (3) We might recognize [A] as a pattern, such as a memcpy or memset.
>> +
>> + (4) [A] has simpler address evolutions, which can help other passes
>> + like loop interchange.
>> +
>> + The optimization is particularly useful for assumed-shape arrays in
>> + Fortran, where the stride of the innermost dimension depends on the
>> + array descriptor but is often equal to 1 in practice. For example:
>> +
>> + subroutine f1(x)
>> + real :: x(:)
>> + x(:) = 100
>> + end subroutine f1
>> +
>> + generates the equivalent of:
>> +
>> + raw_stride = *x.dim[0].stride;
>> + stride = raw_stride != 0 ? raw_stride : 1;
>> + x_base = *x.data;
>> + ...
>> + tmp1 = stride * S;
>> + tmp2 = tmp1 - stride;
>> + *x_base[tmp2] = 1.0e+2;
>> +
>> + but in the common case that stride == 1, the last three statements
>> + simplify to:
>> +
>> + tmp3 = S + -1;
>> + *x_base[tmp3] = 1.0e+2;
>> +
>> + The optimization is in principle very simple. The difficult parts are:
>> +
>> + (a) deciding which parts of a general address calculation correspond
>> + to the inner dimension of an array, since this usually isn't explicit
>> + in the IL, and for C often isn't even explicit in the source code
>> +
>> + (b) estimating when the transformation is worthwhile
>> +
>> + Structure
>> + ---------
>> +
>> + The pass has four phases:
>> +
>> + (1) Walk through the statements looking for and recording potential
>> + versioning opportunities. Stop if there are none.
>> +
>> + (2) Use context-sensitive range information to see whether any versioning
>> + conditions are impossible in practice. Remove them if so, and stop
>> + if no opportunities remain.
>> +
>> + (We do this only after (1) to keep compile time down when no
>> + versioning opportunities exist.)
>> +
>> + (3) Apply the cost model. Decide which versioning opportunities are
>> + worthwhile and at which nesting level they should be applied.
>> +
>> + (4) Attempt to version all the loops selected by (3), so that:
>> +
>> + for (...)
>> + ...
>> +
>> + becomes:
>> +
>> + if (!cond)
>> + for (...) // Original loop
>> + ...
>> + else
>> + for (...) // New loop
>> + ...
>> +
>> + Use the version condition COND to simplify the new loop. */
>> +class loop_versioning {
>> +public:
>> + loop_versioning (function *);
>> + ~loop_versioning ();
>> + unsigned int run ();
>> +
>> +private:
>> + /* Information about the versioning we'd like to apply to a loop. */
>> + struct loop_info {
>> + bool worth_versioning_p () const;
>> +
>> + /* True if we've decided not to version this loop. The remaining
>> + fields are meaningless if so. */
>> + bool rejected_p;
>> +
>> + /* True if at least one subloop of this loop benefits from versioning. */
>> + bool subloops_benefit_p;
>> +
>> + /* An estimate of the total number of instructions in the loop,
>> + excluding those in subloops that benefit from versioning. */
>> + unsigned int num_insns;
>> +
>> + /* The outermost loop that can handle all the version checks
>> + described below. */
>> + struct loop *outermost;
>> +
>> + /* We'd like to version the loop for the case in which these
>> + SSA_NAMEs are all equal to 1 at runtime. */
>> + vec<tree> unity_names;
>> +
>> + /* The set of SSA_NAMEs in UNITY_NAMES, keyed off the SSA_NAME_VERSION. */
>> + bitmap_head unity_name_ids;
>> + };
>> +
>> + /* Used to walk the dominator tree to find loop versioning conditions
>> + that are always false. */
>> + class lv_dom_walker : public dom_walker
>> + {
>> + public:
>> + lv_dom_walker (loop_versioning &);
>> +
>> + edge before_dom_children (basic_block) FINAL OVERRIDE;
>> + void after_dom_children (basic_block) FINAL OVERRIDE;
>> +
>> + private:
>> + /* The parent pass. */
>> + loop_versioning &m_lv;
>> +
>> + /* Used to build context-dependent range information. */
>> + evrp_range_analyzer m_range_analyzer;
>> + };
>> +
>> + /* Used to simplify statements based on conditions that are established
>> + by the version checks. */
>> + class name_prop : public substitute_and_fold_engine
>> + {
>> + public:
>> + name_prop (loop_info &li) : m_li (li) {}
>> + tree get_value (tree) FINAL OVERRIDE;
>> +
>> + private:
>> + /* Information about the versioning we've performed on the loop. */
>> + loop_info &m_li;
>> + };
>> +
>> + loop_info &get_loop_info (struct loop *loop) { return m_loops[loop->num]; }
>> +
>> + unsigned int max_insns_for_loop (struct loop *);
>> + bool expensive_stmt_p (gimple *);
>> +
>> + void version_for_unity (struct loop *, tree);
>> + bool acceptable_scale_p (tree, poly_uint64);
>> + tree get_step_if_innermost (tree, poly_uint64);
>> + tree extract_step (tree, poly_uint64, tree *);
>> + void analyze_evolution (struct loop *, tree, poly_uint64);
>> + bool analyze_product (struct loop *, gassign *, poly_uint64);
>> + bool analyze_sum_of_products (struct loop *, tree, poly_uint64);
>> + void analyze_pointer (struct loop *, tree, tree);
>> + void analyze_expr (struct loop *, tree);
>> + void analyze_stmt (gimple *);
>> + void analyze_block (basic_block);
>> + bool analyze_blocks ();
>> +
>> + void prune_loop_conditions (struct loop *, vr_values *);
>> + bool prune_conditions ();
>> +
>> + void merge_loop_info (struct loop *, struct loop *);
>> + void add_loop_to_queue (struct loop *);
>> + bool decide_whether_loop_is_versionable (struct loop *);
>> + bool make_versioning_decisions ();
>> +
>> + bool version_loop (struct loop *);
>> + bool implement_versioning_decisions ();
>> +
>> + /* The function we're optimizing. */
>> + function *m_fn;
>> +
>> + /* The obstack to use for all pass-specific bitmaps. */
>> + bitmap_obstack m_obstack;
>> +
>> + /* The number of loops in the function. */
>> + unsigned int m_nloops;
>> +
>> + /* The total number of loop version conditions we've found. */
>> + unsigned int m_num_conditions;
>> +
>> + /* Information about each loop. */
>> + auto_vec<loop_info> m_loops;
>> +
>> + /* The list of loops that we've decided to version. */
>> + auto_vec<struct loop *> m_loops_to_version;
>> +};
>> +
>> +/* If EXPR is an SSA name and not a default definition, return the
>> + defining statement, otherwise return null. */
>> +
>> +static gimple *
>> +maybe_get_stmt (tree expr)
>> +{
>> + if (TREE_CODE (expr) == SSA_NAME && !SSA_NAME_IS_DEFAULT_DEF (expr))
>> + return SSA_NAME_DEF_STMT (expr);
>> + return NULL;
>> +}
>> +
>> +/* Like maybe_get_stmt, but also return null if the defining
>> + statement isn't an assignment. */
>> +
>> +static gassign *
>> +maybe_get_assign (tree expr)
>> +{
>> + return safe_dyn_cast <gassign *> (maybe_get_stmt (expr));
>> +}
>> +
>> +/* If EXPR is an SSA name, look through any casts to see whether the
>> + unconverted value is defined in LOOP by a gassign. Return the
>> + gassign if so, otherwise return null. */
>> +
>> +gassign *
>> +maybe_get_assign_strip_casts (struct loop *loop, tree expr)
>> +{
>> + const unsigned int MAX_NITERS = 4;
>> +
>> + tree type = TREE_TYPE (expr);
>> + for (unsigned int niters = 0; niters < MAX_NITERS; ++niters)
>> + {
>> + gassign *assign = maybe_get_assign (expr);
>> + if (!assign || gimple_bb (assign)->loop_father != loop)
>> + return NULL;
>> + expr = gimple_assign_rhs1 (assign);
>> + if (CONVERT_EXPR_CODE_P (gimple_assign_rhs_code (assign))
>> + && INTEGRAL_TYPE_P (TREE_TYPE (expr)) == INTEGRAL_TYPE_P (type)
>> + && POINTER_TYPE_P (TREE_TYPE (expr)) == POINTER_TYPE_P (type))
>> + ;
>> + else
>> + return assign;
>> + }
>> + return NULL;
>> +}
>> +
>> +/* Strip all conversions of integers from EXPR, regardless of whether
>> + the conversions are nops. This is useful in the context of this pass
>> + because we're not trying to fold or simulate the expression; we just
>> + want to see how it's structured. */
>> +
>> +static tree
>> +strip_casts (tree expr)
>> +{
>> + while (CONVERT_EXPR_P (expr)
>> + && INTEGRAL_TYPE_P (TREE_TYPE (TREE_OPERAND (expr, 0))))
>> + expr = TREE_OPERAND (expr, 0);
>> + return expr;
>> +}
>> +
>> +/* Return true if we want to version the loop, i.e. if we have a
>> + specific reason for doing so and no specific reason not to. */
>> +
>> +bool
>> +loop_versioning::loop_info::worth_versioning_p () const
>> +{
>> + return !rejected_p && (!unity_names.is_empty () || subloops_benefit_p);
>> +}
>> +
>> +loop_versioning::lv_dom_walker::lv_dom_walker (loop_versioning &lv)
>> + : dom_walker (CDI_DOMINATORS), m_lv (lv)
>> +{
>> +}
>> +
>> +/* Process BB before processing the blocks it dominates. */
>> +
>> +edge
>> +loop_versioning::lv_dom_walker::before_dom_children (basic_block bb)
>> +{
>> + m_range_analyzer.enter (bb);
>> +
>> + if (bb == bb->loop_father->header)
>> + m_lv.prune_loop_conditions (bb->loop_father,
>> + m_range_analyzer.get_vr_values ());
>> +
>> + for (gimple_stmt_iterator si = gsi_start_bb (bb); !gsi_end_p (si);
>> + gsi_next (&si))
>> + m_range_analyzer.record_ranges_from_stmt (gsi_stmt (si), false);
>> +
>> + return NULL;
>> +}
>> +
>> +/* Process BB after processing the blocks it dominates. */
>> +
>> +void
>> +loop_versioning::lv_dom_walker::after_dom_children (basic_block bb)
>> +{
>> + m_range_analyzer.leave (bb);
>> +}
>> +
>> +/* Decide whether to replace VAL with a new value in a versioned loop.
>> + Return the new value if so, otherwise return null. */
>> +
>> +tree
>> +loop_versioning::name_prop::get_value (tree val)
>> +{
>> + if (TREE_CODE (val) == SSA_NAME
>> + && bitmap_bit_p (&m_li.unity_name_ids, SSA_NAME_VERSION (val)))
>> + return build_one_cst (TREE_TYPE (val));
>> + return NULL_TREE;
>> +}
>> +
>> +/* Initialize the structure to optimize FN. */
>> +
>> +loop_versioning::loop_versioning (function *fn)
>> + : m_fn (fn),
>> + m_nloops (number_of_loops (fn)),
>> + m_num_conditions (0)
>> +{
>> + bitmap_obstack_initialize (&m_obstack);
>> +
>> + m_loops.safe_grow_cleared (m_nloops);
>> + for (unsigned int i = 0; i < m_nloops; ++i)
>> + {
>> + m_loops[i].outermost = get_loop (m_fn, 0);
>> + bitmap_initialize (&m_loops[i].unity_name_ids, &m_obstack);
>> + }
>> +}
>> +
>> +loop_versioning::~loop_versioning ()
>> +{
>> + for (unsigned int i = 0; i < m_nloops; ++i)
>> + m_loops[i].unity_names.release ();
>> + bitmap_obstack_release (&m_obstack);
>> +}
>> +
>> +/* Return the maximum number of instructions allowed in LOOP before
>> + it becomes too big for versioning.
>> +
>> + There are separate limits for inner and outer loops. The limit for
>> + inner loops applies only to loops that benefit directly from versioning.
>> + The limit for outer loops applies to all code in the outer loop and
>> + its subloops that *doesn't* benefit directly from versioning; such code
>> + would be "taken along for the ride". The idea is that if the cost of
>> + the latter is small, it is better to version outer loops rather than
>> + inner loops, both to reduce the number of repeated checks and to enable
>> + more of the loop nest to be optimized as a natural nest (e.g. by loop
>> + interchange or outer-loop vectorization). */
>> +
>> +unsigned int
>> +loop_versioning::max_insns_for_loop (struct loop *loop)
>> +{
>> + return (loop->inner
>> + ? PARAM_VALUE (PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS)
>> + : PARAM_VALUE (PARAM_LOOP_VERSIONING_MAX_INNER_INSNS));
>> +}
>> +
>> +/* Return true if for cost reasons we should avoid versioning any loop
>> + that contains STMT.
>> +
>> + Note that we don't need to check whether versioning is invalid for
>> + correctness reasons, since the versioning process does that for us.
>> + The conditions involved are too rare to be worth duplicating here. */
>> +
>> +bool
>> +loop_versioning::expensive_stmt_p (gimple *stmt)
>> +{
>> + if (gcall *call = dyn_cast <gcall *> (stmt))
>> + /* Assume for now that the time spent in an "expensive" call would
>> + overwhelm any saving from versioning. */
>> + return !gimple_inexpensive_call_p (call);
>> + return false;
>> +}
>> +
>> +/* Record that we want to version LOOP for the case in which SSA name NAME
>> + is equal to 1. We already know that NAME is invariant in LOOP. */
>> +
>> +void
>> +loop_versioning::version_for_unity (struct loop *loop, tree name)
>> +{
>> + loop_info &li = get_loop_info (loop);
>> +
>> + if (bitmap_set_bit (&li.unity_name_ids, SSA_NAME_VERSION (name)))
>> + {
>> + /* This is the first time we've wanted to version LOOP for NAME. */
>> + li.unity_names.safe_push (name);
>> +
>> + /* Keep track of the outermost loop that can handle all versioning
>> + checks in LI. */
>> + struct loop *outermost
>> + = outermost_invariant_loop_for_expr (loop, name);
>> + if (loop_depth (li.outermost) < loop_depth (outermost))
>> + li.outermost = outermost;
>> +
>> + if (dump_file && (dump_flags & TDF_DETAILS))
>> + {
>> + fprintf (dump_file, ";; Want to version loop %d (depth %d)"
>> + " for when ", loop->num, loop_depth (loop));
>> + print_generic_expr (dump_file, name, TDF_SLIM);
>> + fprintf (dump_file, " == 1");
>> + if (outermost == loop)
>> + fprintf (dump_file, "; cannot hoist check further");
>> + else
>> + {
>> + fprintf (dump_file, "; could hoist check to loop %d (depth %d)",
>> + outermost->num, loop_depth (outermost));
>> + if (loop_depth (li.outermost) > loop_depth (outermost))
>> + fprintf (dump_file, ", but that's further than"
>> + " other checks allow");
>> + }
>> + fprintf (dump_file, "\n");
>> + }
>> +
>> + m_num_conditions += 1;
>> + }
>> + else
>> + {
>> + /* This is a duplicate request. */
>> + if (dump_file && (dump_flags & TDF_DETAILS))
>> + {
>> + fprintf (dump_file, ";; Already want to version loop for when ");
>> + print_generic_expr (dump_file, name, TDF_SLIM);
>> + fprintf (dump_file, " == 1\n");
>> + }
>> + }
>> +}
>> +
>> +/* Return true if in principle it is worth versioning an index fragment of
>> + the form:
>> +
>> + (i * b * SCALE) / FACTOR
>> +
>> + for the case in which b == 1. */
>> +
>> +bool
>> +loop_versioning::acceptable_scale_p (tree scale, poly_uint64 factor)
>> +{
>> + /* See whether SCALE is a constant multiple of FACTOR, and if the
>> + multiple is small enough for us to treat it as a potential grouped
>> + access. For example:
>> +
>> + for (auto i : ...)
>> + y[i] = f (x[4 * i * stride],
>> + x[4 * i * stride + 1],
>> + x[4 * i * stride + 2]);
>> +
>> + would benefit from versioning for the case in which stride == 1.
>> + High multiples of i * stride are less likely to benefit, and could
>> + indicate a simulated multi-dimensional array.
>> +
>> + This is just a heuristic, to avoid having to do expensive group
>> + analysis of the data references in a loop. */
>> + poly_uint64 const_scale;
>> + unsigned int multiple;
>> + if (poly_int_tree_p (scale, &const_scale)
>> + && constant_multiple_p (const_scale, factor, &multiple))
>> + {
>> + unsigned int maxval = PARAM_VALUE (PARAM_LOOP_VERSIONING_GROUP_SIZE);
>> + return IN_RANGE (multiple, 1, maxval);
>> + }
>> +
>> + return false;
>> +}
>> +
>> +/* Decide whether an index fragment of the form:
>> +
>> + (i * STEP) / FACTOR
>> +
>> + is likely to be for an innermost dimension. If we think it is,
>> + return one of the constant values that it could have (returning 1 if
>> + that's a possibility). If think it isn't, return null. Otherwise
>> + return STEP, to indicate that it might or might not be an inner
>> + dimension. */
>> +
>> +tree
>> +loop_versioning::get_step_if_innermost (tree step, poly_uint64 factor)
>> +{
>> + const unsigned int MAX_NITERS = 8;
>> +
>> + tree likely = NULL_TREE;
>> + tree unlikely = NULL_TREE;
>> + tree worklist[MAX_NITERS];
>> + unsigned int length = 0;
>> + worklist[length++] = step;
>> + for (unsigned int i = 0; i < length; ++i)
>> + {
>> + tree expr = worklist[i];
>> +
>> + if (TREE_CONSTANT (expr))
>> + {
>> + /* See if multiplying by EXPR applies a scale that would be
>> + consistent with an individual access or a small grouped
>> + access. */
>> + if (acceptable_scale_p (expr, factor))
>> + {
>> + likely = expr;
>> + if (integer_onep (expr))
>> + break;
>> + }
>> + else
>> + unlikely = expr;
>> + continue;
>> + }
>> +
>> + /* Otherwise we can only handle SSA names. */
>> + gimple *stmt = maybe_get_stmt (expr);
>> + if (!stmt)
>> + continue;
>> +
>> + /* If EXPR is set by a PHI node, queue its arguments in case
>> + we find one that is consistent with an inner dimension.
>> +
>> + An important instance of this is the Fortran handling of array
>> + descriptors, which calculates the stride of the inner dimension
>> + using a PHI equivalent of:
>> +
>> + raw_stride = a.dim[0].stride;
>> + stride = raw_stride != 0 ? raw_stride : 1;
>> +
>> + (Strides for outer dimensions do not treat 0 specially.) */
>> + if (gphi *phi = dyn_cast <gphi *> (stmt))
>> + {
>> + unsigned int nargs = gimple_phi_num_args (phi);
>> + for (unsigned int j = 0; j < nargs && length < MAX_NITERS; ++j)
>> + worklist[length++] = gimple_phi_arg_def (phi, j);
>> + continue;
>> + }
>> +
>> + /* If the value is set by an assignment, expect it to be read from
>> + memory (such as an array descriptor) rather than be calculated. */
>> + if (gassign *assign = dyn_cast <gassign *> (stmt))
>> + {
>> + if (gimple_assign_single_p (assign)
>> + && is_gimple_lvalue (gimple_assign_rhs1 (assign)))
>> + continue;
>> +
>> + unlikely = expr;
>> + }
>> +
>> + /* Things like calls don't really tell us anything. */
>> + }
>> +
>> + if (likely)
>> + {
>> + if (dump_file && (dump_flags & TDF_DETAILS))
>> + fprintf (dump_file, "likely to be innermost dimension\n");
>> + return likely;
>> + }
>> +
>> + if (unlikely)
>> + {
>> + if (dump_file && (dump_flags & TDF_DETAILS))
>> + fprintf (dump_file, "probably not innermost dimension\n");
>> + return NULL_TREE;
>> + }
>> +
>> + if (dump_file && (dump_flags & TDF_DETAILS))
>> + {
>> + if (maybe_ne (factor, 1U))
>> + fprintf (dump_file, "didn't find expected scaling factor\n");
>> + else
>> + fprintf (dump_file, "no information about value\n");
>> + }
>> + return step;
>> +}
>> +
>> +/* STEP appears in an index fragment of the form:
>> +
>> + {..., +, STEP}_n / FACTOR
>> +
>> + Remove any conversions and grouping or scaling factors from STEP and
>> + return the underlying index step. Set *STEP_IF_INNERMOST to the value
>> + returned by get_step_if_innermost. */
>> +
>> +tree
>> +loop_versioning::extract_step (tree step, poly_uint64 factor,
>> + tree *step_if_innermost)
>> +{
>> + if (dump_file && (dump_flags & TDF_DETAILS))
>> + {
>> + fprintf (dump_file, ";; step ");
>> + print_generic_expr (dump_file, step, TDF_SLIM);
>> + fprintf (dump_file, ": ");
>> + }
>> +
>> + step = strip_casts (step);
>> +
>> + /* Peel any scaling, which generally happens after conversion to
>> + pointer width. For example, on LP64 systems:
>> +
>> + int *x, i, stride;
>> + ... x[4 * i * stride] ...;
>> +
>> + multiplies i * stride by 4 using ints, then widens the result
>> + to pointer width before multiplying by sizeof (int). */
>> + poly_uint64 scale;
>> + if (TREE_CODE (step) == MULT_EXPR
>> + && poly_int_tree_p (TREE_OPERAND (step, 1), &scale)
>> + && known_eq (scale, factor))
>> + {
>> + step = strip_casts (TREE_OPERAND (step, 0));
>> + if (dump_file && (dump_flags & TDF_DETAILS))
>> + fprintf (dump_file, "scaled, ");
>> + factor = 1;
>> + }
>> +
>> + /* Peel any grouping multiple; see acceptable_scale_p for details. */
>> + if (TREE_CODE (step) == MULT_EXPR
>> + && acceptable_scale_p (TREE_OPERAND (step, 1), factor))
>> + {
>> + step = strip_casts (TREE_OPERAND (step, 0));
>> + if (dump_file && (dump_flags & TDF_DETAILS))
>> + fprintf (dump_file, "%sgrouped, ",
>> + maybe_ne (factor, 1U) ? "scaled, " : "");
>> + factor = 1;
>> + }
>> +
>> + *step_if_innermost = get_step_if_innermost (step, factor);
>> + return step;
>> +}
>> +
>> +/* Analyze the evolution of index fragment EXPR / FACTOR in LOOP and its
>> + containing loops to see whether any part of it could be simplified
>> + by versioning. Register the versioning opportunities if so. */
>> +
>> +void
>> +loop_versioning::analyze_evolution (struct loop *loop, tree expr,
>> + poly_uint64 factor)
>> +{
>> + const unsigned int MAX_NSPLIT = 8;
>> +
>> + if (dump_file && (dump_flags & TDF_DETAILS))
>> + {
>> + fprintf (dump_file, ";; Analyzing use of ");
>> + print_generic_expr (dump_file, expr, TDF_SLIM);
>> + if (maybe_ne (factor, 1U))
>> + {
>> + fprintf (dump_file, " (which addresses ");
>> + print_dec (factor, dump_file);
>> + fprintf (dump_file, " bytes)");
>> + }
>> + fprintf (dump_file, " in loop %d (depth %d)\n",
>> + loop->num, loop_depth (loop));
>> + }
>> +
>> + /* The main problem we have here is that we cannot assume that the
>> + innermost loop iterates over the innermost dimension of an array.
>> + Accidentally adding versioning checks for outer dimensions would
>> + cause the version condition to be false, which as well as bloating
>> + the code would defeat loop versioning benefits for other accesses.
>> +
>> + Unfortunately all we usually see at this stage is general address
>> + arithmetic, with no positive way of identifying how many dimensions
>> + an array access has and which multiplication factors in the address
>> + expression correspond to which array dimensions. In C code this is
>> + often not even explicit in the source, since variable-sized multi-
>> + dimensional arrays are often simulated using one-dimensional arrays.
>> +
>> + The three main ways in which we deal with this are:
>> +
>> + - use heuristics that positively identify steps that are likely
>> + to represent the inner dimension.
>> +
>> + - use heuristics that positively identify steps that are unlikely
>> + to represent the inner dimension.
>> +
>> + - if a part of EXPR is invariant in LOOP, analyze its evolution in
>> + the outer loops to see whether we can positively identify any of
>> + it as iterating over the inner dimension. */
>> + tree best_step = NULL_TREE;
>> + auto_vec<tree, MAX_NSPLIT> worklist;
>> + worklist.quick_push (expr);
>> + unsigned int nsplit = 0;
>> + while (!worklist.is_empty ())
>> + {
>> + expr = strip_casts (worklist.pop ());
>> + tree_code code = TREE_CODE (expr);
>> +
>> + if (code == POLYNOMIAL_CHREC)
>> + {
>> + /* Analyze the CHREC_RIGHT as the step in an index fragment. */
>> + tree step_if_innermost;
>> + tree step = extract_step (CHREC_RIGHT (expr), factor,
>> + &step_if_innermost);
>> + if (!best_step)
>> + {
>> + /* This is the outermost chrec for the original expression.
>> + It's not worth carrying on if the step isn't versionable,
>> + or if we're pretty sure it's not for the inner dimension. */
>> + if (!step_if_innermost
>> + || TREE_CODE (step) != SSA_NAME
>> + || !expr_invariant_in_loop_p (loop, step))
>> + return;
>> +
>> + best_step = step;
>> +
>> + /* We should version for STEP == 1 if we know that that can be
>> + true under some circumstances. */
>> + if (integer_onep (step_if_innermost))
>> + break;
>> +
>> + /* Bail out if this appears to be the step for the innermost
>> + dimension, but isn't likely to be 1.
>> +
>> + ??? We could instead version for when it equals
>> + STEP_IF_INNERMOST, but it's not likely to have as much
>> + benefit as versioning for 1. */
>> + if (step_if_innermost != step)
>> + return;
>> + }
>> + else
>> + {
>> + /* This is an inner chrec. If it looks like it iterates over
>> + the innermost dimension, abort any attempt to version for
>> + the outermost chrec (which if we reach here wasn't itself
>> + obviously iterating over the innermost dimension). */
>> + if (step_if_innermost && TREE_CONSTANT (step_if_innermost))
>> + return;
>> + }
>> + worklist.quick_push (CHREC_LEFT (expr));
>> + continue;
>> + }
>> +
>> + /* Analyze parts of a queued CHREC_LEFT. This is better than just
>> + analyzing the evolution of the whole expression since the value
>> + could include a mixture of analyzable and unanalyzable elements.
>> + Use NSPLIT to count cases in which we add more expressions to
>> + analyze, as opposed to just simplifying the existing one. */
>> + if (code == PLUS_EXPR || code == POINTER_PLUS_EXPR || code == MINUS_EXPR)
>> + {
>> + worklist.quick_push (TREE_OPERAND (expr, 0));
>> + if (nsplit++ < MAX_NSPLIT)
>> + worklist.quick_push (TREE_OPERAND (expr, 1));
>> + continue;
>> + }
>> + if (code == MULT_EXPR)
>> + {
>> + tree op0 = strip_casts (TREE_OPERAND (expr, 0));
>> + tree op1 = TREE_OPERAND (expr, 1);
>> + if (TREE_CODE (op0) == PLUS_EXPR || TREE_CODE (op0) == MINUS_EXPR)
>> + {
>> + tree type = TREE_TYPE (expr);
>> + tree op00 = fold_convert (type, TREE_OPERAND (op0, 0));
>> + worklist.quick_push (fold_build2 (MULT_EXPR, type, op00, op1));
>> + if (nsplit++ < MAX_NSPLIT)
>> + {
>> + tree op01 = fold_convert (type, TREE_OPERAND (op0, 1));
>> + worklist.quick_push (fold_build2 (MULT_EXPR, type,
>> + op01, op1));
>> + }
>> + continue;
>> + }
>> + }
>> +
>> + /* If EXPR is invariant in LOOP, analyze it wrt the innermost loop
>> + for which it could evolve (i.e. the loop containing the outermost
>> + one for which EXPR is invariant). */
>> + struct loop *wrt_loop = outermost_invariant_loop_for_expr (loop, expr);
>> + if (wrt_loop)
>> + {
>> + wrt_loop = loop_outer (wrt_loop);
>> + if (!wrt_loop)
>> + continue;
>> + }
>> + else
>> + wrt_loop = loop;
>> + tree evolution = analyze_scalar_evolution (wrt_loop, expr);
>> + tree chrec = strip_casts (evolution);
>> + if (TREE_CODE (chrec) == POLYNOMIAL_CHREC)
>> + {
>> + if (dump_file && (dump_flags & TDF_DETAILS))
>> + {
>> + fprintf (dump_file, ";; evolution of ");
>> + print_generic_expr (dump_file, expr, TDF_SLIM);
>> + fprintf (dump_file, " in loop %d (depth %d): ",
>> + wrt_loop->num, loop_depth (wrt_loop));
>> + print_generic_expr (dump_file, evolution, TDF_SLIM);
>> + fprintf (dump_file, "\n");
>> + }
>> + worklist.quick_push (chrec);
>> + }
>> + else
>> + {
>> + if (dump_file && (dump_flags & TDF_DETAILS))
>> + {
>> + fprintf (dump_file, ";; cannot analyze ");
>> + print_generic_expr (dump_file, expr, TDF_SLIM);
>> + fprintf (dump_file, " any further\n");
>> + }
>> + }
>> + }
>> + if (best_step)
>> + version_for_unity (loop, best_step);
>> +}
>> +
>> +/* Analyze multiplication MULT to see whether we can identify "gather-like"
>> + versioning opportunities such as:
>> +
>> + for (int i = 0; i < n; ++i)
>> + res += a[index[i] * stride];
>> +
>> + Return true if there was a versioning opportunity.
>> +
>> + LOOP is the loop that contains MULT. Dividing the value of MULT
>> + by FACTOR converts it into an element index. */
>> +
>> +bool
>> +loop_versioning::analyze_product (struct loop *loop, gassign *mult,
>> + poly_uint64 factor)
>> +{
>> + /* Record the original LHS for the dump message below. */
>> + tree lhs = gimple_assign_lhs (mult);
>> +
>> + /* Peel any scaling, which generally happens after conversion to
>> + pointer width. For example, on LP64 systems:
>> +
>> + int *x, i, stride;
>> + ... x[4 * i * stride] ...;
>> +
>> + multiplies i * stride by 4 using ints, then widens the result
>> + to pointer width before multiplying by sizeof (int). */
>> + poly_uint64 scale;
>> + if (poly_int_tree_p (gimple_assign_rhs2 (mult), &scale)
>> + && known_eq (scale, factor))
>> + {
>> + mult = maybe_get_assign_strip_casts (loop, gimple_assign_rhs1 (mult));
>> + if (!mult || gimple_assign_rhs_code (mult) != MULT_EXPR)
>> + return false;
>> + factor = 1;
>> + }
>> +
>> + /* Peel any grouping multiple; see acceptable_scale_p for details. */
>> + if (acceptable_scale_p (gimple_assign_rhs2 (mult), factor))
>> + {
>> + mult = maybe_get_assign_strip_casts (loop, gimple_assign_rhs1 (mult));
>> + if (!mult || gimple_assign_rhs_code (mult) != MULT_EXPR)
>> + return false;
>> + }
>> + else if (maybe_ne (factor, 1U))
>> + /* We expect to see a scaling multiplication when FACTOR is > 1. */
>> + return false;
>> +
>> + /* See whether this multiplication involves a loop-invariant SSA name
>> + and a non-invariant SSA name. */
>> + tree op1 = gimple_assign_rhs1 (mult);
>> + tree op2 = gimple_assign_rhs2 (mult);
>> + if (TREE_CODE (op1) != SSA_NAME || TREE_CODE (op2) != SSA_NAME)
>> + return false;
>> +
>> + bool invariant1_p = expr_invariant_in_loop_p (loop, op1);
>> + bool invariant2_p = expr_invariant_in_loop_p (loop, op2);
>> + if (invariant1_p == invariant2_p)
>> + return false;
>> +
>> + /* Make sure that the invariant is OP1 and the other operand is OP2. */
>> + if (invariant2_p)
>> + std::swap (op1, op2);
>> +
>> + /* Bail out if OP2 can be analyzed as an evolution; we want to use
>> + analyze_evolution in that case instead. There's no point trying
>> + hard to avoid repeating the call to analyze_scalar_evolution since
>> + that function does its own caching. */
>> + if (tree_contains_chrecs (analyze_scalar_evolution (loop, op2), NULL))
>> + return false;
>> +
>> + if (dump_file && (dump_flags & TDF_DETAILS))
>> + {
>> + fprintf (dump_file, ";; Address fragment ");
>> + print_generic_expr (dump_file, lhs, TDF_SLIM);
>> + fprintf (dump_file, " multiplies invariant ");
>> + print_generic_expr (dump_file, op1, TDF_SLIM);
>> + fprintf (dump_file, " by ");
>> + print_generic_expr (dump_file, op2, TDF_SLIM);
>> + fprintf (dump_file, ", which isn't a scalar evolution\n");
>> + }
>> +
>> + version_for_unity (loop, op1);
>> + return true;
>> +}
>> +
>> +/* Treat EXPR as a sum of products and apply analyze_product to each of the
>> + products. Return true if one of the products provides a versioning
>> + opportunity. FACTOR is as for analyze_product. */
>> +
>> +bool
>> +loop_versioning::analyze_sum_of_products (struct loop *loop, tree expr,
>> + poly_uint64 factor)
>> +{
>> + const unsigned int MAX_NITERS = 8;
>> +
>> + tree worklist[MAX_NITERS];
>> + unsigned int length = 0;
>> + worklist[length++] = expr;
>> + for (unsigned int i = 0; i < length; ++i)
>> + {
>> + expr = worklist[i];
>> + gassign *assign = maybe_get_assign_strip_casts (loop, expr);
>> + if (!assign)
>> + continue;
>> +
>> + tree_code code = gimple_assign_rhs_code (assign);
>> + if (code == PLUS_EXPR || code == POINTER_PLUS_EXPR || code == MINUS_EXPR)
>> + {
>> + if (length < MAX_NITERS)
>> + worklist[length++] = gimple_assign_rhs1 (assign);
>> + if (length < MAX_NITERS)
>> + worklist[length++] = gimple_assign_rhs2 (assign);
>> + }
>> + else if (code == MULT_EXPR && analyze_product (loop, assign, factor))
>> + return true;
>> + }
>> + return false;
>> +}
>> +
>> +/* Analyze pointer expression EXPR, which occurs in loop LOOP and which
>> + is used to address a value of type TYPE. */
>> +
>> +void
>> +loop_versioning::analyze_pointer (struct loop *loop, tree expr, tree type)
>> +{
>> + poly_uint64 factor;
>> + if (poly_int_tree_p (TYPE_SIZE_UNIT (type), &factor))
>> + {
>> + if (!analyze_sum_of_products (loop, expr, factor))
>> + analyze_evolution (loop, expr, factor);
>> + }
>> +}
>> +
>> +/* Analyze expression EXPR, which occurs in loop LOOP. */
>> +
>> +void
>> +loop_versioning::analyze_expr (struct loop *loop, tree expr)
>> +{
>> + while (handled_component_p (expr))
>> + {
>> + /* See whether we can use versioning to avoid a multiplication
>> + in the array index. */
>> + if (TREE_CODE (expr) == ARRAY_REF)
>> + {
>> + if (!analyze_sum_of_products (loop, TREE_OPERAND (expr, 1), 1))
>> + analyze_evolution (loop, TREE_OPERAND (expr, 1), 1);
>> + }
>> + expr = TREE_OPERAND (expr, 0);
>> + }
>> +
>> + if (TREE_CODE (expr) == MEM_REF)
>> + {
>> + tree addr = TREE_OPERAND (expr, 0);
>> + /* See whether we can use versioning to avoid a multiplication
>> + in the pointer calculation. This is generally only worth
>> + doing if the multiplication occurs in this loop rather than
>> + an outer loop. */
>> + if (!expr_invariant_in_loop_p (loop, addr))
>> + analyze_pointer (loop, addr, TREE_TYPE (expr));
>> + }
>> +
>> + /* These would be easy to handle if they existed at this stage. */
>> + gcc_checking_assert (TREE_CODE (expr) != TARGET_MEM_REF);
>> +}
>> +
>> +/* Analyze STMT looking for useful version checks. */
>> +
>> +void
>> +loop_versioning::analyze_stmt (gimple *stmt)
>> +{
>> + struct loop *loop = gimple_bb (stmt)->loop_father;
>> +
>> + unsigned int nops = gimple_num_ops (stmt);
>> + for (unsigned int i = 0; i < nops; ++i)
>> + if (tree op = gimple_op (stmt, i))
>> + analyze_expr (loop, op);
>> +}
>> +
>> +/* Analyze all the statements in BB looking for useful version checks. */
>> +
>> +void
>> +loop_versioning::analyze_block (basic_block bb)
>> +{
>> + struct loop *loop = bb->loop_father;
>> + loop_info &li = get_loop_info (loop);
>> + if (li.rejected_p)
>> + return;
>> +
>> + for (gimple_stmt_iterator gsi = gsi_start_bb (bb); !gsi_end_p (gsi);
>> + gsi_next (&gsi))
>> + {
>> + gimple *stmt = gsi_stmt (gsi);
>> + if (expensive_stmt_p (stmt))
>> + {
>> + if (dump_file && (dump_flags & TDF_DETAILS))
>> + {
>> + struct loop *loop = gimple_bb (stmt)->loop_father;
>> + fprintf (dump_file, ";; Loop %d (depth %d) contains expensive"
>> + " stmt: ", loop->num, loop_depth (loop));
>> + print_gimple_stmt (dump_file, stmt, 0, TDF_SLIM);
>> + }
>> + li.rejected_p = true;
>> + break;
>> + }
>> +
>> + /* Only look for direct versioning opportunities in inner loops
>> + since the benefit tends to be much smaller for outer loops. */
>> + if (!loop->inner)
>> + analyze_stmt (stmt);
>> +
>> + /* The point of the instruction limit is to prevent excessive
>> + code growth, so this is a size-based estimate even though
>> + the optimization is aimed at speed. */
>> + li.num_insns += estimate_num_insns (stmt, &eni_size_weights);
>> + }
>> +}
>> +
>> +/* Analyze all the blocks in the function looking for useful version checks.
>> + Return true if we found one. */
>> +
>> +bool
>> +loop_versioning::analyze_blocks ()
>> +{
>> + /* For now we don't try to version the whole function, although
>> + versioning at that level could be useful in some cases. */
>> + get_loop_info (get_loop (m_fn, 0)).rejected_p = true;
>> +
>> + basic_block bb;
>> + FOR_EACH_BB_FN (bb, m_fn)
>> + if (loop_outer (bb->loop_father))
>> + analyze_block (bb);
>> +
>> + return m_num_conditions != 0;
>> +}
>> +
>> +/* Use the ranges in VRS to remove impossible versioning conditions from
>> + LOOP. */
>> +
>> +void
>> +loop_versioning::prune_loop_conditions (struct loop *loop, vr_values *vrs)
>> +{
>> + loop_info &li = get_loop_info (loop);
>> +
>> + unsigned int i = li.unity_names.length ();
>> + while (i > 0)
>> + {
>> + i -= 1;
>> + tree name = li.unity_names[i];
>> + value_range *vr = vrs->get_value_range (name);
>> + if (vr && !range_includes_p (vr, 1))
>> + {
>> + if (dump_file && (dump_flags & TDF_DETAILS))
>> + {
>> + fprintf (dump_file, ";; ");
>> + print_generic_expr (dump_file, name, TDF_SLIM);
>> + fprintf (dump_file, " can never be 1 in loop %d\n", loop->num);
>> + }
>> +
>> + li.unity_names.unordered_remove (i);
>> + bitmap_clear_bit (&li.unity_name_ids, SSA_NAME_VERSION (name));
>> + m_num_conditions -= 1;
>> + }
>> + }
>> +}
>> +
>> +/* Remove any scheduled loop version conditions that will never be true.
>> + Return true if any remain. */
>> +
>> +bool
>> +loop_versioning::prune_conditions ()
>> +{
>> + calculate_dominance_info (CDI_DOMINATORS);
>> + lv_dom_walker dom_walker (*this);
>> + dom_walker.walk (ENTRY_BLOCK_PTR_FOR_FN (m_fn));
>> + return m_num_conditions != 0;
>> +}
>> +
>> +/* Merge the version checks for INNER into immediately-enclosing loop
>> + OUTER. */
>> +
>> +void
>> +loop_versioning::merge_loop_info (struct loop *outer, struct loop *inner)
>> +{
>> + loop_info &inner_li = get_loop_info (inner);
>> + loop_info &outer_li = get_loop_info (outer);
>> +
>> + tree name;
>> + unsigned int i;
>> + FOR_EACH_VEC_ELT (inner_li.unity_names, i, name)
>> + if (bitmap_set_bit (&outer_li.unity_name_ids, SSA_NAME_VERSION (name)))
>> + {
>> + outer_li.unity_names.safe_push (name);
>> + if (dump_file && (dump_flags & TDF_DETAILS))
>> + {
>> + fprintf (dump_file, ";; Hoisting check that ");
>> + print_generic_expr (dump_file, name, TDF_SLIM);
>> + fprintf (dump_file, " == 1 from loop %d (depth %d) to loop %d\n",
>> + inner->num, loop_depth (inner), outer->num);
>> + }
>> + }
>> +
>> + if (loop_depth (outer_li.outermost) < loop_depth (inner_li.outermost))
>> + outer_li.outermost = inner_li.outermost;
>> +}
>> +
>> +/* Add LOOP to the queue of loops to version. */
>> +
>> +void
>> +loop_versioning::add_loop_to_queue (struct loop *loop)
>> +{
>> + loop_info &li = get_loop_info (loop);
>> +
>> + if (dump_file && (dump_flags & TDF_DETAILS))
>> + fprintf (dump_file, ";; Queuing loop %d (depth %d) for versioning\n",
>> + loop->num, loop_depth (loop));
>> + m_loops_to_version.safe_push (loop);
>> +
>> + /* Don't try to version superloops. */
>> + li.rejected_p = true;
>> +}
>> +
>> +/* Decide whether the cost model would allow us to version LOOP,
>> + either directly or as part of a parent loop, and return true if so.
>> + This does not imply that the loop is actually worth versioning in its
>> + own right, just that it would be valid to version it if something
>> + benefited.
>> +
>> + We have already made this decision for all inner loops of LOOP. */
>> +
>> +bool
>> +loop_versioning::decide_whether_loop_is_versionable (struct loop *loop)
>> +{
>> + loop_info &li = get_loop_info (loop);
>> +
>> + if (li.rejected_p)
>> + return false;
>> +
>> + /* Examine the decisions made for inner loops. */
>> + for (struct loop *inner = loop->inner; inner; inner = inner->next)
>> + {
>> + loop_info &inner_li = get_loop_info (inner);
>> + if (inner_li.rejected_p)
>> + {
>> + if (dump_file && (dump_flags & TDF_DETAILS))
>> + fprintf (dump_file, ";; Not versioning loop %d (depth %d)"
>> + " because inner loop %d should not be versioned\n",
>> + loop->num, loop_depth (loop), inner->num);
>> + return false;
>> + }
>> +
>> + if (inner_li.worth_versioning_p ())
>> + li.subloops_benefit_p = true;
>> +
>> + /* Accumulate the number of instructions from subloops that are not
>> + the innermost, or that don't benefit from versioning. Only the
>> + instructions from innermost loops that benefit from versioning
>> + should be weighed against loop-versioning-max-inner-insns;
>> + everything else should be weighed against
>> + loop-versioning-max-outer-insns. */
>> + if (!inner_li.worth_versioning_p () || inner->inner)
>> + {
>> + if (dump_file && (dump_flags & TDF_DETAILS))
>> + fprintf (dump_file, ";; Counting %d instructions from"
>> + " loop %d (depth %d) against parent loop %d\n",
>> + inner_li.num_insns, inner->num, loop_depth (inner),
>> + loop->num);
>> + li.num_insns += inner_li.num_insns;
>> + }
>> + }
>> +
>> + /* Enforce the size limits. */
>> + if (li.worth_versioning_p ())
>> + {
>> + unsigned int max_num_insns = max_insns_for_loop (loop);
>> + if (dump_file && (dump_flags & TDF_DETAILS))
>> + fprintf (dump_file, ";; Loop %d (depth %d) has %d instructions,"
>> + " against a limit of %d\n", loop->num, loop_depth (loop),
>> + li.num_insns, max_num_insns);
>> + if (li.num_insns > max_num_insns)
>> + {
>> + if (dump_file && (dump_flags & TDF_DETAILS))
>> + fprintf (dump_file, ";; Not versioning loop %d (depth %d)"
>> + " because it is too big\n", loop->num, loop_depth (loop));
>> + return false;
>> + }
>> + }
>> +
>> + /* Hoist all version checks for subloops to this loop. */
>> + for (struct loop *subloop = loop->inner; subloop; subloop = subloop->next)
>> + merge_loop_info (loop, subloop);
>> +
>> + return true;
>> +}
>> +
>> +/* Decide which loops to version and add them to the versioning queue.
>> + Return true if there are any loops to version. */
>> +
>> +bool
>> +loop_versioning::make_versioning_decisions ()
>> +{
>> + struct loop *loop;
>> + FOR_EACH_LOOP (loop, LI_FROM_INNERMOST)
>> + {
>> + loop_info &linfo = get_loop_info (loop);
>> + if (decide_whether_loop_is_versionable (loop))
>> + {
>> + /* Commit to versioning LOOP directly if we can't hoist the
>> + version checks any further. */
>> + if (linfo.worth_versioning_p ()
>> + && (loop_depth (loop) == 1 || linfo.outermost == loop))
>> + add_loop_to_queue (loop);
>> + }
>> + else
>> + {
>> + /* We can't version this loop, so individually version any
>> + subloops that would benefit and haven't been versioned yet. */
>> + linfo.rejected_p = true;
>> + for (struct loop *subloop = loop->inner; subloop;
>> + subloop = subloop->next)
>> + if (get_loop_info (subloop).worth_versioning_p ())
>> + add_loop_to_queue (subloop);
>> + }
>> + }
>> +
>> + return !m_loops_to_version.is_empty ();
>> +}
>> +
>> +/* Attempt to implement loop versioning for LOOP, using the information
>> + cached in the associated loop_info. Return true on success. */
>> +
>> +bool
>> +loop_versioning::version_loop (struct loop *loop)
>> +{
>> + loop_info &li = get_loop_info (loop);
>> +
>> + /* Not protected by TDF_DETAILS since this is the main piece of
>> + information. */
>> + if (dump_file)
>> + fprintf (dump_file, ";; Versioning loop %d (depth %d)\n",
>> + loop->num, loop_depth (loop));
>> +
>> + /* Build up a condition that selects the original loop instead of
>> + the simplified loop. */
>> + tree cond = boolean_false_node;
>> + tree name;
>> + unsigned int i;
>> + FOR_EACH_VEC_ELT (li.unity_names, i, name)
>> + {
>> + tree ne_one = fold_build2 (NE_EXPR, boolean_type_node, name,
>> + build_one_cst (TREE_TYPE (name)));
>> + cond = fold_build2 (TRUTH_OR_EXPR, boolean_type_node, cond, ne_one);
>> + }
>> +
>> + /* Convert the condition into a suitable gcond. */
>> + gimple_seq stmts = NULL;
>> + cond = force_gimple_operand_1 (cond, &stmts, is_gimple_condexpr, NULL_TREE);
>> +
>> + /* Version the loop. */
>> + initialize_original_copy_tables ();
>> + basic_block cond_bb;
>> + struct loop *else_loop
>> + = loop_version (loop, cond, &cond_bb,
>> + profile_probability::unlikely (),
>> + profile_probability::likely (),
>> + profile_probability::unlikely (),
>> + profile_probability::likely (), true);
>> + free_original_copy_tables ();
>> + if (!else_loop)
>> + {
>> + if (dump_file)
>> + fprintf (dump_file, ";; Versioning of loop %d failed\n", loop->num);
>> + return false;
>> + }
>> +
>> + /* Insert the statements that feed COND. */
>> + if (stmts)
>> + {
>> + gimple_stmt_iterator gsi = gsi_last_bb (cond_bb);
>> + gsi_insert_seq_before (&gsi, stmts, GSI_SAME_STMT);
>> + }
>> +
>> + /* Simplify the new loop, which is used when COND is false. */
>> + name_prop (li).substitute_and_fold (else_loop->header);
>> + return true;
>> +}
>> +
>> +/* Attempt to version all loops in the versioning queue. Return true
>> + if we succeeded for at least one loop. */
>> +
>> +bool
>> +loop_versioning::implement_versioning_decisions ()
>> +{
>> + bool any_succeeded_p = false;
>> +
>> + struct loop *loop;
>> + unsigned int i;
>> + FOR_EACH_VEC_ELT (m_loops_to_version, i, loop)
>> + if (version_loop (loop))
>> + any_succeeded_p = true;
>> +
>> + return any_succeeded_p;
>> +}
>> +
>> +/* Run the pass and return a set of TODO_* flags. */
>> +
>> +unsigned int
>> +loop_versioning::run ()
>> +{
>> + gcc_assert (scev_initialized_p ());
>> +
>> + if (!analyze_blocks ()
>> + || !prune_conditions ()
>> + || !make_versioning_decisions ()
>> + || !implement_versioning_decisions ())
>> + return 0;
>> +
>> + return TODO_update_ssa;
>> +}
>> +
>> +/* Loop versioningting pass. */
>> +
>> +namespace {
>> +
>> +const pass_data pass_data_loop_versioning =
>> +{
>> + GIMPLE_PASS, /* type */
>> + "lversion", /* name */
>> + OPTGROUP_LOOP, /* optinfo_flags */
>> + TV_LOOP_VERSIONING, /* tv_id */
>> + PROP_cfg, /* properties_required */
>> + 0, /* properties_provided */
>> + 0, /* properties_destroyed */
>> + 0, /* todo_flags_start */
>> + 0, /* todo_flags_finish */
>> +};
>> +
>> +class pass_loop_versioning : public gimple_opt_pass
>> +{
>> +public:
>> + pass_loop_versioning (gcc::context *ctxt)
>> + : gimple_opt_pass (pass_data_loop_versioning, ctxt)
>> + {}
>> +
>> + /* opt_pass methods: */
>> + virtual bool gate (function *) { return flag_version_loops_for_strides; }
>> + virtual unsigned int execute (function *);
>> +};
>> +
>> +unsigned int
>> +pass_loop_versioning::execute (function *fn)
>> +{
>> + if (number_of_loops (fn) <= 1)
>> + return 0;
>> +
>> + return loop_versioning (fn).run ();
>> +}
>> +
>> +} // anon namespace
>> +
>> +gimple_opt_pass *
>> +make_pass_loop_versioning (gcc::context *ctxt)
>> +{
>> + return new pass_loop_versioning (ctxt);
>> +}
>> Index: gcc/testsuite/gcc.dg/loop-versioning-1.c
>> ===================================================================
>> --- /dev/null 2018-09-14 11:16:31.122530289 +0100
>> +++ gcc/testsuite/gcc.dg/loop-versioning-1.c 2018-10-24 14:02:15.184152693 +0100
>> @@ -0,0 +1,20 @@
>> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
>> +
>> +/* The simplest IV case. */
>> +
>> +void
>> +f1 (double *x, int stepx, int n)
>> +{
>> + for (int i = 0; i < n; ++i)
>> + x[stepx * i] = 100;
>> +}
>> +
>> +void
>> +f2 (double *x, int stepx, int limit)
>> +{
>> + for (int i = 0; i < limit; i += stepx)
>> + x[i] = 100;
>> +}
>> +
>> +/* { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 1\)} 2 "lversion" } } */
>> +/* { dg-final { scan-tree-dump-times {Versioning loop} 2 "lversion" } } */
>> Index: gcc/testsuite/gcc.dg/loop-versioning-10.c
>> ===================================================================
>> --- /dev/null 2018-09-14 11:16:31.122530289 +0100
>> +++ gcc/testsuite/gcc.dg/loop-versioning-10.c 2018-10-24 14:02:15.184152693 +0100
>> @@ -0,0 +1,17 @@
>> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
>> +
>> +/* Test that we can version a gather-like operation in which a variable
>> + stride is applied to the index. */
>> +
>> +int
>> +f1 (int *x, int *index, int step, int n)
>> +{
>> + int res = 0;
>> + for (int i = 0; i < n; ++i)
>> + res += x[index[i] * step];
>> + return res;
>> +}
>> +
>> +/* { dg-final { scan-tree-dump-times {Address[^\n]*invariant} 1 "lversion" } } */
>> +/* { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 1\)} 1 "lversion" } } */
>> +/* { dg-final { scan-tree-dump-times {Versioning loop} 1 "lversion" } } */
>> Index: gcc/testsuite/gcc.dg/loop-versioning-11.c
>> ===================================================================
>> --- /dev/null 2018-09-14 11:16:31.122530289 +0100
>> +++ gcc/testsuite/gcc.dg/loop-versioning-11.c 2018-10-24 14:02:15.188152659 +0100
>> @@ -0,0 +1,29 @@
>> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
>> +
>> +/* Test that we don't try to version for something that is never 1. */
>> +
>> +void
>> +f1 (double *x, int stepx, int n)
>> +{
>> + if (stepx == 1)
>> + for (int i = 0; i < n; ++i)
>> + x[i] = 100;
>> + else
>> + for (int i = 0; i < n; ++i)
>> + x[stepx * i] = 100;
>> +}
>> +
>> +void
>> +f2 (double *x, int stepx, int n)
>> +{
>> + if (stepx <= 1)
>> + for (int i = 0; i < n; ++i)
>> + x[i] = 100;
>> + else
>> + for (int i = 0; i < n; ++i)
>> + x[stepx * i] = 100;
>> +}
>> +
>> +/* { dg-final { scan-tree-dump-times {Want to version loop} 2 "lversion" } } */
>> +/* { dg-final { scan-tree-dump-times {can never be 1} 2 "lversion" } } */
>> +/* { dg-final { scan-tree-dump-not {Versioning loop} "lversion" } } */
>> Index: gcc/testsuite/gcc.dg/loop-versioning-2.c
>> ===================================================================
>> --- /dev/null 2018-09-14 11:16:31.122530289 +0100
>> +++ gcc/testsuite/gcc.dg/loop-versioning-2.c 2018-10-24 14:02:15.188152659 +0100
>> @@ -0,0 +1,39 @@
>> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
>> +
>> +/* Versioning these loops allows loop interchange. */
>> +
>> +void
>> +f1 (double x[][100], int step, int n)
>> +{
>> + for (int i = 0; i < n; ++i)
>> + for (int j = 0; j < n; ++j)
>> + x[j * step][i] = 100;
>> +}
>> +
>> +void
>> +f2 (double x[][100], int step, int n)
>> +{
>> + for (int i = 0; i < n; ++i)
>> + for (int j = 0; j < n; ++j)
>> + x[j][i * step] = 100;
>> +}
>> +
>> +void
>> +f3 (double x[][100], int step, int limit)
>> +{
>> + for (int i = 0; i < 100; ++i)
>> + for (int j = 0; j < limit; j += step)
>> + x[j][i] = 100;
>> +}
>> +
>> +void
>> +f4 (double x[][100], int step, int limit)
>> +{
>> + for (int i = 0; i < limit; i += step)
>> + for (int j = 0; j < 100; ++j)
>> + x[j][i] = 100;
>> +}
>> +
>> +/* { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 2\)} 4 "lversion" } } */
>> +/* { dg-final { scan-tree-dump-times {Hoisting check} 4 "lversion" } } */
>> +/* { dg-final { scan-tree-dump-times {Versioning loop [0-9]+ \(depth 1\)} 4 "lversion" } } */
>> Index: gcc/testsuite/gcc.dg/loop-versioning-3.c
>> ============================================================
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Add a loop versioning pass
2018-10-24 13:41 Richard Sandiford
@ 2018-10-25 14:16 ` Richard Biener
2018-10-25 16:03 ` Richard Sandiford
2018-10-26 14:49 ` Richard Biener
2018-10-25 16:16 ` David Malcolm
1 sibling, 2 replies; 17+ messages in thread
From: Richard Biener @ 2018-10-25 14:16 UTC (permalink / raw)
To: GCC Patches, Richard Sandiford
On Wed, Oct 24, 2018 at 3:05 PM Richard Sandiford
<richard.sandiford@arm.com> wrote:
>
> This patch adds a pass that versions loops with variable index strides
> for the case in which the stride is 1. E.g.:
>
> for (int i = 0; i < n; ++i)
> x[i * stride] = ...;
>
> becomes:
>
> if (stepx == 1)
> for (int i = 0; i < n; ++i)
> x[i] = ...;
> else
> for (int i = 0; i < n; ++i)
> x[i * stride] = ...;
>
> This is useful for both vector code and scalar code, and in some cases
> can enable further optimisations like loop interchange or pattern
> recognition.
>
> The pass gives a 7.6% improvement on Cortex-A72 for 554.roms_r at -O3
> and a 2.4% improvement for 465.tonto. I haven't found any SPEC tests
> that regress.
>
> Sizewise, there's a 10% increase in .text for both 554.roms_r and
> 465.tonto. That's obviously a lot, but in tonto's case it's because
> the whole program is written using assumed-shape arrays and pointers,
> so a large number of functions really do benefit from versioning.
> roms likewise makes heavy use of assumed-shape arrays, and that
> improvement in performance IMO justifies the code growth.
>
> The next biggest .text increase is 4.5% for 548.exchange2_r. I did see
> a small (0.4%) speed improvement there, but although both 3-iteration runs
> produced stable results, that might still be noise. There was a slightly
> larger (non-noise) improvement for a 256-bit SVE model.
>
> 481.wrf and 521.wrf_r .text grew by 2.8% and 2.5% respectively, but
> without any noticeable improvement in performance. No other test grew
> by more than 2%.
>
> Although the main SPEC beneficiaries are all Fortran tests, the
> benchmarks we use for SVE also include some C and C++ tests that
> benefit.
>
> Using -frepack-arrays gives the same benefits in many Fortran cases.
> The problem is that using that option inappropriately can force a full
> array copy for arguments that the function only reads once, and so it
> isn't really something we can turn on by default. The new pass is
> supposed to give most of the benefits of -frepack-arrays without
> the risk of unnecessary repacking.
>
> The patch therefore enables the pass by default at -O3.
>
> Tested on aarch64-linux-gnu and x86_64-linux-gnu. OK to install?
I will give this a thorough review tomorror (sorry for the delay), but
one question that comes up is how we avoid excessive versioning.
Consider this pass versioning for stride1, then loop distribution
versioning for aliasing and then vectorization versioning for alignment.
Do we have any idea on how to track these versionings so we can
eventually commonize paths that didn't get any followup optimization
applied?
Also consider the case where loop distribution applies transforms
with versioning for aliasing to _both_ loop nests from your versioning.
Maybe we can at least add some loop->number_of_times_versioned
counter and some dumping/statistics to get an idea how big an
issue this might be?
Thanks,
Richard.
> Richard
>
>
> 2018-10-24 Richard Sandiford <richard.sandiford@arm.com>
>
> gcc/
> * doc/invoke.texi (-fversion-loops-for-strides): Document
> (loop-versioning-group-size, loop-versioning-max-inner-insns)
> (loop-versioning-max-outer-insns): Document new --params.
> * Makefile.in (OBJS): Add gimple-loop-versioning.o.
> * common.opt (fversion-loops-for-strides): New option.
> * opts.c (default_options_table): Enable fversion-loops-for-strides
> at -O3.
> * params.def (PARAM_LOOP_VERSIONING_GROUP_SIZE)
> (PARAM_LOOP_VERSIONING_MAX_INNER_INSNS)
> (PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS): New parameters.
> * passes.def: Add pass_loop_versioning.
> * timevar.def (TV_LOOP_VERSIONING): New time variable.
> * tree-ssa-propagate.h
> (substitute_and_fold_engine::substitute_and_fold): Add an optional
> block parameter.
> * tree-ssa-propagate.c
> (substitute_and_fold_engine::substitute_and_fold): Likewise.
> When passed, only walk blocks dominated by that block.
> * tree-vrp.h (range_includes_p): Declare.
> (range_includes_zero_p): Turn into an inline wrapper around
> range_includes_p.
> * tree-vrp.c (range_includes_p): New function, generalizing...
> (range_includes_zero_p): ...this.
> * tree-pass.h (make_pass_loop_versioning): Declare.
> * gimple-loop-versioning.cc: New file.
>
> gcc/testsuite/
> * gcc.dg/loop-versioning-1.c: New test.
> * gcc.dg/loop-versioning-10.c: Likewise.
> * gcc.dg/loop-versioning-11.c: Likewise.
> * gcc.dg/loop-versioning-2.c: Likewise.
> * gcc.dg/loop-versioning-3.c: Likewise.
> * gcc.dg/loop-versioning-4.c: Likewise.
> * gcc.dg/loop-versioning-5.c: Likewise.
> * gcc.dg/loop-versioning-6.c: Likewise.
> * gcc.dg/loop-versioning-7.c: Likewise.
> * gcc.dg/loop-versioning-8.c: Likewise.
> * gcc.dg/loop-versioning-9.c: Likewise.
> * gfortran.dg/loop_versioning_1.f90: Likewise.
> * gfortran.dg/loop_versioning_2.f90: Likewise.
> * gfortran.dg/loop_versioning_3.f90: Likewise.
> * gfortran.dg/loop_versioning_4.f90: Likewise.
> * gfortran.dg/loop_versioning_5.f90: Likewise.
> * gfortran.dg/loop_versioning_6.f90: Likewise.
> * gfortran.dg/loop_versioning_7.f90: Likewise.
> * gfortran.dg/loop_versioning_8.f90: Likewise.
>
> Index: gcc/doc/invoke.texi
> ===================================================================
> --- gcc/doc/invoke.texi 2018-10-24 14:02:14.000000000 +0100
> +++ gcc/doc/invoke.texi 2018-10-24 14:02:15.184152693 +0100
> @@ -7934,7 +7934,8 @@ by @option{-O2} and also turns on the fo
> -fvect-cost-model @gol
> -ftree-partial-pre @gol
> -fpeel-loops @gol
> --fipa-cp-clone}
> +-fipa-cp-clone @gol
> +-fversion-loops-for-strides}
>
> @item -O0
> @opindex O0
> @@ -10358,6 +10359,29 @@ for one side of the iteration space and
> Move branches with loop invariant conditions out of the loop, with duplicates
> of the loop on both branches (modified according to result of the condition).
>
> +@item -fversion-loops-for-strides
> +@opindex fversion-loops-for-strides
> +If a loop iterates over an array with a variable stride, create another
> +version of the loop that assumes the stride is always 1. For example:
> +
> +@smallexample
> +for (int i = 0; i < n; ++i)
> + x[i * stride] = @dots{};
> +@end smallexample
> +
> +becomes:
> +
> +@smallexample
> +if (stride == 1)
> + for (int i = 0; i < n; ++i)
> + x[i] = @dots{};
> +else
> + for (int i = 0; i < n; ++i)
> + x[i * stride] = @dots{};
> +@end smallexample
> +
> +This is particularly useful for assumed-shape arrays in Fortran.
> +
> @item -ffunction-sections
> @itemx -fdata-sections
> @opindex ffunction-sections
> @@ -11567,6 +11591,20 @@ Hardware autoprefetcher scheduler model
> Number of lookahead cycles the model looks into; at '
> ' only enable instruction sorting heuristic.
>
> +@item loop-versioning-group-size
> +Make the loop versioning pass optimize @samp{a[i * index * @var{N}]}
> +in the same way as it would optimize @samp{a[i * index]} when @var{N}
> +is less than or equal to this value.
> +
> +@item loop-versioning-max-inner-insns
> +The maximum number of instructions that an inner loop can have
> +before the loop versioning pass considers it too big to copy.
> +
> +@item loop-versioning-max-outer-insns
> +The maximum number of instructions that an outer loop can have
> +before the loop versioning pass considers it too big to copy,
> +discounting any instructions in inner loops that directly benefit
> +from versioning.
>
> @end table
> @end table
> Index: gcc/Makefile.in
> ===================================================================
> --- gcc/Makefile.in 2018-10-24 14:02:14.000000000 +0100
> +++ gcc/Makefile.in 2018-10-24 14:02:15.180152727 +0100
> @@ -1312,6 +1312,7 @@ OBJS = \
> gimple-laddress.o \
> gimple-loop-interchange.o \
> gimple-loop-jam.o \
> + gimple-loop-versioning.o \
> gimple-low.o \
> gimple-pretty-print.o \
> gimple-ssa-backprop.o \
> Index: gcc/common.opt
> ===================================================================
> --- gcc/common.opt 2018-10-24 14:02:14.000000000 +0100
> +++ gcc/common.opt 2018-10-24 14:02:15.180152727 +0100
> @@ -2712,6 +2712,10 @@ fsplit-loops
> Common Report Var(flag_split_loops) Optimization
> Perform loop splitting.
>
> +fversion-loops-for-strides
> +Common Report Var(flag_version_loops_for_strides) Optimization
> +Version loops based on whether indices have a stride of 1.
> +
> funwind-tables
> Common Report Var(flag_unwind_tables) Optimization
> Just generate unwind tables for exception handling.
> Index: gcc/opts.c
> ===================================================================
> --- gcc/opts.c 2018-10-24 14:02:14.000000000 +0100
> +++ gcc/opts.c 2018-10-24 14:02:15.184152693 +0100
> @@ -544,6 +544,7 @@ static const struct default_options defa
> { OPT_LEVELS_3_PLUS, OPT_fipa_cp_clone, NULL, 1 },
> { OPT_LEVELS_3_PLUS, OPT_ftree_partial_pre, NULL, 1 },
> { OPT_LEVELS_3_PLUS, OPT_fpeel_loops, NULL, 1 },
> + { OPT_LEVELS_3_PLUS, OPT_fversion_loops_for_strides, NULL, 1 },
>
> /* -Ofast adds optimizations to -O3. */
> { OPT_LEVELS_FAST, OPT_ffast_math, NULL, 1 },
> Index: gcc/params.def
> ===================================================================
> --- gcc/params.def 2018-10-24 14:02:14.000000000 +0100
> +++ gcc/params.def 2018-10-24 14:02:15.184152693 +0100
> @@ -1360,6 +1360,25 @@ DEFPARAM(PARAM_AVOID_FMA_MAX_BITS,
> "Maximum number of bits for which we avoid creating FMAs.",
> 0, 0, 512)
>
> +DEFPARAM(PARAM_LOOP_VERSIONING_GROUP_SIZE,
> + "loop-versioning-group-size",
> + "The maximum constant N for which accesses of the form x[N * step]"
> + " are worth versioning for the case in which step is 1",
> + 4, 1, 0)
> +
> +DEFPARAM(PARAM_LOOP_VERSIONING_MAX_INNER_INSNS,
> + "loop-versioning-max-inner-insns",
> + "The maximum number of instructions in an inner loop that is being"
> + " considered for versioning",
> + 200, 0, 0)
> +
> +DEFPARAM(PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS,
> + "loop-versioning-max-outer-insns",
> + "The maximum number of instructions in an outer loop that is being"
> + " considered for versioning, on top of the instructions in inner"
> + " loops",
> + 100, 0, 0)
> +
> /*
>
> Local variables:
> Index: gcc/passes.def
> ===================================================================
> --- gcc/passes.def 2018-10-24 14:02:14.000000000 +0100
> +++ gcc/passes.def 2018-10-24 14:02:15.184152693 +0100
> @@ -260,6 +260,7 @@ along with GCC; see the file COPYING3.
> NEXT_PASS (pass_tree_loop);
> PUSH_INSERT_PASSES_WITHIN (pass_tree_loop)
> NEXT_PASS (pass_tree_loop_init);
> + NEXT_PASS (pass_loop_versioning);
> NEXT_PASS (pass_tree_unswitch);
> NEXT_PASS (pass_scev_cprop);
> NEXT_PASS (pass_loop_split);
> Index: gcc/timevar.def
> ===================================================================
> --- gcc/timevar.def 2018-10-24 14:02:14.000000000 +0100
> +++ gcc/timevar.def 2018-10-24 14:02:15.188152659 +0100
> @@ -234,6 +234,7 @@ DEFTIMEVAR (TV_DSE1 , "
> DEFTIMEVAR (TV_DSE2 , "dead store elim2")
> DEFTIMEVAR (TV_LOOP , "loop analysis")
> DEFTIMEVAR (TV_LOOP_INIT , "loop init")
> +DEFTIMEVAR (TV_LOOP_VERSIONING , "loop versioning")
> DEFTIMEVAR (TV_LOOP_MOVE_INVARIANTS , "loop invariant motion")
> DEFTIMEVAR (TV_LOOP_UNROLL , "loop unrolling")
> DEFTIMEVAR (TV_LOOP_DOLOOP , "loop doloop")
> Index: gcc/tree-ssa-propagate.h
> ===================================================================
> --- gcc/tree-ssa-propagate.h 2018-10-24 14:02:14.000000000 +0100
> +++ gcc/tree-ssa-propagate.h 2018-10-24 14:02:15.188152659 +0100
> @@ -104,7 +104,7 @@ extern void propagate_tree_value_into_st
> virtual bool fold_stmt (gimple_stmt_iterator *) { return false; }
> virtual tree get_value (tree) { return NULL_TREE; }
>
> - bool substitute_and_fold (void);
> + bool substitute_and_fold (basic_block = NULL);
> bool replace_uses_in (gimple *);
> bool replace_phi_args_in (gphi *);
> };
> Index: gcc/tree-ssa-propagate.c
> ===================================================================
> --- gcc/tree-ssa-propagate.c 2018-10-24 14:02:14.000000000 +0100
> +++ gcc/tree-ssa-propagate.c 2018-10-24 14:02:15.188152659 +0100
> @@ -1152,6 +1152,10 @@ substitute_and_fold_dom_walker::before_d
>
>
> /* Perform final substitution and folding of propagated values.
> + Process the whole function if BLOCK is null, otherwise only
> + process the blocks that BLOCK dominates. In the latter case,
> + it is the caller's responsibility to ensure that dominator
> + information is available and up-to-date.
>
> PROP_VALUE[I] contains the single value that should be substituted
> at every use of SSA name N_I. If PROP_VALUE is NULL, no values are
> @@ -1168,16 +1172,24 @@ substitute_and_fold_dom_walker::before_d
> Return TRUE when something changed. */
>
> bool
> -substitute_and_fold_engine::substitute_and_fold (void)
> +substitute_and_fold_engine::substitute_and_fold (basic_block block)
> {
> if (dump_file && (dump_flags & TDF_DETAILS))
> fprintf (dump_file, "\nSubstituting values and folding statements\n\n");
>
> memset (&prop_stats, 0, sizeof (prop_stats));
>
> - calculate_dominance_info (CDI_DOMINATORS);
> + /* Don't call calculate_dominance_info when iterating over a subgraph.
> + Callers that are using the interface this way are likely to want to
> + iterate over several disjoint subgraphs, and it would be expensive
> + in enable-checking builds to revalidate the whole dominance tree
> + each time. */
> + if (block)
> + gcc_assert (dom_info_state (CDI_DOMINATORS));
> + else
> + calculate_dominance_info (CDI_DOMINATORS);
> substitute_and_fold_dom_walker walker (CDI_DOMINATORS, this);
> - walker.walk (ENTRY_BLOCK_PTR_FOR_FN (cfun));
> + walker.walk (block ? block : ENTRY_BLOCK_PTR_FOR_FN (cfun));
>
> /* We cannot remove stmts during the BB walk, especially not release
> SSA names there as that destroys the lattice of our callers.
> Index: gcc/tree-vrp.h
> ===================================================================
> --- gcc/tree-vrp.h 2018-10-24 14:02:14.000000000 +0100
> +++ gcc/tree-vrp.h 2018-10-24 14:02:15.188152659 +0100
> @@ -86,7 +86,7 @@ extern void register_edge_assert_for (tr
> tree, tree, vec<assert_info> &);
> extern bool stmt_interesting_for_vrp (gimple *);
> extern void set_value_range_to_varying (value_range *);
> -extern bool range_includes_zero_p (const value_range *);
> +extern bool range_includes_p (const value_range *, HOST_WIDE_INT);
> extern bool infer_value_range (gimple *, tree, tree_code *, tree *);
>
> extern void set_value_range_to_nonnull (value_range *, tree);
> @@ -122,4 +122,13 @@ extern int value_inside_range (tree, tre
> extern tree get_single_symbol (tree, bool *, tree *);
> extern void maybe_set_nonzero_bits (edge, tree);
> extern value_range_type determine_value_range (tree, wide_int *, wide_int *);
> +
> +/* Return TRUE if *VR includes the value zero. */
> +
> +inline bool
> +range_includes_zero_p (const value_range *vr)
> +{
> + return range_includes_p (vr, 0);
> +}
> +
> #endif /* GCC_TREE_VRP_H */
> Index: gcc/tree-vrp.c
> ===================================================================
> --- gcc/tree-vrp.c 2018-10-24 14:02:14.000000000 +0100
> +++ gcc/tree-vrp.c 2018-10-24 14:02:15.188152659 +0100
> @@ -844,10 +844,10 @@ value_inside_range (tree val, tree min,
> }
>
>
> -/* Return TRUE if *VR includes the value zero. */
> +/* Return TRUE if *VR includes the value X. */
>
> bool
> -range_includes_zero_p (const value_range *vr)
> +range_includes_p (const value_range *vr, HOST_WIDE_INT x)
> {
> if (vr->type == VR_VARYING)
> return true;
> @@ -856,13 +856,13 @@ range_includes_zero_p (const value_range
> if (vr->type == VR_UNDEFINED)
> return true;
>
> - tree zero = build_int_cst (TREE_TYPE (vr->min), 0);
> + tree x_cst = build_int_cst (TREE_TYPE (vr->min), x);
> if (vr->type == VR_ANTI_RANGE)
> {
> - int res = value_inside_range (zero, vr->min, vr->max);
> + int res = value_inside_range (x_cst, vr->min, vr->max);
> return res == 0 || res == -2;
> }
> - return value_inside_range (zero, vr->min, vr->max) != 0;
> + return value_inside_range (x_cst, vr->min, vr->max) != 0;
> }
>
> /* If *VR has a value rante that is a single constant value return that,
> Index: gcc/tree-pass.h
> ===================================================================
> --- gcc/tree-pass.h 2018-10-24 14:02:14.000000000 +0100
> +++ gcc/tree-pass.h 2018-10-24 14:02:15.188152659 +0100
> @@ -362,6 +362,7 @@ extern gimple_opt_pass *make_pass_fix_lo
> extern gimple_opt_pass *make_pass_tree_loop (gcc::context *ctxt);
> extern gimple_opt_pass *make_pass_tree_no_loop (gcc::context *ctxt);
> extern gimple_opt_pass *make_pass_tree_loop_init (gcc::context *ctxt);
> +extern gimple_opt_pass *make_pass_loop_versioning (gcc::context *ctxt);
> extern gimple_opt_pass *make_pass_lim (gcc::context *ctxt);
> extern gimple_opt_pass *make_pass_linterchange (gcc::context *ctxt);
> extern gimple_opt_pass *make_pass_tree_unswitch (gcc::context *ctxt);
> Index: gcc/gimple-loop-versioning.cc
> ===================================================================
> --- /dev/null 2018-09-14 11:16:31.122530289 +0100
> +++ gcc/gimple-loop-versioning.cc 2018-10-24 14:02:15.184152693 +0100
> @@ -0,0 +1,1417 @@
> +/* Loop versioning pass.
> + Copyright (C) 2018 Free Software Foundation, Inc.
> +
> +This file is part of GCC.
> +
> +GCC is free software; you can redistribute it and/or modify it
> +under the terms of the GNU General Public License as published by the
> +Free Software Foundation; either version 3, or (at your option) any
> +later version.
> +
> +GCC is distributed in the hope that it will be useful, but WITHOUT
> +ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
> +FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
> +for more details.
> +
> +You should have received a copy of the GNU General Public License
> +along with GCC; see the file COPYING3. If not see
> +<http://www.gnu.org/licenses/>. */
> +
> +#include "config.h"
> +#include "system.h"
> +#include "coretypes.h"
> +#include "backend.h"
> +#include "tree.h"
> +#include "gimple.h"
> +#include "gimple-iterator.h"
> +#include "tree-pass.h"
> +#include "gimplify-me.h"
> +#include "cfgloop.h"
> +#include "tree-ssa-loop.h"
> +#include "ssa.h"
> +#include "tree-scalar-evolution.h"
> +#include "tree-chrec.h"
> +#include "tree-ssa-loop-ivopts.h"
> +#include "fold-const.h"
> +#include "tree-ssa-propagate.h"
> +#include "tree-inline.h"
> +#include "domwalk.h"
> +#include "alloc-pool.h"
> +#include "vr-values.h"
> +#include "gimple-ssa-evrp-analyze.h"
> +#include "gimple-pretty-print.h"
> +#include "params.h"
> +
> +/* This pass looks for loops that could be simplified if certain loop
> + invariant conditions were true. It is effectively a form of loop
> + splitting in which the pass produces the split conditions itself,
> + instead of using ones that are already present in the IL.
> +
> + Versioning for when strides are 1
> + ---------------------------------
> +
> + At the moment the only thing the pass looks for are memory references
> + like:
> +
> + for (auto i : ...)
> + ...x[i * stride]...
> +
> + It considers changing such loops to:
> +
> + if (stride == 1)
> + for (auto i : ...) [A]
> + ...x[i]...
> + else
> + for (auto i : ...) [B]
> + ...x[i * stride]...
> +
> + This can have several benefits:
> +
> + (1) [A] is often easier or cheaper to vectorize than [B].
> +
> + (2) The scalar code in [A] is simpler than the scalar code in [B]
> + (if the loops cannot be vectorized or need an epilogue loop).
> +
> + (3) We might recognize [A] as a pattern, such as a memcpy or memset.
> +
> + (4) [A] has simpler address evolutions, which can help other passes
> + like loop interchange.
> +
> + The optimization is particularly useful for assumed-shape arrays in
> + Fortran, where the stride of the innermost dimension depends on the
> + array descriptor but is often equal to 1 in practice. For example:
> +
> + subroutine f1(x)
> + real :: x(:)
> + x(:) = 100
> + end subroutine f1
> +
> + generates the equivalent of:
> +
> + raw_stride = *x.dim[0].stride;
> + stride = raw_stride != 0 ? raw_stride : 1;
> + x_base = *x.data;
> + ...
> + tmp1 = stride * S;
> + tmp2 = tmp1 - stride;
> + *x_base[tmp2] = 1.0e+2;
> +
> + but in the common case that stride == 1, the last three statements
> + simplify to:
> +
> + tmp3 = S + -1;
> + *x_base[tmp3] = 1.0e+2;
> +
> + The optimization is in principle very simple. The difficult parts are:
> +
> + (a) deciding which parts of a general address calculation correspond
> + to the inner dimension of an array, since this usually isn't explicit
> + in the IL, and for C often isn't even explicit in the source code
> +
> + (b) estimating when the transformation is worthwhile
> +
> + Structure
> + ---------
> +
> + The pass has four phases:
> +
> + (1) Walk through the statements looking for and recording potential
> + versioning opportunities. Stop if there are none.
> +
> + (2) Use context-sensitive range information to see whether any versioning
> + conditions are impossible in practice. Remove them if so, and stop
> + if no opportunities remain.
> +
> + (We do this only after (1) to keep compile time down when no
> + versioning opportunities exist.)
> +
> + (3) Apply the cost model. Decide which versioning opportunities are
> + worthwhile and at which nesting level they should be applied.
> +
> + (4) Attempt to version all the loops selected by (3), so that:
> +
> + for (...)
> + ...
> +
> + becomes:
> +
> + if (!cond)
> + for (...) // Original loop
> + ...
> + else
> + for (...) // New loop
> + ...
> +
> + Use the version condition COND to simplify the new loop. */
> +class loop_versioning {
> +public:
> + loop_versioning (function *);
> + ~loop_versioning ();
> + unsigned int run ();
> +
> +private:
> + /* Information about the versioning we'd like to apply to a loop. */
> + struct loop_info {
> + bool worth_versioning_p () const;
> +
> + /* True if we've decided not to version this loop. The remaining
> + fields are meaningless if so. */
> + bool rejected_p;
> +
> + /* True if at least one subloop of this loop benefits from versioning. */
> + bool subloops_benefit_p;
> +
> + /* An estimate of the total number of instructions in the loop,
> + excluding those in subloops that benefit from versioning. */
> + unsigned int num_insns;
> +
> + /* The outermost loop that can handle all the version checks
> + described below. */
> + struct loop *outermost;
> +
> + /* We'd like to version the loop for the case in which these
> + SSA_NAMEs are all equal to 1 at runtime. */
> + vec<tree> unity_names;
> +
> + /* The set of SSA_NAMEs in UNITY_NAMES, keyed off the SSA_NAME_VERSION. */
> + bitmap_head unity_name_ids;
> + };
> +
> + /* Used to walk the dominator tree to find loop versioning conditions
> + that are always false. */
> + class lv_dom_walker : public dom_walker
> + {
> + public:
> + lv_dom_walker (loop_versioning &);
> +
> + edge before_dom_children (basic_block) FINAL OVERRIDE;
> + void after_dom_children (basic_block) FINAL OVERRIDE;
> +
> + private:
> + /* The parent pass. */
> + loop_versioning &m_lv;
> +
> + /* Used to build context-dependent range information. */
> + evrp_range_analyzer m_range_analyzer;
> + };
> +
> + /* Used to simplify statements based on conditions that are established
> + by the version checks. */
> + class name_prop : public substitute_and_fold_engine
> + {
> + public:
> + name_prop (loop_info &li) : m_li (li) {}
> + tree get_value (tree) FINAL OVERRIDE;
> +
> + private:
> + /* Information about the versioning we've performed on the loop. */
> + loop_info &m_li;
> + };
> +
> + loop_info &get_loop_info (struct loop *loop) { return m_loops[loop->num]; }
> +
> + unsigned int max_insns_for_loop (struct loop *);
> + bool expensive_stmt_p (gimple *);
> +
> + void version_for_unity (struct loop *, tree);
> + bool acceptable_scale_p (tree, poly_uint64);
> + tree get_step_if_innermost (tree, poly_uint64);
> + tree extract_step (tree, poly_uint64, tree *);
> + void analyze_evolution (struct loop *, tree, poly_uint64);
> + bool analyze_product (struct loop *, gassign *, poly_uint64);
> + bool analyze_sum_of_products (struct loop *, tree, poly_uint64);
> + void analyze_pointer (struct loop *, tree, tree);
> + void analyze_expr (struct loop *, tree);
> + void analyze_stmt (gimple *);
> + void analyze_block (basic_block);
> + bool analyze_blocks ();
> +
> + void prune_loop_conditions (struct loop *, vr_values *);
> + bool prune_conditions ();
> +
> + void merge_loop_info (struct loop *, struct loop *);
> + void add_loop_to_queue (struct loop *);
> + bool decide_whether_loop_is_versionable (struct loop *);
> + bool make_versioning_decisions ();
> +
> + bool version_loop (struct loop *);
> + bool implement_versioning_decisions ();
> +
> + /* The function we're optimizing. */
> + function *m_fn;
> +
> + /* The obstack to use for all pass-specific bitmaps. */
> + bitmap_obstack m_obstack;
> +
> + /* The number of loops in the function. */
> + unsigned int m_nloops;
> +
> + /* The total number of loop version conditions we've found. */
> + unsigned int m_num_conditions;
> +
> + /* Information about each loop. */
> + auto_vec<loop_info> m_loops;
> +
> + /* The list of loops that we've decided to version. */
> + auto_vec<struct loop *> m_loops_to_version;
> +};
> +
> +/* If EXPR is an SSA name and not a default definition, return the
> + defining statement, otherwise return null. */
> +
> +static gimple *
> +maybe_get_stmt (tree expr)
> +{
> + if (TREE_CODE (expr) == SSA_NAME && !SSA_NAME_IS_DEFAULT_DEF (expr))
> + return SSA_NAME_DEF_STMT (expr);
> + return NULL;
> +}
> +
> +/* Like maybe_get_stmt, but also return null if the defining
> + statement isn't an assignment. */
> +
> +static gassign *
> +maybe_get_assign (tree expr)
> +{
> + return safe_dyn_cast <gassign *> (maybe_get_stmt (expr));
> +}
> +
> +/* If EXPR is an SSA name, look through any casts to see whether the
> + unconverted value is defined in LOOP by a gassign. Return the
> + gassign if so, otherwise return null. */
> +
> +gassign *
> +maybe_get_assign_strip_casts (struct loop *loop, tree expr)
> +{
> + const unsigned int MAX_NITERS = 4;
> +
> + tree type = TREE_TYPE (expr);
> + for (unsigned int niters = 0; niters < MAX_NITERS; ++niters)
> + {
> + gassign *assign = maybe_get_assign (expr);
> + if (!assign || gimple_bb (assign)->loop_father != loop)
> + return NULL;
> + expr = gimple_assign_rhs1 (assign);
> + if (CONVERT_EXPR_CODE_P (gimple_assign_rhs_code (assign))
> + && INTEGRAL_TYPE_P (TREE_TYPE (expr)) == INTEGRAL_TYPE_P (type)
> + && POINTER_TYPE_P (TREE_TYPE (expr)) == POINTER_TYPE_P (type))
> + ;
> + else
> + return assign;
> + }
> + return NULL;
> +}
> +
> +/* Strip all conversions of integers from EXPR, regardless of whether
> + the conversions are nops. This is useful in the context of this pass
> + because we're not trying to fold or simulate the expression; we just
> + want to see how it's structured. */
> +
> +static tree
> +strip_casts (tree expr)
> +{
> + while (CONVERT_EXPR_P (expr)
> + && INTEGRAL_TYPE_P (TREE_TYPE (TREE_OPERAND (expr, 0))))
> + expr = TREE_OPERAND (expr, 0);
> + return expr;
> +}
> +
> +/* Return true if we want to version the loop, i.e. if we have a
> + specific reason for doing so and no specific reason not to. */
> +
> +bool
> +loop_versioning::loop_info::worth_versioning_p () const
> +{
> + return !rejected_p && (!unity_names.is_empty () || subloops_benefit_p);
> +}
> +
> +loop_versioning::lv_dom_walker::lv_dom_walker (loop_versioning &lv)
> + : dom_walker (CDI_DOMINATORS), m_lv (lv)
> +{
> +}
> +
> +/* Process BB before processing the blocks it dominates. */
> +
> +edge
> +loop_versioning::lv_dom_walker::before_dom_children (basic_block bb)
> +{
> + m_range_analyzer.enter (bb);
> +
> + if (bb == bb->loop_father->header)
> + m_lv.prune_loop_conditions (bb->loop_father,
> + m_range_analyzer.get_vr_values ());
> +
> + for (gimple_stmt_iterator si = gsi_start_bb (bb); !gsi_end_p (si);
> + gsi_next (&si))
> + m_range_analyzer.record_ranges_from_stmt (gsi_stmt (si), false);
> +
> + return NULL;
> +}
> +
> +/* Process BB after processing the blocks it dominates. */
> +
> +void
> +loop_versioning::lv_dom_walker::after_dom_children (basic_block bb)
> +{
> + m_range_analyzer.leave (bb);
> +}
> +
> +/* Decide whether to replace VAL with a new value in a versioned loop.
> + Return the new value if so, otherwise return null. */
> +
> +tree
> +loop_versioning::name_prop::get_value (tree val)
> +{
> + if (TREE_CODE (val) == SSA_NAME
> + && bitmap_bit_p (&m_li.unity_name_ids, SSA_NAME_VERSION (val)))
> + return build_one_cst (TREE_TYPE (val));
> + return NULL_TREE;
> +}
> +
> +/* Initialize the structure to optimize FN. */
> +
> +loop_versioning::loop_versioning (function *fn)
> + : m_fn (fn),
> + m_nloops (number_of_loops (fn)),
> + m_num_conditions (0)
> +{
> + bitmap_obstack_initialize (&m_obstack);
> +
> + m_loops.safe_grow_cleared (m_nloops);
> + for (unsigned int i = 0; i < m_nloops; ++i)
> + {
> + m_loops[i].outermost = get_loop (m_fn, 0);
> + bitmap_initialize (&m_loops[i].unity_name_ids, &m_obstack);
> + }
> +}
> +
> +loop_versioning::~loop_versioning ()
> +{
> + for (unsigned int i = 0; i < m_nloops; ++i)
> + m_loops[i].unity_names.release ();
> + bitmap_obstack_release (&m_obstack);
> +}
> +
> +/* Return the maximum number of instructions allowed in LOOP before
> + it becomes too big for versioning.
> +
> + There are separate limits for inner and outer loops. The limit for
> + inner loops applies only to loops that benefit directly from versioning.
> + The limit for outer loops applies to all code in the outer loop and
> + its subloops that *doesn't* benefit directly from versioning; such code
> + would be "taken along for the ride". The idea is that if the cost of
> + the latter is small, it is better to version outer loops rather than
> + inner loops, both to reduce the number of repeated checks and to enable
> + more of the loop nest to be optimized as a natural nest (e.g. by loop
> + interchange or outer-loop vectorization). */
> +
> +unsigned int
> +loop_versioning::max_insns_for_loop (struct loop *loop)
> +{
> + return (loop->inner
> + ? PARAM_VALUE (PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS)
> + : PARAM_VALUE (PARAM_LOOP_VERSIONING_MAX_INNER_INSNS));
> +}
> +
> +/* Return true if for cost reasons we should avoid versioning any loop
> + that contains STMT.
> +
> + Note that we don't need to check whether versioning is invalid for
> + correctness reasons, since the versioning process does that for us.
> + The conditions involved are too rare to be worth duplicating here. */
> +
> +bool
> +loop_versioning::expensive_stmt_p (gimple *stmt)
> +{
> + if (gcall *call = dyn_cast <gcall *> (stmt))
> + /* Assume for now that the time spent in an "expensive" call would
> + overwhelm any saving from versioning. */
> + return !gimple_inexpensive_call_p (call);
> + return false;
> +}
> +
> +/* Record that we want to version LOOP for the case in which SSA name NAME
> + is equal to 1. We already know that NAME is invariant in LOOP. */
> +
> +void
> +loop_versioning::version_for_unity (struct loop *loop, tree name)
> +{
> + loop_info &li = get_loop_info (loop);
> +
> + if (bitmap_set_bit (&li.unity_name_ids, SSA_NAME_VERSION (name)))
> + {
> + /* This is the first time we've wanted to version LOOP for NAME. */
> + li.unity_names.safe_push (name);
> +
> + /* Keep track of the outermost loop that can handle all versioning
> + checks in LI. */
> + struct loop *outermost
> + = outermost_invariant_loop_for_expr (loop, name);
> + if (loop_depth (li.outermost) < loop_depth (outermost))
> + li.outermost = outermost;
> +
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + {
> + fprintf (dump_file, ";; Want to version loop %d (depth %d)"
> + " for when ", loop->num, loop_depth (loop));
> + print_generic_expr (dump_file, name, TDF_SLIM);
> + fprintf (dump_file, " == 1");
> + if (outermost == loop)
> + fprintf (dump_file, "; cannot hoist check further");
> + else
> + {
> + fprintf (dump_file, "; could hoist check to loop %d (depth %d)",
> + outermost->num, loop_depth (outermost));
> + if (loop_depth (li.outermost) > loop_depth (outermost))
> + fprintf (dump_file, ", but that's further than"
> + " other checks allow");
> + }
> + fprintf (dump_file, "\n");
> + }
> +
> + m_num_conditions += 1;
> + }
> + else
> + {
> + /* This is a duplicate request. */
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + {
> + fprintf (dump_file, ";; Already want to version loop for when ");
> + print_generic_expr (dump_file, name, TDF_SLIM);
> + fprintf (dump_file, " == 1\n");
> + }
> + }
> +}
> +
> +/* Return true if in principle it is worth versioning an index fragment of
> + the form:
> +
> + (i * b * SCALE) / FACTOR
> +
> + for the case in which b == 1. */
> +
> +bool
> +loop_versioning::acceptable_scale_p (tree scale, poly_uint64 factor)
> +{
> + /* See whether SCALE is a constant multiple of FACTOR, and if the
> + multiple is small enough for us to treat it as a potential grouped
> + access. For example:
> +
> + for (auto i : ...)
> + y[i] = f (x[4 * i * stride],
> + x[4 * i * stride + 1],
> + x[4 * i * stride + 2]);
> +
> + would benefit from versioning for the case in which stride == 1.
> + High multiples of i * stride are less likely to benefit, and could
> + indicate a simulated multi-dimensional array.
> +
> + This is just a heuristic, to avoid having to do expensive group
> + analysis of the data references in a loop. */
> + poly_uint64 const_scale;
> + unsigned int multiple;
> + if (poly_int_tree_p (scale, &const_scale)
> + && constant_multiple_p (const_scale, factor, &multiple))
> + {
> + unsigned int maxval = PARAM_VALUE (PARAM_LOOP_VERSIONING_GROUP_SIZE);
> + return IN_RANGE (multiple, 1, maxval);
> + }
> +
> + return false;
> +}
> +
> +/* Decide whether an index fragment of the form:
> +
> + (i * STEP) / FACTOR
> +
> + is likely to be for an innermost dimension. If we think it is,
> + return one of the constant values that it could have (returning 1 if
> + that's a possibility). If think it isn't, return null. Otherwise
> + return STEP, to indicate that it might or might not be an inner
> + dimension. */
> +
> +tree
> +loop_versioning::get_step_if_innermost (tree step, poly_uint64 factor)
> +{
> + const unsigned int MAX_NITERS = 8;
> +
> + tree likely = NULL_TREE;
> + tree unlikely = NULL_TREE;
> + tree worklist[MAX_NITERS];
> + unsigned int length = 0;
> + worklist[length++] = step;
> + for (unsigned int i = 0; i < length; ++i)
> + {
> + tree expr = worklist[i];
> +
> + if (TREE_CONSTANT (expr))
> + {
> + /* See if multiplying by EXPR applies a scale that would be
> + consistent with an individual access or a small grouped
> + access. */
> + if (acceptable_scale_p (expr, factor))
> + {
> + likely = expr;
> + if (integer_onep (expr))
> + break;
> + }
> + else
> + unlikely = expr;
> + continue;
> + }
> +
> + /* Otherwise we can only handle SSA names. */
> + gimple *stmt = maybe_get_stmt (expr);
> + if (!stmt)
> + continue;
> +
> + /* If EXPR is set by a PHI node, queue its arguments in case
> + we find one that is consistent with an inner dimension.
> +
> + An important instance of this is the Fortran handling of array
> + descriptors, which calculates the stride of the inner dimension
> + using a PHI equivalent of:
> +
> + raw_stride = a.dim[0].stride;
> + stride = raw_stride != 0 ? raw_stride : 1;
> +
> + (Strides for outer dimensions do not treat 0 specially.) */
> + if (gphi *phi = dyn_cast <gphi *> (stmt))
> + {
> + unsigned int nargs = gimple_phi_num_args (phi);
> + for (unsigned int j = 0; j < nargs && length < MAX_NITERS; ++j)
> + worklist[length++] = gimple_phi_arg_def (phi, j);
> + continue;
> + }
> +
> + /* If the value is set by an assignment, expect it to be read from
> + memory (such as an array descriptor) rather than be calculated. */
> + if (gassign *assign = dyn_cast <gassign *> (stmt))
> + {
> + if (gimple_assign_single_p (assign)
> + && is_gimple_lvalue (gimple_assign_rhs1 (assign)))
> + continue;
> +
> + unlikely = expr;
> + }
> +
> + /* Things like calls don't really tell us anything. */
> + }
> +
> + if (likely)
> + {
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + fprintf (dump_file, "likely to be innermost dimension\n");
> + return likely;
> + }
> +
> + if (unlikely)
> + {
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + fprintf (dump_file, "probably not innermost dimension\n");
> + return NULL_TREE;
> + }
> +
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + {
> + if (maybe_ne (factor, 1U))
> + fprintf (dump_file, "didn't find expected scaling factor\n");
> + else
> + fprintf (dump_file, "no information about value\n");
> + }
> + return step;
> +}
> +
> +/* STEP appears in an index fragment of the form:
> +
> + {..., +, STEP}_n / FACTOR
> +
> + Remove any conversions and grouping or scaling factors from STEP and
> + return the underlying index step. Set *STEP_IF_INNERMOST to the value
> + returned by get_step_if_innermost. */
> +
> +tree
> +loop_versioning::extract_step (tree step, poly_uint64 factor,
> + tree *step_if_innermost)
> +{
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + {
> + fprintf (dump_file, ";; step ");
> + print_generic_expr (dump_file, step, TDF_SLIM);
> + fprintf (dump_file, ": ");
> + }
> +
> + step = strip_casts (step);
> +
> + /* Peel any scaling, which generally happens after conversion to
> + pointer width. For example, on LP64 systems:
> +
> + int *x, i, stride;
> + ... x[4 * i * stride] ...;
> +
> + multiplies i * stride by 4 using ints, then widens the result
> + to pointer width before multiplying by sizeof (int). */
> + poly_uint64 scale;
> + if (TREE_CODE (step) == MULT_EXPR
> + && poly_int_tree_p (TREE_OPERAND (step, 1), &scale)
> + && known_eq (scale, factor))
> + {
> + step = strip_casts (TREE_OPERAND (step, 0));
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + fprintf (dump_file, "scaled, ");
> + factor = 1;
> + }
> +
> + /* Peel any grouping multiple; see acceptable_scale_p for details. */
> + if (TREE_CODE (step) == MULT_EXPR
> + && acceptable_scale_p (TREE_OPERAND (step, 1), factor))
> + {
> + step = strip_casts (TREE_OPERAND (step, 0));
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + fprintf (dump_file, "%sgrouped, ",
> + maybe_ne (factor, 1U) ? "scaled, " : "");
> + factor = 1;
> + }
> +
> + *step_if_innermost = get_step_if_innermost (step, factor);
> + return step;
> +}
> +
> +/* Analyze the evolution of index fragment EXPR / FACTOR in LOOP and its
> + containing loops to see whether any part of it could be simplified
> + by versioning. Register the versioning opportunities if so. */
> +
> +void
> +loop_versioning::analyze_evolution (struct loop *loop, tree expr,
> + poly_uint64 factor)
> +{
> + const unsigned int MAX_NSPLIT = 8;
> +
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + {
> + fprintf (dump_file, ";; Analyzing use of ");
> + print_generic_expr (dump_file, expr, TDF_SLIM);
> + if (maybe_ne (factor, 1U))
> + {
> + fprintf (dump_file, " (which addresses ");
> + print_dec (factor, dump_file);
> + fprintf (dump_file, " bytes)");
> + }
> + fprintf (dump_file, " in loop %d (depth %d)\n",
> + loop->num, loop_depth (loop));
> + }
> +
> + /* The main problem we have here is that we cannot assume that the
> + innermost loop iterates over the innermost dimension of an array.
> + Accidentally adding versioning checks for outer dimensions would
> + cause the version condition to be false, which as well as bloating
> + the code would defeat loop versioning benefits for other accesses.
> +
> + Unfortunately all we usually see at this stage is general address
> + arithmetic, with no positive way of identifying how many dimensions
> + an array access has and which multiplication factors in the address
> + expression correspond to which array dimensions. In C code this is
> + often not even explicit in the source, since variable-sized multi-
> + dimensional arrays are often simulated using one-dimensional arrays.
> +
> + The three main ways in which we deal with this are:
> +
> + - use heuristics that positively identify steps that are likely
> + to represent the inner dimension.
> +
> + - use heuristics that positively identify steps that are unlikely
> + to represent the inner dimension.
> +
> + - if a part of EXPR is invariant in LOOP, analyze its evolution in
> + the outer loops to see whether we can positively identify any of
> + it as iterating over the inner dimension. */
> + tree best_step = NULL_TREE;
> + auto_vec<tree, MAX_NSPLIT> worklist;
> + worklist.quick_push (expr);
> + unsigned int nsplit = 0;
> + while (!worklist.is_empty ())
> + {
> + expr = strip_casts (worklist.pop ());
> + tree_code code = TREE_CODE (expr);
> +
> + if (code == POLYNOMIAL_CHREC)
> + {
> + /* Analyze the CHREC_RIGHT as the step in an index fragment. */
> + tree step_if_innermost;
> + tree step = extract_step (CHREC_RIGHT (expr), factor,
> + &step_if_innermost);
> + if (!best_step)
> + {
> + /* This is the outermost chrec for the original expression.
> + It's not worth carrying on if the step isn't versionable,
> + or if we're pretty sure it's not for the inner dimension. */
> + if (!step_if_innermost
> + || TREE_CODE (step) != SSA_NAME
> + || !expr_invariant_in_loop_p (loop, step))
> + return;
> +
> + best_step = step;
> +
> + /* We should version for STEP == 1 if we know that that can be
> + true under some circumstances. */
> + if (integer_onep (step_if_innermost))
> + break;
> +
> + /* Bail out if this appears to be the step for the innermost
> + dimension, but isn't likely to be 1.
> +
> + ??? We could instead version for when it equals
> + STEP_IF_INNERMOST, but it's not likely to have as much
> + benefit as versioning for 1. */
> + if (step_if_innermost != step)
> + return;
> + }
> + else
> + {
> + /* This is an inner chrec. If it looks like it iterates over
> + the innermost dimension, abort any attempt to version for
> + the outermost chrec (which if we reach here wasn't itself
> + obviously iterating over the innermost dimension). */
> + if (step_if_innermost && TREE_CONSTANT (step_if_innermost))
> + return;
> + }
> + worklist.quick_push (CHREC_LEFT (expr));
> + continue;
> + }
> +
> + /* Analyze parts of a queued CHREC_LEFT. This is better than just
> + analyzing the evolution of the whole expression since the value
> + could include a mixture of analyzable and unanalyzable elements.
> + Use NSPLIT to count cases in which we add more expressions to
> + analyze, as opposed to just simplifying the existing one. */
> + if (code == PLUS_EXPR || code == POINTER_PLUS_EXPR || code == MINUS_EXPR)
> + {
> + worklist.quick_push (TREE_OPERAND (expr, 0));
> + if (nsplit++ < MAX_NSPLIT)
> + worklist.quick_push (TREE_OPERAND (expr, 1));
> + continue;
> + }
> + if (code == MULT_EXPR)
> + {
> + tree op0 = strip_casts (TREE_OPERAND (expr, 0));
> + tree op1 = TREE_OPERAND (expr, 1);
> + if (TREE_CODE (op0) == PLUS_EXPR || TREE_CODE (op0) == MINUS_EXPR)
> + {
> + tree type = TREE_TYPE (expr);
> + tree op00 = fold_convert (type, TREE_OPERAND (op0, 0));
> + worklist.quick_push (fold_build2 (MULT_EXPR, type, op00, op1));
> + if (nsplit++ < MAX_NSPLIT)
> + {
> + tree op01 = fold_convert (type, TREE_OPERAND (op0, 1));
> + worklist.quick_push (fold_build2 (MULT_EXPR, type,
> + op01, op1));
> + }
> + continue;
> + }
> + }
> +
> + /* If EXPR is invariant in LOOP, analyze it wrt the innermost loop
> + for which it could evolve (i.e. the loop containing the outermost
> + one for which EXPR is invariant). */
> + struct loop *wrt_loop = outermost_invariant_loop_for_expr (loop, expr);
> + if (wrt_loop)
> + {
> + wrt_loop = loop_outer (wrt_loop);
> + if (!wrt_loop)
> + continue;
> + }
> + else
> + wrt_loop = loop;
> + tree evolution = analyze_scalar_evolution (wrt_loop, expr);
> + tree chrec = strip_casts (evolution);
> + if (TREE_CODE (chrec) == POLYNOMIAL_CHREC)
> + {
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + {
> + fprintf (dump_file, ";; evolution of ");
> + print_generic_expr (dump_file, expr, TDF_SLIM);
> + fprintf (dump_file, " in loop %d (depth %d): ",
> + wrt_loop->num, loop_depth (wrt_loop));
> + print_generic_expr (dump_file, evolution, TDF_SLIM);
> + fprintf (dump_file, "\n");
> + }
> + worklist.quick_push (chrec);
> + }
> + else
> + {
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + {
> + fprintf (dump_file, ";; cannot analyze ");
> + print_generic_expr (dump_file, expr, TDF_SLIM);
> + fprintf (dump_file, " any further\n");
> + }
> + }
> + }
> + if (best_step)
> + version_for_unity (loop, best_step);
> +}
> +
> +/* Analyze multiplication MULT to see whether we can identify "gather-like"
> + versioning opportunities such as:
> +
> + for (int i = 0; i < n; ++i)
> + res += a[index[i] * stride];
> +
> + Return true if there was a versioning opportunity.
> +
> + LOOP is the loop that contains MULT. Dividing the value of MULT
> + by FACTOR converts it into an element index. */
> +
> +bool
> +loop_versioning::analyze_product (struct loop *loop, gassign *mult,
> + poly_uint64 factor)
> +{
> + /* Record the original LHS for the dump message below. */
> + tree lhs = gimple_assign_lhs (mult);
> +
> + /* Peel any scaling, which generally happens after conversion to
> + pointer width. For example, on LP64 systems:
> +
> + int *x, i, stride;
> + ... x[4 * i * stride] ...;
> +
> + multiplies i * stride by 4 using ints, then widens the result
> + to pointer width before multiplying by sizeof (int). */
> + poly_uint64 scale;
> + if (poly_int_tree_p (gimple_assign_rhs2 (mult), &scale)
> + && known_eq (scale, factor))
> + {
> + mult = maybe_get_assign_strip_casts (loop, gimple_assign_rhs1 (mult));
> + if (!mult || gimple_assign_rhs_code (mult) != MULT_EXPR)
> + return false;
> + factor = 1;
> + }
> +
> + /* Peel any grouping multiple; see acceptable_scale_p for details. */
> + if (acceptable_scale_p (gimple_assign_rhs2 (mult), factor))
> + {
> + mult = maybe_get_assign_strip_casts (loop, gimple_assign_rhs1 (mult));
> + if (!mult || gimple_assign_rhs_code (mult) != MULT_EXPR)
> + return false;
> + }
> + else if (maybe_ne (factor, 1U))
> + /* We expect to see a scaling multiplication when FACTOR is > 1. */
> + return false;
> +
> + /* See whether this multiplication involves a loop-invariant SSA name
> + and a non-invariant SSA name. */
> + tree op1 = gimple_assign_rhs1 (mult);
> + tree op2 = gimple_assign_rhs2 (mult);
> + if (TREE_CODE (op1) != SSA_NAME || TREE_CODE (op2) != SSA_NAME)
> + return false;
> +
> + bool invariant1_p = expr_invariant_in_loop_p (loop, op1);
> + bool invariant2_p = expr_invariant_in_loop_p (loop, op2);
> + if (invariant1_p == invariant2_p)
> + return false;
> +
> + /* Make sure that the invariant is OP1 and the other operand is OP2. */
> + if (invariant2_p)
> + std::swap (op1, op2);
> +
> + /* Bail out if OP2 can be analyzed as an evolution; we want to use
> + analyze_evolution in that case instead. There's no point trying
> + hard to avoid repeating the call to analyze_scalar_evolution since
> + that function does its own caching. */
> + if (tree_contains_chrecs (analyze_scalar_evolution (loop, op2), NULL))
> + return false;
> +
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + {
> + fprintf (dump_file, ";; Address fragment ");
> + print_generic_expr (dump_file, lhs, TDF_SLIM);
> + fprintf (dump_file, " multiplies invariant ");
> + print_generic_expr (dump_file, op1, TDF_SLIM);
> + fprintf (dump_file, " by ");
> + print_generic_expr (dump_file, op2, TDF_SLIM);
> + fprintf (dump_file, ", which isn't a scalar evolution\n");
> + }
> +
> + version_for_unity (loop, op1);
> + return true;
> +}
> +
> +/* Treat EXPR as a sum of products and apply analyze_product to each of the
> + products. Return true if one of the products provides a versioning
> + opportunity. FACTOR is as for analyze_product. */
> +
> +bool
> +loop_versioning::analyze_sum_of_products (struct loop *loop, tree expr,
> + poly_uint64 factor)
> +{
> + const unsigned int MAX_NITERS = 8;
> +
> + tree worklist[MAX_NITERS];
> + unsigned int length = 0;
> + worklist[length++] = expr;
> + for (unsigned int i = 0; i < length; ++i)
> + {
> + expr = worklist[i];
> + gassign *assign = maybe_get_assign_strip_casts (loop, expr);
> + if (!assign)
> + continue;
> +
> + tree_code code = gimple_assign_rhs_code (assign);
> + if (code == PLUS_EXPR || code == POINTER_PLUS_EXPR || code == MINUS_EXPR)
> + {
> + if (length < MAX_NITERS)
> + worklist[length++] = gimple_assign_rhs1 (assign);
> + if (length < MAX_NITERS)
> + worklist[length++] = gimple_assign_rhs2 (assign);
> + }
> + else if (code == MULT_EXPR && analyze_product (loop, assign, factor))
> + return true;
> + }
> + return false;
> +}
> +
> +/* Analyze pointer expression EXPR, which occurs in loop LOOP and which
> + is used to address a value of type TYPE. */
> +
> +void
> +loop_versioning::analyze_pointer (struct loop *loop, tree expr, tree type)
> +{
> + poly_uint64 factor;
> + if (poly_int_tree_p (TYPE_SIZE_UNIT (type), &factor))
> + {
> + if (!analyze_sum_of_products (loop, expr, factor))
> + analyze_evolution (loop, expr, factor);
> + }
> +}
> +
> +/* Analyze expression EXPR, which occurs in loop LOOP. */
> +
> +void
> +loop_versioning::analyze_expr (struct loop *loop, tree expr)
> +{
> + while (handled_component_p (expr))
> + {
> + /* See whether we can use versioning to avoid a multiplication
> + in the array index. */
> + if (TREE_CODE (expr) == ARRAY_REF)
> + {
> + if (!analyze_sum_of_products (loop, TREE_OPERAND (expr, 1), 1))
> + analyze_evolution (loop, TREE_OPERAND (expr, 1), 1);
> + }
> + expr = TREE_OPERAND (expr, 0);
> + }
> +
> + if (TREE_CODE (expr) == MEM_REF)
> + {
> + tree addr = TREE_OPERAND (expr, 0);
> + /* See whether we can use versioning to avoid a multiplication
> + in the pointer calculation. This is generally only worth
> + doing if the multiplication occurs in this loop rather than
> + an outer loop. */
> + if (!expr_invariant_in_loop_p (loop, addr))
> + analyze_pointer (loop, addr, TREE_TYPE (expr));
> + }
> +
> + /* These would be easy to handle if they existed at this stage. */
> + gcc_checking_assert (TREE_CODE (expr) != TARGET_MEM_REF);
> +}
> +
> +/* Analyze STMT looking for useful version checks. */
> +
> +void
> +loop_versioning::analyze_stmt (gimple *stmt)
> +{
> + struct loop *loop = gimple_bb (stmt)->loop_father;
> +
> + unsigned int nops = gimple_num_ops (stmt);
> + for (unsigned int i = 0; i < nops; ++i)
> + if (tree op = gimple_op (stmt, i))
> + analyze_expr (loop, op);
> +}
> +
> +/* Analyze all the statements in BB looking for useful version checks. */
> +
> +void
> +loop_versioning::analyze_block (basic_block bb)
> +{
> + struct loop *loop = bb->loop_father;
> + loop_info &li = get_loop_info (loop);
> + if (li.rejected_p)
> + return;
> +
> + for (gimple_stmt_iterator gsi = gsi_start_bb (bb); !gsi_end_p (gsi);
> + gsi_next (&gsi))
> + {
> + gimple *stmt = gsi_stmt (gsi);
> + if (expensive_stmt_p (stmt))
> + {
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + {
> + struct loop *loop = gimple_bb (stmt)->loop_father;
> + fprintf (dump_file, ";; Loop %d (depth %d) contains expensive"
> + " stmt: ", loop->num, loop_depth (loop));
> + print_gimple_stmt (dump_file, stmt, 0, TDF_SLIM);
> + }
> + li.rejected_p = true;
> + break;
> + }
> +
> + /* Only look for direct versioning opportunities in inner loops
> + since the benefit tends to be much smaller for outer loops. */
> + if (!loop->inner)
> + analyze_stmt (stmt);
> +
> + /* The point of the instruction limit is to prevent excessive
> + code growth, so this is a size-based estimate even though
> + the optimization is aimed at speed. */
> + li.num_insns += estimate_num_insns (stmt, &eni_size_weights);
> + }
> +}
> +
> +/* Analyze all the blocks in the function looking for useful version checks.
> + Return true if we found one. */
> +
> +bool
> +loop_versioning::analyze_blocks ()
> +{
> + /* For now we don't try to version the whole function, although
> + versioning at that level could be useful in some cases. */
> + get_loop_info (get_loop (m_fn, 0)).rejected_p = true;
> +
> + basic_block bb;
> + FOR_EACH_BB_FN (bb, m_fn)
> + if (loop_outer (bb->loop_father))
> + analyze_block (bb);
> +
> + return m_num_conditions != 0;
> +}
> +
> +/* Use the ranges in VRS to remove impossible versioning conditions from
> + LOOP. */
> +
> +void
> +loop_versioning::prune_loop_conditions (struct loop *loop, vr_values *vrs)
> +{
> + loop_info &li = get_loop_info (loop);
> +
> + unsigned int i = li.unity_names.length ();
> + while (i > 0)
> + {
> + i -= 1;
> + tree name = li.unity_names[i];
> + value_range *vr = vrs->get_value_range (name);
> + if (vr && !range_includes_p (vr, 1))
> + {
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + {
> + fprintf (dump_file, ";; ");
> + print_generic_expr (dump_file, name, TDF_SLIM);
> + fprintf (dump_file, " can never be 1 in loop %d\n", loop->num);
> + }
> +
> + li.unity_names.unordered_remove (i);
> + bitmap_clear_bit (&li.unity_name_ids, SSA_NAME_VERSION (name));
> + m_num_conditions -= 1;
> + }
> + }
> +}
> +
> +/* Remove any scheduled loop version conditions that will never be true.
> + Return true if any remain. */
> +
> +bool
> +loop_versioning::prune_conditions ()
> +{
> + calculate_dominance_info (CDI_DOMINATORS);
> + lv_dom_walker dom_walker (*this);
> + dom_walker.walk (ENTRY_BLOCK_PTR_FOR_FN (m_fn));
> + return m_num_conditions != 0;
> +}
> +
> +/* Merge the version checks for INNER into immediately-enclosing loop
> + OUTER. */
> +
> +void
> +loop_versioning::merge_loop_info (struct loop *outer, struct loop *inner)
> +{
> + loop_info &inner_li = get_loop_info (inner);
> + loop_info &outer_li = get_loop_info (outer);
> +
> + tree name;
> + unsigned int i;
> + FOR_EACH_VEC_ELT (inner_li.unity_names, i, name)
> + if (bitmap_set_bit (&outer_li.unity_name_ids, SSA_NAME_VERSION (name)))
> + {
> + outer_li.unity_names.safe_push (name);
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + {
> + fprintf (dump_file, ";; Hoisting check that ");
> + print_generic_expr (dump_file, name, TDF_SLIM);
> + fprintf (dump_file, " == 1 from loop %d (depth %d) to loop %d\n",
> + inner->num, loop_depth (inner), outer->num);
> + }
> + }
> +
> + if (loop_depth (outer_li.outermost) < loop_depth (inner_li.outermost))
> + outer_li.outermost = inner_li.outermost;
> +}
> +
> +/* Add LOOP to the queue of loops to version. */
> +
> +void
> +loop_versioning::add_loop_to_queue (struct loop *loop)
> +{
> + loop_info &li = get_loop_info (loop);
> +
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + fprintf (dump_file, ";; Queuing loop %d (depth %d) for versioning\n",
> + loop->num, loop_depth (loop));
> + m_loops_to_version.safe_push (loop);
> +
> + /* Don't try to version superloops. */
> + li.rejected_p = true;
> +}
> +
> +/* Decide whether the cost model would allow us to version LOOP,
> + either directly or as part of a parent loop, and return true if so.
> + This does not imply that the loop is actually worth versioning in its
> + own right, just that it would be valid to version it if something
> + benefited.
> +
> + We have already made this decision for all inner loops of LOOP. */
> +
> +bool
> +loop_versioning::decide_whether_loop_is_versionable (struct loop *loop)
> +{
> + loop_info &li = get_loop_info (loop);
> +
> + if (li.rejected_p)
> + return false;
> +
> + /* Examine the decisions made for inner loops. */
> + for (struct loop *inner = loop->inner; inner; inner = inner->next)
> + {
> + loop_info &inner_li = get_loop_info (inner);
> + if (inner_li.rejected_p)
> + {
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + fprintf (dump_file, ";; Not versioning loop %d (depth %d)"
> + " because inner loop %d should not be versioned\n",
> + loop->num, loop_depth (loop), inner->num);
> + return false;
> + }
> +
> + if (inner_li.worth_versioning_p ())
> + li.subloops_benefit_p = true;
> +
> + /* Accumulate the number of instructions from subloops that are not
> + the innermost, or that don't benefit from versioning. Only the
> + instructions from innermost loops that benefit from versioning
> + should be weighed against loop-versioning-max-inner-insns;
> + everything else should be weighed against
> + loop-versioning-max-outer-insns. */
> + if (!inner_li.worth_versioning_p () || inner->inner)
> + {
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + fprintf (dump_file, ";; Counting %d instructions from"
> + " loop %d (depth %d) against parent loop %d\n",
> + inner_li.num_insns, inner->num, loop_depth (inner),
> + loop->num);
> + li.num_insns += inner_li.num_insns;
> + }
> + }
> +
> + /* Enforce the size limits. */
> + if (li.worth_versioning_p ())
> + {
> + unsigned int max_num_insns = max_insns_for_loop (loop);
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + fprintf (dump_file, ";; Loop %d (depth %d) has %d instructions,"
> + " against a limit of %d\n", loop->num, loop_depth (loop),
> + li.num_insns, max_num_insns);
> + if (li.num_insns > max_num_insns)
> + {
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + fprintf (dump_file, ";; Not versioning loop %d (depth %d)"
> + " because it is too big\n", loop->num, loop_depth (loop));
> + return false;
> + }
> + }
> +
> + /* Hoist all version checks for subloops to this loop. */
> + for (struct loop *subloop = loop->inner; subloop; subloop = subloop->next)
> + merge_loop_info (loop, subloop);
> +
> + return true;
> +}
> +
> +/* Decide which loops to version and add them to the versioning queue.
> + Return true if there are any loops to version. */
> +
> +bool
> +loop_versioning::make_versioning_decisions ()
> +{
> + struct loop *loop;
> + FOR_EACH_LOOP (loop, LI_FROM_INNERMOST)
> + {
> + loop_info &linfo = get_loop_info (loop);
> + if (decide_whether_loop_is_versionable (loop))
> + {
> + /* Commit to versioning LOOP directly if we can't hoist the
> + version checks any further. */
> + if (linfo.worth_versioning_p ()
> + && (loop_depth (loop) == 1 || linfo.outermost == loop))
> + add_loop_to_queue (loop);
> + }
> + else
> + {
> + /* We can't version this loop, so individually version any
> + subloops that would benefit and haven't been versioned yet. */
> + linfo.rejected_p = true;
> + for (struct loop *subloop = loop->inner; subloop;
> + subloop = subloop->next)
> + if (get_loop_info (subloop).worth_versioning_p ())
> + add_loop_to_queue (subloop);
> + }
> + }
> +
> + return !m_loops_to_version.is_empty ();
> +}
> +
> +/* Attempt to implement loop versioning for LOOP, using the information
> + cached in the associated loop_info. Return true on success. */
> +
> +bool
> +loop_versioning::version_loop (struct loop *loop)
> +{
> + loop_info &li = get_loop_info (loop);
> +
> + /* Not protected by TDF_DETAILS since this is the main piece of
> + information. */
> + if (dump_file)
> + fprintf (dump_file, ";; Versioning loop %d (depth %d)\n",
> + loop->num, loop_depth (loop));
> +
> + /* Build up a condition that selects the original loop instead of
> + the simplified loop. */
> + tree cond = boolean_false_node;
> + tree name;
> + unsigned int i;
> + FOR_EACH_VEC_ELT (li.unity_names, i, name)
> + {
> + tree ne_one = fold_build2 (NE_EXPR, boolean_type_node, name,
> + build_one_cst (TREE_TYPE (name)));
> + cond = fold_build2 (TRUTH_OR_EXPR, boolean_type_node, cond, ne_one);
> + }
> +
> + /* Convert the condition into a suitable gcond. */
> + gimple_seq stmts = NULL;
> + cond = force_gimple_operand_1 (cond, &stmts, is_gimple_condexpr, NULL_TREE);
> +
> + /* Version the loop. */
> + initialize_original_copy_tables ();
> + basic_block cond_bb;
> + struct loop *else_loop
> + = loop_version (loop, cond, &cond_bb,
> + profile_probability::unlikely (),
> + profile_probability::likely (),
> + profile_probability::unlikely (),
> + profile_probability::likely (), true);
> + free_original_copy_tables ();
> + if (!else_loop)
> + {
> + if (dump_file)
> + fprintf (dump_file, ";; Versioning of loop %d failed\n", loop->num);
> + return false;
> + }
> +
> + /* Insert the statements that feed COND. */
> + if (stmts)
> + {
> + gimple_stmt_iterator gsi = gsi_last_bb (cond_bb);
> + gsi_insert_seq_before (&gsi, stmts, GSI_SAME_STMT);
> + }
> +
> + /* Simplify the new loop, which is used when COND is false. */
> + name_prop (li).substitute_and_fold (else_loop->header);
> + return true;
> +}
> +
> +/* Attempt to version all loops in the versioning queue. Return true
> + if we succeeded for at least one loop. */
> +
> +bool
> +loop_versioning::implement_versioning_decisions ()
> +{
> + bool any_succeeded_p = false;
> +
> + struct loop *loop;
> + unsigned int i;
> + FOR_EACH_VEC_ELT (m_loops_to_version, i, loop)
> + if (version_loop (loop))
> + any_succeeded_p = true;
> +
> + return any_succeeded_p;
> +}
> +
> +/* Run the pass and return a set of TODO_* flags. */
> +
> +unsigned int
> +loop_versioning::run ()
> +{
> + gcc_assert (scev_initialized_p ());
> +
> + if (!analyze_blocks ()
> + || !prune_conditions ()
> + || !make_versioning_decisions ()
> + || !implement_versioning_decisions ())
> + return 0;
> +
> + return TODO_update_ssa;
> +}
> +
> +/* Loop versioningting pass. */
> +
> +namespace {
> +
> +const pass_data pass_data_loop_versioning =
> +{
> + GIMPLE_PASS, /* type */
> + "lversion", /* name */
> + OPTGROUP_LOOP, /* optinfo_flags */
> + TV_LOOP_VERSIONING, /* tv_id */
> + PROP_cfg, /* properties_required */
> + 0, /* properties_provided */
> + 0, /* properties_destroyed */
> + 0, /* todo_flags_start */
> + 0, /* todo_flags_finish */
> +};
> +
> +class pass_loop_versioning : public gimple_opt_pass
> +{
> +public:
> + pass_loop_versioning (gcc::context *ctxt)
> + : gimple_opt_pass (pass_data_loop_versioning, ctxt)
> + {}
> +
> + /* opt_pass methods: */
> + virtual bool gate (function *) { return flag_version_loops_for_strides; }
> + virtual unsigned int execute (function *);
> +};
> +
> +unsigned int
> +pass_loop_versioning::execute (function *fn)
> +{
> + if (number_of_loops (fn) <= 1)
> + return 0;
> +
> + return loop_versioning (fn).run ();
> +}
> +
> +} // anon namespace
> +
> +gimple_opt_pass *
> +make_pass_loop_versioning (gcc::context *ctxt)
> +{
> + return new pass_loop_versioning (ctxt);
> +}
> Index: gcc/testsuite/gcc.dg/loop-versioning-1.c
> ===================================================================
> --- /dev/null 2018-09-14 11:16:31.122530289 +0100
> +++ gcc/testsuite/gcc.dg/loop-versioning-1.c 2018-10-24 14:02:15.184152693 +0100
> @@ -0,0 +1,20 @@
> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
> +
> +/* The simplest IV case. */
> +
> +void
> +f1 (double *x, int stepx, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + x[stepx * i] = 100;
> +}
> +
> +void
> +f2 (double *x, int stepx, int limit)
> +{
> + for (int i = 0; i < limit; i += stepx)
> + x[i] = 100;
> +}
> +
> +/* { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 1\)} 2 "lversion" } } */
> +/* { dg-final { scan-tree-dump-times {Versioning loop} 2 "lversion" } } */
> Index: gcc/testsuite/gcc.dg/loop-versioning-10.c
> ===================================================================
> --- /dev/null 2018-09-14 11:16:31.122530289 +0100
> +++ gcc/testsuite/gcc.dg/loop-versioning-10.c 2018-10-24 14:02:15.184152693 +0100
> @@ -0,0 +1,17 @@
> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
> +
> +/* Test that we can version a gather-like operation in which a variable
> + stride is applied to the index. */
> +
> +int
> +f1 (int *x, int *index, int step, int n)
> +{
> + int res = 0;
> + for (int i = 0; i < n; ++i)
> + res += x[index[i] * step];
> + return res;
> +}
> +
> +/* { dg-final { scan-tree-dump-times {Address[^\n]*invariant} 1 "lversion" } } */
> +/* { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 1\)} 1 "lversion" } } */
> +/* { dg-final { scan-tree-dump-times {Versioning loop} 1 "lversion" } } */
> Index: gcc/testsuite/gcc.dg/loop-versioning-11.c
> ===================================================================
> --- /dev/null 2018-09-14 11:16:31.122530289 +0100
> +++ gcc/testsuite/gcc.dg/loop-versioning-11.c 2018-10-24 14:02:15.188152659 +0100
> @@ -0,0 +1,29 @@
> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
> +
> +/* Test that we don't try to version for something that is never 1. */
> +
> +void
> +f1 (double *x, int stepx, int n)
> +{
> + if (stepx == 1)
> + for (int i = 0; i < n; ++i)
> + x[i] = 100;
> + else
> + for (int i = 0; i < n; ++i)
> + x[stepx * i] = 100;
> +}
> +
> +void
> +f2 (double *x, int stepx, int n)
> +{
> + if (stepx <= 1)
> + for (int i = 0; i < n; ++i)
> + x[i] = 100;
> + else
> + for (int i = 0; i < n; ++i)
> + x[stepx * i] = 100;
> +}
> +
> +/* { dg-final { scan-tree-dump-times {Want to version loop} 2 "lversion" } } */
> +/* { dg-final { scan-tree-dump-times {can never be 1} 2 "lversion" } } */
> +/* { dg-final { scan-tree-dump-not {Versioning loop} "lversion" } } */
> Index: gcc/testsuite/gcc.dg/loop-versioning-2.c
> ===================================================================
> --- /dev/null 2018-09-14 11:16:31.122530289 +0100
> +++ gcc/testsuite/gcc.dg/loop-versioning-2.c 2018-10-24 14:02:15.188152659 +0100
> @@ -0,0 +1,39 @@
> +/* { dg-options "-O3 -fdump-tree-lversion-details" } */
> +
> +/* Versioning these loops allows loop interchange. */
> +
> +void
> +f1 (double x[][100], int step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + x[j * step][i] = 100;
> +}
> +
> +void
> +f2 (double x[][100], int step, int n)
> +{
> + for (int i = 0; i < n; ++i)
> + for (int j = 0; j < n; ++j)
> + x[j][i * step] = 100;
> +}
> +
> +void
> +f3 (double x[][100], int step, int limit)
> +{
> + for (int i = 0; i < 100; ++i)
> + for (int j = 0; j < limit; j += step)
> + x[j][i] = 100;
> +}
> +
> +void
> +f4 (double x[][100], int step, int limit)
> +{
> + for (int i = 0; i < limit; i += step)
> + for (int j = 0; j < 100; ++j)
> + x[j][i] = 100;
> +}
> +
> +/* { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 2\)} 4 "lversion" } } */
> +/* { dg-final { scan-tree-dump-times {Hoisting check} 4 "lversion" } } */
> +/* { dg-final { scan-tree-dump-times {Versioning loop [0-9]+ \(depth 1\)} 4 "lversion" } } */
> Index: gcc/testsuite/gcc.dg/loop-versioning-3.c
> ============================================================
^ permalink raw reply [flat|nested] 17+ messages in thread
* Add a loop versioning pass
@ 2018-10-24 13:41 Richard Sandiford
2018-10-25 14:16 ` Richard Biener
2018-10-25 16:16 ` David Malcolm
0 siblings, 2 replies; 17+ messages in thread
From: Richard Sandiford @ 2018-10-24 13:41 UTC (permalink / raw)
To: gcc-patches
This patch adds a pass that versions loops with variable index strides
for the case in which the stride is 1. E.g.:
for (int i = 0; i < n; ++i)
x[i * stride] = ...;
becomes:
if (stepx == 1)
for (int i = 0; i < n; ++i)
x[i] = ...;
else
for (int i = 0; i < n; ++i)
x[i * stride] = ...;
This is useful for both vector code and scalar code, and in some cases
can enable further optimisations like loop interchange or pattern
recognition.
The pass gives a 7.6% improvement on Cortex-A72 for 554.roms_r at -O3
and a 2.4% improvement for 465.tonto. I haven't found any SPEC tests
that regress.
Sizewise, there's a 10% increase in .text for both 554.roms_r and
465.tonto. That's obviously a lot, but in tonto's case it's because
the whole program is written using assumed-shape arrays and pointers,
so a large number of functions really do benefit from versioning.
roms likewise makes heavy use of assumed-shape arrays, and that
improvement in performance IMO justifies the code growth.
The next biggest .text increase is 4.5% for 548.exchange2_r. I did see
a small (0.4%) speed improvement there, but although both 3-iteration runs
produced stable results, that might still be noise. There was a slightly
larger (non-noise) improvement for a 256-bit SVE model.
481.wrf and 521.wrf_r .text grew by 2.8% and 2.5% respectively, but
without any noticeable improvement in performance. No other test grew
by more than 2%.
Although the main SPEC beneficiaries are all Fortran tests, the
benchmarks we use for SVE also include some C and C++ tests that
benefit.
Using -frepack-arrays gives the same benefits in many Fortran cases.
The problem is that using that option inappropriately can force a full
array copy for arguments that the function only reads once, and so it
isn't really something we can turn on by default. The new pass is
supposed to give most of the benefits of -frepack-arrays without
the risk of unnecessary repacking.
The patch therefore enables the pass by default at -O3.
Tested on aarch64-linux-gnu and x86_64-linux-gnu. OK to install?
Richard
2018-10-24 Richard Sandiford <richard.sandiford@arm.com>
gcc/
* doc/invoke.texi (-fversion-loops-for-strides): Document
(loop-versioning-group-size, loop-versioning-max-inner-insns)
(loop-versioning-max-outer-insns): Document new --params.
* Makefile.in (OBJS): Add gimple-loop-versioning.o.
* common.opt (fversion-loops-for-strides): New option.
* opts.c (default_options_table): Enable fversion-loops-for-strides
at -O3.
* params.def (PARAM_LOOP_VERSIONING_GROUP_SIZE)
(PARAM_LOOP_VERSIONING_MAX_INNER_INSNS)
(PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS): New parameters.
* passes.def: Add pass_loop_versioning.
* timevar.def (TV_LOOP_VERSIONING): New time variable.
* tree-ssa-propagate.h
(substitute_and_fold_engine::substitute_and_fold): Add an optional
block parameter.
* tree-ssa-propagate.c
(substitute_and_fold_engine::substitute_and_fold): Likewise.
When passed, only walk blocks dominated by that block.
* tree-vrp.h (range_includes_p): Declare.
(range_includes_zero_p): Turn into an inline wrapper around
range_includes_p.
* tree-vrp.c (range_includes_p): New function, generalizing...
(range_includes_zero_p): ...this.
* tree-pass.h (make_pass_loop_versioning): Declare.
* gimple-loop-versioning.cc: New file.
gcc/testsuite/
* gcc.dg/loop-versioning-1.c: New test.
* gcc.dg/loop-versioning-10.c: Likewise.
* gcc.dg/loop-versioning-11.c: Likewise.
* gcc.dg/loop-versioning-2.c: Likewise.
* gcc.dg/loop-versioning-3.c: Likewise.
* gcc.dg/loop-versioning-4.c: Likewise.
* gcc.dg/loop-versioning-5.c: Likewise.
* gcc.dg/loop-versioning-6.c: Likewise.
* gcc.dg/loop-versioning-7.c: Likewise.
* gcc.dg/loop-versioning-8.c: Likewise.
* gcc.dg/loop-versioning-9.c: Likewise.
* gfortran.dg/loop_versioning_1.f90: Likewise.
* gfortran.dg/loop_versioning_2.f90: Likewise.
* gfortran.dg/loop_versioning_3.f90: Likewise.
* gfortran.dg/loop_versioning_4.f90: Likewise.
* gfortran.dg/loop_versioning_5.f90: Likewise.
* gfortran.dg/loop_versioning_6.f90: Likewise.
* gfortran.dg/loop_versioning_7.f90: Likewise.
* gfortran.dg/loop_versioning_8.f90: Likewise.
Index: gcc/doc/invoke.texi
===================================================================
--- gcc/doc/invoke.texi 2018-10-24 14:02:14.000000000 +0100
+++ gcc/doc/invoke.texi 2018-10-24 14:02:15.184152693 +0100
@@ -7934,7 +7934,8 @@ by @option{-O2} and also turns on the fo
-fvect-cost-model @gol
-ftree-partial-pre @gol
-fpeel-loops @gol
--fipa-cp-clone}
+-fipa-cp-clone @gol
+-fversion-loops-for-strides}
@item -O0
@opindex O0
@@ -10358,6 +10359,29 @@ for one side of the iteration space and
Move branches with loop invariant conditions out of the loop, with duplicates
of the loop on both branches (modified according to result of the condition).
+@item -fversion-loops-for-strides
+@opindex fversion-loops-for-strides
+If a loop iterates over an array with a variable stride, create another
+version of the loop that assumes the stride is always 1. For example:
+
+@smallexample
+for (int i = 0; i < n; ++i)
+ x[i * stride] = @dots{};
+@end smallexample
+
+becomes:
+
+@smallexample
+if (stride == 1)
+ for (int i = 0; i < n; ++i)
+ x[i] = @dots{};
+else
+ for (int i = 0; i < n; ++i)
+ x[i * stride] = @dots{};
+@end smallexample
+
+This is particularly useful for assumed-shape arrays in Fortran.
+
@item -ffunction-sections
@itemx -fdata-sections
@opindex ffunction-sections
@@ -11567,6 +11591,20 @@ Hardware autoprefetcher scheduler model
Number of lookahead cycles the model looks into; at '
' only enable instruction sorting heuristic.
+@item loop-versioning-group-size
+Make the loop versioning pass optimize @samp{a[i * index * @var{N}]}
+in the same way as it would optimize @samp{a[i * index]} when @var{N}
+is less than or equal to this value.
+
+@item loop-versioning-max-inner-insns
+The maximum number of instructions that an inner loop can have
+before the loop versioning pass considers it too big to copy.
+
+@item loop-versioning-max-outer-insns
+The maximum number of instructions that an outer loop can have
+before the loop versioning pass considers it too big to copy,
+discounting any instructions in inner loops that directly benefit
+from versioning.
@end table
@end table
Index: gcc/Makefile.in
===================================================================
--- gcc/Makefile.in 2018-10-24 14:02:14.000000000 +0100
+++ gcc/Makefile.in 2018-10-24 14:02:15.180152727 +0100
@@ -1312,6 +1312,7 @@ OBJS = \
gimple-laddress.o \
gimple-loop-interchange.o \
gimple-loop-jam.o \
+ gimple-loop-versioning.o \
gimple-low.o \
gimple-pretty-print.o \
gimple-ssa-backprop.o \
Index: gcc/common.opt
===================================================================
--- gcc/common.opt 2018-10-24 14:02:14.000000000 +0100
+++ gcc/common.opt 2018-10-24 14:02:15.180152727 +0100
@@ -2712,6 +2712,10 @@ fsplit-loops
Common Report Var(flag_split_loops) Optimization
Perform loop splitting.
+fversion-loops-for-strides
+Common Report Var(flag_version_loops_for_strides) Optimization
+Version loops based on whether indices have a stride of 1.
+
funwind-tables
Common Report Var(flag_unwind_tables) Optimization
Just generate unwind tables for exception handling.
Index: gcc/opts.c
===================================================================
--- gcc/opts.c 2018-10-24 14:02:14.000000000 +0100
+++ gcc/opts.c 2018-10-24 14:02:15.184152693 +0100
@@ -544,6 +544,7 @@ static const struct default_options defa
{ OPT_LEVELS_3_PLUS, OPT_fipa_cp_clone, NULL, 1 },
{ OPT_LEVELS_3_PLUS, OPT_ftree_partial_pre, NULL, 1 },
{ OPT_LEVELS_3_PLUS, OPT_fpeel_loops, NULL, 1 },
+ { OPT_LEVELS_3_PLUS, OPT_fversion_loops_for_strides, NULL, 1 },
/* -Ofast adds optimizations to -O3. */
{ OPT_LEVELS_FAST, OPT_ffast_math, NULL, 1 },
Index: gcc/params.def
===================================================================
--- gcc/params.def 2018-10-24 14:02:14.000000000 +0100
+++ gcc/params.def 2018-10-24 14:02:15.184152693 +0100
@@ -1360,6 +1360,25 @@ DEFPARAM(PARAM_AVOID_FMA_MAX_BITS,
"Maximum number of bits for which we avoid creating FMAs.",
0, 0, 512)
+DEFPARAM(PARAM_LOOP_VERSIONING_GROUP_SIZE,
+ "loop-versioning-group-size",
+ "The maximum constant N for which accesses of the form x[N * step]"
+ " are worth versioning for the case in which step is 1",
+ 4, 1, 0)
+
+DEFPARAM(PARAM_LOOP_VERSIONING_MAX_INNER_INSNS,
+ "loop-versioning-max-inner-insns",
+ "The maximum number of instructions in an inner loop that is being"
+ " considered for versioning",
+ 200, 0, 0)
+
+DEFPARAM(PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS,
+ "loop-versioning-max-outer-insns",
+ "The maximum number of instructions in an outer loop that is being"
+ " considered for versioning, on top of the instructions in inner"
+ " loops",
+ 100, 0, 0)
+
/*
Local variables:
Index: gcc/passes.def
===================================================================
--- gcc/passes.def 2018-10-24 14:02:14.000000000 +0100
+++ gcc/passes.def 2018-10-24 14:02:15.184152693 +0100
@@ -260,6 +260,7 @@ along with GCC; see the file COPYING3.
NEXT_PASS (pass_tree_loop);
PUSH_INSERT_PASSES_WITHIN (pass_tree_loop)
NEXT_PASS (pass_tree_loop_init);
+ NEXT_PASS (pass_loop_versioning);
NEXT_PASS (pass_tree_unswitch);
NEXT_PASS (pass_scev_cprop);
NEXT_PASS (pass_loop_split);
Index: gcc/timevar.def
===================================================================
--- gcc/timevar.def 2018-10-24 14:02:14.000000000 +0100
+++ gcc/timevar.def 2018-10-24 14:02:15.188152659 +0100
@@ -234,6 +234,7 @@ DEFTIMEVAR (TV_DSE1 , "
DEFTIMEVAR (TV_DSE2 , "dead store elim2")
DEFTIMEVAR (TV_LOOP , "loop analysis")
DEFTIMEVAR (TV_LOOP_INIT , "loop init")
+DEFTIMEVAR (TV_LOOP_VERSIONING , "loop versioning")
DEFTIMEVAR (TV_LOOP_MOVE_INVARIANTS , "loop invariant motion")
DEFTIMEVAR (TV_LOOP_UNROLL , "loop unrolling")
DEFTIMEVAR (TV_LOOP_DOLOOP , "loop doloop")
Index: gcc/tree-ssa-propagate.h
===================================================================
--- gcc/tree-ssa-propagate.h 2018-10-24 14:02:14.000000000 +0100
+++ gcc/tree-ssa-propagate.h 2018-10-24 14:02:15.188152659 +0100
@@ -104,7 +104,7 @@ extern void propagate_tree_value_into_st
virtual bool fold_stmt (gimple_stmt_iterator *) { return false; }
virtual tree get_value (tree) { return NULL_TREE; }
- bool substitute_and_fold (void);
+ bool substitute_and_fold (basic_block = NULL);
bool replace_uses_in (gimple *);
bool replace_phi_args_in (gphi *);
};
Index: gcc/tree-ssa-propagate.c
===================================================================
--- gcc/tree-ssa-propagate.c 2018-10-24 14:02:14.000000000 +0100
+++ gcc/tree-ssa-propagate.c 2018-10-24 14:02:15.188152659 +0100
@@ -1152,6 +1152,10 @@ substitute_and_fold_dom_walker::before_d
/* Perform final substitution and folding of propagated values.
+ Process the whole function if BLOCK is null, otherwise only
+ process the blocks that BLOCK dominates. In the latter case,
+ it is the caller's responsibility to ensure that dominator
+ information is available and up-to-date.
PROP_VALUE[I] contains the single value that should be substituted
at every use of SSA name N_I. If PROP_VALUE is NULL, no values are
@@ -1168,16 +1172,24 @@ substitute_and_fold_dom_walker::before_d
Return TRUE when something changed. */
bool
-substitute_and_fold_engine::substitute_and_fold (void)
+substitute_and_fold_engine::substitute_and_fold (basic_block block)
{
if (dump_file && (dump_flags & TDF_DETAILS))
fprintf (dump_file, "\nSubstituting values and folding statements\n\n");
memset (&prop_stats, 0, sizeof (prop_stats));
- calculate_dominance_info (CDI_DOMINATORS);
+ /* Don't call calculate_dominance_info when iterating over a subgraph.
+ Callers that are using the interface this way are likely to want to
+ iterate over several disjoint subgraphs, and it would be expensive
+ in enable-checking builds to revalidate the whole dominance tree
+ each time. */
+ if (block)
+ gcc_assert (dom_info_state (CDI_DOMINATORS));
+ else
+ calculate_dominance_info (CDI_DOMINATORS);
substitute_and_fold_dom_walker walker (CDI_DOMINATORS, this);
- walker.walk (ENTRY_BLOCK_PTR_FOR_FN (cfun));
+ walker.walk (block ? block : ENTRY_BLOCK_PTR_FOR_FN (cfun));
/* We cannot remove stmts during the BB walk, especially not release
SSA names there as that destroys the lattice of our callers.
Index: gcc/tree-vrp.h
===================================================================
--- gcc/tree-vrp.h 2018-10-24 14:02:14.000000000 +0100
+++ gcc/tree-vrp.h 2018-10-24 14:02:15.188152659 +0100
@@ -86,7 +86,7 @@ extern void register_edge_assert_for (tr
tree, tree, vec<assert_info> &);
extern bool stmt_interesting_for_vrp (gimple *);
extern void set_value_range_to_varying (value_range *);
-extern bool range_includes_zero_p (const value_range *);
+extern bool range_includes_p (const value_range *, HOST_WIDE_INT);
extern bool infer_value_range (gimple *, tree, tree_code *, tree *);
extern void set_value_range_to_nonnull (value_range *, tree);
@@ -122,4 +122,13 @@ extern int value_inside_range (tree, tre
extern tree get_single_symbol (tree, bool *, tree *);
extern void maybe_set_nonzero_bits (edge, tree);
extern value_range_type determine_value_range (tree, wide_int *, wide_int *);
+
+/* Return TRUE if *VR includes the value zero. */
+
+inline bool
+range_includes_zero_p (const value_range *vr)
+{
+ return range_includes_p (vr, 0);
+}
+
#endif /* GCC_TREE_VRP_H */
Index: gcc/tree-vrp.c
===================================================================
--- gcc/tree-vrp.c 2018-10-24 14:02:14.000000000 +0100
+++ gcc/tree-vrp.c 2018-10-24 14:02:15.188152659 +0100
@@ -844,10 +844,10 @@ value_inside_range (tree val, tree min,
}
-/* Return TRUE if *VR includes the value zero. */
+/* Return TRUE if *VR includes the value X. */
bool
-range_includes_zero_p (const value_range *vr)
+range_includes_p (const value_range *vr, HOST_WIDE_INT x)
{
if (vr->type == VR_VARYING)
return true;
@@ -856,13 +856,13 @@ range_includes_zero_p (const value_range
if (vr->type == VR_UNDEFINED)
return true;
- tree zero = build_int_cst (TREE_TYPE (vr->min), 0);
+ tree x_cst = build_int_cst (TREE_TYPE (vr->min), x);
if (vr->type == VR_ANTI_RANGE)
{
- int res = value_inside_range (zero, vr->min, vr->max);
+ int res = value_inside_range (x_cst, vr->min, vr->max);
return res == 0 || res == -2;
}
- return value_inside_range (zero, vr->min, vr->max) != 0;
+ return value_inside_range (x_cst, vr->min, vr->max) != 0;
}
/* If *VR has a value rante that is a single constant value return that,
Index: gcc/tree-pass.h
===================================================================
--- gcc/tree-pass.h 2018-10-24 14:02:14.000000000 +0100
+++ gcc/tree-pass.h 2018-10-24 14:02:15.188152659 +0100
@@ -362,6 +362,7 @@ extern gimple_opt_pass *make_pass_fix_lo
extern gimple_opt_pass *make_pass_tree_loop (gcc::context *ctxt);
extern gimple_opt_pass *make_pass_tree_no_loop (gcc::context *ctxt);
extern gimple_opt_pass *make_pass_tree_loop_init (gcc::context *ctxt);
+extern gimple_opt_pass *make_pass_loop_versioning (gcc::context *ctxt);
extern gimple_opt_pass *make_pass_lim (gcc::context *ctxt);
extern gimple_opt_pass *make_pass_linterchange (gcc::context *ctxt);
extern gimple_opt_pass *make_pass_tree_unswitch (gcc::context *ctxt);
Index: gcc/gimple-loop-versioning.cc
===================================================================
--- /dev/null 2018-09-14 11:16:31.122530289 +0100
+++ gcc/gimple-loop-versioning.cc 2018-10-24 14:02:15.184152693 +0100
@@ -0,0 +1,1417 @@
+/* Loop versioning pass.
+ Copyright (C) 2018 Free Software Foundation, Inc.
+
+This file is part of GCC.
+
+GCC is free software; you can redistribute it and/or modify it
+under the terms of the GNU General Public License as published by the
+Free Software Foundation; either version 3, or (at your option) any
+later version.
+
+GCC is distributed in the hope that it will be useful, but WITHOUT
+ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
+for more details.
+
+You should have received a copy of the GNU General Public License
+along with GCC; see the file COPYING3. If not see
+<http://www.gnu.org/licenses/>. */
+
+#include "config.h"
+#include "system.h"
+#include "coretypes.h"
+#include "backend.h"
+#include "tree.h"
+#include "gimple.h"
+#include "gimple-iterator.h"
+#include "tree-pass.h"
+#include "gimplify-me.h"
+#include "cfgloop.h"
+#include "tree-ssa-loop.h"
+#include "ssa.h"
+#include "tree-scalar-evolution.h"
+#include "tree-chrec.h"
+#include "tree-ssa-loop-ivopts.h"
+#include "fold-const.h"
+#include "tree-ssa-propagate.h"
+#include "tree-inline.h"
+#include "domwalk.h"
+#include "alloc-pool.h"
+#include "vr-values.h"
+#include "gimple-ssa-evrp-analyze.h"
+#include "gimple-pretty-print.h"
+#include "params.h"
+
+/* This pass looks for loops that could be simplified if certain loop
+ invariant conditions were true. It is effectively a form of loop
+ splitting in which the pass produces the split conditions itself,
+ instead of using ones that are already present in the IL.
+
+ Versioning for when strides are 1
+ ---------------------------------
+
+ At the moment the only thing the pass looks for are memory references
+ like:
+
+ for (auto i : ...)
+ ...x[i * stride]...
+
+ It considers changing such loops to:
+
+ if (stride == 1)
+ for (auto i : ...) [A]
+ ...x[i]...
+ else
+ for (auto i : ...) [B]
+ ...x[i * stride]...
+
+ This can have several benefits:
+
+ (1) [A] is often easier or cheaper to vectorize than [B].
+
+ (2) The scalar code in [A] is simpler than the scalar code in [B]
+ (if the loops cannot be vectorized or need an epilogue loop).
+
+ (3) We might recognize [A] as a pattern, such as a memcpy or memset.
+
+ (4) [A] has simpler address evolutions, which can help other passes
+ like loop interchange.
+
+ The optimization is particularly useful for assumed-shape arrays in
+ Fortran, where the stride of the innermost dimension depends on the
+ array descriptor but is often equal to 1 in practice. For example:
+
+ subroutine f1(x)
+ real :: x(:)
+ x(:) = 100
+ end subroutine f1
+
+ generates the equivalent of:
+
+ raw_stride = *x.dim[0].stride;
+ stride = raw_stride != 0 ? raw_stride : 1;
+ x_base = *x.data;
+ ...
+ tmp1 = stride * S;
+ tmp2 = tmp1 - stride;
+ *x_base[tmp2] = 1.0e+2;
+
+ but in the common case that stride == 1, the last three statements
+ simplify to:
+
+ tmp3 = S + -1;
+ *x_base[tmp3] = 1.0e+2;
+
+ The optimization is in principle very simple. The difficult parts are:
+
+ (a) deciding which parts of a general address calculation correspond
+ to the inner dimension of an array, since this usually isn't explicit
+ in the IL, and for C often isn't even explicit in the source code
+
+ (b) estimating when the transformation is worthwhile
+
+ Structure
+ ---------
+
+ The pass has four phases:
+
+ (1) Walk through the statements looking for and recording potential
+ versioning opportunities. Stop if there are none.
+
+ (2) Use context-sensitive range information to see whether any versioning
+ conditions are impossible in practice. Remove them if so, and stop
+ if no opportunities remain.
+
+ (We do this only after (1) to keep compile time down when no
+ versioning opportunities exist.)
+
+ (3) Apply the cost model. Decide which versioning opportunities are
+ worthwhile and at which nesting level they should be applied.
+
+ (4) Attempt to version all the loops selected by (3), so that:
+
+ for (...)
+ ...
+
+ becomes:
+
+ if (!cond)
+ for (...) // Original loop
+ ...
+ else
+ for (...) // New loop
+ ...
+
+ Use the version condition COND to simplify the new loop. */
+class loop_versioning {
+public:
+ loop_versioning (function *);
+ ~loop_versioning ();
+ unsigned int run ();
+
+private:
+ /* Information about the versioning we'd like to apply to a loop. */
+ struct loop_info {
+ bool worth_versioning_p () const;
+
+ /* True if we've decided not to version this loop. The remaining
+ fields are meaningless if so. */
+ bool rejected_p;
+
+ /* True if at least one subloop of this loop benefits from versioning. */
+ bool subloops_benefit_p;
+
+ /* An estimate of the total number of instructions in the loop,
+ excluding those in subloops that benefit from versioning. */
+ unsigned int num_insns;
+
+ /* The outermost loop that can handle all the version checks
+ described below. */
+ struct loop *outermost;
+
+ /* We'd like to version the loop for the case in which these
+ SSA_NAMEs are all equal to 1 at runtime. */
+ vec<tree> unity_names;
+
+ /* The set of SSA_NAMEs in UNITY_NAMES, keyed off the SSA_NAME_VERSION. */
+ bitmap_head unity_name_ids;
+ };
+
+ /* Used to walk the dominator tree to find loop versioning conditions
+ that are always false. */
+ class lv_dom_walker : public dom_walker
+ {
+ public:
+ lv_dom_walker (loop_versioning &);
+
+ edge before_dom_children (basic_block) FINAL OVERRIDE;
+ void after_dom_children (basic_block) FINAL OVERRIDE;
+
+ private:
+ /* The parent pass. */
+ loop_versioning &m_lv;
+
+ /* Used to build context-dependent range information. */
+ evrp_range_analyzer m_range_analyzer;
+ };
+
+ /* Used to simplify statements based on conditions that are established
+ by the version checks. */
+ class name_prop : public substitute_and_fold_engine
+ {
+ public:
+ name_prop (loop_info &li) : m_li (li) {}
+ tree get_value (tree) FINAL OVERRIDE;
+
+ private:
+ /* Information about the versioning we've performed on the loop. */
+ loop_info &m_li;
+ };
+
+ loop_info &get_loop_info (struct loop *loop) { return m_loops[loop->num]; }
+
+ unsigned int max_insns_for_loop (struct loop *);
+ bool expensive_stmt_p (gimple *);
+
+ void version_for_unity (struct loop *, tree);
+ bool acceptable_scale_p (tree, poly_uint64);
+ tree get_step_if_innermost (tree, poly_uint64);
+ tree extract_step (tree, poly_uint64, tree *);
+ void analyze_evolution (struct loop *, tree, poly_uint64);
+ bool analyze_product (struct loop *, gassign *, poly_uint64);
+ bool analyze_sum_of_products (struct loop *, tree, poly_uint64);
+ void analyze_pointer (struct loop *, tree, tree);
+ void analyze_expr (struct loop *, tree);
+ void analyze_stmt (gimple *);
+ void analyze_block (basic_block);
+ bool analyze_blocks ();
+
+ void prune_loop_conditions (struct loop *, vr_values *);
+ bool prune_conditions ();
+
+ void merge_loop_info (struct loop *, struct loop *);
+ void add_loop_to_queue (struct loop *);
+ bool decide_whether_loop_is_versionable (struct loop *);
+ bool make_versioning_decisions ();
+
+ bool version_loop (struct loop *);
+ bool implement_versioning_decisions ();
+
+ /* The function we're optimizing. */
+ function *m_fn;
+
+ /* The obstack to use for all pass-specific bitmaps. */
+ bitmap_obstack m_obstack;
+
+ /* The number of loops in the function. */
+ unsigned int m_nloops;
+
+ /* The total number of loop version conditions we've found. */
+ unsigned int m_num_conditions;
+
+ /* Information about each loop. */
+ auto_vec<loop_info> m_loops;
+
+ /* The list of loops that we've decided to version. */
+ auto_vec<struct loop *> m_loops_to_version;
+};
+
+/* If EXPR is an SSA name and not a default definition, return the
+ defining statement, otherwise return null. */
+
+static gimple *
+maybe_get_stmt (tree expr)
+{
+ if (TREE_CODE (expr) == SSA_NAME && !SSA_NAME_IS_DEFAULT_DEF (expr))
+ return SSA_NAME_DEF_STMT (expr);
+ return NULL;
+}
+
+/* Like maybe_get_stmt, but also return null if the defining
+ statement isn't an assignment. */
+
+static gassign *
+maybe_get_assign (tree expr)
+{
+ return safe_dyn_cast <gassign *> (maybe_get_stmt (expr));
+}
+
+/* If EXPR is an SSA name, look through any casts to see whether the
+ unconverted value is defined in LOOP by a gassign. Return the
+ gassign if so, otherwise return null. */
+
+gassign *
+maybe_get_assign_strip_casts (struct loop *loop, tree expr)
+{
+ const unsigned int MAX_NITERS = 4;
+
+ tree type = TREE_TYPE (expr);
+ for (unsigned int niters = 0; niters < MAX_NITERS; ++niters)
+ {
+ gassign *assign = maybe_get_assign (expr);
+ if (!assign || gimple_bb (assign)->loop_father != loop)
+ return NULL;
+ expr = gimple_assign_rhs1 (assign);
+ if (CONVERT_EXPR_CODE_P (gimple_assign_rhs_code (assign))
+ && INTEGRAL_TYPE_P (TREE_TYPE (expr)) == INTEGRAL_TYPE_P (type)
+ && POINTER_TYPE_P (TREE_TYPE (expr)) == POINTER_TYPE_P (type))
+ ;
+ else
+ return assign;
+ }
+ return NULL;
+}
+
+/* Strip all conversions of integers from EXPR, regardless of whether
+ the conversions are nops. This is useful in the context of this pass
+ because we're not trying to fold or simulate the expression; we just
+ want to see how it's structured. */
+
+static tree
+strip_casts (tree expr)
+{
+ while (CONVERT_EXPR_P (expr)
+ && INTEGRAL_TYPE_P (TREE_TYPE (TREE_OPERAND (expr, 0))))
+ expr = TREE_OPERAND (expr, 0);
+ return expr;
+}
+
+/* Return true if we want to version the loop, i.e. if we have a
+ specific reason for doing so and no specific reason not to. */
+
+bool
+loop_versioning::loop_info::worth_versioning_p () const
+{
+ return !rejected_p && (!unity_names.is_empty () || subloops_benefit_p);
+}
+
+loop_versioning::lv_dom_walker::lv_dom_walker (loop_versioning &lv)
+ : dom_walker (CDI_DOMINATORS), m_lv (lv)
+{
+}
+
+/* Process BB before processing the blocks it dominates. */
+
+edge
+loop_versioning::lv_dom_walker::before_dom_children (basic_block bb)
+{
+ m_range_analyzer.enter (bb);
+
+ if (bb == bb->loop_father->header)
+ m_lv.prune_loop_conditions (bb->loop_father,
+ m_range_analyzer.get_vr_values ());
+
+ for (gimple_stmt_iterator si = gsi_start_bb (bb); !gsi_end_p (si);
+ gsi_next (&si))
+ m_range_analyzer.record_ranges_from_stmt (gsi_stmt (si), false);
+
+ return NULL;
+}
+
+/* Process BB after processing the blocks it dominates. */
+
+void
+loop_versioning::lv_dom_walker::after_dom_children (basic_block bb)
+{
+ m_range_analyzer.leave (bb);
+}
+
+/* Decide whether to replace VAL with a new value in a versioned loop.
+ Return the new value if so, otherwise return null. */
+
+tree
+loop_versioning::name_prop::get_value (tree val)
+{
+ if (TREE_CODE (val) == SSA_NAME
+ && bitmap_bit_p (&m_li.unity_name_ids, SSA_NAME_VERSION (val)))
+ return build_one_cst (TREE_TYPE (val));
+ return NULL_TREE;
+}
+
+/* Initialize the structure to optimize FN. */
+
+loop_versioning::loop_versioning (function *fn)
+ : m_fn (fn),
+ m_nloops (number_of_loops (fn)),
+ m_num_conditions (0)
+{
+ bitmap_obstack_initialize (&m_obstack);
+
+ m_loops.safe_grow_cleared (m_nloops);
+ for (unsigned int i = 0; i < m_nloops; ++i)
+ {
+ m_loops[i].outermost = get_loop (m_fn, 0);
+ bitmap_initialize (&m_loops[i].unity_name_ids, &m_obstack);
+ }
+}
+
+loop_versioning::~loop_versioning ()
+{
+ for (unsigned int i = 0; i < m_nloops; ++i)
+ m_loops[i].unity_names.release ();
+ bitmap_obstack_release (&m_obstack);
+}
+
+/* Return the maximum number of instructions allowed in LOOP before
+ it becomes too big for versioning.
+
+ There are separate limits for inner and outer loops. The limit for
+ inner loops applies only to loops that benefit directly from versioning.
+ The limit for outer loops applies to all code in the outer loop and
+ its subloops that *doesn't* benefit directly from versioning; such code
+ would be "taken along for the ride". The idea is that if the cost of
+ the latter is small, it is better to version outer loops rather than
+ inner loops, both to reduce the number of repeated checks and to enable
+ more of the loop nest to be optimized as a natural nest (e.g. by loop
+ interchange or outer-loop vectorization). */
+
+unsigned int
+loop_versioning::max_insns_for_loop (struct loop *loop)
+{
+ return (loop->inner
+ ? PARAM_VALUE (PARAM_LOOP_VERSIONING_MAX_OUTER_INSNS)
+ : PARAM_VALUE (PARAM_LOOP_VERSIONING_MAX_INNER_INSNS));
+}
+
+/* Return true if for cost reasons we should avoid versioning any loop
+ that contains STMT.
+
+ Note that we don't need to check whether versioning is invalid for
+ correctness reasons, since the versioning process does that for us.
+ The conditions involved are too rare to be worth duplicating here. */
+
+bool
+loop_versioning::expensive_stmt_p (gimple *stmt)
+{
+ if (gcall *call = dyn_cast <gcall *> (stmt))
+ /* Assume for now that the time spent in an "expensive" call would
+ overwhelm any saving from versioning. */
+ return !gimple_inexpensive_call_p (call);
+ return false;
+}
+
+/* Record that we want to version LOOP for the case in which SSA name NAME
+ is equal to 1. We already know that NAME is invariant in LOOP. */
+
+void
+loop_versioning::version_for_unity (struct loop *loop, tree name)
+{
+ loop_info &li = get_loop_info (loop);
+
+ if (bitmap_set_bit (&li.unity_name_ids, SSA_NAME_VERSION (name)))
+ {
+ /* This is the first time we've wanted to version LOOP for NAME. */
+ li.unity_names.safe_push (name);
+
+ /* Keep track of the outermost loop that can handle all versioning
+ checks in LI. */
+ struct loop *outermost
+ = outermost_invariant_loop_for_expr (loop, name);
+ if (loop_depth (li.outermost) < loop_depth (outermost))
+ li.outermost = outermost;
+
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ {
+ fprintf (dump_file, ";; Want to version loop %d (depth %d)"
+ " for when ", loop->num, loop_depth (loop));
+ print_generic_expr (dump_file, name, TDF_SLIM);
+ fprintf (dump_file, " == 1");
+ if (outermost == loop)
+ fprintf (dump_file, "; cannot hoist check further");
+ else
+ {
+ fprintf (dump_file, "; could hoist check to loop %d (depth %d)",
+ outermost->num, loop_depth (outermost));
+ if (loop_depth (li.outermost) > loop_depth (outermost))
+ fprintf (dump_file, ", but that's further than"
+ " other checks allow");
+ }
+ fprintf (dump_file, "\n");
+ }
+
+ m_num_conditions += 1;
+ }
+ else
+ {
+ /* This is a duplicate request. */
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ {
+ fprintf (dump_file, ";; Already want to version loop for when ");
+ print_generic_expr (dump_file, name, TDF_SLIM);
+ fprintf (dump_file, " == 1\n");
+ }
+ }
+}
+
+/* Return true if in principle it is worth versioning an index fragment of
+ the form:
+
+ (i * b * SCALE) / FACTOR
+
+ for the case in which b == 1. */
+
+bool
+loop_versioning::acceptable_scale_p (tree scale, poly_uint64 factor)
+{
+ /* See whether SCALE is a constant multiple of FACTOR, and if the
+ multiple is small enough for us to treat it as a potential grouped
+ access. For example:
+
+ for (auto i : ...)
+ y[i] = f (x[4 * i * stride],
+ x[4 * i * stride + 1],
+ x[4 * i * stride + 2]);
+
+ would benefit from versioning for the case in which stride == 1.
+ High multiples of i * stride are less likely to benefit, and could
+ indicate a simulated multi-dimensional array.
+
+ This is just a heuristic, to avoid having to do expensive group
+ analysis of the data references in a loop. */
+ poly_uint64 const_scale;
+ unsigned int multiple;
+ if (poly_int_tree_p (scale, &const_scale)
+ && constant_multiple_p (const_scale, factor, &multiple))
+ {
+ unsigned int maxval = PARAM_VALUE (PARAM_LOOP_VERSIONING_GROUP_SIZE);
+ return IN_RANGE (multiple, 1, maxval);
+ }
+
+ return false;
+}
+
+/* Decide whether an index fragment of the form:
+
+ (i * STEP) / FACTOR
+
+ is likely to be for an innermost dimension. If we think it is,
+ return one of the constant values that it could have (returning 1 if
+ that's a possibility). If think it isn't, return null. Otherwise
+ return STEP, to indicate that it might or might not be an inner
+ dimension. */
+
+tree
+loop_versioning::get_step_if_innermost (tree step, poly_uint64 factor)
+{
+ const unsigned int MAX_NITERS = 8;
+
+ tree likely = NULL_TREE;
+ tree unlikely = NULL_TREE;
+ tree worklist[MAX_NITERS];
+ unsigned int length = 0;
+ worklist[length++] = step;
+ for (unsigned int i = 0; i < length; ++i)
+ {
+ tree expr = worklist[i];
+
+ if (TREE_CONSTANT (expr))
+ {
+ /* See if multiplying by EXPR applies a scale that would be
+ consistent with an individual access or a small grouped
+ access. */
+ if (acceptable_scale_p (expr, factor))
+ {
+ likely = expr;
+ if (integer_onep (expr))
+ break;
+ }
+ else
+ unlikely = expr;
+ continue;
+ }
+
+ /* Otherwise we can only handle SSA names. */
+ gimple *stmt = maybe_get_stmt (expr);
+ if (!stmt)
+ continue;
+
+ /* If EXPR is set by a PHI node, queue its arguments in case
+ we find one that is consistent with an inner dimension.
+
+ An important instance of this is the Fortran handling of array
+ descriptors, which calculates the stride of the inner dimension
+ using a PHI equivalent of:
+
+ raw_stride = a.dim[0].stride;
+ stride = raw_stride != 0 ? raw_stride : 1;
+
+ (Strides for outer dimensions do not treat 0 specially.) */
+ if (gphi *phi = dyn_cast <gphi *> (stmt))
+ {
+ unsigned int nargs = gimple_phi_num_args (phi);
+ for (unsigned int j = 0; j < nargs && length < MAX_NITERS; ++j)
+ worklist[length++] = gimple_phi_arg_def (phi, j);
+ continue;
+ }
+
+ /* If the value is set by an assignment, expect it to be read from
+ memory (such as an array descriptor) rather than be calculated. */
+ if (gassign *assign = dyn_cast <gassign *> (stmt))
+ {
+ if (gimple_assign_single_p (assign)
+ && is_gimple_lvalue (gimple_assign_rhs1 (assign)))
+ continue;
+
+ unlikely = expr;
+ }
+
+ /* Things like calls don't really tell us anything. */
+ }
+
+ if (likely)
+ {
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file, "likely to be innermost dimension\n");
+ return likely;
+ }
+
+ if (unlikely)
+ {
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file, "probably not innermost dimension\n");
+ return NULL_TREE;
+ }
+
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ {
+ if (maybe_ne (factor, 1U))
+ fprintf (dump_file, "didn't find expected scaling factor\n");
+ else
+ fprintf (dump_file, "no information about value\n");
+ }
+ return step;
+}
+
+/* STEP appears in an index fragment of the form:
+
+ {..., +, STEP}_n / FACTOR
+
+ Remove any conversions and grouping or scaling factors from STEP and
+ return the underlying index step. Set *STEP_IF_INNERMOST to the value
+ returned by get_step_if_innermost. */
+
+tree
+loop_versioning::extract_step (tree step, poly_uint64 factor,
+ tree *step_if_innermost)
+{
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ {
+ fprintf (dump_file, ";; step ");
+ print_generic_expr (dump_file, step, TDF_SLIM);
+ fprintf (dump_file, ": ");
+ }
+
+ step = strip_casts (step);
+
+ /* Peel any scaling, which generally happens after conversion to
+ pointer width. For example, on LP64 systems:
+
+ int *x, i, stride;
+ ... x[4 * i * stride] ...;
+
+ multiplies i * stride by 4 using ints, then widens the result
+ to pointer width before multiplying by sizeof (int). */
+ poly_uint64 scale;
+ if (TREE_CODE (step) == MULT_EXPR
+ && poly_int_tree_p (TREE_OPERAND (step, 1), &scale)
+ && known_eq (scale, factor))
+ {
+ step = strip_casts (TREE_OPERAND (step, 0));
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file, "scaled, ");
+ factor = 1;
+ }
+
+ /* Peel any grouping multiple; see acceptable_scale_p for details. */
+ if (TREE_CODE (step) == MULT_EXPR
+ && acceptable_scale_p (TREE_OPERAND (step, 1), factor))
+ {
+ step = strip_casts (TREE_OPERAND (step, 0));
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file, "%sgrouped, ",
+ maybe_ne (factor, 1U) ? "scaled, " : "");
+ factor = 1;
+ }
+
+ *step_if_innermost = get_step_if_innermost (step, factor);
+ return step;
+}
+
+/* Analyze the evolution of index fragment EXPR / FACTOR in LOOP and its
+ containing loops to see whether any part of it could be simplified
+ by versioning. Register the versioning opportunities if so. */
+
+void
+loop_versioning::analyze_evolution (struct loop *loop, tree expr,
+ poly_uint64 factor)
+{
+ const unsigned int MAX_NSPLIT = 8;
+
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ {
+ fprintf (dump_file, ";; Analyzing use of ");
+ print_generic_expr (dump_file, expr, TDF_SLIM);
+ if (maybe_ne (factor, 1U))
+ {
+ fprintf (dump_file, " (which addresses ");
+ print_dec (factor, dump_file);
+ fprintf (dump_file, " bytes)");
+ }
+ fprintf (dump_file, " in loop %d (depth %d)\n",
+ loop->num, loop_depth (loop));
+ }
+
+ /* The main problem we have here is that we cannot assume that the
+ innermost loop iterates over the innermost dimension of an array.
+ Accidentally adding versioning checks for outer dimensions would
+ cause the version condition to be false, which as well as bloating
+ the code would defeat loop versioning benefits for other accesses.
+
+ Unfortunately all we usually see at this stage is general address
+ arithmetic, with no positive way of identifying how many dimensions
+ an array access has and which multiplication factors in the address
+ expression correspond to which array dimensions. In C code this is
+ often not even explicit in the source, since variable-sized multi-
+ dimensional arrays are often simulated using one-dimensional arrays.
+
+ The three main ways in which we deal with this are:
+
+ - use heuristics that positively identify steps that are likely
+ to represent the inner dimension.
+
+ - use heuristics that positively identify steps that are unlikely
+ to represent the inner dimension.
+
+ - if a part of EXPR is invariant in LOOP, analyze its evolution in
+ the outer loops to see whether we can positively identify any of
+ it as iterating over the inner dimension. */
+ tree best_step = NULL_TREE;
+ auto_vec<tree, MAX_NSPLIT> worklist;
+ worklist.quick_push (expr);
+ unsigned int nsplit = 0;
+ while (!worklist.is_empty ())
+ {
+ expr = strip_casts (worklist.pop ());
+ tree_code code = TREE_CODE (expr);
+
+ if (code == POLYNOMIAL_CHREC)
+ {
+ /* Analyze the CHREC_RIGHT as the step in an index fragment. */
+ tree step_if_innermost;
+ tree step = extract_step (CHREC_RIGHT (expr), factor,
+ &step_if_innermost);
+ if (!best_step)
+ {
+ /* This is the outermost chrec for the original expression.
+ It's not worth carrying on if the step isn't versionable,
+ or if we're pretty sure it's not for the inner dimension. */
+ if (!step_if_innermost
+ || TREE_CODE (step) != SSA_NAME
+ || !expr_invariant_in_loop_p (loop, step))
+ return;
+
+ best_step = step;
+
+ /* We should version for STEP == 1 if we know that that can be
+ true under some circumstances. */
+ if (integer_onep (step_if_innermost))
+ break;
+
+ /* Bail out if this appears to be the step for the innermost
+ dimension, but isn't likely to be 1.
+
+ ??? We could instead version for when it equals
+ STEP_IF_INNERMOST, but it's not likely to have as much
+ benefit as versioning for 1. */
+ if (step_if_innermost != step)
+ return;
+ }
+ else
+ {
+ /* This is an inner chrec. If it looks like it iterates over
+ the innermost dimension, abort any attempt to version for
+ the outermost chrec (which if we reach here wasn't itself
+ obviously iterating over the innermost dimension). */
+ if (step_if_innermost && TREE_CONSTANT (step_if_innermost))
+ return;
+ }
+ worklist.quick_push (CHREC_LEFT (expr));
+ continue;
+ }
+
+ /* Analyze parts of a queued CHREC_LEFT. This is better than just
+ analyzing the evolution of the whole expression since the value
+ could include a mixture of analyzable and unanalyzable elements.
+ Use NSPLIT to count cases in which we add more expressions to
+ analyze, as opposed to just simplifying the existing one. */
+ if (code == PLUS_EXPR || code == POINTER_PLUS_EXPR || code == MINUS_EXPR)
+ {
+ worklist.quick_push (TREE_OPERAND (expr, 0));
+ if (nsplit++ < MAX_NSPLIT)
+ worklist.quick_push (TREE_OPERAND (expr, 1));
+ continue;
+ }
+ if (code == MULT_EXPR)
+ {
+ tree op0 = strip_casts (TREE_OPERAND (expr, 0));
+ tree op1 = TREE_OPERAND (expr, 1);
+ if (TREE_CODE (op0) == PLUS_EXPR || TREE_CODE (op0) == MINUS_EXPR)
+ {
+ tree type = TREE_TYPE (expr);
+ tree op00 = fold_convert (type, TREE_OPERAND (op0, 0));
+ worklist.quick_push (fold_build2 (MULT_EXPR, type, op00, op1));
+ if (nsplit++ < MAX_NSPLIT)
+ {
+ tree op01 = fold_convert (type, TREE_OPERAND (op0, 1));
+ worklist.quick_push (fold_build2 (MULT_EXPR, type,
+ op01, op1));
+ }
+ continue;
+ }
+ }
+
+ /* If EXPR is invariant in LOOP, analyze it wrt the innermost loop
+ for which it could evolve (i.e. the loop containing the outermost
+ one for which EXPR is invariant). */
+ struct loop *wrt_loop = outermost_invariant_loop_for_expr (loop, expr);
+ if (wrt_loop)
+ {
+ wrt_loop = loop_outer (wrt_loop);
+ if (!wrt_loop)
+ continue;
+ }
+ else
+ wrt_loop = loop;
+ tree evolution = analyze_scalar_evolution (wrt_loop, expr);
+ tree chrec = strip_casts (evolution);
+ if (TREE_CODE (chrec) == POLYNOMIAL_CHREC)
+ {
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ {
+ fprintf (dump_file, ";; evolution of ");
+ print_generic_expr (dump_file, expr, TDF_SLIM);
+ fprintf (dump_file, " in loop %d (depth %d): ",
+ wrt_loop->num, loop_depth (wrt_loop));
+ print_generic_expr (dump_file, evolution, TDF_SLIM);
+ fprintf (dump_file, "\n");
+ }
+ worklist.quick_push (chrec);
+ }
+ else
+ {
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ {
+ fprintf (dump_file, ";; cannot analyze ");
+ print_generic_expr (dump_file, expr, TDF_SLIM);
+ fprintf (dump_file, " any further\n");
+ }
+ }
+ }
+ if (best_step)
+ version_for_unity (loop, best_step);
+}
+
+/* Analyze multiplication MULT to see whether we can identify "gather-like"
+ versioning opportunities such as:
+
+ for (int i = 0; i < n; ++i)
+ res += a[index[i] * stride];
+
+ Return true if there was a versioning opportunity.
+
+ LOOP is the loop that contains MULT. Dividing the value of MULT
+ by FACTOR converts it into an element index. */
+
+bool
+loop_versioning::analyze_product (struct loop *loop, gassign *mult,
+ poly_uint64 factor)
+{
+ /* Record the original LHS for the dump message below. */
+ tree lhs = gimple_assign_lhs (mult);
+
+ /* Peel any scaling, which generally happens after conversion to
+ pointer width. For example, on LP64 systems:
+
+ int *x, i, stride;
+ ... x[4 * i * stride] ...;
+
+ multiplies i * stride by 4 using ints, then widens the result
+ to pointer width before multiplying by sizeof (int). */
+ poly_uint64 scale;
+ if (poly_int_tree_p (gimple_assign_rhs2 (mult), &scale)
+ && known_eq (scale, factor))
+ {
+ mult = maybe_get_assign_strip_casts (loop, gimple_assign_rhs1 (mult));
+ if (!mult || gimple_assign_rhs_code (mult) != MULT_EXPR)
+ return false;
+ factor = 1;
+ }
+
+ /* Peel any grouping multiple; see acceptable_scale_p for details. */
+ if (acceptable_scale_p (gimple_assign_rhs2 (mult), factor))
+ {
+ mult = maybe_get_assign_strip_casts (loop, gimple_assign_rhs1 (mult));
+ if (!mult || gimple_assign_rhs_code (mult) != MULT_EXPR)
+ return false;
+ }
+ else if (maybe_ne (factor, 1U))
+ /* We expect to see a scaling multiplication when FACTOR is > 1. */
+ return false;
+
+ /* See whether this multiplication involves a loop-invariant SSA name
+ and a non-invariant SSA name. */
+ tree op1 = gimple_assign_rhs1 (mult);
+ tree op2 = gimple_assign_rhs2 (mult);
+ if (TREE_CODE (op1) != SSA_NAME || TREE_CODE (op2) != SSA_NAME)
+ return false;
+
+ bool invariant1_p = expr_invariant_in_loop_p (loop, op1);
+ bool invariant2_p = expr_invariant_in_loop_p (loop, op2);
+ if (invariant1_p == invariant2_p)
+ return false;
+
+ /* Make sure that the invariant is OP1 and the other operand is OP2. */
+ if (invariant2_p)
+ std::swap (op1, op2);
+
+ /* Bail out if OP2 can be analyzed as an evolution; we want to use
+ analyze_evolution in that case instead. There's no point trying
+ hard to avoid repeating the call to analyze_scalar_evolution since
+ that function does its own caching. */
+ if (tree_contains_chrecs (analyze_scalar_evolution (loop, op2), NULL))
+ return false;
+
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ {
+ fprintf (dump_file, ";; Address fragment ");
+ print_generic_expr (dump_file, lhs, TDF_SLIM);
+ fprintf (dump_file, " multiplies invariant ");
+ print_generic_expr (dump_file, op1, TDF_SLIM);
+ fprintf (dump_file, " by ");
+ print_generic_expr (dump_file, op2, TDF_SLIM);
+ fprintf (dump_file, ", which isn't a scalar evolution\n");
+ }
+
+ version_for_unity (loop, op1);
+ return true;
+}
+
+/* Treat EXPR as a sum of products and apply analyze_product to each of the
+ products. Return true if one of the products provides a versioning
+ opportunity. FACTOR is as for analyze_product. */
+
+bool
+loop_versioning::analyze_sum_of_products (struct loop *loop, tree expr,
+ poly_uint64 factor)
+{
+ const unsigned int MAX_NITERS = 8;
+
+ tree worklist[MAX_NITERS];
+ unsigned int length = 0;
+ worklist[length++] = expr;
+ for (unsigned int i = 0; i < length; ++i)
+ {
+ expr = worklist[i];
+ gassign *assign = maybe_get_assign_strip_casts (loop, expr);
+ if (!assign)
+ continue;
+
+ tree_code code = gimple_assign_rhs_code (assign);
+ if (code == PLUS_EXPR || code == POINTER_PLUS_EXPR || code == MINUS_EXPR)
+ {
+ if (length < MAX_NITERS)
+ worklist[length++] = gimple_assign_rhs1 (assign);
+ if (length < MAX_NITERS)
+ worklist[length++] = gimple_assign_rhs2 (assign);
+ }
+ else if (code == MULT_EXPR && analyze_product (loop, assign, factor))
+ return true;
+ }
+ return false;
+}
+
+/* Analyze pointer expression EXPR, which occurs in loop LOOP and which
+ is used to address a value of type TYPE. */
+
+void
+loop_versioning::analyze_pointer (struct loop *loop, tree expr, tree type)
+{
+ poly_uint64 factor;
+ if (poly_int_tree_p (TYPE_SIZE_UNIT (type), &factor))
+ {
+ if (!analyze_sum_of_products (loop, expr, factor))
+ analyze_evolution (loop, expr, factor);
+ }
+}
+
+/* Analyze expression EXPR, which occurs in loop LOOP. */
+
+void
+loop_versioning::analyze_expr (struct loop *loop, tree expr)
+{
+ while (handled_component_p (expr))
+ {
+ /* See whether we can use versioning to avoid a multiplication
+ in the array index. */
+ if (TREE_CODE (expr) == ARRAY_REF)
+ {
+ if (!analyze_sum_of_products (loop, TREE_OPERAND (expr, 1), 1))
+ analyze_evolution (loop, TREE_OPERAND (expr, 1), 1);
+ }
+ expr = TREE_OPERAND (expr, 0);
+ }
+
+ if (TREE_CODE (expr) == MEM_REF)
+ {
+ tree addr = TREE_OPERAND (expr, 0);
+ /* See whether we can use versioning to avoid a multiplication
+ in the pointer calculation. This is generally only worth
+ doing if the multiplication occurs in this loop rather than
+ an outer loop. */
+ if (!expr_invariant_in_loop_p (loop, addr))
+ analyze_pointer (loop, addr, TREE_TYPE (expr));
+ }
+
+ /* These would be easy to handle if they existed at this stage. */
+ gcc_checking_assert (TREE_CODE (expr) != TARGET_MEM_REF);
+}
+
+/* Analyze STMT looking for useful version checks. */
+
+void
+loop_versioning::analyze_stmt (gimple *stmt)
+{
+ struct loop *loop = gimple_bb (stmt)->loop_father;
+
+ unsigned int nops = gimple_num_ops (stmt);
+ for (unsigned int i = 0; i < nops; ++i)
+ if (tree op = gimple_op (stmt, i))
+ analyze_expr (loop, op);
+}
+
+/* Analyze all the statements in BB looking for useful version checks. */
+
+void
+loop_versioning::analyze_block (basic_block bb)
+{
+ struct loop *loop = bb->loop_father;
+ loop_info &li = get_loop_info (loop);
+ if (li.rejected_p)
+ return;
+
+ for (gimple_stmt_iterator gsi = gsi_start_bb (bb); !gsi_end_p (gsi);
+ gsi_next (&gsi))
+ {
+ gimple *stmt = gsi_stmt (gsi);
+ if (expensive_stmt_p (stmt))
+ {
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ {
+ struct loop *loop = gimple_bb (stmt)->loop_father;
+ fprintf (dump_file, ";; Loop %d (depth %d) contains expensive"
+ " stmt: ", loop->num, loop_depth (loop));
+ print_gimple_stmt (dump_file, stmt, 0, TDF_SLIM);
+ }
+ li.rejected_p = true;
+ break;
+ }
+
+ /* Only look for direct versioning opportunities in inner loops
+ since the benefit tends to be much smaller for outer loops. */
+ if (!loop->inner)
+ analyze_stmt (stmt);
+
+ /* The point of the instruction limit is to prevent excessive
+ code growth, so this is a size-based estimate even though
+ the optimization is aimed at speed. */
+ li.num_insns += estimate_num_insns (stmt, &eni_size_weights);
+ }
+}
+
+/* Analyze all the blocks in the function looking for useful version checks.
+ Return true if we found one. */
+
+bool
+loop_versioning::analyze_blocks ()
+{
+ /* For now we don't try to version the whole function, although
+ versioning at that level could be useful in some cases. */
+ get_loop_info (get_loop (m_fn, 0)).rejected_p = true;
+
+ basic_block bb;
+ FOR_EACH_BB_FN (bb, m_fn)
+ if (loop_outer (bb->loop_father))
+ analyze_block (bb);
+
+ return m_num_conditions != 0;
+}
+
+/* Use the ranges in VRS to remove impossible versioning conditions from
+ LOOP. */
+
+void
+loop_versioning::prune_loop_conditions (struct loop *loop, vr_values *vrs)
+{
+ loop_info &li = get_loop_info (loop);
+
+ unsigned int i = li.unity_names.length ();
+ while (i > 0)
+ {
+ i -= 1;
+ tree name = li.unity_names[i];
+ value_range *vr = vrs->get_value_range (name);
+ if (vr && !range_includes_p (vr, 1))
+ {
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ {
+ fprintf (dump_file, ";; ");
+ print_generic_expr (dump_file, name, TDF_SLIM);
+ fprintf (dump_file, " can never be 1 in loop %d\n", loop->num);
+ }
+
+ li.unity_names.unordered_remove (i);
+ bitmap_clear_bit (&li.unity_name_ids, SSA_NAME_VERSION (name));
+ m_num_conditions -= 1;
+ }
+ }
+}
+
+/* Remove any scheduled loop version conditions that will never be true.
+ Return true if any remain. */
+
+bool
+loop_versioning::prune_conditions ()
+{
+ calculate_dominance_info (CDI_DOMINATORS);
+ lv_dom_walker dom_walker (*this);
+ dom_walker.walk (ENTRY_BLOCK_PTR_FOR_FN (m_fn));
+ return m_num_conditions != 0;
+}
+
+/* Merge the version checks for INNER into immediately-enclosing loop
+ OUTER. */
+
+void
+loop_versioning::merge_loop_info (struct loop *outer, struct loop *inner)
+{
+ loop_info &inner_li = get_loop_info (inner);
+ loop_info &outer_li = get_loop_info (outer);
+
+ tree name;
+ unsigned int i;
+ FOR_EACH_VEC_ELT (inner_li.unity_names, i, name)
+ if (bitmap_set_bit (&outer_li.unity_name_ids, SSA_NAME_VERSION (name)))
+ {
+ outer_li.unity_names.safe_push (name);
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ {
+ fprintf (dump_file, ";; Hoisting check that ");
+ print_generic_expr (dump_file, name, TDF_SLIM);
+ fprintf (dump_file, " == 1 from loop %d (depth %d) to loop %d\n",
+ inner->num, loop_depth (inner), outer->num);
+ }
+ }
+
+ if (loop_depth (outer_li.outermost) < loop_depth (inner_li.outermost))
+ outer_li.outermost = inner_li.outermost;
+}
+
+/* Add LOOP to the queue of loops to version. */
+
+void
+loop_versioning::add_loop_to_queue (struct loop *loop)
+{
+ loop_info &li = get_loop_info (loop);
+
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file, ";; Queuing loop %d (depth %d) for versioning\n",
+ loop->num, loop_depth (loop));
+ m_loops_to_version.safe_push (loop);
+
+ /* Don't try to version superloops. */
+ li.rejected_p = true;
+}
+
+/* Decide whether the cost model would allow us to version LOOP,
+ either directly or as part of a parent loop, and return true if so.
+ This does not imply that the loop is actually worth versioning in its
+ own right, just that it would be valid to version it if something
+ benefited.
+
+ We have already made this decision for all inner loops of LOOP. */
+
+bool
+loop_versioning::decide_whether_loop_is_versionable (struct loop *loop)
+{
+ loop_info &li = get_loop_info (loop);
+
+ if (li.rejected_p)
+ return false;
+
+ /* Examine the decisions made for inner loops. */
+ for (struct loop *inner = loop->inner; inner; inner = inner->next)
+ {
+ loop_info &inner_li = get_loop_info (inner);
+ if (inner_li.rejected_p)
+ {
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file, ";; Not versioning loop %d (depth %d)"
+ " because inner loop %d should not be versioned\n",
+ loop->num, loop_depth (loop), inner->num);
+ return false;
+ }
+
+ if (inner_li.worth_versioning_p ())
+ li.subloops_benefit_p = true;
+
+ /* Accumulate the number of instructions from subloops that are not
+ the innermost, or that don't benefit from versioning. Only the
+ instructions from innermost loops that benefit from versioning
+ should be weighed against loop-versioning-max-inner-insns;
+ everything else should be weighed against
+ loop-versioning-max-outer-insns. */
+ if (!inner_li.worth_versioning_p () || inner->inner)
+ {
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file, ";; Counting %d instructions from"
+ " loop %d (depth %d) against parent loop %d\n",
+ inner_li.num_insns, inner->num, loop_depth (inner),
+ loop->num);
+ li.num_insns += inner_li.num_insns;
+ }
+ }
+
+ /* Enforce the size limits. */
+ if (li.worth_versioning_p ())
+ {
+ unsigned int max_num_insns = max_insns_for_loop (loop);
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file, ";; Loop %d (depth %d) has %d instructions,"
+ " against a limit of %d\n", loop->num, loop_depth (loop),
+ li.num_insns, max_num_insns);
+ if (li.num_insns > max_num_insns)
+ {
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file, ";; Not versioning loop %d (depth %d)"
+ " because it is too big\n", loop->num, loop_depth (loop));
+ return false;
+ }
+ }
+
+ /* Hoist all version checks for subloops to this loop. */
+ for (struct loop *subloop = loop->inner; subloop; subloop = subloop->next)
+ merge_loop_info (loop, subloop);
+
+ return true;
+}
+
+/* Decide which loops to version and add them to the versioning queue.
+ Return true if there are any loops to version. */
+
+bool
+loop_versioning::make_versioning_decisions ()
+{
+ struct loop *loop;
+ FOR_EACH_LOOP (loop, LI_FROM_INNERMOST)
+ {
+ loop_info &linfo = get_loop_info (loop);
+ if (decide_whether_loop_is_versionable (loop))
+ {
+ /* Commit to versioning LOOP directly if we can't hoist the
+ version checks any further. */
+ if (linfo.worth_versioning_p ()
+ && (loop_depth (loop) == 1 || linfo.outermost == loop))
+ add_loop_to_queue (loop);
+ }
+ else
+ {
+ /* We can't version this loop, so individually version any
+ subloops that would benefit and haven't been versioned yet. */
+ linfo.rejected_p = true;
+ for (struct loop *subloop = loop->inner; subloop;
+ subloop = subloop->next)
+ if (get_loop_info (subloop).worth_versioning_p ())
+ add_loop_to_queue (subloop);
+ }
+ }
+
+ return !m_loops_to_version.is_empty ();
+}
+
+/* Attempt to implement loop versioning for LOOP, using the information
+ cached in the associated loop_info. Return true on success. */
+
+bool
+loop_versioning::version_loop (struct loop *loop)
+{
+ loop_info &li = get_loop_info (loop);
+
+ /* Not protected by TDF_DETAILS since this is the main piece of
+ information. */
+ if (dump_file)
+ fprintf (dump_file, ";; Versioning loop %d (depth %d)\n",
+ loop->num, loop_depth (loop));
+
+ /* Build up a condition that selects the original loop instead of
+ the simplified loop. */
+ tree cond = boolean_false_node;
+ tree name;
+ unsigned int i;
+ FOR_EACH_VEC_ELT (li.unity_names, i, name)
+ {
+ tree ne_one = fold_build2 (NE_EXPR, boolean_type_node, name,
+ build_one_cst (TREE_TYPE (name)));
+ cond = fold_build2 (TRUTH_OR_EXPR, boolean_type_node, cond, ne_one);
+ }
+
+ /* Convert the condition into a suitable gcond. */
+ gimple_seq stmts = NULL;
+ cond = force_gimple_operand_1 (cond, &stmts, is_gimple_condexpr, NULL_TREE);
+
+ /* Version the loop. */
+ initialize_original_copy_tables ();
+ basic_block cond_bb;
+ struct loop *else_loop
+ = loop_version (loop, cond, &cond_bb,
+ profile_probability::unlikely (),
+ profile_probability::likely (),
+ profile_probability::unlikely (),
+ profile_probability::likely (), true);
+ free_original_copy_tables ();
+ if (!else_loop)
+ {
+ if (dump_file)
+ fprintf (dump_file, ";; Versioning of loop %d failed\n", loop->num);
+ return false;
+ }
+
+ /* Insert the statements that feed COND. */
+ if (stmts)
+ {
+ gimple_stmt_iterator gsi = gsi_last_bb (cond_bb);
+ gsi_insert_seq_before (&gsi, stmts, GSI_SAME_STMT);
+ }
+
+ /* Simplify the new loop, which is used when COND is false. */
+ name_prop (li).substitute_and_fold (else_loop->header);
+ return true;
+}
+
+/* Attempt to version all loops in the versioning queue. Return true
+ if we succeeded for at least one loop. */
+
+bool
+loop_versioning::implement_versioning_decisions ()
+{
+ bool any_succeeded_p = false;
+
+ struct loop *loop;
+ unsigned int i;
+ FOR_EACH_VEC_ELT (m_loops_to_version, i, loop)
+ if (version_loop (loop))
+ any_succeeded_p = true;
+
+ return any_succeeded_p;
+}
+
+/* Run the pass and return a set of TODO_* flags. */
+
+unsigned int
+loop_versioning::run ()
+{
+ gcc_assert (scev_initialized_p ());
+
+ if (!analyze_blocks ()
+ || !prune_conditions ()
+ || !make_versioning_decisions ()
+ || !implement_versioning_decisions ())
+ return 0;
+
+ return TODO_update_ssa;
+}
+
+/* Loop versioningting pass. */
+
+namespace {
+
+const pass_data pass_data_loop_versioning =
+{
+ GIMPLE_PASS, /* type */
+ "lversion", /* name */
+ OPTGROUP_LOOP, /* optinfo_flags */
+ TV_LOOP_VERSIONING, /* tv_id */
+ PROP_cfg, /* properties_required */
+ 0, /* properties_provided */
+ 0, /* properties_destroyed */
+ 0, /* todo_flags_start */
+ 0, /* todo_flags_finish */
+};
+
+class pass_loop_versioning : public gimple_opt_pass
+{
+public:
+ pass_loop_versioning (gcc::context *ctxt)
+ : gimple_opt_pass (pass_data_loop_versioning, ctxt)
+ {}
+
+ /* opt_pass methods: */
+ virtual bool gate (function *) { return flag_version_loops_for_strides; }
+ virtual unsigned int execute (function *);
+};
+
+unsigned int
+pass_loop_versioning::execute (function *fn)
+{
+ if (number_of_loops (fn) <= 1)
+ return 0;
+
+ return loop_versioning (fn).run ();
+}
+
+} // anon namespace
+
+gimple_opt_pass *
+make_pass_loop_versioning (gcc::context *ctxt)
+{
+ return new pass_loop_versioning (ctxt);
+}
Index: gcc/testsuite/gcc.dg/loop-versioning-1.c
===================================================================
--- /dev/null 2018-09-14 11:16:31.122530289 +0100
+++ gcc/testsuite/gcc.dg/loop-versioning-1.c 2018-10-24 14:02:15.184152693 +0100
@@ -0,0 +1,20 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* The simplest IV case. */
+
+void
+f1 (double *x, int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ x[stepx * i] = 100;
+}
+
+void
+f2 (double *x, int stepx, int limit)
+{
+ for (int i = 0; i < limit; i += stepx)
+ x[i] = 100;
+}
+
+/* { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 1\)} 2 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {Versioning loop} 2 "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-10.c
===================================================================
--- /dev/null 2018-09-14 11:16:31.122530289 +0100
+++ gcc/testsuite/gcc.dg/loop-versioning-10.c 2018-10-24 14:02:15.184152693 +0100
@@ -0,0 +1,17 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Test that we can version a gather-like operation in which a variable
+ stride is applied to the index. */
+
+int
+f1 (int *x, int *index, int step, int n)
+{
+ int res = 0;
+ for (int i = 0; i < n; ++i)
+ res += x[index[i] * step];
+ return res;
+}
+
+/* { dg-final { scan-tree-dump-times {Address[^\n]*invariant} 1 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 1\)} 1 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {Versioning loop} 1 "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-11.c
===================================================================
--- /dev/null 2018-09-14 11:16:31.122530289 +0100
+++ gcc/testsuite/gcc.dg/loop-versioning-11.c 2018-10-24 14:02:15.188152659 +0100
@@ -0,0 +1,29 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Test that we don't try to version for something that is never 1. */
+
+void
+f1 (double *x, int stepx, int n)
+{
+ if (stepx == 1)
+ for (int i = 0; i < n; ++i)
+ x[i] = 100;
+ else
+ for (int i = 0; i < n; ++i)
+ x[stepx * i] = 100;
+}
+
+void
+f2 (double *x, int stepx, int n)
+{
+ if (stepx <= 1)
+ for (int i = 0; i < n; ++i)
+ x[i] = 100;
+ else
+ for (int i = 0; i < n; ++i)
+ x[stepx * i] = 100;
+}
+
+/* { dg-final { scan-tree-dump-times {Want to version loop} 2 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {can never be 1} 2 "lversion" } } */
+/* { dg-final { scan-tree-dump-not {Versioning loop} "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-2.c
===================================================================
--- /dev/null 2018-09-14 11:16:31.122530289 +0100
+++ gcc/testsuite/gcc.dg/loop-versioning-2.c 2018-10-24 14:02:15.188152659 +0100
@@ -0,0 +1,39 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Versioning these loops allows loop interchange. */
+
+void
+f1 (double x[][100], int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j * step][i] = 100;
+}
+
+void
+f2 (double x[][100], int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j][i * step] = 100;
+}
+
+void
+f3 (double x[][100], int step, int limit)
+{
+ for (int i = 0; i < 100; ++i)
+ for (int j = 0; j < limit; j += step)
+ x[j][i] = 100;
+}
+
+void
+f4 (double x[][100], int step, int limit)
+{
+ for (int i = 0; i < limit; i += step)
+ for (int j = 0; j < 100; ++j)
+ x[j][i] = 100;
+}
+
+/* { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 2\)} 4 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {Hoisting check} 4 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {Versioning loop [0-9]+ \(depth 1\)} 4 "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-3.c
===================================================================
--- /dev/null 2018-09-14 11:16:31.122530289 +0100
+++ gcc/testsuite/gcc.dg/loop-versioning-3.c 2018-10-24 14:02:15.188152659 +0100
@@ -0,0 +1,24 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Versioning these loops for when both steps are 1 allows loop
+ interchange. */
+
+void
+f1 (double x[][100], int step1, int step2, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j * step1][i * step2] = 100;
+}
+
+void
+f2 (double x[][100], int step1, int step2, int limit)
+{
+ for (int i = 0; i < limit; i += step1)
+ for (int j = 0; j < limit; j += step2)
+ x[j][i] = 100;
+}
+
+/* { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 2\)} 4 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {Hoisting check} 4 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {Versioning loop} 2 "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-4.c
===================================================================
--- /dev/null 2018-09-14 11:16:31.122530289 +0100
+++ gcc/testsuite/gcc.dg/loop-versioning-4.c 2018-10-24 14:02:15.188152659 +0100
@@ -0,0 +1,38 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* These shouldn't be versioned; it's extremely likely that the code
+ is emulating two-dimensional arrays. */
+
+void
+f1 (double *x, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[i * step + j] = 100;
+}
+
+void
+f2 (double *x, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j * step + i] = 100;
+}
+
+void
+f3 (double *x, int *offsets, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[i * step + j + offsets[i]] = 100;
+}
+
+void
+f4 (double *x, int *offsets, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j * step + i + offsets[i]] = 100;
+}
+
+/* { dg-final { scan-tree-dump-not {Versioning loop} "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-5.c
===================================================================
--- /dev/null 2018-09-14 11:16:31.122530289 +0100
+++ gcc/testsuite/gcc.dg/loop-versioning-5.c 2018-10-24 14:02:15.188152659 +0100
@@ -0,0 +1,17 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* There's no information about whether STEP1 or STEP2 is innermost,
+ so we should assume the code is sensible and version for the inner
+ evolution, i.e. when STEP2 is 1. */
+
+void
+f1 (double *x, int step1, int step2, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[i * step1 + j * step2] = 100;
+}
+
+/* { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 2\) for when step2} 1 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {Want to version loop} 1 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {Versioning loop} 1 "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-6.c
===================================================================
--- /dev/null 2018-09-14 11:16:31.122530289 +0100
+++ gcc/testsuite/gcc.dg/loop-versioning-6.c 2018-10-24 14:02:15.188152659 +0100
@@ -0,0 +1,30 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* The read from y in f1 will be hoisted to the outer loop. In general
+ it's not worth versioning outer loops when the inner loops don't also
+ benefit.
+
+ This test is meant to be a slight counterexample, since versioning
+ does lead to cheaper outer-loop vectorization. However, the benefit
+ isn't enough to justify the cost. */
+
+void
+f1 (double *restrict x, double *restrict y, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[i + j] = y[i * step];
+}
+
+/* A similar example in which the read can't be hoisted, but could
+ for example be handled by vectorizer alias checks. */
+
+void
+f2 (double *x, double *y, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[i + j] = y[i * step];
+}
+
+/* { dg-final { scan-tree-dump-not {Versioning loop} "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-7.c
===================================================================
--- /dev/null 2018-09-14 11:16:31.122530289 +0100
+++ gcc/testsuite/gcc.dg/loop-versioning-7.c 2018-10-24 14:02:15.188152659 +0100
@@ -0,0 +1,32 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Check that versioning can handle arrays of structures. */
+
+struct foo {
+ int a, b, c;
+};
+
+void
+f1 (struct foo *x, int stepx, int n)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ x[stepx * i].a = 1;
+ x[stepx * i].b = 2;
+ x[stepx * i].c = 3;
+ }
+}
+
+void
+f2 (struct foo *x, int stepx, int limit)
+{
+ for (int i = 0; i < limit; i += stepx)
+ {
+ x[i].a = 1;
+ x[i].b = 2;
+ x[i].c = 3;
+ }
+}
+
+/* { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 1\)} 2 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {Versioning loop} 2 "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-8.c
===================================================================
--- /dev/null 2018-09-14 11:16:31.122530289 +0100
+++ gcc/testsuite/gcc.dg/loop-versioning-8.c 2018-10-24 14:02:15.188152659 +0100
@@ -0,0 +1,44 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Check that versioning can handle arrays of structures that wrap
+ subarrays. */
+
+struct foo {
+ int a[100];
+};
+
+void
+f1 (struct foo *x, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j * step].a[i] = 100;
+}
+
+void
+f2 (struct foo *x, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ for (int j = 0; j < n; ++j)
+ x[j].a[i * step] = 100;
+}
+
+void
+f3 (struct foo *x, int step, int limit)
+{
+ for (int i = 0; i < 100; ++i)
+ for (int j = 0; j < limit; j += step)
+ x[j].a[i] = 100;
+}
+
+void
+f4 (struct foo *x, int step, int limit)
+{
+ for (int i = 0; i < limit; i += step)
+ for (int j = 0; j < 100; ++j)
+ x[j].a[i] = 100;
+}
+
+/* { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 2\)} 4 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {Hoisting check} 4 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {Versioning loop [0-9]+ \(depth 1\)} 4 "lversion" } } */
Index: gcc/testsuite/gcc.dg/loop-versioning-9.c
===================================================================
--- /dev/null 2018-09-14 11:16:31.122530289 +0100
+++ gcc/testsuite/gcc.dg/loop-versioning-9.c 2018-10-24 14:02:15.188152659 +0100
@@ -0,0 +1,48 @@
+/* { dg-options "-O3 -fdump-tree-lversion-details" } */
+
+/* Check that versioning can handle small groups of accesses. */
+
+void
+f1 (int *x, int *y, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ x[i] = y[i * step * 2] + y[i * step * 2 + 1];
+}
+
+void
+f2 (int *x, int *y, __INTPTR_TYPE__ step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ x[i] = y[i * step * 2] + y[i * step * 2 + 1];
+}
+
+void
+f3 (int *x, int *y, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ x[i] = y[i * step * 3] + y[i * step * 3 + 2];
+}
+
+void
+f4 (int *x, int *y, __INTPTR_TYPE__ step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ x[i] = y[i * step * 3] + y[i * step * 3 + 2];
+}
+
+void
+f5 (int *x, int *y, int step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ x[i] = y[i * step * 4] + y[i * step * 4 + 3];
+}
+
+void
+f6 (int *x, int *y, __INTPTR_TYPE__ step, int n)
+{
+ for (int i = 0; i < n; ++i)
+ x[i] = y[i * step * 4] + y[i * step * 4 + 3];
+}
+
+/* { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 1\)} 6 "lversion" } } */
+/* { dg-final { scan-tree-dump-times {Versioning loop} 6 "lversion" } } */
Index: gcc/testsuite/gfortran.dg/loop_versioning_1.f90
===================================================================
--- /dev/null 2018-09-14 11:16:31.122530289 +0100
+++ gcc/testsuite/gfortran.dg/loop_versioning_1.f90 2018-10-24 14:02:15.188152659 +0100
@@ -0,0 +1,28 @@
+! { dg-options "-O3 -fdump-tree-lversion-details" }
+
+! The simplest IV case.
+
+subroutine f1(x)
+ real :: x(:)
+ x(:) = 100
+end subroutine f1
+
+subroutine f2(x, n, step)
+ integer :: n, step
+ real :: x(n * step)
+ do i = 1, n
+ x(i * step) = 100
+ end do
+end subroutine f2
+
+subroutine f3(x, limit, step)
+ integer :: limit, step
+ real :: x(limit)
+ do i = 1, limit, step
+ x(i) = 100
+ end do
+end subroutine f3
+
+! { dg-final { scan-tree-dump-times {likely to be innermost dimension} 1 "lversion" } }
+! { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 1\)} 3 "lversion" } }
+! { dg-final { scan-tree-dump-times {Versioning loop} 3 "lversion" } }
Index: gcc/testsuite/gfortran.dg/loop_versioning_2.f90
===================================================================
--- /dev/null 2018-09-14 11:16:31.122530289 +0100
+++ gcc/testsuite/gfortran.dg/loop_versioning_2.f90 2018-10-24 14:02:15.188152659 +0100
@@ -0,0 +1,38 @@
+! { dg-options "-O3 -fdump-tree-lversion-details -fno-frontend-loop-interchange" }
+
+! We could version the loop for when the first dimension has a stride
+! of 1, but at present there's no real benefit. The gimple loop
+! interchange pass couldn't handle the versioned loop, and interchange
+! is instead done by the frontend (but disabled by the options above).
+
+subroutine f1(x)
+ real :: x(:, :)
+ do i = lbound(x, 1), ubound(x, 1)
+ do j = lbound(x, 2), ubound(x, 2)
+ x(i, j) = 100
+ end do
+ end do
+end subroutine f1
+
+subroutine f2(x, n, step)
+ integer :: n, step
+ real :: x(100, 100)
+ do i = 1, n
+ do j = 1, n
+ x(i * step, j) = 100
+ end do
+ end do
+end subroutine f2
+
+subroutine f3(x, n, step)
+ integer :: n, step
+ real :: x(n * step, n)
+ do i = 1, n
+ do j = 1, n
+ x(i * step, j) = 100
+ end do
+ end do
+end subroutine f3
+
+! { dg-final { scan-tree-dump-times {likely to be innermost dimension} 1 "lversion" } }
+! { dg-final { scan-tree-dump-not {Versioning loop} "lversion" } }
Index: gcc/testsuite/gfortran.dg/loop_versioning_3.f90
===================================================================
--- /dev/null 2018-09-14 11:16:31.122530289 +0100
+++ gcc/testsuite/gfortran.dg/loop_versioning_3.f90 2018-10-24 14:02:15.188152659 +0100
@@ -0,0 +1,29 @@
+! { dg-options "-O3 -fdump-tree-lversion-details -fno-frontend-loop-interchange" }
+
+! Test a case in which the outer loop iterates over the inner dimension.
+! The options above prevent the frontend from interchanging the loops.
+
+subroutine f1(x, limit, step, n)
+ integer :: limit, step, n
+ real :: x(limit, n)
+ do i = 1, limit, step
+ do j = 1, n
+ x(i, j) = 100
+ end do
+ end do
+end subroutine f1
+
+subroutine f2(x, n, limit, step)
+ integer :: n, limit, step
+ real :: x(limit, n)
+ do i = 1, n
+ do j = 1, limit, step
+ x(j, i) = 100
+ end do
+ end do
+end subroutine f2
+
+! FIXME: The frontend doesn't give us enough information to tell which loop
+! is iterating over the innermost dimension, so we optimistically
+! assume the inner one is.
+! { dg-final { scan-tree-dump-not {Versioning loop} "lversion" { xfail *-*-* } } }
Index: gcc/testsuite/gfortran.dg/loop_versioning_4.f90
===================================================================
--- /dev/null 2018-09-14 11:16:31.122530289 +0100
+++ gcc/testsuite/gfortran.dg/loop_versioning_4.f90 2018-10-24 14:02:15.188152659 +0100
@@ -0,0 +1,52 @@
+! { dg-options "-O3 -fdump-tree-lversion-details -fno-frontend-loop-interchange" }
+
+! Test cases in which versioning is useful for a two-dimensional array.
+
+subroutine f1(x)
+ real :: x(:, :)
+ x(:, :) = 100
+end subroutine f1
+
+subroutine f2(x)
+ real :: x(:, :)
+ do i = lbound(x, 1), ubound(x, 1)
+ do j = lbound(x, 2), ubound(x, 2)
+ x(j, i) = 100
+ end do
+ end do
+end subroutine f2
+
+subroutine f3(x, n, step)
+ integer :: n, step
+ real :: x(100, 100)
+ do i = 1, n
+ do j = 1, n
+ x(j * step, i) = 100
+ end do
+ end do
+end subroutine f3
+
+subroutine f4(x, n, step)
+ integer :: n, step
+ real :: x(n * step, n)
+ do i = 1, n
+ do j = 1, n
+ x(j * step, i) = 100
+ end do
+ end do
+end subroutine f4
+
+subroutine f5(x, n, limit, step)
+ integer :: n, limit, step
+ real :: x(limit, n)
+ do i = 1, n
+ do j = 1, limit, step
+ x(j, i) = 100
+ end do
+ end do
+end subroutine f5
+
+! { dg-final { scan-tree-dump-times {likely to be innermost dimension} 2 "lversion" } }
+! { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 2\)} 5 "lversion" } }
+! { dg-final { scan-tree-dump-times {Hoisting check} 5 "lversion" } }
+! { dg-final { scan-tree-dump-times {Versioning loop [0-9]+ \(depth 1\)} 5 "lversion" } }
Index: gcc/testsuite/gfortran.dg/loop_versioning_5.f90
===================================================================
--- /dev/null 2018-09-14 11:16:31.122530289 +0100
+++ gcc/testsuite/gfortran.dg/loop_versioning_5.f90 2018-10-24 14:02:15.188152659 +0100
@@ -0,0 +1,56 @@
+! { dg-options "-O3 -fdump-tree-lversion-details -fno-frontend-loop-interchange" }
+
+! Make sure that in a "badly nested" loop, we don't treat the inner loop
+! as iterating over the inner dimension with a variable stride.
+
+subroutine f1(x, n)
+ integer :: n
+ real :: x(100, 100)
+ do i = 1, n
+ do j = 1, n
+ x(i, j) = 100
+ end do
+ end do
+end subroutine f1
+
+subroutine f2(x, n, step)
+ integer :: n, step
+ real :: x(100, 100)
+ do i = 1, n
+ do j = 1, n
+ x(i, j * step) = 100
+ end do
+ end do
+end subroutine f2
+
+subroutine f3(x, n)
+ integer :: n
+ real :: x(n, n)
+ do i = 1, n
+ do j = 1, n
+ x(i, j) = 100
+ end do
+ end do
+end subroutine f3
+
+subroutine f4(x, n, step)
+ integer :: n, step
+ real :: x(n, n * step)
+ do i = 1, n
+ do j = 1, n
+ x(i, j * step) = 100
+ end do
+ end do
+end subroutine f4
+
+subroutine f5(x, n, limit, step)
+ integer :: n, limit, step
+ real :: x(n, limit)
+ do i = 1, n
+ do j = 1, limit, step
+ x(i, j) = 100
+ end do
+ end do
+end subroutine f5
+
+! { dg-final { scan-tree-dump-not {Versioning loop} "lversion" } }
Index: gcc/testsuite/gfortran.dg/loop_versioning_6.f90
===================================================================
--- /dev/null 2018-09-14 11:16:31.122530289 +0100
+++ gcc/testsuite/gfortran.dg/loop_versioning_6.f90 2018-10-24 14:02:15.188152659 +0100
@@ -0,0 +1,93 @@
+! { dg-options "-O3 -fdump-tree-lversion-details" }
+
+! Check that versioning can handle small groups of accesses.
+
+subroutine f1(x)
+ real :: x(:)
+ do i = lbound(x, 1), ubound(x, 1) / 2
+ x(i * 2) = 100
+ x(i * 2 + 1) = 101
+ end do
+end subroutine f1
+
+subroutine f2(x, n, step)
+ integer :: n, step
+ real :: x(n * step * 2)
+ do i = 1, n
+ x(i * step * 2) = 100
+ x(i * step * 2 + 1) = 101
+ end do
+end subroutine f2
+
+subroutine f3(x, limit, step)
+ integer :: limit, step
+ real :: x(limit * 2)
+ do i = 1, limit, step
+ x(i * 2) = 100
+ x(i * 2 + 1) = 101
+ end do
+end subroutine f3
+
+subroutine f4(x)
+ real :: x(:)
+ do i = lbound(x, 1), ubound(x, 1) / 3
+ x(i * 3) = 100
+ x(i * 3 + 1) = 101
+ x(i * 3 + 2) = 102
+ end do
+end subroutine f4
+
+subroutine f5(x, n, step)
+ integer :: n, step
+ real :: x(n * step * 3)
+ do i = 1, n
+ x(i * step * 3) = 100
+ x(i * step * 3 + 1) = 101
+ x(i * step * 3 + 2) = 102
+ end do
+end subroutine f5
+
+subroutine f6(x, limit, step)
+ integer :: limit, step
+ real :: x(limit * 3)
+ do i = 1, limit, step
+ x(i * 3) = 100
+ x(i * 3 + 1) = 101
+ x(i * 3 + 2) = 102
+ end do
+end subroutine f6
+
+subroutine f7(x)
+ real :: x(:)
+ do i = lbound(x, 1), ubound(x, 1) / 4
+ x(i * 4) = 100
+ x(i * 4 + 1) = 101
+ x(i * 4 + 2) = 102
+ x(i * 4 + 3) = 103
+ end do
+end subroutine f7
+
+subroutine f8(x, n, step)
+ integer :: n, step
+ real :: x(n * step * 4)
+ do i = 1, n
+ x(i * step * 4) = 100
+ x(i * step * 4 + 1) = 101
+ x(i * step * 4 + 2) = 102
+ x(i * step * 4 + 3) = 103
+ end do
+end subroutine f8
+
+subroutine f9(x, limit, step)
+ integer :: limit, step
+ real :: x(limit * 4)
+ do i = 1, limit, step
+ x(i * 4) = 100
+ x(i * 4 + 1) = 101
+ x(i * 4 + 2) = 102
+ x(i * 4 + 3) = 103
+ end do
+end subroutine f9
+
+! { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 1\)} 9 "lversion" } }
+! { dg-final { scan-tree-dump-times {Versioning loop} 9 "lversion" } }
Index: gcc/testsuite/gfortran.dg/loop_versioning_7.f90
===================================================================
--- /dev/null 2018-09-14 11:16:31.122530289 +0100
+++ gcc/testsuite/gfortran.dg/loop_versioning_7.f90 2018-10-24 14:02:15.188152659 +0100
@@ -0,0 +1,67 @@
+! { dg-options "-O3 -fdump-tree-lversion-details" }
+
+! Check that versioning can handle small groups of accesses, with the
+! group being a separate array dimension.
+
+subroutine f1(x, n, step)
+ integer :: n, step
+ real :: x(2, n * step)
+ do i = 1, n
+ x(1, i * step) = 100
+ x(2, i * step) = 101
+ end do
+end subroutine f1
+
+subroutine f2(x, limit, step)
+ integer :: limit, step
+ real :: x(2, limit)
+ do i = 1, limit, step
+ x(1, i) = 100
+ x(2, i) = 101
+ end do
+end subroutine f2
+
+subroutine f3(x, n, step)
+ integer :: n, step
+ real :: x(3, n * step)
+ do i = 1, n
+ x(1, i * step) = 100
+ x(2, i * step) = 101
+ x(3, i * step) = 102
+ end do
+end subroutine f3
+
+subroutine f4(x, limit, step)
+ integer :: limit, step
+ real :: x(3, limit)
+ do i = 1, limit, step
+ x(1, i) = 100
+ x(2, i) = 101
+ x(3, i) = 102
+ end do
+end subroutine f4
+
+subroutine f5(x, n, step)
+ integer :: n, step
+ real :: x(4, n * step)
+ do i = 1, n
+ x(1, i * step) = 100
+ x(2, i * step) = 101
+ x(3, i * step) = 102
+ x(4, i * step) = 103
+ end do
+end subroutine f5
+
+subroutine f6(x, limit, step)
+ integer :: limit, step
+ real :: x(4, limit)
+ do i = 1, limit, step
+ x(1, i) = 100
+ x(2, i) = 101
+ x(3, i) = 102
+ x(4, i) = 103
+ end do
+end subroutine f6
+
+! { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 1\)} 6 "lversion" } }
+! { dg-final { scan-tree-dump-times {Versioning loop} 6 "lversion" } }
Index: gcc/testsuite/gfortran.dg/loop_versioning_8.f90
===================================================================
--- /dev/null 2018-09-14 11:16:31.122530289 +0100
+++ gcc/testsuite/gfortran.dg/loop_versioning_8.f90 2018-10-24 14:02:15.188152659 +0100
@@ -0,0 +1,13 @@
+! { dg-options "-O3 -fdump-tree-lversion-details" }
+
+! Check that versioning is applied to a gather-like reduction operation.
+
+function f(x, index, n)
+ integer :: n
+ real :: x(:)
+ integer :: index(n)
+ f = sum(x(index(:)))
+end function f
+
+! { dg-final { scan-tree-dump-times {Want to version loop [0-9]+ \(depth 1\)} 1 "lversion" } }
+! { dg-final { scan-tree-dump-times {Versioning loop} 1 "lversion" } }
^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2018-12-15 10:27 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-10-30 10:55 Add a loop versioning pass Richard Biener
2018-11-09 10:46 ` Kyrill Tkachov
2018-11-28 16:48 ` Richard Sandiford
2018-11-29 11:31 ` Martin Jambor
2018-12-03 13:16 ` Richard Biener
2018-12-06 13:19 ` Richard Sandiford
2018-12-12 12:06 ` Richard Biener
2018-12-12 18:43 ` Richard Sandiford
2018-12-13 16:08 ` Richard Biener
2018-12-14 16:18 ` Richard Sandiford
2018-12-15 10:27 ` Richard Biener
-- strict thread matches above, loose matches on Subject: below --
2018-10-24 13:41 Richard Sandiford
2018-10-25 14:16 ` Richard Biener
2018-10-25 16:03 ` Richard Sandiford
2018-10-26 14:49 ` Richard Biener
2018-10-25 16:16 ` David Malcolm
2018-11-28 14:16 ` Richard Sandiford
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).