* [patch] vectorizer cleanups (and prep for outer-loop vectorization)
@ 2007-04-02 8:43 Dorit Nuzman
2007-04-02 9:48 ` Zdenek Dvorak
0 siblings, 1 reply; 18+ messages in thread
From: Dorit Nuzman @ 2007-04-02 8:43 UTC (permalink / raw)
To: gcc-patches
[-- Attachment #1: Type: text/plain, Size: 6151 bytes --]
Hi,
This is exactly the same patch as was committed to autovect last week:
http://gcc.gnu.org/ml/gcc-patches/2007-03/msg01904.html
(but relative to mainline, and after testig on mainline).
These are a few cleanups (really technical things, no new functionality),
some of which will be also useful for outer-loop vectorization:
o Remove the field exit_cond from loop_vec_info, which used to hold the
loop-exit stmt. Instead, a stmt that is identified as a loop-exit control
stmt is marked as a 'loop_exit_ctrl_vec_info_type' (new enum value for
stmt_vec_info_type). This will be useful when looking at outer-loops
because we have more than one stmt like that in the loop (there's also one
in the inner-loop).
o Vectorization of induction is currently done upon encountering a use of
the induction variable. Instead, we now vectorize the induction earlier,
when we encounter the respective induction phi-node. This simplifies things
a bit cause it's more in-line with how we vectorize other things (when we
encounter a use of the induction we just get the vector def from the
STMT_VINFO_VEC_STMT field of the defining phi-node), and also we don't need
to worry about vectorizing the same induction multiple times. When looking
at outer-loops this will be even more useful cause when we vectorize a
phi-node we know exactly relative to which loop-nest is the induction, and
don't need any extra logic to check for that.
o Reorganize the calls to the vectorizable_* functions: In the drivers that
call these functions (vect_analyze_operations and vect_transform_stmt), we
currently call most of these functions only for RELEVANT_P stmts, and then
call vectorizable_reduction only for LIVE_P stmts. Now, instead, each of
the vectorizable_* functions checks that STMT_VINFO_RELEVANT/LIVE_P are as
expected, and in the two driver functions we don't check for
STMT_VINFO_RELEVANT_P anymore, but apply the vectorizable_* functions to
all stmts. When looking at outer-loops this is useful because a reduction
(in the inner-loop) may now be marked RELEVANT (if it is used in the
outer-loop), and not only LIVE as it is currently.
o A bunch of other cleanups (that are not necessarily inspired by
outer-loop vectorization): Factor out code from
vect_mark_stmts_to_be_vectorized to process_use; Remove the '#ifdef
ADJUST_IN_EPILOG' and the code path that was disabled by it (which removes
unused code and simplifies the function); Add a flag to allow disabling
tree-reassociation, to help test reductions that are otherwise transformed
by reassociation to something we currently don't recognize; Fix
indentation; A few ther small miscellaneous stuff.
Bootstrapped and tested (full regression testing) on powerpc64-linux and
i386-linux.
Also bootstrapped with vectorization enabled on powerpc64-linux.
ok for mainline?
:ADDPATCH vectorizer,non-algorithmic:
ChangeLog:
* tree-vectorizer.h (_loop_vec_info): Remove unused field
exit_cond.
(stmt_vec_info_type): Add enum value loop_exit_ctrl_vec_info_type.
* tree-vectorizer.c (new_loop_vec_info): Remove setting of
LOOP_VINFO_EXIT_COND.
* tree-vect-analyze.c (vect_stmt_relevant_p): Replace use of
LOOP_VINFO_EXIT_COND with use of loop_exit_ctrl_vec_info_type.
(vect_analyze_loop_form): Likewise.
* tree-vectorizer.h (stmt_vec_info_type): Add enum value
induc_vec_info_type.
(vectorizable_induction): New function declaration.
* tree-vect-transform.c (get_initial_def_for_induction): No need to
check if already vectorized. Find first place in BB where nre
stmts
can be inserted.
(vectorizable_induction): New function.
(vect_transform_stmt): Add case for induc_vec_info_type to call
vectorizable_induction.
(vect_transform_loop): Consider phis for vectorization.
* tree-vect-analyze.c (analyze_operations): Call
vectorizable_induction when analyzing phis and when analyzing
stmts.
* tree-vectorizer.c (vect_is_simple_use): Remove redundant check.
(destroy_loop_vec_info): Set loop->aux to NULL.
* tree-vect-analyze.c (vect_determine_vectorization_factor):
Simplify
condition. Add dump print.
(vect_analyze_operations): Fix indenetation. Fix a comment. Fix a
print message.
(vect_analyze_scalar_cycles): Fix indentation.
(vect_enhance_data_refs_alignment): Fix check in case of peeling.
(vect_mark_relevant): Include phis in relevance analysis.
(process_use): New function.
(vect_mark_stmts_to_be_vectorized): Factor out code to process_use.
Check phis in all bbs.
* tree-vect-patterns.c (widened_name_p): Remove redundant check.
* tree-vect-analyze.c (vect_analyze_operations): Reorganize calls
to
vectorizable_* functions.
* tree-vect-transform.c (vectorizable_call): Add check for
STMT_VINFO_RELEVANT_P, STMT_VINFO_DEF_TYPE and STMT_VINFO_LIVE_P.
(vectorizable_store): likewise.
(vectorizable_conversion): Add check for STMT_VINFO_DEF_TYPE.
Add comments.
(vectorizable_operation, vectorizable_type_demotion): Likewise.
(vectorizable_type_promotion, vectorizable_load): Likewise.
(vectorizable_live_operation, vectorizable_condition): Likewise.
(vectorizable_assignment): Add check for STMT_VINFO_DEF_TYPE and
STMT_VINFO_LIVE_P.
(vect_transform_stmt): Reorganize calls to vectorizable_*
functions.
(vect_analyze_loop_form): Use the new NITERS_KNOWN_P macro so that
this check could be done without creating a loop_vinfo first.
Record
loop_vinfo in loop->aux.
* tree-vect-transform.c (get_initial_def_for_reduction): Clean away
the unused code for reduction without adjust-in-epilog to simplify
the
function.
* tree-vect-transform.c (vect_transform_loop): Add an assert.
* tree-ssa-reassoc.c (gate_reassoc): New function.
* common.opt (tree-reassoc): New flag.
(pass_reassoc): Add gate.
(See attached file: mainline-vect-cleanups.mar27.txt)
[-- Attachment #2: mainline-vect-cleanups.mar27.txt --]
[-- Type: text/plain, Size: 65441 bytes --]
Index: ChangeLog
===================================================================
*** ChangeLog (revision 123103)
--- ChangeLog (working copy)
***************
*** 1,3 ****
--- 1,70 ----
+ 2007-03-20 Dorit Nuzman <dorit@il.ibm.com>
+
+ * tree-vectorizer.h (_loop_vec_info): Remove unused field exit_cond.
+ (stmt_vec_info_type): Add enum value loop_exit_ctrl_vec_info_type.
+ * tree-vectorizer.c (new_loop_vec_info): Remove setting of
+ LOOP_VINFO_EXIT_COND.
+ * tree-vect-analyze.c (vect_stmt_relevant_p): Replace use of
+ LOOP_VINFO_EXIT_COND with use of loop_exit_ctrl_vec_info_type.
+ (vect_analyze_loop_form): Likewise.
+
+ * tree-vectorizer.h (stmt_vec_info_type): Add enum value
+ induc_vec_info_type.
+ (vectorizable_induction): New function declaration.
+ * tree-vect-transform.c (get_initial_def_for_induction): No need to
+ check if already vectorized. Find first place in BB where nre stmts
+ can be inserted.
+ (vectorizable_induction): New function.
+ (vect_transform_stmt): Add case for induc_vec_info_type to call
+ vectorizable_induction.
+ (vect_transform_loop): Consider phis for vectorization.
+ * tree-vect-analyze.c (analyze_operations): Call
+ vectorizable_induction when analyzing phis and when analyzing
+ stmts.
+
+ * tree-vectorizer.c (vect_is_simple_use): Remove redundant check.
+ (destroy_loop_vec_info): Set loop->aux to NULL.
+
+ * tree-vect-analyze.c (vect_determine_vectorization_factor): Simplify
+ condition. Add dump print.
+ (vect_analyze_operations): Fix indenetation. Fix a comment. Fix a
+ print message.
+ (vect_analyze_scalar_cycles): Fix indentation.
+ (vect_enhance_data_refs_alignment): Fix check in case of peeling.
+ (vect_mark_relevant): Include phis in relevance analysis.
+ (process_use): New function.
+ (vect_mark_stmts_to_be_vectorized): Factor out code to process_use.
+ Check phis in all bbs.
+
+ * tree-vect-patterns.c (widened_name_p): Remove redundant check.
+
+ * tree-vect-analyze.c (vect_analyze_operations): Reorganize calls to
+ vectorizable_* functions.
+ * tree-vect-transform.c (vectorizable_call): Add check for
+ STMT_VINFO_RELEVANT_P, STMT_VINFO_DEF_TYPE and STMT_VINFO_LIVE_P.
+ (vectorizable_store): likewise.
+ (vectorizable_conversion): Add check for STMT_VINFO_DEF_TYPE.
+ Add comments.
+ (vectorizable_operation, vectorizable_type_demotion): Likewise.
+ (vectorizable_type_promotion, vectorizable_load): Likewise.
+ (vectorizable_live_operation, vectorizable_condition): Likewise.
+ (vectorizable_assignment): Add check for STMT_VINFO_DEF_TYPE and
+ STMT_VINFO_LIVE_P.
+ (vect_transform_stmt): Reorganize calls to vectorizable_* functions.
+ (vect_analyze_loop_form): Use the new NITERS_KNOWN_P macro so that
+ this check could be done without creating a loop_vinfo first. Record
+ loop_vinfo in loop->aux.
+
+ * tree-vect-transform.c (get_initial_def_for_reduction): Clean away
+ the unused code for reduction without adjust-in-epilog to simplify the
+ function.
+
+ * tree-vect-transform.c (vect_transform_loop): Add an assert.
+
+ * tree-ssa-reassoc.c (gate_reassoc): New function.
+ * common.opt (tree-reassoc): New flag.
+ (pass_reassoc): Add gate.
+
2007-03-20 Mark Mitchell <mark@codesourcery.com>
* config/arm/elf.h (TARGET_ASM_DESTRUCTOR): Define.
Index: testsuite/gcc.dg/vect/vect-1.c
===================================================================
*** testsuite/gcc.dg/vect/vect-1.c (revision 123103)
--- testsuite/gcc.dg/vect/vect-1.c (working copy)
*************** foo (int n)
*** 38,44 ****
inner-loop: cross iteration cycle; multi-dimensional arrays). */
diff = 0;
for (i = 0; i < N; i++) {
! for (i = 0; i < N; i++) {
diff += (image[i][j] - block[i][j]);
}
}
--- 38,44 ----
inner-loop: cross iteration cycle; multi-dimensional arrays). */
diff = 0;
for (i = 0; i < N; i++) {
! for (j = 0; j < N; j++) {
diff += (image[i][j] - block[i][j]);
}
}
Index: testsuite/gcc.dg/vect/vect.exp
===================================================================
*** testsuite/gcc.dg/vect/vect.exp (revision 123103)
--- testsuite/gcc.dg/vect/vect.exp (working copy)
*************** dg-runtest [lsort [glob -nocomplain $src
*** 169,174 ****
--- 169,175 ----
"" $DEFAULT_VECTCFLAGS
# With -Os
+ set DEFAULT_VECTCFLAGS $SAVED_DEFAULT_VECTCFLAGS
lappend DEFAULT_VECTCFLAGS "-Os"
dg-runtest [lsort [glob -nocomplain $srcdir/$subdir/Os-vect-*.\[cS\]]] \
"" $DEFAULT_VECTCFLAGS
Index: tree-vectorizer.c
===================================================================
*** tree-vectorizer.c (revision 123103)
--- tree-vectorizer.c (working copy)
*************** new_loop_vec_info (struct loop *loop)
*** 1420,1426 ****
LOOP_VINFO_LOOP (res) = loop;
LOOP_VINFO_BBS (res) = bbs;
- LOOP_VINFO_EXIT_COND (res) = NULL;
LOOP_VINFO_NITERS (res) = NULL;
LOOP_VINFO_VECTORIZABLE_P (res) = 0;
LOOP_PEELING_FOR_ALIGNMENT (res) = 0;
--- 1420,1425 ----
*************** destroy_loop_vec_info (loop_vec_info loo
*** 1511,1516 ****
--- 1510,1516 ----
VEC_free (tree, heap, LOOP_VINFO_MAY_MISALIGN_STMTS (loop_vinfo));
free (loop_vinfo);
+ loop->aux = NULL;
}
*************** vect_is_simple_use (tree operand, loop_v
*** 1714,1728 ****
return false;
}
- /* stmts inside the loop that have been identified as performing
- a reduction operation cannot have uses in the loop. */
- if (*dt == vect_reduction_def && TREE_CODE (*def_stmt) != PHI_NODE)
- {
- if (vect_print_dump_info (REPORT_DETAILS))
- fprintf (vect_dump, "reduction used in loop.");
- return false;
- }
-
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "type of def: %d.",*dt);
--- 1714,1719 ----
*************** vect_is_simple_use (tree operand, loop_v
*** 1730,1742 ****
{
case PHI_NODE:
*def = PHI_RESULT (*def_stmt);
- gcc_assert (*dt == vect_induction_def || *dt == vect_reduction_def
- || *dt == vect_invariant_def);
break;
case GIMPLE_MODIFY_STMT:
*def = GIMPLE_STMT_OPERAND (*def_stmt, 0);
- gcc_assert (*dt == vect_loop_def || *dt == vect_invariant_def);
break;
default:
--- 1721,1730 ----
Index: tree-vectorizer.h
===================================================================
*** tree-vectorizer.h (revision 123103)
--- tree-vectorizer.h (working copy)
*************** typedef struct _loop_vec_info {
*** 93,101 ****
/* The loop basic blocks. */
basic_block *bbs;
- /* The loop exit_condition. */
- tree exit_cond;
-
/* Number of iterations. */
tree num_iters;
--- 93,98 ----
*************** typedef struct _loop_vec_info {
*** 138,144 ****
/* Access Functions. */
#define LOOP_VINFO_LOOP(L) (L)->loop
#define LOOP_VINFO_BBS(L) (L)->bbs
- #define LOOP_VINFO_EXIT_COND(L) (L)->exit_cond
#define LOOP_VINFO_NITERS(L) (L)->num_iters
#define LOOP_VINFO_VECTORIZABLE_P(L) (L)->vectorizable
#define LOOP_VINFO_VECT_FACTOR(L) (L)->vectorization_factor
--- 135,140 ----
*************** typedef struct _loop_vec_info {
*** 151,159 ****
#define LOOP_VINFO_MAY_MISALIGN_STMTS(L) (L)->may_misalign_stmts
#define LOOP_VINFO_LOC(L) (L)->loop_line_number
#define LOOP_VINFO_NITERS_KNOWN_P(L) \
! (host_integerp ((L)->num_iters,0) \
! && TREE_INT_CST_LOW ((L)->num_iters) > 0)
/*-----------------------------------------------------------------*/
/* Info on vectorized defs. */
--- 147,158 ----
#define LOOP_VINFO_MAY_MISALIGN_STMTS(L) (L)->may_misalign_stmts
#define LOOP_VINFO_LOC(L) (L)->loop_line_number
+ #define NITERS_KNOWN_P(n) \
+ (host_integerp ((n),0) \
+ && TREE_INT_CST_LOW ((n)) > 0)
+
#define LOOP_VINFO_NITERS_KNOWN_P(L) \
! NITERS_KNOWN_P((L)->num_iters)
/*-----------------------------------------------------------------*/
/* Info on vectorized defs. */
*************** enum stmt_vec_info_type {
*** 167,175 ****
assignment_vec_info_type,
condition_vec_info_type,
reduc_vec_info_type,
type_promotion_vec_info_type,
type_demotion_vec_info_type,
! type_conversion_vec_info_type
};
/* Indicates whether/how a variable is used in the loop. */
--- 166,176 ----
assignment_vec_info_type,
condition_vec_info_type,
reduc_vec_info_type,
+ induc_vec_info_type,
type_promotion_vec_info_type,
type_demotion_vec_info_type,
! type_conversion_vec_info_type,
! loop_exit_ctrl_vec_info_type
};
/* Indicates whether/how a variable is used in the loop. */
*************** extern bool vectorizable_call (tree, blo
*** 428,433 ****
--- 429,435 ----
extern bool vectorizable_condition (tree, block_stmt_iterator *, tree *);
extern bool vectorizable_live_operation (tree, block_stmt_iterator *, tree *);
extern bool vectorizable_reduction (tree, block_stmt_iterator *, tree *);
+ extern bool vectorizable_induction (tree, block_stmt_iterator *, tree *);
/* Driver for transformation stage. */
extern void vect_transform_loop (loop_vec_info);
Index: tree-vect-analyze.c
===================================================================
*** tree-vect-analyze.c (revision 123103)
--- tree-vect-analyze.c (working copy)
*************** vect_determine_vectorization_factor (loo
*** 122,134 ****
gcc_assert (stmt_info);
! /* Two cases of "relevant" phis: those that define an
! induction that is used in the loop, and those that
! define a reduction. */
! if ((STMT_VINFO_RELEVANT (stmt_info) == vect_used_in_loop
! && STMT_VINFO_DEF_TYPE (stmt_info) == vect_induction_def)
! || (STMT_VINFO_RELEVANT (stmt_info) == vect_used_by_reduction
! && STMT_VINFO_DEF_TYPE (stmt_info) == vect_reduction_def))
{
gcc_assert (!STMT_VINFO_VECTYPE (stmt_info));
scalar_type = TREE_TYPE (PHI_RESULT (phi));
--- 122,128 ----
gcc_assert (stmt_info);
! if (STMT_VINFO_RELEVANT_P (stmt_info))
{
gcc_assert (!STMT_VINFO_VECTYPE (stmt_info));
scalar_type = TREE_TYPE (PHI_RESULT (phi));
*************** vect_determine_vectorization_factor (loo
*** 268,274 ****
}
/* TODO: Analyze cost. Decide if worth while to vectorize. */
!
if (vectorization_factor <= 1)
{
if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
--- 262,269 ----
}
/* TODO: Analyze cost. Decide if worth while to vectorize. */
! if (vect_print_dump_info (REPORT_DETAILS))
! fprintf (vect_dump, "vectorization factor = %d", vectorization_factor);
if (vectorization_factor <= 1)
{
if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
*************** vect_analyze_operations (loop_vec_info l
*** 311,316 ****
--- 306,313 ----
for (phi = phi_nodes (bb); phi; phi = PHI_CHAIN (phi))
{
+ ok = true;
+
stmt_info = vinfo_for_stmt (phi);
if (vect_print_dump_info (REPORT_DETAILS))
{
*************** vect_analyze_operations (loop_vec_info l
*** 319,341 ****
}
gcc_assert (stmt_info);
-
if (STMT_VINFO_LIVE_P (stmt_info))
{
/* FORNOW: not yet supported. */
if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
fprintf (vect_dump, "not vectorized: value used after loop.");
! return false;
}
if (STMT_VINFO_RELEVANT (stmt_info) == vect_used_in_loop
&& STMT_VINFO_DEF_TYPE (stmt_info) != vect_induction_def)
{
! /* Most likely a reduction-like computation that is used
! in the loop. */
if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
! fprintf (vect_dump, "not vectorized: unsupported pattern.");
! return false;
}
}
--- 316,351 ----
}
gcc_assert (stmt_info);
if (STMT_VINFO_LIVE_P (stmt_info))
{
/* FORNOW: not yet supported. */
if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
fprintf (vect_dump, "not vectorized: value used after loop.");
! return false;
}
if (STMT_VINFO_RELEVANT (stmt_info) == vect_used_in_loop
&& STMT_VINFO_DEF_TYPE (stmt_info) != vect_induction_def)
{
! /* A scalar-dependence cycle that we don't support. */
! if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
! fprintf (vect_dump, "not vectorized: scalar dependence cycle.");
! return false;
! }
!
! if (STMT_VINFO_RELEVANT_P (stmt_info)
! && STMT_VINFO_DEF_TYPE (stmt_info) == vect_induction_def)
! ok = vectorizable_induction (phi, NULL, NULL);
!
! if (!ok)
! {
if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
! {
! fprintf (vect_dump,
! "not vectorized: relevant phi not supported: ");
! print_generic_expr (vect_dump, phi, TDF_SLIM);
! }
! return false;
}
}
*************** vect_analyze_operations (loop_vec_info l
*** 343,348 ****
--- 353,359 ----
{
tree stmt = bsi_stmt (si);
stmt_vec_info stmt_info = vinfo_for_stmt (stmt);
+ enum vect_def_type relevance = STMT_VINFO_RELEVANT (stmt_info);
if (vect_print_dump_info (REPORT_DETAILS))
{
*************** vect_analyze_operations (loop_vec_info l
*** 367,421 ****
continue;
}
if (STMT_VINFO_RELEVANT_P (stmt_info))
{
gcc_assert (GIMPLE_STMT_P (stmt)
|| !VECTOR_MODE_P (TYPE_MODE (TREE_TYPE (stmt))));
gcc_assert (STMT_VINFO_VECTYPE (stmt_info));
-
- ok = (vectorizable_type_promotion (stmt, NULL, NULL)
- || vectorizable_type_demotion (stmt, NULL, NULL)
- || vectorizable_conversion (stmt, NULL, NULL)
- || vectorizable_operation (stmt, NULL, NULL)
- || vectorizable_assignment (stmt, NULL, NULL)
- || vectorizable_load (stmt, NULL, NULL)
- || vectorizable_call (stmt, NULL, NULL)
- || vectorizable_store (stmt, NULL, NULL)
- || vectorizable_condition (stmt, NULL, NULL));
-
- if (!ok)
- {
- if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
- {
- fprintf (vect_dump,
- "not vectorized: relevant stmt not supported: ");
- print_generic_expr (vect_dump, stmt, TDF_SLIM);
- }
- return false;
- }
need_to_vectorize = true;
! }
!
! if (STMT_VINFO_LIVE_P (stmt_info))
! {
! ok = vectorizable_reduction (stmt, NULL, NULL);
! if (ok)
! need_to_vectorize = true;
! else
! ok = vectorizable_live_operation (stmt, NULL, NULL);
! if (!ok)
{
! if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
! {
! fprintf (vect_dump,
! "not vectorized: live stmt not supported: ");
! print_generic_expr (vect_dump, stmt, TDF_SLIM);
! }
! return false;
}
! }
} /* stmts in bb */
} /* bbs */
--- 378,434 ----
continue;
}
+ switch (STMT_VINFO_DEF_TYPE (stmt_info))
+ {
+ case vect_loop_def:
+ break;
+
+ case vect_reduction_def:
+ gcc_assert (relevance == vect_unused_in_loop);
+ break;
+
+ case vect_induction_def:
+ case vect_constant_def:
+ case vect_invariant_def:
+ case vect_unknown_def_type:
+ default:
+ gcc_unreachable ();
+ }
+
if (STMT_VINFO_RELEVANT_P (stmt_info))
{
gcc_assert (GIMPLE_STMT_P (stmt)
|| !VECTOR_MODE_P (TYPE_MODE (TREE_TYPE (stmt))));
gcc_assert (STMT_VINFO_VECTYPE (stmt_info));
need_to_vectorize = true;
! }
! ok = (vectorizable_type_promotion (stmt, NULL, NULL)
! || vectorizable_type_demotion (stmt, NULL, NULL)
! || vectorizable_conversion (stmt, NULL, NULL)
! || vectorizable_operation (stmt, NULL, NULL)
! || vectorizable_assignment (stmt, NULL, NULL)
! || vectorizable_load (stmt, NULL, NULL)
! || vectorizable_call (stmt, NULL, NULL)
! || vectorizable_store (stmt, NULL, NULL)
! || vectorizable_condition (stmt, NULL, NULL)
! || vectorizable_reduction (stmt, NULL, NULL));
!
! /* Stmts that are (also) "live" (i.e. - that are used out of the loop)
! need extra handling, except for vectorizable reductions. */
! if (STMT_VINFO_LIVE_P (stmt_info)
! && STMT_VINFO_TYPE (stmt_info) != reduc_vec_info_type)
! ok |= vectorizable_live_operation (stmt, NULL, NULL);
! if (!ok)
! {
! if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
{
! fprintf (vect_dump, "not vectorized: stmt not supported: ");
! print_generic_expr (vect_dump, stmt, TDF_SLIM);
}
! return false;
! }
} /* stmts in bb */
} /* bbs */
*************** vect_analyze_scalar_cycles (loop_vec_inf
*** 605,618 ****
}
if (!access_fn
! || !vect_is_simple_iv_evolution (loop->num, access_fn, &dumy, &dumy))
{
! VEC_safe_push (tree, heap, worklist, phi);
continue;
}
if (vect_print_dump_info (REPORT_DETAILS))
! fprintf (vect_dump, "Detected induction.");
STMT_VINFO_DEF_TYPE (stmt_vinfo) = vect_induction_def;
}
--- 618,631 ----
}
if (!access_fn
! || !vect_is_simple_iv_evolution (loop->num, access_fn, &dumy, &dumy))
{
! VEC_safe_push (tree, heap, worklist, phi);
continue;
}
if (vect_print_dump_info (REPORT_DETAILS))
! fprintf (vect_dump, "Detected induction.");
STMT_VINFO_DEF_TYPE (stmt_vinfo) = vect_induction_def;
}
*************** static bool
*** 1418,1423 ****
--- 1431,1437 ----
vect_enhance_data_refs_alignment (loop_vec_info loop_vinfo)
{
VEC (data_reference_p, heap) *datarefs = LOOP_VINFO_DATAREFS (loop_vinfo);
+ struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
enum dr_alignment_support supportable_dr_alignment;
struct data_reference *dr0 = NULL;
struct data_reference *dr;
*************** vect_enhance_data_refs_alignment (loop_v
*** 1515,1521 ****
/* Often peeling for alignment will require peeling for loop-bound, which in
turn requires that we know how to adjust the loop ivs after the loop. */
! if (!vect_can_advance_ivs_p (loop_vinfo))
do_peeling = false;
if (do_peeling)
--- 1529,1536 ----
/* Often peeling for alignment will require peeling for loop-bound, which in
turn requires that we know how to adjust the loop ivs after the loop. */
! if (!vect_can_advance_ivs_p (loop_vinfo)
! || !slpeel_can_duplicate_loop_p (loop, single_exit (loop)))
do_peeling = false;
if (do_peeling)
*************** vect_mark_relevant (VEC(tree,heap) **wor
*** 2100,2110 ****
if (relevant > STMT_VINFO_RELEVANT (stmt_info))
STMT_VINFO_RELEVANT (stmt_info) = relevant;
- if (TREE_CODE (stmt) == PHI_NODE)
- /* Don't put phi-nodes in the worklist. Phis that are marked relevant
- or live will fail vectorization later on. */
- return;
-
if (STMT_VINFO_RELEVANT (stmt_info) == save_relevant
&& STMT_VINFO_LIVE_P (stmt_info) == save_live_p)
{
--- 2115,2120 ----
*************** vect_stmt_relevant_p (tree stmt, loop_ve
*** 2143,2149 ****
*live_p = false;
/* cond stmt other than loop exit cond. */
! if (is_ctrl_stmt (stmt) && (stmt != LOOP_VINFO_EXIT_COND (loop_vinfo)))
*relevant = vect_used_in_loop;
/* changing memory. */
--- 2153,2160 ----
*live_p = false;
/* cond stmt other than loop exit cond. */
! if (is_ctrl_stmt (stmt)
! && STMT_VINFO_TYPE (vinfo_for_stmt (stmt)) != loop_exit_ctrl_vec_info_type)
*relevant = vect_used_in_loop;
/* changing memory. */
*************** vect_stmt_relevant_p (tree stmt, loop_ve
*** 2180,2185 ****
--- 2191,2271 ----
}
+ /*
+ Function process_use.
+
+ Inputs:
+ - a USE in STMT in a loop represented by LOOP_VINFO
+ - LIVE_P, RELEVANT - enum values to be set in the STMT_VINFO of the stmt
+ that defined USE. This is dont by calling mark_relevant and passing it
+ the WORKLIST (to add DEF_STMT to the WORKlist in case itis relevant).
+
+ Outputs:
+ Generally, LIVE_P and RELEVANT are used to define the liveness and
+ relevance info of the DEF_STMT of this USE:
+ STMT_VINFO_LIVE_P (DEF_STMT_info) <-- live_p
+ STMT_VINFO_RELEVANT (DEF_STMT_info) <-- relevant
+ Exceptions:
+ - case 1: If USE is used only for address computations (e.g. array indexing),
+ which does not need to be directly vectorized, then the liveness/relevance
+ of the respective DEF_STMT is left unchanged.
+ - case 2: If STMT is a reduction phi and DEF_STMT is a reduction stmt, we
+ skip DEF_STMT cause it had already been processed.
+
+ Return true if everyting is as expected. Return false otherwise. */
+
+ static bool
+ process_use (tree stmt, tree use, loop_vec_info loop_vinfo, bool live_p,
+ enum vect_relevant relevant, VEC(tree,heap) **worklist)
+ {
+ struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
+ stmt_vec_info stmt_vinfo = vinfo_for_stmt (stmt);
+ stmt_vec_info dstmt_vinfo;
+ basic_block def_bb;
+ tree def, def_stmt;
+ enum vect_def_type dt;
+
+ /* case 1: we are only interested in uses that need to be vectorized. Uses
+ that are used for address computation are not considered relevant. */
+ if (!exist_non_indexing_operands_for_use_p (use, stmt))
+ return true;
+
+ if (!vect_is_simple_use (use, loop_vinfo, &def_stmt, &def, &dt))
+ {
+ if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
+ fprintf (vect_dump, "not vectorized: unsupported use in stmt.");
+ return false;
+ }
+
+ if (!def_stmt || IS_EMPTY_STMT (def_stmt))
+ return true;
+
+ def_bb = bb_for_stmt (def_stmt);
+ if (!flow_bb_inside_loop_p (loop, def_bb))
+ return true;
+
+ /* case 2: A reduction phi defining a reduction stmt (DEF_STMT). DEF_STMT
+ must have already been processed, so we just check that everything is as
+ expected, and we are done. */
+ dstmt_vinfo = vinfo_for_stmt (def_stmt);
+ if (TREE_CODE (stmt) == PHI_NODE
+ && STMT_VINFO_DEF_TYPE (stmt_vinfo) == vect_reduction_def
+ && TREE_CODE (def_stmt) != PHI_NODE
+ && STMT_VINFO_DEF_TYPE (dstmt_vinfo) == vect_reduction_def)
+ {
+ if (STMT_VINFO_IN_PATTERN_P (dstmt_vinfo))
+ dstmt_vinfo = vinfo_for_stmt (STMT_VINFO_RELATED_STMT (dstmt_vinfo));
+ gcc_assert (STMT_VINFO_RELEVANT (dstmt_vinfo) < vect_used_by_reduction);
+ gcc_assert (STMT_VINFO_LIVE_P (dstmt_vinfo)
+ || STMT_VINFO_RELEVANT (dstmt_vinfo) > vect_unused_in_loop);
+ return true;
+ }
+
+ vect_mark_relevant (worklist, def_stmt, relevant, live_p);
+ return true;
+ }
+
+
/* Function vect_mark_stmts_to_be_vectorized.
Not all stmts in the loop need to be vectorized. For example:
*************** vect_mark_stmts_to_be_vectorized (loop_v
*** 2204,2220 ****
basic_block *bbs = LOOP_VINFO_BBS (loop_vinfo);
unsigned int nbbs = loop->num_nodes;
block_stmt_iterator si;
! tree stmt, use;
stmt_ann_t ann;
- ssa_op_iter iter;
unsigned int i;
stmt_vec_info stmt_vinfo;
basic_block bb;
tree phi;
bool live_p;
enum vect_relevant relevant;
- tree def, def_stmt;
- enum vect_def_type dt;
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "=== vect_mark_stmts_to_be_vectorized ===");
--- 2290,2303 ----
basic_block *bbs = LOOP_VINFO_BBS (loop_vinfo);
unsigned int nbbs = loop->num_nodes;
block_stmt_iterator si;
! tree stmt;
stmt_ann_t ann;
unsigned int i;
stmt_vec_info stmt_vinfo;
basic_block bb;
tree phi;
bool live_p;
enum vect_relevant relevant;
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "=== vect_mark_stmts_to_be_vectorized ===");
*************** vect_mark_stmts_to_be_vectorized (loop_v
*** 2222,2248 ****
worklist = VEC_alloc (tree, heap, 64);
/* 1. Init worklist. */
-
- bb = loop->header;
- for (phi = phi_nodes (bb); phi; phi = PHI_CHAIN (phi))
- {
- if (vect_print_dump_info (REPORT_DETAILS))
- {
- fprintf (vect_dump, "init: phi relevant? ");
- print_generic_expr (vect_dump, phi, TDF_SLIM);
- }
-
- if (vect_stmt_relevant_p (phi, loop_vinfo, &relevant, &live_p))
- vect_mark_relevant (&worklist, phi, relevant, live_p);
- }
-
for (i = 0; i < nbbs; i++)
{
bb = bbs[i];
for (si = bsi_start (bb); !bsi_end_p (si); bsi_next (&si))
{
stmt = bsi_stmt (si);
-
if (vect_print_dump_info (REPORT_DETAILS))
{
fprintf (vect_dump, "init: stmt relevant? ");
--- 2305,2327 ----
worklist = VEC_alloc (tree, heap, 64);
/* 1. Init worklist. */
for (i = 0; i < nbbs; i++)
{
bb = bbs[i];
+ for (phi = phi_nodes (bb); phi; phi = PHI_CHAIN (phi))
+ {
+ if (vect_print_dump_info (REPORT_DETAILS))
+ {
+ fprintf (vect_dump, "init: phi relevant? ");
+ print_generic_expr (vect_dump, phi, TDF_SLIM);
+ }
+
+ if (vect_stmt_relevant_p (phi, loop_vinfo, &relevant, &live_p))
+ vect_mark_relevant (&worklist, phi, relevant, live_p);
+ }
for (si = bsi_start (bb); !bsi_end_p (si); bsi_next (&si))
{
stmt = bsi_stmt (si);
if (vect_print_dump_info (REPORT_DETAILS))
{
fprintf (vect_dump, "init: stmt relevant? ");
*************** vect_mark_stmts_to_be_vectorized (loop_v
*** 2254,2355 ****
}
}
-
/* 2. Process_worklist */
-
while (VEC_length (tree, worklist) > 0)
{
! stmt = VEC_pop (tree, worklist);
if (vect_print_dump_info (REPORT_DETAILS))
{
fprintf (vect_dump, "worklist: examine stmt: ");
print_generic_expr (vect_dump, stmt, TDF_SLIM);
}
! /* Examine the USEs of STMT. For each ssa-name USE that is defined
! in the loop, mark the stmt that defines it (DEF_STMT) as
! relevant/irrelevant and live/dead according to the liveness and
! relevance properties of STMT.
! */
!
! gcc_assert (TREE_CODE (stmt) != PHI_NODE);
!
ann = stmt_ann (stmt);
stmt_vinfo = vinfo_for_stmt (stmt);
-
relevant = STMT_VINFO_RELEVANT (stmt_vinfo);
live_p = STMT_VINFO_LIVE_P (stmt_vinfo);
/* Generally, the liveness and relevance properties of STMT are
! propagated to the DEF_STMTs of its USEs:
! STMT_VINFO_LIVE_P (DEF_STMT_info) <-- live_p
! STMT_VINFO_RELEVANT (DEF_STMT_info) <-- relevant
!
! Exceptions:
!
! (case 1)
! If USE is used only for address computations (e.g. array indexing),
! which does not need to be directly vectorized, then the
! liveness/relevance of the respective DEF_STMT is left unchanged.
!
! (case 2)
! If STMT has been identified as defining a reduction variable, then
! we want to set liveness/relevance as follows:
! STMT_VINFO_LIVE_P (DEF_STMT_info) <-- false
! STMT_VINFO_RELEVANT (DEF_STMT_info) <-- vect_used_by_reduction
! because even though STMT is classified as live (since it defines a
! value that is used across loop iterations) and irrelevant (since it
! is not used inside the loop), it will be vectorized, and therefore
! the corresponding DEF_STMTs need to marked as relevant.
! We distinguish between two kinds of relevant stmts - those that are
! used by a reduction computation, and those that are (also) used by
! a regular computation. This allows us later on to identify stmts
! that are used solely by a reduction, and therefore the order of
! the results that they produce does not have to be kept.
! */
- /* case 2.2: */
if (STMT_VINFO_DEF_TYPE (stmt_vinfo) == vect_reduction_def)
! {
! gcc_assert (relevant == vect_unused_in_loop && live_p);
relevant = vect_used_by_reduction;
! live_p = false;
}
! i = 0;
! FOR_EACH_SSA_TREE_OPERAND (use, stmt, iter, SSA_OP_USE)
{
! if (vect_print_dump_info (REPORT_DETAILS))
{
- fprintf (vect_dump, "worklist: examine use %d: ", i++);
- print_generic_expr (vect_dump, use, TDF_SLIM);
- }
-
- /* case 1: we are only interested in uses that need to be vectorized.
- Uses that are used for address computation are not considered
- relevant.
- */
- if (!exist_non_indexing_operands_for_use_p (use, stmt))
- continue;
-
- if (!vect_is_simple_use (use, loop_vinfo, &def_stmt, &def, &dt))
- {
- if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
- fprintf (vect_dump, "not vectorized: unsupported use in stmt.");
VEC_free (tree, heap, worklist);
return false;
! }
!
! if (!def_stmt || IS_EMPTY_STMT (def_stmt))
! continue;
!
! bb = bb_for_stmt (def_stmt);
! if (!flow_bb_inside_loop_p (loop, bb))
! continue;
! vect_mark_relevant (&worklist, def_stmt, relevant, live_p);
}
! } /* while worklist */
VEC_free (tree, heap, worklist);
return true;
--- 2333,2414 ----
}
}
/* 2. Process_worklist */
while (VEC_length (tree, worklist) > 0)
{
! use_operand_p use_p;
! ssa_op_iter iter;
+ stmt = VEC_pop (tree, worklist);
if (vect_print_dump_info (REPORT_DETAILS))
{
fprintf (vect_dump, "worklist: examine stmt: ");
print_generic_expr (vect_dump, stmt, TDF_SLIM);
}
! /* Examine the USEs of STMT. For each USE, mark the stmt that defines it
! (DEF_STMT) as relevant/irrelevant and live/dead according to the
! liveness and relevance properties of STMT. */
ann = stmt_ann (stmt);
stmt_vinfo = vinfo_for_stmt (stmt);
relevant = STMT_VINFO_RELEVANT (stmt_vinfo);
live_p = STMT_VINFO_LIVE_P (stmt_vinfo);
/* Generally, the liveness and relevance properties of STMT are
! propagated as is to the DEF_STMTs of its USEs:
! live_p <-- STMT_VINFO_LIVE_P (STMT_VINFO)
! relevant <-- STMT_VINFO_RELEVANT (STMT_VINFO)
!
! One exception is when STMT has been identified as defining a reduction
! variable; in this case we set the liveness/relevance as follows:
! live_p = false
! relevant = vect_used_by_reduction
! This is because we distinguish between two kinds of relevant stmts -
! those that are used by a reduction computation, and those that are
! (also) used by a regular computation. This allows us later on to
! identify stmts that are used solely by a reduction, and therefore the
! order of the results that they produce does not have to be kept.
!
! Reduction phis are expected to be used by a reduction stmt; Other
! reduction stmts are expected to be unused in the loop. These are the
! expected values of "relevant" for reduction phis/stmts in the loop:
!
! relevance: phi stmt
! vect_unused_in_loop ok
! vect_used_by_reduction ok
! vect_used_in_loop */
if (STMT_VINFO_DEF_TYPE (stmt_vinfo) == vect_reduction_def)
! {
! switch (relevant)
! {
! case vect_unused_in_loop:
! gcc_assert (TREE_CODE (stmt) != PHI_NODE);
! break;
! case vect_used_by_reduction:
! if (TREE_CODE (stmt) == PHI_NODE)
! break;
! case vect_used_in_loop:
! default:
! if (vect_print_dump_info (REPORT_DETAILS))
! fprintf (vect_dump, "unsupported use of reduction.");
! VEC_free (tree, heap, worklist);
! return false;
! }
relevant = vect_used_by_reduction;
! live_p = false;
}
! FOR_EACH_PHI_OR_STMT_USE (use_p, stmt, iter, SSA_OP_USE)
{
! tree op = USE_FROM_PTR (use_p);
! if (!process_use (stmt, op, loop_vinfo, live_p, relevant, &worklist))
{
VEC_free (tree, heap, worklist);
return false;
! }
}
! } /* while worklist */
VEC_free (tree, heap, worklist);
return true;
*************** vect_analyze_loop_form (struct loop *loo
*** 2582,2591 ****
return false;
}
! loop_vinfo = new_loop_vec_info (loop);
! LOOP_VINFO_NITERS (loop_vinfo) = number_of_iterations;
!
! if (!LOOP_VINFO_NITERS_KNOWN_P (loop_vinfo))
{
if (vect_print_dump_info (REPORT_DETAILS))
{
--- 2641,2647 ----
return false;
}
! if (!NITERS_KNOWN_P (number_of_iterations))
{
if (vect_print_dump_info (REPORT_DETAILS))
{
*************** vect_analyze_loop_form (struct loop *loo
*** 2594,2608 ****
}
}
else
! if (LOOP_VINFO_INT_NITERS (loop_vinfo) == 0)
{
if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
fprintf (vect_dump, "not vectorized: number of iterations = 0.");
return NULL;
}
! LOOP_VINFO_EXIT_COND (loop_vinfo) = loop_cond;
return loop_vinfo;
}
--- 2650,2669 ----
}
}
else
! if (TREE_INT_CST_LOW (number_of_iterations) == 0)
{
if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
fprintf (vect_dump, "not vectorized: number of iterations = 0.");
return NULL;
}
! loop_vinfo = new_loop_vec_info (loop);
! LOOP_VINFO_NITERS (loop_vinfo) = number_of_iterations;
!
! STMT_VINFO_TYPE (vinfo_for_stmt (loop_cond)) = loop_exit_ctrl_vec_info_type;
+ gcc_assert (!loop->aux);
+ loop->aux = loop_vinfo;
return loop_vinfo;
}
Index: common.opt
===================================================================
*** common.opt (revision 123103)
--- common.opt (working copy)
*************** funroll-all-loops
*** 1070,1075 ****
--- 1070,1079 ----
Common Report Var(flag_unroll_all_loops) Optimization
Perform loop unrolling for all loops
+ ftree-reassoc
+ Common Report Var(flag_tree_reassoc) Init(1) Optimization
+ Perform reassociation on trees
+
; Nonzero means that loop optimizer may assume that the induction variables
; that control loops do not overflow and that the loops with nontrivial
; exit condition are not infinite
Index: tree-vect-patterns.c
===================================================================
*** tree-vect-patterns.c (revision 123103)
--- tree-vect-patterns.c (working copy)
*************** widened_name_p (tree name, tree use_stmt
*** 109,118 ****
if (!vect_is_simple_use (oprnd0, loop_vinfo, &dummy, &dummy, &dt))
return false;
- if (dt != vect_invariant_def && dt != vect_constant_def
- && dt != vect_loop_def)
- return false;
-
return true;
}
--- 109,114 ----
Index: tree-vect-transform.c
===================================================================
*** tree-vect-transform.c (revision 123103)
--- tree-vect-transform.c (working copy)
*************** vect_init_vector (tree stmt, tree vector
*** 514,520 ****
/* Function get_initial_def_for_induction
Input:
- STMT - a stmt that performs an induction operation in the loop.
IV_PHI - the initial value of the induction variable
Output:
--- 514,519 ----
*************** vect_init_vector (tree stmt, tree vector
*** 524,532 ****
[X, X + S, X + 2*S, X + 3*S]. */
static tree
! get_initial_def_for_induction (tree stmt, tree iv_phi)
{
! stmt_vec_info stmt_vinfo = vinfo_for_stmt (stmt);
loop_vec_info loop_vinfo = STMT_VINFO_LOOP_VINFO (stmt_vinfo);
struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
tree scalar_type = TREE_TYPE (iv_phi);
--- 523,531 ----
[X, X + S, X + 2*S, X + 3*S]. */
static tree
! get_initial_def_for_induction (tree iv_phi)
{
! stmt_vec_info stmt_vinfo = vinfo_for_stmt (iv_phi);
loop_vec_info loop_vinfo = STMT_VINFO_LOOP_VINFO (stmt_vinfo);
struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
tree scalar_type = TREE_TYPE (iv_phi);
*************** get_initial_def_for_induction (tree stmt
*** 549,575 ****
tree expr;
stmt_vec_info phi_info = vinfo_for_stmt (iv_phi);
tree stmts;
gcc_assert (phi_info);
! if (STMT_VINFO_VEC_STMT (phi_info))
{
! induction_phi = STMT_VINFO_VEC_STMT (phi_info);
! gcc_assert (TREE_CODE (induction_phi) == PHI_NODE);
!
! if (vect_print_dump_info (REPORT_DETAILS))
! {
! fprintf (vect_dump, "induction already vectorized:");
! print_generic_expr (vect_dump, iv_phi, TDF_SLIM);
! fprintf (vect_dump, "\n");
! print_generic_expr (vect_dump, induction_phi, TDF_SLIM);
! }
!
! return PHI_RESULT (induction_phi);
}
- gcc_assert (ncopies >= 1);
-
access_fn = analyze_scalar_evolution (loop, PHI_RESULT (iv_phi));
gcc_assert (access_fn);
ok = vect_is_simple_iv_evolution (loop->num, access_fn, &init_expr, &step_expr);
--- 548,568 ----
tree expr;
stmt_vec_info phi_info = vinfo_for_stmt (iv_phi);
tree stmts;
+ tree stmt = NULL_TREE;
+ block_stmt_iterator si;
+ basic_block bb = bb_for_stmt (iv_phi);
gcc_assert (phi_info);
+ gcc_assert (ncopies >= 1);
! /* Find the first insertion point in the BB. */
! for (si = bsi_start (bb); !bsi_end_p (si); bsi_next (&si))
{
! stmt = bsi_stmt (si);
! if (TREE_CODE (stmt) != LABEL_EXPR)
! break;
}
access_fn = analyze_scalar_evolution (loop, PHI_RESULT (iv_phi));
gcc_assert (access_fn);
ok = vect_is_simple_iv_evolution (loop->num, access_fn, &init_expr, &step_expr);
*************** vect_get_vec_def_for_operand (tree op, t
*** 833,839 ****
gcc_assert (TREE_CODE (def_stmt) == PHI_NODE);
/* Get the def before the loop */
! return get_initial_def_for_induction (stmt, def_stmt);
}
default:
--- 826,832 ----
gcc_assert (TREE_CODE (def_stmt) == PHI_NODE);
/* Get the def before the loop */
! return get_initial_def_for_induction (def_stmt);
}
default:
*************** vect_finish_stmt_generation (tree stmt,
*** 951,958 ****
}
- #define ADJUST_IN_EPILOG 1
-
/* Function get_initial_def_for_reduction
Input:
--- 944,949 ----
*************** vect_finish_stmt_generation (tree stmt,
*** 960,977 ****
INIT_VAL - the initial value of the reduction variable
Output:
! SCALAR_DEF - a tree that holds a value to be added to the final result
! of the reduction (used for "ADJUST_IN_EPILOG" - see below).
Return a vector variable, initialized according to the operation that STMT
! performs. This vector will be used as the initial value of the
! vector of partial results.
! Option1 ("ADJUST_IN_EPILOG"): Initialize the vector as follows:
add: [0,0,...,0,0]
mult: [1,1,...,1,1]
min/max: [init_val,init_val,..,init_val,init_val]
bit and/or: [init_val,init_val,..,init_val,init_val]
! and when necessary (e.g. add/mult case) let the caller know
that it needs to adjust the result by init_val.
Option2: Initialize the vector as follows:
--- 951,968 ----
INIT_VAL - the initial value of the reduction variable
Output:
! ADJUSTMENT_DEF - a tree that holds a value to be added to the final result
! of the reduction (used for adjusting the epilog - see below).
Return a vector variable, initialized according to the operation that STMT
! performs. This vector will be used as the initial value of the
! vector of partial results.
! Option1 (adjust in epilog): Initialize the vector as follows:
add: [0,0,...,0,0]
mult: [1,1,...,1,1]
min/max: [init_val,init_val,..,init_val,init_val]
bit and/or: [init_val,init_val,..,init_val,init_val]
! and when necessary (e.g. add/mult case) let the caller know
that it needs to adjust the result by init_val.
Option2: Initialize the vector as follows:
*************** vect_finish_stmt_generation (tree stmt,
*** 992,1075 ****
or [0,0,0,0] and let the caller know that it needs to adjust
the result at the end by 'init_val'.
! FORNOW: We use the "ADJUST_IN_EPILOG" scheme.
! TODO: Use some cost-model to estimate which scheme is more profitable.
! */
static tree
! get_initial_def_for_reduction (tree stmt, tree init_val, tree *scalar_def)
{
stmt_vec_info stmt_vinfo = vinfo_for_stmt (stmt);
tree vectype = STMT_VINFO_VECTYPE (stmt_vinfo);
int nunits = TYPE_VECTOR_SUBPARTS (vectype);
- int nelements;
enum tree_code code = TREE_CODE (GIMPLE_STMT_OPERAND (stmt, 1));
tree type = TREE_TYPE (init_val);
! tree def;
! tree vec, t = NULL_TREE;
! bool need_epilog_adjust;
int i;
tree vector_type;
gcc_assert (INTEGRAL_TYPE_P (type) || SCALAR_FLOAT_TYPE_P (type));
switch (code)
{
case WIDEN_SUM_EXPR:
case DOT_PROD_EXPR:
case PLUS_EXPR:
if (INTEGRAL_TYPE_P (type))
! def = build_int_cst (type, 0);
else
! def = build_real (type, dconst0);
!
! #ifdef ADJUST_IN_EPILOG
! /* All the 'nunits' elements are set to 0. The final result will be
! adjusted by 'init_val' at the loop epilog. */
! nelements = nunits;
! need_epilog_adjust = true;
! #else
! /* 'nunits - 1' elements are set to 0; The last element is set to
! 'init_val'. No further adjustments at the epilog are needed. */
! nelements = nunits - 1;
! need_epilog_adjust = false;
! #endif
break;
case MIN_EXPR:
case MAX_EXPR:
! def = init_val;
! nelements = nunits;
! need_epilog_adjust = false;
break;
default:
gcc_unreachable ();
}
! for (i = nelements - 1; i >= 0; --i)
! t = tree_cons (NULL_TREE, def, t);
!
! if (nelements == nunits - 1)
! {
! /* Set the last element of the vector. */
! t = tree_cons (NULL_TREE, init_val, t);
! nelements += 1;
! }
! gcc_assert (nelements == nunits);
!
! vector_type = get_vectype_for_scalar_type (TREE_TYPE (def));
! if (TREE_CODE (init_val) == INTEGER_CST || TREE_CODE (init_val) == REAL_CST)
! vec = build_vector (vector_type, t);
! else
! vec = build_constructor_from_list (vector_type, t);
!
! if (!need_epilog_adjust)
! *scalar_def = NULL_TREE;
! else
! *scalar_def = init_val;
!
! return vect_init_vector (stmt, vec, vector_type);
}
--- 983,1038 ----
or [0,0,0,0] and let the caller know that it needs to adjust
the result at the end by 'init_val'.
! FORNOW, we are using the 'adjust in epilog' scheme, because this way the
! initialization vector is simpler (same element in all entries).
! A cost model should help decide between these two schemes. */
static tree
! get_initial_def_for_reduction (tree stmt, tree init_val, tree *adjustment_def)
{
stmt_vec_info stmt_vinfo = vinfo_for_stmt (stmt);
tree vectype = STMT_VINFO_VECTYPE (stmt_vinfo);
int nunits = TYPE_VECTOR_SUBPARTS (vectype);
enum tree_code code = TREE_CODE (GIMPLE_STMT_OPERAND (stmt, 1));
tree type = TREE_TYPE (init_val);
! tree vecdef;
! tree def_for_init;
! tree init_def;
! tree t = NULL_TREE;
int i;
tree vector_type;
gcc_assert (INTEGRAL_TYPE_P (type) || SCALAR_FLOAT_TYPE_P (type));
+ vecdef = vect_get_vec_def_for_operand (init_val, stmt, NULL);
switch (code)
{
case WIDEN_SUM_EXPR:
case DOT_PROD_EXPR:
case PLUS_EXPR:
+ *adjustment_def = init_val;
+ /* Create a vector of zeros for init_def. */
if (INTEGRAL_TYPE_P (type))
! def_for_init = build_int_cst (type, 0);
else
! def_for_init = build_real (type, dconst0);
! for (i = nunits - 1; i >= 0; --i)
! t = tree_cons (NULL_TREE, def_for_init, t);
! vector_type = get_vectype_for_scalar_type (TREE_TYPE (def_for_init));
! init_def = build_vector (vector_type, t);
break;
case MIN_EXPR:
case MAX_EXPR:
! *adjustment_def = NULL_TREE;
! init_def = vecdef;
break;
default:
gcc_unreachable ();
}
! return init_def;
}
*************** vectorizable_call (tree stmt, block_stmt
*** 1816,1821 ****
--- 1779,1798 ----
int ncopies, j, nargs;
call_expr_arg_iterator iter;
+ if (!STMT_VINFO_RELEVANT_P (stmt_info))
+ return false;
+
+ if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
+ return false;
+
+ /* FORNOW: not yet supported. */
+ if (STMT_VINFO_LIVE_P (stmt_info))
+ {
+ if (vect_print_dump_info (REPORT_DETAILS))
+ fprintf (vect_dump, "value used after loop.");
+ return false;
+ }
+
/* Is STMT a vectorizable call? */
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
return false;
*************** vectorizable_conversion (tree stmt, bloc
*** 1994,2000 ****
if (!STMT_VINFO_RELEVANT_P (stmt_info))
return false;
! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
if (STMT_VINFO_LIVE_P (stmt_info))
{
--- 1971,1978 ----
if (!STMT_VINFO_RELEVANT_P (stmt_info))
return false;
! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
! return false;
if (STMT_VINFO_LIVE_P (stmt_info))
{
*************** vectorizable_assignment (tree stmt, bloc
*** 2134,2145 ****
if (ncopies > 1)
return false; /* FORNOW */
- /* Is vectorizable assignment? */
if (!STMT_VINFO_RELEVANT_P (stmt_info))
return false;
! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
return false;
--- 2112,2132 ----
if (ncopies > 1)
return false; /* FORNOW */
if (!STMT_VINFO_RELEVANT_P (stmt_info))
return false;
! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
! return false;
!
! /* FORNOW: not yet supported. */
! if (STMT_VINFO_LIVE_P (stmt_info))
! {
! if (vect_print_dump_info (REPORT_DETAILS))
! fprintf (vect_dump, "value used after loop.");
! return false;
! }
+ /* Is vectorizable assignment? */
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
return false;
*************** vect_min_worthwhile_factor (enum tree_co
*** 2209,2214 ****
--- 2196,2254 ----
}
+ /* Function vectorizable_induction
+
+ Check if PHI performs an induction computation that can be vectorized.
+ If VEC_STMT is also passed, vectorize the induction PHI: create a vectorized
+ phi to replace it, put it in VEC_STMT, and add it to the same basic block.
+ Return FALSE if not a vectorizable STMT, TRUE otherwise. */
+
+ bool
+ vectorizable_induction (tree phi, block_stmt_iterator *bsi ATTRIBUTE_UNUSED,
+ tree *vec_stmt)
+ {
+ stmt_vec_info stmt_info = vinfo_for_stmt (phi);
+ tree vectype = STMT_VINFO_VECTYPE (stmt_info);
+ loop_vec_info loop_vinfo = STMT_VINFO_LOOP_VINFO (stmt_info);
+ int nunits = TYPE_VECTOR_SUBPARTS (vectype);
+ int ncopies = LOOP_VINFO_VECT_FACTOR (loop_vinfo) / nunits;
+ tree vec_def;
+
+ gcc_assert (ncopies >= 1);
+
+ if (!STMT_VINFO_RELEVANT_P (stmt_info))
+ return false;
+
+ gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_induction_def);
+
+ if (STMT_VINFO_LIVE_P (stmt_info))
+ {
+ /* FORNOW: not yet supported. */
+ if (vect_print_dump_info (REPORT_DETAILS))
+ fprintf (vect_dump, "value used after loop.");
+ return false;
+ }
+
+ if (TREE_CODE (phi) != PHI_NODE)
+ return false;
+
+ if (!vec_stmt) /* transformation not required. */
+ {
+ STMT_VINFO_TYPE (stmt_info) = induc_vec_info_type;
+ return true;
+ }
+
+ /** Transform. **/
+
+ if (vect_print_dump_info (REPORT_DETAILS))
+ fprintf (vect_dump, "transform induction phi.");
+
+ vec_def = get_initial_def_for_induction (phi);
+ *vec_stmt = SSA_NAME_DEF_STMT (vec_def);
+ return true;
+ }
+
+
/* Function vectorizable_operation.
Check if STMT performs a binary or unary operation that can be vectorized.
*************** vectorizable_operation (tree stmt, block
*** 2246,2265 ****
gcc_assert (ncopies >= 1);
- /* Is STMT a vectorizable binary/unary operation? */
if (!STMT_VINFO_RELEVANT_P (stmt_info))
return false;
! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
if (STMT_VINFO_LIVE_P (stmt_info))
{
- /* FORNOW: not yet supported. */
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "value used after loop.");
return false;
}
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
return false;
--- 2286,2306 ----
gcc_assert (ncopies >= 1);
if (!STMT_VINFO_RELEVANT_P (stmt_info))
return false;
! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
! return false;
+ /* FORNOW: not yet supported. */
if (STMT_VINFO_LIVE_P (stmt_info))
{
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "value used after loop.");
return false;
}
+ /* Is STMT a vectorizable binary/unary operation? */
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
return false;
*************** vectorizable_type_demotion (tree stmt, b
*** 2514,2534 ****
optab optab;
enum machine_mode vec_mode;
- /* Is STMT a vectorizable type-demotion operation? */
-
if (!STMT_VINFO_RELEVANT_P (stmt_info))
return false;
!
! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
!
if (STMT_VINFO_LIVE_P (stmt_info))
{
- /* FORNOW: not yet supported. */
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "value used after loop.");
return false;
}
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
return false;
--- 2555,2575 ----
optab optab;
enum machine_mode vec_mode;
if (!STMT_VINFO_RELEVANT_P (stmt_info))
return false;
!
! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
! return false;
!
! /* FORNOW: not yet supported. */
if (STMT_VINFO_LIVE_P (stmt_info))
{
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "value used after loop.");
return false;
}
+ /* Is STMT a vectorizable type-demotion operation? */
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
return false;
*************** vectorizable_type_promotion (tree stmt,
*** 2723,2743 ****
int j;
tree vectype_in;
- /* Is STMT a vectorizable type-promotion operation? */
-
if (!STMT_VINFO_RELEVANT_P (stmt_info))
return false;
! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
if (STMT_VINFO_LIVE_P (stmt_info))
{
- /* FORNOW: not yet supported. */
if (vect_print_dump_info (REPORT_DETAILS))
! fprintf (vect_dump, "value used after loop.");
return false;
}
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
return false;
--- 2764,2784 ----
int j;
tree vectype_in;
if (!STMT_VINFO_RELEVANT_P (stmt_info))
return false;
! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
! return false;
+ /* FORNOW: not yet supported. */
if (STMT_VINFO_LIVE_P (stmt_info))
{
if (vect_print_dump_info (REPORT_DETAILS))
! fprintf (vect_dump, "value used after loop.");
return false;
}
+ /* Is STMT a vectorizable type-promotion operation? */
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
return false;
*************** vectorizable_store (tree stmt, block_stm
*** 3064,3069 ****
--- 3105,3123 ----
VEC(tree,heap) *dr_chain = NULL, *oprnds = NULL, *result_chain = NULL;
gcc_assert (ncopies >= 1);
+ if (!STMT_VINFO_RELEVANT_P (stmt_info))
+ return false;
+
+ if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
+ return false;
+
+ if (STMT_VINFO_LIVE_P (stmt_info))
+ {
+ if (vect_print_dump_info (REPORT_DETAILS))
+ fprintf (vect_dump, "value used after loop.");
+ return false;
+ }
+
/* Is vectorizable store? */
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
*************** vectorizable_load (tree stmt, block_stmt
*** 3710,3729 ****
bool strided_load = false;
tree first_stmt;
- /* Is vectorizable load? */
if (!STMT_VINFO_RELEVANT_P (stmt_info))
return false;
! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
if (STMT_VINFO_LIVE_P (stmt_info))
{
- /* FORNOW: not yet supported. */
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "value used after loop.");
return false;
}
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
return false;
--- 3764,3784 ----
bool strided_load = false;
tree first_stmt;
if (!STMT_VINFO_RELEVANT_P (stmt_info))
return false;
! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
! return false;
+ /* FORNOW: not yet supported. */
if (STMT_VINFO_LIVE_P (stmt_info))
{
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "value used after loop.");
return false;
}
+ /* Is vectorizable load? */
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
return false;
*************** vectorizable_live_operation (tree stmt,
*** 4010,4016 ****
tree def, def_stmt;
enum vect_def_type dt;
! if (!STMT_VINFO_LIVE_P (stmt_info))
return false;
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
--- 4065,4073 ----
tree def, def_stmt;
enum vect_def_type dt;
! gcc_assert (STMT_VINFO_LIVE_P (stmt_info));
!
! if (STMT_VINFO_DEF_TYPE (stmt_info) == vect_reduction_def)
return false;
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
*************** vectorizable_condition (tree stmt, block
*** 4123,4138 ****
if (!STMT_VINFO_RELEVANT_P (stmt_info))
return false;
! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
if (STMT_VINFO_LIVE_P (stmt_info))
{
- /* FORNOW: not yet supported. */
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "value used after loop.");
return false;
}
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
return false;
--- 4180,4197 ----
if (!STMT_VINFO_RELEVANT_P (stmt_info))
return false;
! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
! return false;
+ /* FORNOW: not yet supported. */
if (STMT_VINFO_LIVE_P (stmt_info))
{
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "value used after loop.");
return false;
}
+ /* Is vectorizable conditional operation? */
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
return false;
*************** vect_transform_stmt (tree stmt, block_st
*** 4225,4334 ****
tree orig_stmt_in_pattern;
bool done;
! if (STMT_VINFO_RELEVANT_P (stmt_info))
{
! switch (STMT_VINFO_TYPE (stmt_info))
! {
! case type_demotion_vec_info_type:
! done = vectorizable_type_demotion (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
! case type_promotion_vec_info_type:
! done = vectorizable_type_promotion (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
!
! case type_conversion_vec_info_type:
! done = vectorizable_conversion (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
!
! case op_vec_info_type:
! done = vectorizable_operation (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
!
! case assignment_vec_info_type:
! done = vectorizable_assignment (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
!
! case load_vec_info_type:
! done = vectorizable_load (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
!
! case store_vec_info_type:
! done = vectorizable_store (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! if (DR_GROUP_FIRST_DR (stmt_info))
! {
! /* In case of interleaving, the whole chain is vectorized when the
! last store in the chain is reached. Store stmts before the last
! one are skipped, and there vec_stmt_info shouldn't be freed
! meanwhile. */
! *strided_store = true;
! if (STMT_VINFO_VEC_STMT (stmt_info))
! is_store = true;
}
! else
! is_store = true;
! break;
!
! case condition_vec_info_type:
! done = vectorizable_condition (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
! case call_vec_info_type:
! done = vectorizable_call (stmt, bsi, &vec_stmt);
! break;
! default:
! if (vect_print_dump_info (REPORT_DETAILS))
! fprintf (vect_dump, "stmt not supported.");
! gcc_unreachable ();
! }
!
! gcc_assert (vec_stmt || *strided_store);
! if (vec_stmt)
{
! STMT_VINFO_VEC_STMT (stmt_info) = vec_stmt;
! orig_stmt_in_pattern = STMT_VINFO_RELATED_STMT (stmt_info);
! if (orig_stmt_in_pattern)
! {
! stmt_vec_info stmt_vinfo = vinfo_for_stmt (orig_stmt_in_pattern);
! if (STMT_VINFO_IN_PATTERN_P (stmt_vinfo))
! {
! gcc_assert (STMT_VINFO_RELATED_STMT (stmt_vinfo) == stmt);
!
! /* STMT was inserted by the vectorizer to replace a
! computation idiom. ORIG_STMT_IN_PATTERN is a stmt in the
! original sequence that computed this idiom. We need to
! record a pointer to VEC_STMT in the stmt_info of
! ORIG_STMT_IN_PATTERN. See more details in the
! documentation of vect_pattern_recog. */
!
! STMT_VINFO_VEC_STMT (stmt_vinfo) = vec_stmt;
! }
! }
}
}
! if (STMT_VINFO_LIVE_P (stmt_info))
{
! switch (STMT_VINFO_TYPE (stmt_info))
! {
! case reduc_vec_info_type:
! done = vectorizable_reduction (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
! default:
! done = vectorizable_live_operation (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! }
}
return is_store;
--- 4284,4391 ----
tree orig_stmt_in_pattern;
bool done;
! switch (STMT_VINFO_TYPE (stmt_info))
{
! case type_demotion_vec_info_type:
! done = vectorizable_type_demotion (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
! case type_promotion_vec_info_type:
! done = vectorizable_type_promotion (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
!
! case type_conversion_vec_info_type:
! done = vectorizable_conversion (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
!
! case induc_vec_info_type:
! done = vectorizable_induction (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
!
! case op_vec_info_type:
! done = vectorizable_operation (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
!
! case assignment_vec_info_type:
! done = vectorizable_assignment (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
!
! case load_vec_info_type:
! done = vectorizable_load (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
!
! case store_vec_info_type:
! done = vectorizable_store (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! if (DR_GROUP_FIRST_DR (stmt_info))
! {
! /* In case of interleaving, the whole chain is vectorized when the
! last store in the chain is reached. Store stmts before the last
! one are skipped, and there vec_stmt_info shouldn't be freed
! meanwhile. */
! *strided_store = true;
! if (STMT_VINFO_VEC_STMT (stmt_info))
! is_store = true;
}
! else
! is_store = true;
! break;
! case condition_vec_info_type:
! done = vectorizable_condition (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
!
! case call_vec_info_type:
! done = vectorizable_call (stmt, bsi, &vec_stmt);
! break;
!
! case reduc_vec_info_type:
! done = vectorizable_reduction (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
! default:
! if (!STMT_VINFO_LIVE_P (stmt_info))
{
! if (vect_print_dump_info (REPORT_DETAILS))
! fprintf (vect_dump, "stmt not supported.");
! gcc_unreachable ();
}
}
! if (STMT_VINFO_LIVE_P (stmt_info)
! && STMT_VINFO_TYPE (stmt_info) != reduc_vec_info_type)
{
! done = vectorizable_live_operation (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! }
! if (vec_stmt)
! {
! STMT_VINFO_VEC_STMT (stmt_info) = vec_stmt;
! orig_stmt_in_pattern = STMT_VINFO_RELATED_STMT (stmt_info);
! if (orig_stmt_in_pattern)
! {
! stmt_vec_info stmt_vinfo = vinfo_for_stmt (orig_stmt_in_pattern);
! /* STMT was inserted by the vectorizer to replace a computation idiom.
! ORIG_STMT_IN_PATTERN is a stmt in the original sequence that
! computed this idiom. We need to record a pointer to VEC_STMT in
! the stmt_info of ORIG_STMT_IN_PATTERN. See more details in the
! documentation of vect_pattern_recog. */
! if (STMT_VINFO_IN_PATTERN_P (stmt_vinfo))
! {
! gcc_assert (STMT_VINFO_RELATED_STMT (stmt_vinfo) == stmt);
! STMT_VINFO_VEC_STMT (stmt_vinfo) = vec_stmt;
! }
! }
}
return is_store;
*************** vect_transform_loop (loop_vec_info loop_
*** 5156,5166 ****
for (i = 0; i < nbbs; i++)
{
basic_block bb = bbs[i];
for (si = bsi_start (bb); !bsi_end_p (si);)
{
tree stmt = bsi_stmt (si);
- stmt_vec_info stmt_info;
bool is_store;
if (vect_print_dump_info (REPORT_DETAILS))
--- 5213,5251 ----
for (i = 0; i < nbbs; i++)
{
basic_block bb = bbs[i];
+ stmt_vec_info stmt_info;
+ tree phi;
+
+ for (phi = phi_nodes (bb); phi; phi = PHI_CHAIN (phi))
+ {
+ if (vect_print_dump_info (REPORT_DETAILS))
+ {
+ fprintf (vect_dump, "------>vectorizing phi: ");
+ print_generic_expr (vect_dump, phi, TDF_SLIM);
+ }
+ stmt_info = vinfo_for_stmt (phi);
+ if (!stmt_info)
+ continue;
+ if (!STMT_VINFO_RELEVANT_P (stmt_info)
+ && !STMT_VINFO_LIVE_P (stmt_info))
+ continue;
+
+ if ((TYPE_VECTOR_SUBPARTS (STMT_VINFO_VECTYPE (stmt_info))
+ != (unsigned HOST_WIDE_INT) vectorization_factor)
+ && vect_print_dump_info (REPORT_DETAILS))
+ fprintf (vect_dump, "multiple-types.");
+
+ if (STMT_VINFO_DEF_TYPE (stmt_info) == vect_induction_def)
+ {
+ if (vect_print_dump_info (REPORT_DETAILS))
+ fprintf (vect_dump, "transform phi.");
+ vect_transform_stmt (phi, NULL, NULL);
+ }
+ }
for (si = bsi_start (bb); !bsi_end_p (si);)
{
tree stmt = bsi_stmt (si);
bool is_store;
if (vect_print_dump_info (REPORT_DETAILS))
*************** vect_transform_loop (loop_vec_info loop_
*** 5177,5182 ****
--- 5262,5268 ----
continue;
}
+ gcc_assert (STMT_VINFO_VECTYPE (stmt_info));
if ((TYPE_VECTOR_SUBPARTS (STMT_VINFO_VECTYPE (stmt_info))
!= (unsigned HOST_WIDE_INT) vectorization_factor)
&& vect_print_dump_info (REPORT_DETAILS))
Index: tree-ssa-reassoc.c
===================================================================
*** tree-ssa-reassoc.c (revision 123103)
--- tree-ssa-reassoc.c (working copy)
*************** execute_reassoc (void)
*** 1476,1485 ****
return 0;
}
struct tree_opt_pass pass_reassoc =
{
"reassoc", /* name */
! NULL, /* gate */
execute_reassoc, /* execute */
NULL, /* sub */
NULL, /* next */
--- 1476,1491 ----
return 0;
}
+ static bool
+ gate_reassoc (void)
+ {
+ return flag_tree_reassoc != 0;
+ }
+
struct tree_opt_pass pass_reassoc =
{
"reassoc", /* name */
! gate_reassoc, /* gate */
execute_reassoc, /* execute */
NULL, /* sub */
NULL, /* next */
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [patch] vectorizer cleanups (and prep for outer-loop vectorization)
2007-04-02 8:43 [patch] vectorizer cleanups (and prep for outer-loop vectorization) Dorit Nuzman
@ 2007-04-02 9:48 ` Zdenek Dvorak
2007-04-02 9:55 ` Zdenek Dvorak
` (6 more replies)
0 siblings, 7 replies; 18+ messages in thread
From: Zdenek Dvorak @ 2007-04-02 9:48 UTC (permalink / raw)
To: Dorit Nuzman; +Cc: gcc-patches
Hello,
I will look at the patch. However, would it be possible to split it
into smaller parts? The changes seem to be quite independent,
Zdenek
> This is exactly the same patch as was committed to autovect last week:
> http://gcc.gnu.org/ml/gcc-patches/2007-03/msg01904.html
> (but relative to mainline, and after testig on mainline).
>
> These are a few cleanups (really technical things, no new functionality),
> some of which will be also useful for outer-loop vectorization:
>
> o Remove the field exit_cond from loop_vec_info, which used to hold the
> loop-exit stmt. Instead, a stmt that is identified as a loop-exit control
> stmt is marked as a 'loop_exit_ctrl_vec_info_type' (new enum value for
> stmt_vec_info_type). This will be useful when looking at outer-loops
> because we have more than one stmt like that in the loop (there's also one
> in the inner-loop).
>
> o Vectorization of induction is currently done upon encountering a use of
> the induction variable. Instead, we now vectorize the induction earlier,
> when we encounter the respective induction phi-node. This simplifies things
> a bit cause it's more in-line with how we vectorize other things (when we
> encounter a use of the induction we just get the vector def from the
> STMT_VINFO_VEC_STMT field of the defining phi-node), and also we don't need
> to worry about vectorizing the same induction multiple times. When looking
> at outer-loops this will be even more useful cause when we vectorize a
> phi-node we know exactly relative to which loop-nest is the induction, and
> don't need any extra logic to check for that.
>
> o Reorganize the calls to the vectorizable_* functions: In the drivers that
> call these functions (vect_analyze_operations and vect_transform_stmt), we
> currently call most of these functions only for RELEVANT_P stmts, and then
> call vectorizable_reduction only for LIVE_P stmts. Now, instead, each of
> the vectorizable_* functions checks that STMT_VINFO_RELEVANT/LIVE_P are as
> expected, and in the two driver functions we don't check for
> STMT_VINFO_RELEVANT_P anymore, but apply the vectorizable_* functions to
> all stmts. When looking at outer-loops this is useful because a reduction
> (in the inner-loop) may now be marked RELEVANT (if it is used in the
> outer-loop), and not only LIVE as it is currently.
>
> o A bunch of other cleanups (that are not necessarily inspired by
> outer-loop vectorization): Factor out code from
> vect_mark_stmts_to_be_vectorized to process_use; Remove the '#ifdef
> ADJUST_IN_EPILOG' and the code path that was disabled by it (which removes
> unused code and simplifies the function); Add a flag to allow disabling
> tree-reassociation, to help test reductions that are otherwise transformed
> by reassociation to something we currently don't recognize; Fix
> indentation; A few ther small miscellaneous stuff.
>
> Bootstrapped and tested (full regression testing) on powerpc64-linux and
> i386-linux.
> Also bootstrapped with vectorization enabled on powerpc64-linux.
>
> ok for mainline?
>
> :ADDPATCH vectorizer,non-algorithmic:
>
> ChangeLog:
>
> * tree-vectorizer.h (_loop_vec_info): Remove unused field
> exit_cond.
> (stmt_vec_info_type): Add enum value loop_exit_ctrl_vec_info_type.
> * tree-vectorizer.c (new_loop_vec_info): Remove setting of
> LOOP_VINFO_EXIT_COND.
> * tree-vect-analyze.c (vect_stmt_relevant_p): Replace use of
> LOOP_VINFO_EXIT_COND with use of loop_exit_ctrl_vec_info_type.
> (vect_analyze_loop_form): Likewise.
>
> * tree-vectorizer.h (stmt_vec_info_type): Add enum value
> induc_vec_info_type.
> (vectorizable_induction): New function declaration.
> * tree-vect-transform.c (get_initial_def_for_induction): No need to
> check if already vectorized. Find first place in BB where nre
> stmts
> can be inserted.
> (vectorizable_induction): New function.
> (vect_transform_stmt): Add case for induc_vec_info_type to call
> vectorizable_induction.
> (vect_transform_loop): Consider phis for vectorization.
> * tree-vect-analyze.c (analyze_operations): Call
> vectorizable_induction when analyzing phis and when analyzing
> stmts.
>
> * tree-vectorizer.c (vect_is_simple_use): Remove redundant check.
> (destroy_loop_vec_info): Set loop->aux to NULL.
>
> * tree-vect-analyze.c (vect_determine_vectorization_factor):
> Simplify
> condition. Add dump print.
> (vect_analyze_operations): Fix indenetation. Fix a comment. Fix a
> print message.
> (vect_analyze_scalar_cycles): Fix indentation.
> (vect_enhance_data_refs_alignment): Fix check in case of peeling.
> (vect_mark_relevant): Include phis in relevance analysis.
> (process_use): New function.
> (vect_mark_stmts_to_be_vectorized): Factor out code to process_use.
> Check phis in all bbs.
>
> * tree-vect-patterns.c (widened_name_p): Remove redundant check.
>
> * tree-vect-analyze.c (vect_analyze_operations): Reorganize calls
> to
> vectorizable_* functions.
> * tree-vect-transform.c (vectorizable_call): Add check for
> STMT_VINFO_RELEVANT_P, STMT_VINFO_DEF_TYPE and STMT_VINFO_LIVE_P.
> (vectorizable_store): likewise.
> (vectorizable_conversion): Add check for STMT_VINFO_DEF_TYPE.
> Add comments.
> (vectorizable_operation, vectorizable_type_demotion): Likewise.
> (vectorizable_type_promotion, vectorizable_load): Likewise.
> (vectorizable_live_operation, vectorizable_condition): Likewise.
> (vectorizable_assignment): Add check for STMT_VINFO_DEF_TYPE and
> STMT_VINFO_LIVE_P.
> (vect_transform_stmt): Reorganize calls to vectorizable_*
> functions.
> (vect_analyze_loop_form): Use the new NITERS_KNOWN_P macro so that
> this check could be done without creating a loop_vinfo first.
> Record
> loop_vinfo in loop->aux.
>
> * tree-vect-transform.c (get_initial_def_for_reduction): Clean away
> the unused code for reduction without adjust-in-epilog to simplify
> the
> function.
>
> * tree-vect-transform.c (vect_transform_loop): Add an assert.
>
> * tree-ssa-reassoc.c (gate_reassoc): New function.
> * common.opt (tree-reassoc): New flag.
> (pass_reassoc): Add gate.
>
>
> (See attached file: mainline-vect-cleanups.mar27.txt)
> Index: ChangeLog
> ===================================================================
> *** ChangeLog (revision 123103)
> --- ChangeLog (working copy)
> ***************
> *** 1,3 ****
> --- 1,70 ----
> + 2007-03-20 Dorit Nuzman <dorit@il.ibm.com>
> +
> + * tree-vectorizer.h (_loop_vec_info): Remove unused field exit_cond.
> + (stmt_vec_info_type): Add enum value loop_exit_ctrl_vec_info_type.
> + * tree-vectorizer.c (new_loop_vec_info): Remove setting of
> + LOOP_VINFO_EXIT_COND.
> + * tree-vect-analyze.c (vect_stmt_relevant_p): Replace use of
> + LOOP_VINFO_EXIT_COND with use of loop_exit_ctrl_vec_info_type.
> + (vect_analyze_loop_form): Likewise.
> +
> + * tree-vectorizer.h (stmt_vec_info_type): Add enum value
> + induc_vec_info_type.
> + (vectorizable_induction): New function declaration.
> + * tree-vect-transform.c (get_initial_def_for_induction): No need to
> + check if already vectorized. Find first place in BB where nre stmts
> + can be inserted.
> + (vectorizable_induction): New function.
> + (vect_transform_stmt): Add case for induc_vec_info_type to call
> + vectorizable_induction.
> + (vect_transform_loop): Consider phis for vectorization.
> + * tree-vect-analyze.c (analyze_operations): Call
> + vectorizable_induction when analyzing phis and when analyzing
> + stmts.
> +
> + * tree-vectorizer.c (vect_is_simple_use): Remove redundant check.
> + (destroy_loop_vec_info): Set loop->aux to NULL.
> +
> + * tree-vect-analyze.c (vect_determine_vectorization_factor): Simplify
> + condition. Add dump print.
> + (vect_analyze_operations): Fix indenetation. Fix a comment. Fix a
> + print message.
> + (vect_analyze_scalar_cycles): Fix indentation.
> + (vect_enhance_data_refs_alignment): Fix check in case of peeling.
> + (vect_mark_relevant): Include phis in relevance analysis.
> + (process_use): New function.
> + (vect_mark_stmts_to_be_vectorized): Factor out code to process_use.
> + Check phis in all bbs.
> +
> + * tree-vect-patterns.c (widened_name_p): Remove redundant check.
> +
> + * tree-vect-analyze.c (vect_analyze_operations): Reorganize calls to
> + vectorizable_* functions.
> + * tree-vect-transform.c (vectorizable_call): Add check for
> + STMT_VINFO_RELEVANT_P, STMT_VINFO_DEF_TYPE and STMT_VINFO_LIVE_P.
> + (vectorizable_store): likewise.
> + (vectorizable_conversion): Add check for STMT_VINFO_DEF_TYPE.
> + Add comments.
> + (vectorizable_operation, vectorizable_type_demotion): Likewise.
> + (vectorizable_type_promotion, vectorizable_load): Likewise.
> + (vectorizable_live_operation, vectorizable_condition): Likewise.
> + (vectorizable_assignment): Add check for STMT_VINFO_DEF_TYPE and
> + STMT_VINFO_LIVE_P.
> + (vect_transform_stmt): Reorganize calls to vectorizable_* functions.
> + (vect_analyze_loop_form): Use the new NITERS_KNOWN_P macro so that
> + this check could be done without creating a loop_vinfo first. Record
> + loop_vinfo in loop->aux.
> +
> + * tree-vect-transform.c (get_initial_def_for_reduction): Clean away
> + the unused code for reduction without adjust-in-epilog to simplify the
> + function.
> +
> + * tree-vect-transform.c (vect_transform_loop): Add an assert.
> +
> + * tree-ssa-reassoc.c (gate_reassoc): New function.
> + * common.opt (tree-reassoc): New flag.
> + (pass_reassoc): Add gate.
> +
> 2007-03-20 Mark Mitchell <mark@codesourcery.com>
>
> * config/arm/elf.h (TARGET_ASM_DESTRUCTOR): Define.
> Index: testsuite/gcc.dg/vect/vect-1.c
> ===================================================================
> *** testsuite/gcc.dg/vect/vect-1.c (revision 123103)
> --- testsuite/gcc.dg/vect/vect-1.c (working copy)
> *************** foo (int n)
> *** 38,44 ****
> inner-loop: cross iteration cycle; multi-dimensional arrays). */
> diff = 0;
> for (i = 0; i < N; i++) {
> ! for (i = 0; i < N; i++) {
> diff += (image[i][j] - block[i][j]);
> }
> }
> --- 38,44 ----
> inner-loop: cross iteration cycle; multi-dimensional arrays). */
> diff = 0;
> for (i = 0; i < N; i++) {
> ! for (j = 0; j < N; j++) {
> diff += (image[i][j] - block[i][j]);
> }
> }
> Index: testsuite/gcc.dg/vect/vect.exp
> ===================================================================
> *** testsuite/gcc.dg/vect/vect.exp (revision 123103)
> --- testsuite/gcc.dg/vect/vect.exp (working copy)
> *************** dg-runtest [lsort [glob -nocomplain $src
> *** 169,174 ****
> --- 169,175 ----
> "" $DEFAULT_VECTCFLAGS
>
> # With -Os
> + set DEFAULT_VECTCFLAGS $SAVED_DEFAULT_VECTCFLAGS
> lappend DEFAULT_VECTCFLAGS "-Os"
> dg-runtest [lsort [glob -nocomplain $srcdir/$subdir/Os-vect-*.\[cS\]]] \
> "" $DEFAULT_VECTCFLAGS
> Index: tree-vectorizer.c
> ===================================================================
> *** tree-vectorizer.c (revision 123103)
> --- tree-vectorizer.c (working copy)
> *************** new_loop_vec_info (struct loop *loop)
> *** 1420,1426 ****
>
> LOOP_VINFO_LOOP (res) = loop;
> LOOP_VINFO_BBS (res) = bbs;
> - LOOP_VINFO_EXIT_COND (res) = NULL;
> LOOP_VINFO_NITERS (res) = NULL;
> LOOP_VINFO_VECTORIZABLE_P (res) = 0;
> LOOP_PEELING_FOR_ALIGNMENT (res) = 0;
> --- 1420,1425 ----
> *************** destroy_loop_vec_info (loop_vec_info loo
> *** 1511,1516 ****
> --- 1510,1516 ----
> VEC_free (tree, heap, LOOP_VINFO_MAY_MISALIGN_STMTS (loop_vinfo));
>
> free (loop_vinfo);
> + loop->aux = NULL;
> }
>
>
> *************** vect_is_simple_use (tree operand, loop_v
> *** 1714,1728 ****
> return false;
> }
>
> - /* stmts inside the loop that have been identified as performing
> - a reduction operation cannot have uses in the loop. */
> - if (*dt == vect_reduction_def && TREE_CODE (*def_stmt) != PHI_NODE)
> - {
> - if (vect_print_dump_info (REPORT_DETAILS))
> - fprintf (vect_dump, "reduction used in loop.");
> - return false;
> - }
> -
> if (vect_print_dump_info (REPORT_DETAILS))
> fprintf (vect_dump, "type of def: %d.",*dt);
>
> --- 1714,1719 ----
> *************** vect_is_simple_use (tree operand, loop_v
> *** 1730,1742 ****
> {
> case PHI_NODE:
> *def = PHI_RESULT (*def_stmt);
> - gcc_assert (*dt == vect_induction_def || *dt == vect_reduction_def
> - || *dt == vect_invariant_def);
> break;
>
> case GIMPLE_MODIFY_STMT:
> *def = GIMPLE_STMT_OPERAND (*def_stmt, 0);
> - gcc_assert (*dt == vect_loop_def || *dt == vect_invariant_def);
> break;
>
> default:
> --- 1721,1730 ----
> Index: tree-vectorizer.h
> ===================================================================
> *** tree-vectorizer.h (revision 123103)
> --- tree-vectorizer.h (working copy)
> *************** typedef struct _loop_vec_info {
> *** 93,101 ****
> /* The loop basic blocks. */
> basic_block *bbs;
>
> - /* The loop exit_condition. */
> - tree exit_cond;
> -
> /* Number of iterations. */
> tree num_iters;
>
> --- 93,98 ----
> *************** typedef struct _loop_vec_info {
> *** 138,144 ****
> /* Access Functions. */
> #define LOOP_VINFO_LOOP(L) (L)->loop
> #define LOOP_VINFO_BBS(L) (L)->bbs
> - #define LOOP_VINFO_EXIT_COND(L) (L)->exit_cond
> #define LOOP_VINFO_NITERS(L) (L)->num_iters
> #define LOOP_VINFO_VECTORIZABLE_P(L) (L)->vectorizable
> #define LOOP_VINFO_VECT_FACTOR(L) (L)->vectorization_factor
> --- 135,140 ----
> *************** typedef struct _loop_vec_info {
> *** 151,159 ****
> #define LOOP_VINFO_MAY_MISALIGN_STMTS(L) (L)->may_misalign_stmts
> #define LOOP_VINFO_LOC(L) (L)->loop_line_number
>
> #define LOOP_VINFO_NITERS_KNOWN_P(L) \
> ! (host_integerp ((L)->num_iters,0) \
> ! && TREE_INT_CST_LOW ((L)->num_iters) > 0)
>
> /*-----------------------------------------------------------------*/
> /* Info on vectorized defs. */
> --- 147,158 ----
> #define LOOP_VINFO_MAY_MISALIGN_STMTS(L) (L)->may_misalign_stmts
> #define LOOP_VINFO_LOC(L) (L)->loop_line_number
>
> + #define NITERS_KNOWN_P(n) \
> + (host_integerp ((n),0) \
> + && TREE_INT_CST_LOW ((n)) > 0)
> +
> #define LOOP_VINFO_NITERS_KNOWN_P(L) \
> ! NITERS_KNOWN_P((L)->num_iters)
>
> /*-----------------------------------------------------------------*/
> /* Info on vectorized defs. */
> *************** enum stmt_vec_info_type {
> *** 167,175 ****
> assignment_vec_info_type,
> condition_vec_info_type,
> reduc_vec_info_type,
> type_promotion_vec_info_type,
> type_demotion_vec_info_type,
> ! type_conversion_vec_info_type
> };
>
> /* Indicates whether/how a variable is used in the loop. */
> --- 166,176 ----
> assignment_vec_info_type,
> condition_vec_info_type,
> reduc_vec_info_type,
> + induc_vec_info_type,
> type_promotion_vec_info_type,
> type_demotion_vec_info_type,
> ! type_conversion_vec_info_type,
> ! loop_exit_ctrl_vec_info_type
> };
>
> /* Indicates whether/how a variable is used in the loop. */
> *************** extern bool vectorizable_call (tree, blo
> *** 428,433 ****
> --- 429,435 ----
> extern bool vectorizable_condition (tree, block_stmt_iterator *, tree *);
> extern bool vectorizable_live_operation (tree, block_stmt_iterator *, tree *);
> extern bool vectorizable_reduction (tree, block_stmt_iterator *, tree *);
> + extern bool vectorizable_induction (tree, block_stmt_iterator *, tree *);
> /* Driver for transformation stage. */
> extern void vect_transform_loop (loop_vec_info);
>
> Index: tree-vect-analyze.c
> ===================================================================
> *** tree-vect-analyze.c (revision 123103)
> --- tree-vect-analyze.c (working copy)
> *************** vect_determine_vectorization_factor (loo
> *** 122,134 ****
>
> gcc_assert (stmt_info);
>
> ! /* Two cases of "relevant" phis: those that define an
> ! induction that is used in the loop, and those that
> ! define a reduction. */
> ! if ((STMT_VINFO_RELEVANT (stmt_info) == vect_used_in_loop
> ! && STMT_VINFO_DEF_TYPE (stmt_info) == vect_induction_def)
> ! || (STMT_VINFO_RELEVANT (stmt_info) == vect_used_by_reduction
> ! && STMT_VINFO_DEF_TYPE (stmt_info) == vect_reduction_def))
> {
> gcc_assert (!STMT_VINFO_VECTYPE (stmt_info));
> scalar_type = TREE_TYPE (PHI_RESULT (phi));
> --- 122,128 ----
>
> gcc_assert (stmt_info);
>
> ! if (STMT_VINFO_RELEVANT_P (stmt_info))
> {
> gcc_assert (!STMT_VINFO_VECTYPE (stmt_info));
> scalar_type = TREE_TYPE (PHI_RESULT (phi));
> *************** vect_determine_vectorization_factor (loo
> *** 268,274 ****
> }
>
> /* TODO: Analyze cost. Decide if worth while to vectorize. */
> !
> if (vectorization_factor <= 1)
> {
> if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> --- 262,269 ----
> }
>
> /* TODO: Analyze cost. Decide if worth while to vectorize. */
> ! if (vect_print_dump_info (REPORT_DETAILS))
> ! fprintf (vect_dump, "vectorization factor = %d", vectorization_factor);
> if (vectorization_factor <= 1)
> {
> if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> *************** vect_analyze_operations (loop_vec_info l
> *** 311,316 ****
> --- 306,313 ----
>
> for (phi = phi_nodes (bb); phi; phi = PHI_CHAIN (phi))
> {
> + ok = true;
> +
> stmt_info = vinfo_for_stmt (phi);
> if (vect_print_dump_info (REPORT_DETAILS))
> {
> *************** vect_analyze_operations (loop_vec_info l
> *** 319,341 ****
> }
>
> gcc_assert (stmt_info);
> -
> if (STMT_VINFO_LIVE_P (stmt_info))
> {
> /* FORNOW: not yet supported. */
> if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> fprintf (vect_dump, "not vectorized: value used after loop.");
> ! return false;
> }
>
> if (STMT_VINFO_RELEVANT (stmt_info) == vect_used_in_loop
> && STMT_VINFO_DEF_TYPE (stmt_info) != vect_induction_def)
> {
> ! /* Most likely a reduction-like computation that is used
> ! in the loop. */
> if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> ! fprintf (vect_dump, "not vectorized: unsupported pattern.");
> ! return false;
> }
> }
>
> --- 316,351 ----
> }
>
> gcc_assert (stmt_info);
> if (STMT_VINFO_LIVE_P (stmt_info))
> {
> /* FORNOW: not yet supported. */
> if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> fprintf (vect_dump, "not vectorized: value used after loop.");
> ! return false;
> }
>
> if (STMT_VINFO_RELEVANT (stmt_info) == vect_used_in_loop
> && STMT_VINFO_DEF_TYPE (stmt_info) != vect_induction_def)
> {
> ! /* A scalar-dependence cycle that we don't support. */
> ! if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> ! fprintf (vect_dump, "not vectorized: scalar dependence cycle.");
> ! return false;
> ! }
> !
> ! if (STMT_VINFO_RELEVANT_P (stmt_info)
> ! && STMT_VINFO_DEF_TYPE (stmt_info) == vect_induction_def)
> ! ok = vectorizable_induction (phi, NULL, NULL);
> !
> ! if (!ok)
> ! {
> if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> ! {
> ! fprintf (vect_dump,
> ! "not vectorized: relevant phi not supported: ");
> ! print_generic_expr (vect_dump, phi, TDF_SLIM);
> ! }
> ! return false;
> }
> }
>
> *************** vect_analyze_operations (loop_vec_info l
> *** 343,348 ****
> --- 353,359 ----
> {
> tree stmt = bsi_stmt (si);
> stmt_vec_info stmt_info = vinfo_for_stmt (stmt);
> + enum vect_def_type relevance = STMT_VINFO_RELEVANT (stmt_info);
>
> if (vect_print_dump_info (REPORT_DETAILS))
> {
> *************** vect_analyze_operations (loop_vec_info l
> *** 367,421 ****
> continue;
> }
>
> if (STMT_VINFO_RELEVANT_P (stmt_info))
> {
> gcc_assert (GIMPLE_STMT_P (stmt)
> || !VECTOR_MODE_P (TYPE_MODE (TREE_TYPE (stmt))));
> gcc_assert (STMT_VINFO_VECTYPE (stmt_info));
> -
> - ok = (vectorizable_type_promotion (stmt, NULL, NULL)
> - || vectorizable_type_demotion (stmt, NULL, NULL)
> - || vectorizable_conversion (stmt, NULL, NULL)
> - || vectorizable_operation (stmt, NULL, NULL)
> - || vectorizable_assignment (stmt, NULL, NULL)
> - || vectorizable_load (stmt, NULL, NULL)
> - || vectorizable_call (stmt, NULL, NULL)
> - || vectorizable_store (stmt, NULL, NULL)
> - || vectorizable_condition (stmt, NULL, NULL));
> -
> - if (!ok)
> - {
> - if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> - {
> - fprintf (vect_dump,
> - "not vectorized: relevant stmt not supported: ");
> - print_generic_expr (vect_dump, stmt, TDF_SLIM);
> - }
> - return false;
> - }
> need_to_vectorize = true;
> ! }
> !
> ! if (STMT_VINFO_LIVE_P (stmt_info))
> ! {
> ! ok = vectorizable_reduction (stmt, NULL, NULL);
>
> ! if (ok)
> ! need_to_vectorize = true;
> ! else
> ! ok = vectorizable_live_operation (stmt, NULL, NULL);
>
> ! if (!ok)
> {
> ! if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> ! {
> ! fprintf (vect_dump,
> ! "not vectorized: live stmt not supported: ");
> ! print_generic_expr (vect_dump, stmt, TDF_SLIM);
> ! }
> ! return false;
> }
> ! }
> } /* stmts in bb */
> } /* bbs */
>
> --- 378,434 ----
> continue;
> }
>
> + switch (STMT_VINFO_DEF_TYPE (stmt_info))
> + {
> + case vect_loop_def:
> + break;
> +
> + case vect_reduction_def:
> + gcc_assert (relevance == vect_unused_in_loop);
> + break;
> +
> + case vect_induction_def:
> + case vect_constant_def:
> + case vect_invariant_def:
> + case vect_unknown_def_type:
> + default:
> + gcc_unreachable ();
> + }
> +
> if (STMT_VINFO_RELEVANT_P (stmt_info))
> {
> gcc_assert (GIMPLE_STMT_P (stmt)
> || !VECTOR_MODE_P (TYPE_MODE (TREE_TYPE (stmt))));
> gcc_assert (STMT_VINFO_VECTYPE (stmt_info));
> need_to_vectorize = true;
> ! }
>
> ! ok = (vectorizable_type_promotion (stmt, NULL, NULL)
> ! || vectorizable_type_demotion (stmt, NULL, NULL)
> ! || vectorizable_conversion (stmt, NULL, NULL)
> ! || vectorizable_operation (stmt, NULL, NULL)
> ! || vectorizable_assignment (stmt, NULL, NULL)
> ! || vectorizable_load (stmt, NULL, NULL)
> ! || vectorizable_call (stmt, NULL, NULL)
> ! || vectorizable_store (stmt, NULL, NULL)
> ! || vectorizable_condition (stmt, NULL, NULL)
> ! || vectorizable_reduction (stmt, NULL, NULL));
> !
> ! /* Stmts that are (also) "live" (i.e. - that are used out of the loop)
> ! need extra handling, except for vectorizable reductions. */
> ! if (STMT_VINFO_LIVE_P (stmt_info)
> ! && STMT_VINFO_TYPE (stmt_info) != reduc_vec_info_type)
> ! ok |= vectorizable_live_operation (stmt, NULL, NULL);
>
> ! if (!ok)
> ! {
> ! if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> {
> ! fprintf (vect_dump, "not vectorized: stmt not supported: ");
> ! print_generic_expr (vect_dump, stmt, TDF_SLIM);
> }
> ! return false;
> ! }
> } /* stmts in bb */
> } /* bbs */
>
> *************** vect_analyze_scalar_cycles (loop_vec_inf
> *** 605,618 ****
> }
>
> if (!access_fn
> ! || !vect_is_simple_iv_evolution (loop->num, access_fn, &dumy, &dumy))
> {
> ! VEC_safe_push (tree, heap, worklist, phi);
> continue;
> }
>
> if (vect_print_dump_info (REPORT_DETAILS))
> ! fprintf (vect_dump, "Detected induction.");
> STMT_VINFO_DEF_TYPE (stmt_vinfo) = vect_induction_def;
> }
>
> --- 618,631 ----
> }
>
> if (!access_fn
> ! || !vect_is_simple_iv_evolution (loop->num, access_fn, &dumy, &dumy))
> {
> ! VEC_safe_push (tree, heap, worklist, phi);
> continue;
> }
>
> if (vect_print_dump_info (REPORT_DETAILS))
> ! fprintf (vect_dump, "Detected induction.");
> STMT_VINFO_DEF_TYPE (stmt_vinfo) = vect_induction_def;
> }
>
> *************** static bool
> *** 1418,1423 ****
> --- 1431,1437 ----
> vect_enhance_data_refs_alignment (loop_vec_info loop_vinfo)
> {
> VEC (data_reference_p, heap) *datarefs = LOOP_VINFO_DATAREFS (loop_vinfo);
> + struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
> enum dr_alignment_support supportable_dr_alignment;
> struct data_reference *dr0 = NULL;
> struct data_reference *dr;
> *************** vect_enhance_data_refs_alignment (loop_v
> *** 1515,1521 ****
>
> /* Often peeling for alignment will require peeling for loop-bound, which in
> turn requires that we know how to adjust the loop ivs after the loop. */
> ! if (!vect_can_advance_ivs_p (loop_vinfo))
> do_peeling = false;
>
> if (do_peeling)
> --- 1529,1536 ----
>
> /* Often peeling for alignment will require peeling for loop-bound, which in
> turn requires that we know how to adjust the loop ivs after the loop. */
> ! if (!vect_can_advance_ivs_p (loop_vinfo)
> ! || !slpeel_can_duplicate_loop_p (loop, single_exit (loop)))
> do_peeling = false;
>
> if (do_peeling)
> *************** vect_mark_relevant (VEC(tree,heap) **wor
> *** 2100,2110 ****
> if (relevant > STMT_VINFO_RELEVANT (stmt_info))
> STMT_VINFO_RELEVANT (stmt_info) = relevant;
>
> - if (TREE_CODE (stmt) == PHI_NODE)
> - /* Don't put phi-nodes in the worklist. Phis that are marked relevant
> - or live will fail vectorization later on. */
> - return;
> -
> if (STMT_VINFO_RELEVANT (stmt_info) == save_relevant
> && STMT_VINFO_LIVE_P (stmt_info) == save_live_p)
> {
> --- 2115,2120 ----
> *************** vect_stmt_relevant_p (tree stmt, loop_ve
> *** 2143,2149 ****
> *live_p = false;
>
> /* cond stmt other than loop exit cond. */
> ! if (is_ctrl_stmt (stmt) && (stmt != LOOP_VINFO_EXIT_COND (loop_vinfo)))
> *relevant = vect_used_in_loop;
>
> /* changing memory. */
> --- 2153,2160 ----
> *live_p = false;
>
> /* cond stmt other than loop exit cond. */
> ! if (is_ctrl_stmt (stmt)
> ! && STMT_VINFO_TYPE (vinfo_for_stmt (stmt)) != loop_exit_ctrl_vec_info_type)
> *relevant = vect_used_in_loop;
>
> /* changing memory. */
> *************** vect_stmt_relevant_p (tree stmt, loop_ve
> *** 2180,2185 ****
> --- 2191,2271 ----
> }
>
>
> + /*
> + Function process_use.
> +
> + Inputs:
> + - a USE in STMT in a loop represented by LOOP_VINFO
> + - LIVE_P, RELEVANT - enum values to be set in the STMT_VINFO of the stmt
> + that defined USE. This is dont by calling mark_relevant and passing it
> + the WORKLIST (to add DEF_STMT to the WORKlist in case itis relevant).
> +
> + Outputs:
> + Generally, LIVE_P and RELEVANT are used to define the liveness and
> + relevance info of the DEF_STMT of this USE:
> + STMT_VINFO_LIVE_P (DEF_STMT_info) <-- live_p
> + STMT_VINFO_RELEVANT (DEF_STMT_info) <-- relevant
> + Exceptions:
> + - case 1: If USE is used only for address computations (e.g. array indexing),
> + which does not need to be directly vectorized, then the liveness/relevance
> + of the respective DEF_STMT is left unchanged.
> + - case 2: If STMT is a reduction phi and DEF_STMT is a reduction stmt, we
> + skip DEF_STMT cause it had already been processed.
> +
> + Return true if everyting is as expected. Return false otherwise. */
> +
> + static bool
> + process_use (tree stmt, tree use, loop_vec_info loop_vinfo, bool live_p,
> + enum vect_relevant relevant, VEC(tree,heap) **worklist)
> + {
> + struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
> + stmt_vec_info stmt_vinfo = vinfo_for_stmt (stmt);
> + stmt_vec_info dstmt_vinfo;
> + basic_block def_bb;
> + tree def, def_stmt;
> + enum vect_def_type dt;
> +
> + /* case 1: we are only interested in uses that need to be vectorized. Uses
> + that are used for address computation are not considered relevant. */
> + if (!exist_non_indexing_operands_for_use_p (use, stmt))
> + return true;
> +
> + if (!vect_is_simple_use (use, loop_vinfo, &def_stmt, &def, &dt))
> + {
> + if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> + fprintf (vect_dump, "not vectorized: unsupported use in stmt.");
> + return false;
> + }
> +
> + if (!def_stmt || IS_EMPTY_STMT (def_stmt))
> + return true;
> +
> + def_bb = bb_for_stmt (def_stmt);
> + if (!flow_bb_inside_loop_p (loop, def_bb))
> + return true;
> +
> + /* case 2: A reduction phi defining a reduction stmt (DEF_STMT). DEF_STMT
> + must have already been processed, so we just check that everything is as
> + expected, and we are done. */
> + dstmt_vinfo = vinfo_for_stmt (def_stmt);
> + if (TREE_CODE (stmt) == PHI_NODE
> + && STMT_VINFO_DEF_TYPE (stmt_vinfo) == vect_reduction_def
> + && TREE_CODE (def_stmt) != PHI_NODE
> + && STMT_VINFO_DEF_TYPE (dstmt_vinfo) == vect_reduction_def)
> + {
> + if (STMT_VINFO_IN_PATTERN_P (dstmt_vinfo))
> + dstmt_vinfo = vinfo_for_stmt (STMT_VINFO_RELATED_STMT (dstmt_vinfo));
> + gcc_assert (STMT_VINFO_RELEVANT (dstmt_vinfo) < vect_used_by_reduction);
> + gcc_assert (STMT_VINFO_LIVE_P (dstmt_vinfo)
> + || STMT_VINFO_RELEVANT (dstmt_vinfo) > vect_unused_in_loop);
> + return true;
> + }
> +
> + vect_mark_relevant (worklist, def_stmt, relevant, live_p);
> + return true;
> + }
> +
> +
> /* Function vect_mark_stmts_to_be_vectorized.
>
> Not all stmts in the loop need to be vectorized. For example:
> *************** vect_mark_stmts_to_be_vectorized (loop_v
> *** 2204,2220 ****
> basic_block *bbs = LOOP_VINFO_BBS (loop_vinfo);
> unsigned int nbbs = loop->num_nodes;
> block_stmt_iterator si;
> ! tree stmt, use;
> stmt_ann_t ann;
> - ssa_op_iter iter;
> unsigned int i;
> stmt_vec_info stmt_vinfo;
> basic_block bb;
> tree phi;
> bool live_p;
> enum vect_relevant relevant;
> - tree def, def_stmt;
> - enum vect_def_type dt;
>
> if (vect_print_dump_info (REPORT_DETAILS))
> fprintf (vect_dump, "=== vect_mark_stmts_to_be_vectorized ===");
> --- 2290,2303 ----
> basic_block *bbs = LOOP_VINFO_BBS (loop_vinfo);
> unsigned int nbbs = loop->num_nodes;
> block_stmt_iterator si;
> ! tree stmt;
> stmt_ann_t ann;
> unsigned int i;
> stmt_vec_info stmt_vinfo;
> basic_block bb;
> tree phi;
> bool live_p;
> enum vect_relevant relevant;
>
> if (vect_print_dump_info (REPORT_DETAILS))
> fprintf (vect_dump, "=== vect_mark_stmts_to_be_vectorized ===");
> *************** vect_mark_stmts_to_be_vectorized (loop_v
> *** 2222,2248 ****
> worklist = VEC_alloc (tree, heap, 64);
>
> /* 1. Init worklist. */
> -
> - bb = loop->header;
> - for (phi = phi_nodes (bb); phi; phi = PHI_CHAIN (phi))
> - {
> - if (vect_print_dump_info (REPORT_DETAILS))
> - {
> - fprintf (vect_dump, "init: phi relevant? ");
> - print_generic_expr (vect_dump, phi, TDF_SLIM);
> - }
> -
> - if (vect_stmt_relevant_p (phi, loop_vinfo, &relevant, &live_p))
> - vect_mark_relevant (&worklist, phi, relevant, live_p);
> - }
> -
> for (i = 0; i < nbbs; i++)
> {
> bb = bbs[i];
> for (si = bsi_start (bb); !bsi_end_p (si); bsi_next (&si))
> {
> stmt = bsi_stmt (si);
> -
> if (vect_print_dump_info (REPORT_DETAILS))
> {
> fprintf (vect_dump, "init: stmt relevant? ");
> --- 2305,2327 ----
> worklist = VEC_alloc (tree, heap, 64);
>
> /* 1. Init worklist. */
> for (i = 0; i < nbbs; i++)
> {
> bb = bbs[i];
> + for (phi = phi_nodes (bb); phi; phi = PHI_CHAIN (phi))
> + {
> + if (vect_print_dump_info (REPORT_DETAILS))
> + {
> + fprintf (vect_dump, "init: phi relevant? ");
> + print_generic_expr (vect_dump, phi, TDF_SLIM);
> + }
> +
> + if (vect_stmt_relevant_p (phi, loop_vinfo, &relevant, &live_p))
> + vect_mark_relevant (&worklist, phi, relevant, live_p);
> + }
> for (si = bsi_start (bb); !bsi_end_p (si); bsi_next (&si))
> {
> stmt = bsi_stmt (si);
> if (vect_print_dump_info (REPORT_DETAILS))
> {
> fprintf (vect_dump, "init: stmt relevant? ");
> *************** vect_mark_stmts_to_be_vectorized (loop_v
> *** 2254,2355 ****
> }
> }
>
> -
> /* 2. Process_worklist */
> -
> while (VEC_length (tree, worklist) > 0)
> {
> ! stmt = VEC_pop (tree, worklist);
>
> if (vect_print_dump_info (REPORT_DETAILS))
> {
> fprintf (vect_dump, "worklist: examine stmt: ");
> print_generic_expr (vect_dump, stmt, TDF_SLIM);
> }
>
> ! /* Examine the USEs of STMT. For each ssa-name USE that is defined
> ! in the loop, mark the stmt that defines it (DEF_STMT) as
> ! relevant/irrelevant and live/dead according to the liveness and
> ! relevance properties of STMT.
> ! */
> !
> ! gcc_assert (TREE_CODE (stmt) != PHI_NODE);
> !
> ann = stmt_ann (stmt);
> stmt_vinfo = vinfo_for_stmt (stmt);
> -
> relevant = STMT_VINFO_RELEVANT (stmt_vinfo);
> live_p = STMT_VINFO_LIVE_P (stmt_vinfo);
>
> /* Generally, the liveness and relevance properties of STMT are
> ! propagated to the DEF_STMTs of its USEs:
> ! STMT_VINFO_LIVE_P (DEF_STMT_info) <-- live_p
> ! STMT_VINFO_RELEVANT (DEF_STMT_info) <-- relevant
> !
> ! Exceptions:
> !
> ! (case 1)
> ! If USE is used only for address computations (e.g. array indexing),
> ! which does not need to be directly vectorized, then the
> ! liveness/relevance of the respective DEF_STMT is left unchanged.
> !
> ! (case 2)
> ! If STMT has been identified as defining a reduction variable, then
> ! we want to set liveness/relevance as follows:
> ! STMT_VINFO_LIVE_P (DEF_STMT_info) <-- false
> ! STMT_VINFO_RELEVANT (DEF_STMT_info) <-- vect_used_by_reduction
> ! because even though STMT is classified as live (since it defines a
> ! value that is used across loop iterations) and irrelevant (since it
> ! is not used inside the loop), it will be vectorized, and therefore
> ! the corresponding DEF_STMTs need to marked as relevant.
> ! We distinguish between two kinds of relevant stmts - those that are
> ! used by a reduction computation, and those that are (also) used by
> ! a regular computation. This allows us later on to identify stmts
> ! that are used solely by a reduction, and therefore the order of
> ! the results that they produce does not have to be kept.
> ! */
>
> - /* case 2.2: */
> if (STMT_VINFO_DEF_TYPE (stmt_vinfo) == vect_reduction_def)
> ! {
> ! gcc_assert (relevant == vect_unused_in_loop && live_p);
> relevant = vect_used_by_reduction;
> ! live_p = false;
> }
>
> ! i = 0;
> ! FOR_EACH_SSA_TREE_OPERAND (use, stmt, iter, SSA_OP_USE)
> {
> ! if (vect_print_dump_info (REPORT_DETAILS))
> {
> - fprintf (vect_dump, "worklist: examine use %d: ", i++);
> - print_generic_expr (vect_dump, use, TDF_SLIM);
> - }
> -
> - /* case 1: we are only interested in uses that need to be vectorized.
> - Uses that are used for address computation are not considered
> - relevant.
> - */
> - if (!exist_non_indexing_operands_for_use_p (use, stmt))
> - continue;
> -
> - if (!vect_is_simple_use (use, loop_vinfo, &def_stmt, &def, &dt))
> - {
> - if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> - fprintf (vect_dump, "not vectorized: unsupported use in stmt.");
> VEC_free (tree, heap, worklist);
> return false;
> ! }
> !
> ! if (!def_stmt || IS_EMPTY_STMT (def_stmt))
> ! continue;
> !
> ! bb = bb_for_stmt (def_stmt);
> ! if (!flow_bb_inside_loop_p (loop, bb))
> ! continue;
> ! vect_mark_relevant (&worklist, def_stmt, relevant, live_p);
> }
> ! } /* while worklist */
>
> VEC_free (tree, heap, worklist);
> return true;
> --- 2333,2414 ----
> }
> }
>
> /* 2. Process_worklist */
> while (VEC_length (tree, worklist) > 0)
> {
> ! use_operand_p use_p;
> ! ssa_op_iter iter;
>
> + stmt = VEC_pop (tree, worklist);
> if (vect_print_dump_info (REPORT_DETAILS))
> {
> fprintf (vect_dump, "worklist: examine stmt: ");
> print_generic_expr (vect_dump, stmt, TDF_SLIM);
> }
>
> ! /* Examine the USEs of STMT. For each USE, mark the stmt that defines it
> ! (DEF_STMT) as relevant/irrelevant and live/dead according to the
> ! liveness and relevance properties of STMT. */
> ann = stmt_ann (stmt);
> stmt_vinfo = vinfo_for_stmt (stmt);
> relevant = STMT_VINFO_RELEVANT (stmt_vinfo);
> live_p = STMT_VINFO_LIVE_P (stmt_vinfo);
>
> /* Generally, the liveness and relevance properties of STMT are
> ! propagated as is to the DEF_STMTs of its USEs:
> ! live_p <-- STMT_VINFO_LIVE_P (STMT_VINFO)
> ! relevant <-- STMT_VINFO_RELEVANT (STMT_VINFO)
> !
> ! One exception is when STMT has been identified as defining a reduction
> ! variable; in this case we set the liveness/relevance as follows:
> ! live_p = false
> ! relevant = vect_used_by_reduction
> ! This is because we distinguish between two kinds of relevant stmts -
> ! those that are used by a reduction computation, and those that are
> ! (also) used by a regular computation. This allows us later on to
> ! identify stmts that are used solely by a reduction, and therefore the
> ! order of the results that they produce does not have to be kept.
> !
> ! Reduction phis are expected to be used by a reduction stmt; Other
> ! reduction stmts are expected to be unused in the loop. These are the
> ! expected values of "relevant" for reduction phis/stmts in the loop:
> !
> ! relevance: phi stmt
> ! vect_unused_in_loop ok
> ! vect_used_by_reduction ok
> ! vect_used_in_loop */
>
> if (STMT_VINFO_DEF_TYPE (stmt_vinfo) == vect_reduction_def)
> ! {
> ! switch (relevant)
> ! {
> ! case vect_unused_in_loop:
> ! gcc_assert (TREE_CODE (stmt) != PHI_NODE);
> ! break;
> ! case vect_used_by_reduction:
> ! if (TREE_CODE (stmt) == PHI_NODE)
> ! break;
> ! case vect_used_in_loop:
> ! default:
> ! if (vect_print_dump_info (REPORT_DETAILS))
> ! fprintf (vect_dump, "unsupported use of reduction.");
> ! VEC_free (tree, heap, worklist);
> ! return false;
> ! }
> relevant = vect_used_by_reduction;
> ! live_p = false;
> }
>
> ! FOR_EACH_PHI_OR_STMT_USE (use_p, stmt, iter, SSA_OP_USE)
> {
> ! tree op = USE_FROM_PTR (use_p);
> ! if (!process_use (stmt, op, loop_vinfo, live_p, relevant, &worklist))
> {
> VEC_free (tree, heap, worklist);
> return false;
> ! }
> }
> ! } /* while worklist */
>
> VEC_free (tree, heap, worklist);
> return true;
> *************** vect_analyze_loop_form (struct loop *loo
> *** 2582,2591 ****
> return false;
> }
>
> ! loop_vinfo = new_loop_vec_info (loop);
> ! LOOP_VINFO_NITERS (loop_vinfo) = number_of_iterations;
> !
> ! if (!LOOP_VINFO_NITERS_KNOWN_P (loop_vinfo))
> {
> if (vect_print_dump_info (REPORT_DETAILS))
> {
> --- 2641,2647 ----
> return false;
> }
>
> ! if (!NITERS_KNOWN_P (number_of_iterations))
> {
> if (vect_print_dump_info (REPORT_DETAILS))
> {
> *************** vect_analyze_loop_form (struct loop *loo
> *** 2594,2608 ****
> }
> }
> else
> ! if (LOOP_VINFO_INT_NITERS (loop_vinfo) == 0)
> {
> if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> fprintf (vect_dump, "not vectorized: number of iterations = 0.");
> return NULL;
> }
>
> ! LOOP_VINFO_EXIT_COND (loop_vinfo) = loop_cond;
>
> return loop_vinfo;
> }
>
> --- 2650,2669 ----
> }
> }
> else
> ! if (TREE_INT_CST_LOW (number_of_iterations) == 0)
> {
> if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> fprintf (vect_dump, "not vectorized: number of iterations = 0.");
> return NULL;
> }
>
> ! loop_vinfo = new_loop_vec_info (loop);
> ! LOOP_VINFO_NITERS (loop_vinfo) = number_of_iterations;
> !
> ! STMT_VINFO_TYPE (vinfo_for_stmt (loop_cond)) = loop_exit_ctrl_vec_info_type;
>
> + gcc_assert (!loop->aux);
> + loop->aux = loop_vinfo;
> return loop_vinfo;
> }
>
> Index: common.opt
> ===================================================================
> *** common.opt (revision 123103)
> --- common.opt (working copy)
> *************** funroll-all-loops
> *** 1070,1075 ****
> --- 1070,1079 ----
> Common Report Var(flag_unroll_all_loops) Optimization
> Perform loop unrolling for all loops
>
> + ftree-reassoc
> + Common Report Var(flag_tree_reassoc) Init(1) Optimization
> + Perform reassociation on trees
> +
> ; Nonzero means that loop optimizer may assume that the induction variables
> ; that control loops do not overflow and that the loops with nontrivial
> ; exit condition are not infinite
> Index: tree-vect-patterns.c
> ===================================================================
> *** tree-vect-patterns.c (revision 123103)
> --- tree-vect-patterns.c (working copy)
> *************** widened_name_p (tree name, tree use_stmt
> *** 109,118 ****
> if (!vect_is_simple_use (oprnd0, loop_vinfo, &dummy, &dummy, &dt))
> return false;
>
> - if (dt != vect_invariant_def && dt != vect_constant_def
> - && dt != vect_loop_def)
> - return false;
> -
> return true;
> }
>
> --- 109,114 ----
> Index: tree-vect-transform.c
> ===================================================================
> *** tree-vect-transform.c (revision 123103)
> --- tree-vect-transform.c (working copy)
> *************** vect_init_vector (tree stmt, tree vector
> *** 514,520 ****
> /* Function get_initial_def_for_induction
>
> Input:
> - STMT - a stmt that performs an induction operation in the loop.
> IV_PHI - the initial value of the induction variable
>
> Output:
> --- 514,519 ----
> *************** vect_init_vector (tree stmt, tree vector
> *** 524,532 ****
> [X, X + S, X + 2*S, X + 3*S]. */
>
> static tree
> ! get_initial_def_for_induction (tree stmt, tree iv_phi)
> {
> ! stmt_vec_info stmt_vinfo = vinfo_for_stmt (stmt);
> loop_vec_info loop_vinfo = STMT_VINFO_LOOP_VINFO (stmt_vinfo);
> struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
> tree scalar_type = TREE_TYPE (iv_phi);
> --- 523,531 ----
> [X, X + S, X + 2*S, X + 3*S]. */
>
> static tree
> ! get_initial_def_for_induction (tree iv_phi)
> {
> ! stmt_vec_info stmt_vinfo = vinfo_for_stmt (iv_phi);
> loop_vec_info loop_vinfo = STMT_VINFO_LOOP_VINFO (stmt_vinfo);
> struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
> tree scalar_type = TREE_TYPE (iv_phi);
> *************** get_initial_def_for_induction (tree stmt
> *** 549,575 ****
> tree expr;
> stmt_vec_info phi_info = vinfo_for_stmt (iv_phi);
> tree stmts;
>
> gcc_assert (phi_info);
>
> ! if (STMT_VINFO_VEC_STMT (phi_info))
> {
> ! induction_phi = STMT_VINFO_VEC_STMT (phi_info);
> ! gcc_assert (TREE_CODE (induction_phi) == PHI_NODE);
> !
> ! if (vect_print_dump_info (REPORT_DETAILS))
> ! {
> ! fprintf (vect_dump, "induction already vectorized:");
> ! print_generic_expr (vect_dump, iv_phi, TDF_SLIM);
> ! fprintf (vect_dump, "\n");
> ! print_generic_expr (vect_dump, induction_phi, TDF_SLIM);
> ! }
> !
> ! return PHI_RESULT (induction_phi);
> }
>
> - gcc_assert (ncopies >= 1);
> -
> access_fn = analyze_scalar_evolution (loop, PHI_RESULT (iv_phi));
> gcc_assert (access_fn);
> ok = vect_is_simple_iv_evolution (loop->num, access_fn, &init_expr, &step_expr);
> --- 548,568 ----
> tree expr;
> stmt_vec_info phi_info = vinfo_for_stmt (iv_phi);
> tree stmts;
> + tree stmt = NULL_TREE;
> + block_stmt_iterator si;
> + basic_block bb = bb_for_stmt (iv_phi);
>
> gcc_assert (phi_info);
> + gcc_assert (ncopies >= 1);
>
> ! /* Find the first insertion point in the BB. */
> ! for (si = bsi_start (bb); !bsi_end_p (si); bsi_next (&si))
> {
> ! stmt = bsi_stmt (si);
> ! if (TREE_CODE (stmt) != LABEL_EXPR)
> ! break;
> }
>
> access_fn = analyze_scalar_evolution (loop, PHI_RESULT (iv_phi));
> gcc_assert (access_fn);
> ok = vect_is_simple_iv_evolution (loop->num, access_fn, &init_expr, &step_expr);
> *************** vect_get_vec_def_for_operand (tree op, t
> *** 833,839 ****
> gcc_assert (TREE_CODE (def_stmt) == PHI_NODE);
>
> /* Get the def before the loop */
> ! return get_initial_def_for_induction (stmt, def_stmt);
> }
>
> default:
> --- 826,832 ----
> gcc_assert (TREE_CODE (def_stmt) == PHI_NODE);
>
> /* Get the def before the loop */
> ! return get_initial_def_for_induction (def_stmt);
> }
>
> default:
> *************** vect_finish_stmt_generation (tree stmt,
> *** 951,958 ****
> }
>
>
> - #define ADJUST_IN_EPILOG 1
> -
> /* Function get_initial_def_for_reduction
>
> Input:
> --- 944,949 ----
> *************** vect_finish_stmt_generation (tree stmt,
> *** 960,977 ****
> INIT_VAL - the initial value of the reduction variable
>
> Output:
> ! SCALAR_DEF - a tree that holds a value to be added to the final result
> ! of the reduction (used for "ADJUST_IN_EPILOG" - see below).
> Return a vector variable, initialized according to the operation that STMT
> ! performs. This vector will be used as the initial value of the
> ! vector of partial results.
>
> ! Option1 ("ADJUST_IN_EPILOG"): Initialize the vector as follows:
> add: [0,0,...,0,0]
> mult: [1,1,...,1,1]
> min/max: [init_val,init_val,..,init_val,init_val]
> bit and/or: [init_val,init_val,..,init_val,init_val]
> ! and when necessary (e.g. add/mult case) let the caller know
> that it needs to adjust the result by init_val.
>
> Option2: Initialize the vector as follows:
> --- 951,968 ----
> INIT_VAL - the initial value of the reduction variable
>
> Output:
> ! ADJUSTMENT_DEF - a tree that holds a value to be added to the final result
> ! of the reduction (used for adjusting the epilog - see below).
> Return a vector variable, initialized according to the operation that STMT
> ! performs. This vector will be used as the initial value of the
> ! vector of partial results.
>
> ! Option1 (adjust in epilog): Initialize the vector as follows:
> add: [0,0,...,0,0]
> mult: [1,1,...,1,1]
> min/max: [init_val,init_val,..,init_val,init_val]
> bit and/or: [init_val,init_val,..,init_val,init_val]
> ! and when necessary (e.g. add/mult case) let the caller know
> that it needs to adjust the result by init_val.
>
> Option2: Initialize the vector as follows:
> *************** vect_finish_stmt_generation (tree stmt,
> *** 992,1075 ****
> or [0,0,0,0] and let the caller know that it needs to adjust
> the result at the end by 'init_val'.
>
> ! FORNOW: We use the "ADJUST_IN_EPILOG" scheme.
> ! TODO: Use some cost-model to estimate which scheme is more profitable.
> ! */
>
> static tree
> ! get_initial_def_for_reduction (tree stmt, tree init_val, tree *scalar_def)
> {
> stmt_vec_info stmt_vinfo = vinfo_for_stmt (stmt);
> tree vectype = STMT_VINFO_VECTYPE (stmt_vinfo);
> int nunits = TYPE_VECTOR_SUBPARTS (vectype);
> - int nelements;
> enum tree_code code = TREE_CODE (GIMPLE_STMT_OPERAND (stmt, 1));
> tree type = TREE_TYPE (init_val);
> ! tree def;
> ! tree vec, t = NULL_TREE;
> ! bool need_epilog_adjust;
> int i;
> tree vector_type;
>
> gcc_assert (INTEGRAL_TYPE_P (type) || SCALAR_FLOAT_TYPE_P (type));
>
> switch (code)
> {
> case WIDEN_SUM_EXPR:
> case DOT_PROD_EXPR:
> case PLUS_EXPR:
> if (INTEGRAL_TYPE_P (type))
> ! def = build_int_cst (type, 0);
> else
> ! def = build_real (type, dconst0);
> !
> ! #ifdef ADJUST_IN_EPILOG
> ! /* All the 'nunits' elements are set to 0. The final result will be
> ! adjusted by 'init_val' at the loop epilog. */
> ! nelements = nunits;
> ! need_epilog_adjust = true;
> ! #else
> ! /* 'nunits - 1' elements are set to 0; The last element is set to
> ! 'init_val'. No further adjustments at the epilog are needed. */
> ! nelements = nunits - 1;
> ! need_epilog_adjust = false;
> ! #endif
> break;
>
> case MIN_EXPR:
> case MAX_EXPR:
> ! def = init_val;
> ! nelements = nunits;
> ! need_epilog_adjust = false;
> break;
>
> default:
> gcc_unreachable ();
> }
>
> ! for (i = nelements - 1; i >= 0; --i)
> ! t = tree_cons (NULL_TREE, def, t);
> !
> ! if (nelements == nunits - 1)
> ! {
> ! /* Set the last element of the vector. */
> ! t = tree_cons (NULL_TREE, init_val, t);
> ! nelements += 1;
> ! }
> ! gcc_assert (nelements == nunits);
> !
> ! vector_type = get_vectype_for_scalar_type (TREE_TYPE (def));
> ! if (TREE_CODE (init_val) == INTEGER_CST || TREE_CODE (init_val) == REAL_CST)
> ! vec = build_vector (vector_type, t);
> ! else
> ! vec = build_constructor_from_list (vector_type, t);
> !
> ! if (!need_epilog_adjust)
> ! *scalar_def = NULL_TREE;
> ! else
> ! *scalar_def = init_val;
> !
> ! return vect_init_vector (stmt, vec, vector_type);
> }
>
>
> --- 983,1038 ----
> or [0,0,0,0] and let the caller know that it needs to adjust
> the result at the end by 'init_val'.
>
> ! FORNOW, we are using the 'adjust in epilog' scheme, because this way the
> ! initialization vector is simpler (same element in all entries).
> ! A cost model should help decide between these two schemes. */
>
> static tree
> ! get_initial_def_for_reduction (tree stmt, tree init_val, tree *adjustment_def)
> {
> stmt_vec_info stmt_vinfo = vinfo_for_stmt (stmt);
> tree vectype = STMT_VINFO_VECTYPE (stmt_vinfo);
> int nunits = TYPE_VECTOR_SUBPARTS (vectype);
> enum tree_code code = TREE_CODE (GIMPLE_STMT_OPERAND (stmt, 1));
> tree type = TREE_TYPE (init_val);
> ! tree vecdef;
> ! tree def_for_init;
> ! tree init_def;
> ! tree t = NULL_TREE;
> int i;
> tree vector_type;
>
> gcc_assert (INTEGRAL_TYPE_P (type) || SCALAR_FLOAT_TYPE_P (type));
> + vecdef = vect_get_vec_def_for_operand (init_val, stmt, NULL);
>
> switch (code)
> {
> case WIDEN_SUM_EXPR:
> case DOT_PROD_EXPR:
> case PLUS_EXPR:
> + *adjustment_def = init_val;
> + /* Create a vector of zeros for init_def. */
> if (INTEGRAL_TYPE_P (type))
> ! def_for_init = build_int_cst (type, 0);
> else
> ! def_for_init = build_real (type, dconst0);
> ! for (i = nunits - 1; i >= 0; --i)
> ! t = tree_cons (NULL_TREE, def_for_init, t);
> ! vector_type = get_vectype_for_scalar_type (TREE_TYPE (def_for_init));
> ! init_def = build_vector (vector_type, t);
> break;
>
> case MIN_EXPR:
> case MAX_EXPR:
> ! *adjustment_def = NULL_TREE;
> ! init_def = vecdef;
> break;
>
> default:
> gcc_unreachable ();
> }
>
> ! return init_def;
> }
>
>
> *************** vectorizable_call (tree stmt, block_stmt
> *** 1816,1821 ****
> --- 1779,1798 ----
> int ncopies, j, nargs;
> call_expr_arg_iterator iter;
>
> + if (!STMT_VINFO_RELEVANT_P (stmt_info))
> + return false;
> +
> + if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
> + return false;
> +
> + /* FORNOW: not yet supported. */
> + if (STMT_VINFO_LIVE_P (stmt_info))
> + {
> + if (vect_print_dump_info (REPORT_DETAILS))
> + fprintf (vect_dump, "value used after loop.");
> + return false;
> + }
> +
> /* Is STMT a vectorizable call? */
> if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> return false;
> *************** vectorizable_conversion (tree stmt, bloc
> *** 1994,2000 ****
> if (!STMT_VINFO_RELEVANT_P (stmt_info))
> return false;
>
> ! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
>
> if (STMT_VINFO_LIVE_P (stmt_info))
> {
> --- 1971,1978 ----
> if (!STMT_VINFO_RELEVANT_P (stmt_info))
> return false;
>
> ! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
> ! return false;
>
> if (STMT_VINFO_LIVE_P (stmt_info))
> {
> *************** vectorizable_assignment (tree stmt, bloc
> *** 2134,2145 ****
> if (ncopies > 1)
> return false; /* FORNOW */
>
> - /* Is vectorizable assignment? */
> if (!STMT_VINFO_RELEVANT_P (stmt_info))
> return false;
>
> ! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
>
> if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> return false;
>
> --- 2112,2132 ----
> if (ncopies > 1)
> return false; /* FORNOW */
>
> if (!STMT_VINFO_RELEVANT_P (stmt_info))
> return false;
>
> ! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
> ! return false;
> !
> ! /* FORNOW: not yet supported. */
> ! if (STMT_VINFO_LIVE_P (stmt_info))
> ! {
> ! if (vect_print_dump_info (REPORT_DETAILS))
> ! fprintf (vect_dump, "value used after loop.");
> ! return false;
> ! }
>
> + /* Is vectorizable assignment? */
> if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> return false;
>
> *************** vect_min_worthwhile_factor (enum tree_co
> *** 2209,2214 ****
> --- 2196,2254 ----
> }
>
>
> + /* Function vectorizable_induction
> +
> + Check if PHI performs an induction computation that can be vectorized.
> + If VEC_STMT is also passed, vectorize the induction PHI: create a vectorized
> + phi to replace it, put it in VEC_STMT, and add it to the same basic block.
> + Return FALSE if not a vectorizable STMT, TRUE otherwise. */
> +
> + bool
> + vectorizable_induction (tree phi, block_stmt_iterator *bsi ATTRIBUTE_UNUSED,
> + tree *vec_stmt)
> + {
> + stmt_vec_info stmt_info = vinfo_for_stmt (phi);
> + tree vectype = STMT_VINFO_VECTYPE (stmt_info);
> + loop_vec_info loop_vinfo = STMT_VINFO_LOOP_VINFO (stmt_info);
> + int nunits = TYPE_VECTOR_SUBPARTS (vectype);
> + int ncopies = LOOP_VINFO_VECT_FACTOR (loop_vinfo) / nunits;
> + tree vec_def;
> +
> + gcc_assert (ncopies >= 1);
> +
> + if (!STMT_VINFO_RELEVANT_P (stmt_info))
> + return false;
> +
> + gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_induction_def);
> +
> + if (STMT_VINFO_LIVE_P (stmt_info))
> + {
> + /* FORNOW: not yet supported. */
> + if (vect_print_dump_info (REPORT_DETAILS))
> + fprintf (vect_dump, "value used after loop.");
> + return false;
> + }
> +
> + if (TREE_CODE (phi) != PHI_NODE)
> + return false;
> +
> + if (!vec_stmt) /* transformation not required. */
> + {
> + STMT_VINFO_TYPE (stmt_info) = induc_vec_info_type;
> + return true;
> + }
> +
> + /** Transform. **/
> +
> + if (vect_print_dump_info (REPORT_DETAILS))
> + fprintf (vect_dump, "transform induction phi.");
> +
> + vec_def = get_initial_def_for_induction (phi);
> + *vec_stmt = SSA_NAME_DEF_STMT (vec_def);
> + return true;
> + }
> +
> +
> /* Function vectorizable_operation.
>
> Check if STMT performs a binary or unary operation that can be vectorized.
> *************** vectorizable_operation (tree stmt, block
> *** 2246,2265 ****
>
> gcc_assert (ncopies >= 1);
>
> - /* Is STMT a vectorizable binary/unary operation? */
> if (!STMT_VINFO_RELEVANT_P (stmt_info))
> return false;
>
> ! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
>
> if (STMT_VINFO_LIVE_P (stmt_info))
> {
> - /* FORNOW: not yet supported. */
> if (vect_print_dump_info (REPORT_DETAILS))
> fprintf (vect_dump, "value used after loop.");
> return false;
> }
>
> if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> return false;
>
> --- 2286,2306 ----
>
> gcc_assert (ncopies >= 1);
>
> if (!STMT_VINFO_RELEVANT_P (stmt_info))
> return false;
>
> ! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
> ! return false;
>
> + /* FORNOW: not yet supported. */
> if (STMT_VINFO_LIVE_P (stmt_info))
> {
> if (vect_print_dump_info (REPORT_DETAILS))
> fprintf (vect_dump, "value used after loop.");
> return false;
> }
>
> + /* Is STMT a vectorizable binary/unary operation? */
> if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> return false;
>
> *************** vectorizable_type_demotion (tree stmt, b
> *** 2514,2534 ****
> optab optab;
> enum machine_mode vec_mode;
>
> - /* Is STMT a vectorizable type-demotion operation? */
> -
> if (!STMT_VINFO_RELEVANT_P (stmt_info))
> return false;
> !
> ! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
> !
> if (STMT_VINFO_LIVE_P (stmt_info))
> {
> - /* FORNOW: not yet supported. */
> if (vect_print_dump_info (REPORT_DETAILS))
> fprintf (vect_dump, "value used after loop.");
> return false;
> }
>
> if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> return false;
>
> --- 2555,2575 ----
> optab optab;
> enum machine_mode vec_mode;
>
> if (!STMT_VINFO_RELEVANT_P (stmt_info))
> return false;
> !
> ! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
> ! return false;
> !
> ! /* FORNOW: not yet supported. */
> if (STMT_VINFO_LIVE_P (stmt_info))
> {
> if (vect_print_dump_info (REPORT_DETAILS))
> fprintf (vect_dump, "value used after loop.");
> return false;
> }
>
> + /* Is STMT a vectorizable type-demotion operation? */
> if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> return false;
>
> *************** vectorizable_type_promotion (tree stmt,
> *** 2723,2743 ****
> int j;
> tree vectype_in;
>
> - /* Is STMT a vectorizable type-promotion operation? */
> -
> if (!STMT_VINFO_RELEVANT_P (stmt_info))
> return false;
>
> ! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
>
> if (STMT_VINFO_LIVE_P (stmt_info))
> {
> - /* FORNOW: not yet supported. */
> if (vect_print_dump_info (REPORT_DETAILS))
> ! fprintf (vect_dump, "value used after loop.");
> return false;
> }
>
> if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> return false;
>
> --- 2764,2784 ----
> int j;
> tree vectype_in;
>
> if (!STMT_VINFO_RELEVANT_P (stmt_info))
> return false;
>
> ! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
> ! return false;
>
> + /* FORNOW: not yet supported. */
> if (STMT_VINFO_LIVE_P (stmt_info))
> {
> if (vect_print_dump_info (REPORT_DETAILS))
> ! fprintf (vect_dump, "value used after loop.");
> return false;
> }
>
> + /* Is STMT a vectorizable type-promotion operation? */
> if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> return false;
>
> *************** vectorizable_store (tree stmt, block_stm
> *** 3064,3069 ****
> --- 3105,3123 ----
> VEC(tree,heap) *dr_chain = NULL, *oprnds = NULL, *result_chain = NULL;
> gcc_assert (ncopies >= 1);
>
> + if (!STMT_VINFO_RELEVANT_P (stmt_info))
> + return false;
> +
> + if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
> + return false;
> +
> + if (STMT_VINFO_LIVE_P (stmt_info))
> + {
> + if (vect_print_dump_info (REPORT_DETAILS))
> + fprintf (vect_dump, "value used after loop.");
> + return false;
> + }
> +
> /* Is vectorizable store? */
>
> if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> *************** vectorizable_load (tree stmt, block_stmt
> *** 3710,3729 ****
> bool strided_load = false;
> tree first_stmt;
>
> - /* Is vectorizable load? */
> if (!STMT_VINFO_RELEVANT_P (stmt_info))
> return false;
>
> ! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
>
> if (STMT_VINFO_LIVE_P (stmt_info))
> {
> - /* FORNOW: not yet supported. */
> if (vect_print_dump_info (REPORT_DETAILS))
> fprintf (vect_dump, "value used after loop.");
> return false;
> }
>
> if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> return false;
>
> --- 3764,3784 ----
> bool strided_load = false;
> tree first_stmt;
>
> if (!STMT_VINFO_RELEVANT_P (stmt_info))
> return false;
>
> ! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
> ! return false;
>
> + /* FORNOW: not yet supported. */
> if (STMT_VINFO_LIVE_P (stmt_info))
> {
> if (vect_print_dump_info (REPORT_DETAILS))
> fprintf (vect_dump, "value used after loop.");
> return false;
> }
>
> + /* Is vectorizable load? */
> if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> return false;
>
> *************** vectorizable_live_operation (tree stmt,
> *** 4010,4016 ****
> tree def, def_stmt;
> enum vect_def_type dt;
>
> ! if (!STMT_VINFO_LIVE_P (stmt_info))
> return false;
>
> if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> --- 4065,4073 ----
> tree def, def_stmt;
> enum vect_def_type dt;
>
> ! gcc_assert (STMT_VINFO_LIVE_P (stmt_info));
> !
> ! if (STMT_VINFO_DEF_TYPE (stmt_info) == vect_reduction_def)
> return false;
>
> if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> *************** vectorizable_condition (tree stmt, block
> *** 4123,4138 ****
> if (!STMT_VINFO_RELEVANT_P (stmt_info))
> return false;
>
> ! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
>
> if (STMT_VINFO_LIVE_P (stmt_info))
> {
> - /* FORNOW: not yet supported. */
> if (vect_print_dump_info (REPORT_DETAILS))
> fprintf (vect_dump, "value used after loop.");
> return false;
> }
>
> if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> return false;
>
> --- 4180,4197 ----
> if (!STMT_VINFO_RELEVANT_P (stmt_info))
> return false;
>
> ! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
> ! return false;
>
> + /* FORNOW: not yet supported. */
> if (STMT_VINFO_LIVE_P (stmt_info))
> {
> if (vect_print_dump_info (REPORT_DETAILS))
> fprintf (vect_dump, "value used after loop.");
> return false;
> }
>
> + /* Is vectorizable conditional operation? */
> if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> return false;
>
> *************** vect_transform_stmt (tree stmt, block_st
> *** 4225,4334 ****
> tree orig_stmt_in_pattern;
> bool done;
>
> ! if (STMT_VINFO_RELEVANT_P (stmt_info))
> {
> ! switch (STMT_VINFO_TYPE (stmt_info))
> ! {
> ! case type_demotion_vec_info_type:
> ! done = vectorizable_type_demotion (stmt, bsi, &vec_stmt);
> ! gcc_assert (done);
> ! break;
>
> ! case type_promotion_vec_info_type:
> ! done = vectorizable_type_promotion (stmt, bsi, &vec_stmt);
> ! gcc_assert (done);
> ! break;
> !
> ! case type_conversion_vec_info_type:
> ! done = vectorizable_conversion (stmt, bsi, &vec_stmt);
> ! gcc_assert (done);
> ! break;
> !
> ! case op_vec_info_type:
> ! done = vectorizable_operation (stmt, bsi, &vec_stmt);
> ! gcc_assert (done);
> ! break;
> !
> ! case assignment_vec_info_type:
> ! done = vectorizable_assignment (stmt, bsi, &vec_stmt);
> ! gcc_assert (done);
> ! break;
> !
> ! case load_vec_info_type:
> ! done = vectorizable_load (stmt, bsi, &vec_stmt);
> ! gcc_assert (done);
> ! break;
> !
> ! case store_vec_info_type:
> ! done = vectorizable_store (stmt, bsi, &vec_stmt);
> ! gcc_assert (done);
> ! if (DR_GROUP_FIRST_DR (stmt_info))
> ! {
> ! /* In case of interleaving, the whole chain is vectorized when the
> ! last store in the chain is reached. Store stmts before the last
> ! one are skipped, and there vec_stmt_info shouldn't be freed
> ! meanwhile. */
> ! *strided_store = true;
> ! if (STMT_VINFO_VEC_STMT (stmt_info))
> ! is_store = true;
> }
> ! else
> ! is_store = true;
> ! break;
> !
> ! case condition_vec_info_type:
> ! done = vectorizable_condition (stmt, bsi, &vec_stmt);
> ! gcc_assert (done);
> ! break;
>
> ! case call_vec_info_type:
> ! done = vectorizable_call (stmt, bsi, &vec_stmt);
> ! break;
>
> ! default:
> ! if (vect_print_dump_info (REPORT_DETAILS))
> ! fprintf (vect_dump, "stmt not supported.");
> ! gcc_unreachable ();
> ! }
> !
> ! gcc_assert (vec_stmt || *strided_store);
> ! if (vec_stmt)
> {
> ! STMT_VINFO_VEC_STMT (stmt_info) = vec_stmt;
> ! orig_stmt_in_pattern = STMT_VINFO_RELATED_STMT (stmt_info);
> ! if (orig_stmt_in_pattern)
> ! {
> ! stmt_vec_info stmt_vinfo = vinfo_for_stmt (orig_stmt_in_pattern);
> ! if (STMT_VINFO_IN_PATTERN_P (stmt_vinfo))
> ! {
> ! gcc_assert (STMT_VINFO_RELATED_STMT (stmt_vinfo) == stmt);
> !
> ! /* STMT was inserted by the vectorizer to replace a
> ! computation idiom. ORIG_STMT_IN_PATTERN is a stmt in the
> ! original sequence that computed this idiom. We need to
> ! record a pointer to VEC_STMT in the stmt_info of
> ! ORIG_STMT_IN_PATTERN. See more details in the
> ! documentation of vect_pattern_recog. */
> !
> ! STMT_VINFO_VEC_STMT (stmt_vinfo) = vec_stmt;
> ! }
> ! }
> }
> }
>
> ! if (STMT_VINFO_LIVE_P (stmt_info))
> {
> ! switch (STMT_VINFO_TYPE (stmt_info))
> ! {
> ! case reduc_vec_info_type:
> ! done = vectorizable_reduction (stmt, bsi, &vec_stmt);
> ! gcc_assert (done);
> ! break;
>
> ! default:
> ! done = vectorizable_live_operation (stmt, bsi, &vec_stmt);
> ! gcc_assert (done);
> ! }
> }
>
> return is_store;
> --- 4284,4391 ----
> tree orig_stmt_in_pattern;
> bool done;
>
> ! switch (STMT_VINFO_TYPE (stmt_info))
> {
> ! case type_demotion_vec_info_type:
> ! done = vectorizable_type_demotion (stmt, bsi, &vec_stmt);
> ! gcc_assert (done);
> ! break;
>
> ! case type_promotion_vec_info_type:
> ! done = vectorizable_type_promotion (stmt, bsi, &vec_stmt);
> ! gcc_assert (done);
> ! break;
> !
> ! case type_conversion_vec_info_type:
> ! done = vectorizable_conversion (stmt, bsi, &vec_stmt);
> ! gcc_assert (done);
> ! break;
> !
> ! case induc_vec_info_type:
> ! done = vectorizable_induction (stmt, bsi, &vec_stmt);
> ! gcc_assert (done);
> ! break;
> !
> ! case op_vec_info_type:
> ! done = vectorizable_operation (stmt, bsi, &vec_stmt);
> ! gcc_assert (done);
> ! break;
> !
> ! case assignment_vec_info_type:
> ! done = vectorizable_assignment (stmt, bsi, &vec_stmt);
> ! gcc_assert (done);
> ! break;
> !
> ! case load_vec_info_type:
> ! done = vectorizable_load (stmt, bsi, &vec_stmt);
> ! gcc_assert (done);
> ! break;
> !
> ! case store_vec_info_type:
> ! done = vectorizable_store (stmt, bsi, &vec_stmt);
> ! gcc_assert (done);
> ! if (DR_GROUP_FIRST_DR (stmt_info))
> ! {
> ! /* In case of interleaving, the whole chain is vectorized when the
> ! last store in the chain is reached. Store stmts before the last
> ! one are skipped, and there vec_stmt_info shouldn't be freed
> ! meanwhile. */
> ! *strided_store = true;
> ! if (STMT_VINFO_VEC_STMT (stmt_info))
> ! is_store = true;
> }
> ! else
> ! is_store = true;
> ! break;
>
> ! case condition_vec_info_type:
> ! done = vectorizable_condition (stmt, bsi, &vec_stmt);
> ! gcc_assert (done);
> ! break;
> !
> ! case call_vec_info_type:
> ! done = vectorizable_call (stmt, bsi, &vec_stmt);
> ! break;
> !
> ! case reduc_vec_info_type:
> ! done = vectorizable_reduction (stmt, bsi, &vec_stmt);
> ! gcc_assert (done);
> ! break;
>
> ! default:
> ! if (!STMT_VINFO_LIVE_P (stmt_info))
> {
> ! if (vect_print_dump_info (REPORT_DETAILS))
> ! fprintf (vect_dump, "stmt not supported.");
> ! gcc_unreachable ();
> }
> }
>
> ! if (STMT_VINFO_LIVE_P (stmt_info)
> ! && STMT_VINFO_TYPE (stmt_info) != reduc_vec_info_type)
> {
> ! done = vectorizable_live_operation (stmt, bsi, &vec_stmt);
> ! gcc_assert (done);
> ! }
>
> ! if (vec_stmt)
> ! {
> ! STMT_VINFO_VEC_STMT (stmt_info) = vec_stmt;
> ! orig_stmt_in_pattern = STMT_VINFO_RELATED_STMT (stmt_info);
> ! if (orig_stmt_in_pattern)
> ! {
> ! stmt_vec_info stmt_vinfo = vinfo_for_stmt (orig_stmt_in_pattern);
> ! /* STMT was inserted by the vectorizer to replace a computation idiom.
> ! ORIG_STMT_IN_PATTERN is a stmt in the original sequence that
> ! computed this idiom. We need to record a pointer to VEC_STMT in
> ! the stmt_info of ORIG_STMT_IN_PATTERN. See more details in the
> ! documentation of vect_pattern_recog. */
> ! if (STMT_VINFO_IN_PATTERN_P (stmt_vinfo))
> ! {
> ! gcc_assert (STMT_VINFO_RELATED_STMT (stmt_vinfo) == stmt);
> ! STMT_VINFO_VEC_STMT (stmt_vinfo) = vec_stmt;
> ! }
> ! }
> }
>
> return is_store;
> *************** vect_transform_loop (loop_vec_info loop_
> *** 5156,5166 ****
> for (i = 0; i < nbbs; i++)
> {
> basic_block bb = bbs[i];
>
> for (si = bsi_start (bb); !bsi_end_p (si);)
> {
> tree stmt = bsi_stmt (si);
> - stmt_vec_info stmt_info;
> bool is_store;
>
> if (vect_print_dump_info (REPORT_DETAILS))
> --- 5213,5251 ----
> for (i = 0; i < nbbs; i++)
> {
> basic_block bb = bbs[i];
> + stmt_vec_info stmt_info;
> + tree phi;
> +
> + for (phi = phi_nodes (bb); phi; phi = PHI_CHAIN (phi))
> + {
> + if (vect_print_dump_info (REPORT_DETAILS))
> + {
> + fprintf (vect_dump, "------>vectorizing phi: ");
> + print_generic_expr (vect_dump, phi, TDF_SLIM);
> + }
> + stmt_info = vinfo_for_stmt (phi);
> + if (!stmt_info)
> + continue;
> + if (!STMT_VINFO_RELEVANT_P (stmt_info)
> + && !STMT_VINFO_LIVE_P (stmt_info))
> + continue;
> +
> + if ((TYPE_VECTOR_SUBPARTS (STMT_VINFO_VECTYPE (stmt_info))
> + != (unsigned HOST_WIDE_INT) vectorization_factor)
> + && vect_print_dump_info (REPORT_DETAILS))
> + fprintf (vect_dump, "multiple-types.");
> +
> + if (STMT_VINFO_DEF_TYPE (stmt_info) == vect_induction_def)
> + {
> + if (vect_print_dump_info (REPORT_DETAILS))
> + fprintf (vect_dump, "transform phi.");
> + vect_transform_stmt (phi, NULL, NULL);
> + }
> + }
>
> for (si = bsi_start (bb); !bsi_end_p (si);)
> {
> tree stmt = bsi_stmt (si);
> bool is_store;
>
> if (vect_print_dump_info (REPORT_DETAILS))
> *************** vect_transform_loop (loop_vec_info loop_
> *** 5177,5182 ****
> --- 5262,5268 ----
> continue;
> }
>
> + gcc_assert (STMT_VINFO_VECTYPE (stmt_info));
> if ((TYPE_VECTOR_SUBPARTS (STMT_VINFO_VECTYPE (stmt_info))
> != (unsigned HOST_WIDE_INT) vectorization_factor)
> && vect_print_dump_info (REPORT_DETAILS))
> Index: tree-ssa-reassoc.c
> ===================================================================
> *** tree-ssa-reassoc.c (revision 123103)
> --- tree-ssa-reassoc.c (working copy)
> *************** execute_reassoc (void)
> *** 1476,1485 ****
> return 0;
> }
>
> struct tree_opt_pass pass_reassoc =
> {
> "reassoc", /* name */
> ! NULL, /* gate */
> execute_reassoc, /* execute */
> NULL, /* sub */
> NULL, /* next */
> --- 1476,1491 ----
> return 0;
> }
>
> + static bool
> + gate_reassoc (void)
> + {
> + return flag_tree_reassoc != 0;
> + }
> +
> struct tree_opt_pass pass_reassoc =
> {
> "reassoc", /* name */
> ! gate_reassoc, /* gate */
> execute_reassoc, /* execute */
> NULL, /* sub */
> NULL, /* next */
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [patch] vectorizer cleanups (and prep for outer-loop vectorization)
2007-04-02 9:48 ` Zdenek Dvorak
@ 2007-04-02 9:55 ` Zdenek Dvorak
2007-04-11 11:51 ` [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 2/5 Dorit Nuzman
` (5 subsequent siblings)
6 siblings, 0 replies; 18+ messages in thread
From: Zdenek Dvorak @ 2007-04-02 9:55 UTC (permalink / raw)
To: Dorit Nuzman; +Cc: gcc-patches
> I will look at the patch. However, would it be possible to split it
> into smaller parts? The changes seem to be quite independent,
sorry for posting this reply together with the patch :-(
Zdenek
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 1/5
2007-04-02 9:48 ` Zdenek Dvorak
` (2 preceding siblings ...)
2007-04-11 11:51 ` [patch] vectorizer cleanups (and prep for outer-loop vectorization) Dorit Nuzman
@ 2007-04-11 11:51 ` Dorit Nuzman
2007-04-11 11:51 ` [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 4/5 Dorit Nuzman
` (2 subsequent siblings)
6 siblings, 0 replies; 18+ messages in thread
From: Dorit Nuzman @ 2007-04-11 11:51 UTC (permalink / raw)
To: Zdenek Dvorak; +Cc: gcc-patches
[-- Attachment #1: Type: text/plain, Size: 2326 bytes --]
Zdenek Dvorak <rakdver@atrey.karlin.mff.cuni.cz> wrote on 02/04/2007
12:48:12:
> Hello,
>
> I will look at the patch. However, would it be possible to split it
> into smaller parts? The changes seem to be quite independent,
>
> Zdenek
>
> > This is exactly the same patch as was committed to autovect last week:
> > http://gcc.gnu.org/ml/gcc-patches/2007-03/msg01904.html
> > (but relative to mainline, and after testig on mainline).
> >
> > These are a few cleanups (really technical things, no new
functionality),
> > some of which will be also useful for outer-loop vectorization:
> >
...
> >
> > o Reorganize the calls to the vectorizable_* functions: In the drivers
that
> > call these functions (vect_analyze_operations and vect_transform_stmt),
we
> > currently call most of these functions only for RELEVANT_P stmts, and
then
> > call vectorizable_reduction only for LIVE_P stmts. Now, instead, each
of
> > the vectorizable_* functions checks that STMT_VINFO_RELEVANT/LIVE_P are
as
> > expected, and in the two driver functions we don't check for
> > STMT_VINFO_RELEVANT_P anymore, but apply the vectorizable_* functions
to
> > all stmts. When looking at outer-loops this is useful because a
reduction
> > (in the inner-loop) may now be marked RELEVANT (if it is used in the
> > outer-loop), and not only LIVE as it is currently.
> >
Tested on the vectorizer testcases
Bootstrapped and passed full testing on powerpc-linux together with the
other parts of this patch.
thanks,
dorit
ChangeLog:
* tree-vect-analyze.c (vect_analyze_operations): Reorganize calls to
vectorizable_* functions.
* tree-vect-transform.c (vectorizable_call): Add check for
STMT_VINFO_RELEVANT_P, STMT_VINFO_DEF_TYPE and STMT_VINFO_LIVE_P.
(vectorizable_store): likewise.
(vectorizable_conversion): Add check for STMT_VINFO_DEF_TYPE.
Add comments.
(vectorizable_operation, vectorizable_type_demotion): Likewise.
(vectorizable_type_promotion, vectorizable_load): Likewise.
(vectorizable_live_operation, vectorizable_condition): Likewise.
(vectorizable_assignment): Add check for STMT_VINFO_DEF_TYPE and
STMT_VINFO_LIVE_P.
(vect_transform_stmt): Reorganize calls to vectorizable_* functions.
Patch:
(See attached file: vect_cleanups.1.reorg.txt)
[-- Attachment #2: vect_cleanups.1.reorg.txt --]
[-- Type: text/plain, Size: 21457 bytes --]
Index: tree-vect-analyze.c
===================================================================
*** tree-vect-analyze.c (revision 123694)
--- tree-vect-analyze.c (working copy)
*************** vect_analyze_operations (loop_vec_info l
*** 325,332 ****
/* FORNOW: not yet supported. */
if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
fprintf (vect_dump, "not vectorized: value used after loop.");
! return false;
! }
if (STMT_VINFO_RELEVANT (stmt_info) == vect_used_in_loop
&& STMT_VINFO_DEF_TYPE (stmt_info) != vect_induction_def)
--- 325,332 ----
/* FORNOW: not yet supported. */
if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
fprintf (vect_dump, "not vectorized: value used after loop.");
! return false;
! }
if (STMT_VINFO_RELEVANT (stmt_info) == vect_used_in_loop
&& STMT_VINFO_DEF_TYPE (stmt_info) != vect_induction_def)
*************** vect_analyze_operations (loop_vec_info l
*** 337,348 ****
--- 337,352 ----
fprintf (vect_dump, "not vectorized: unsupported pattern.");
return false;
}
+
+ if (STMT_VINFO_RELEVANT_P (stmt_info))
+ need_to_vectorize = true;
}
for (si = bsi_start (bb); !bsi_end_p (si); bsi_next (&si))
{
tree stmt = bsi_stmt (si);
stmt_vec_info stmt_info = vinfo_for_stmt (stmt);
+ enum vect_def_type relevance = STMT_VINFO_RELEVANT (stmt_info);
if (vect_print_dump_info (REPORT_DETAILS))
{
*************** vect_analyze_operations (loop_vec_info l
*** 367,421 ****
continue;
}
! if (STMT_VINFO_RELEVANT_P (stmt_info))
! {
! gcc_assert (GIMPLE_STMT_P (stmt)
! || !VECTOR_MODE_P (TYPE_MODE (TREE_TYPE (stmt))));
! gcc_assert (STMT_VINFO_VECTYPE (stmt_info));
!
! ok = (vectorizable_type_promotion (stmt, NULL, NULL)
! || vectorizable_type_demotion (stmt, NULL, NULL)
! || vectorizable_conversion (stmt, NULL, NULL)
! || vectorizable_operation (stmt, NULL, NULL)
! || vectorizable_assignment (stmt, NULL, NULL)
! || vectorizable_load (stmt, NULL, NULL)
! || vectorizable_call (stmt, NULL, NULL)
! || vectorizable_store (stmt, NULL, NULL)
! || vectorizable_condition (stmt, NULL, NULL));
!
! if (!ok)
! {
! if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
! {
! fprintf (vect_dump,
! "not vectorized: relevant stmt not supported: ");
! print_generic_expr (vect_dump, stmt, TDF_SLIM);
! }
! return false;
! }
! need_to_vectorize = true;
! }
! if (STMT_VINFO_LIVE_P (stmt_info))
{
! ok = vectorizable_reduction (stmt, NULL, NULL);
! if (ok)
! need_to_vectorize = true;
! else
! ok = vectorizable_live_operation (stmt, NULL, NULL);
! if (!ok)
{
! if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
! {
! fprintf (vect_dump,
! "not vectorized: live stmt not supported: ");
! print_generic_expr (vect_dump, stmt, TDF_SLIM);
! }
! return false;
}
! }
} /* stmts in bb */
} /* bbs */
--- 371,427 ----
continue;
}
! switch (STMT_VINFO_DEF_TYPE (stmt_info))
! {
! case vect_loop_def:
! break;
!
! case vect_reduction_def:
! gcc_assert (relevance == vect_unused_in_loop);
! break;
!
! case vect_induction_def:
! case vect_constant_def:
! case vect_invariant_def:
! case vect_unknown_def_type:
! default:
! gcc_unreachable ();
! }
! if (STMT_VINFO_RELEVANT_P (stmt_info))
{
! gcc_assert (GIMPLE_STMT_P (stmt)
! || !VECTOR_MODE_P (TYPE_MODE (TREE_TYPE (stmt))));
! gcc_assert (STMT_VINFO_VECTYPE (stmt_info));
! need_to_vectorize = true;
! }
! ok = (vectorizable_type_promotion (stmt, NULL, NULL)
! || vectorizable_type_demotion (stmt, NULL, NULL)
! || vectorizable_conversion (stmt, NULL, NULL)
! || vectorizable_operation (stmt, NULL, NULL)
! || vectorizable_assignment (stmt, NULL, NULL)
! || vectorizable_load (stmt, NULL, NULL)
! || vectorizable_call (stmt, NULL, NULL)
! || vectorizable_store (stmt, NULL, NULL)
! || vectorizable_condition (stmt, NULL, NULL)
! || vectorizable_reduction (stmt, NULL, NULL));
!
! /* Stmts that are (also) "live" (i.e. - that are used out of the loop)
! need extra handling, except for vectorizable reductions. */
! if (STMT_VINFO_LIVE_P (stmt_info)
! && STMT_VINFO_TYPE (stmt_info) != reduc_vec_info_type)
! ok |= vectorizable_live_operation (stmt, NULL, NULL);
! if (!ok)
! {
! if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
{
! fprintf (vect_dump, "not vectorized: stmt not supported: ");
! print_generic_expr (vect_dump, stmt, TDF_SLIM);
}
! return false;
! }
} /* stmts in bb */
} /* bbs */
Index: tree-vect-transform.c
===================================================================
*** tree-vect-transform.c (revision 123694)
--- tree-vect-transform.c (working copy)
*************** vectorizable_call (tree stmt, block_stmt
*** 1816,1821 ****
--- 1816,1835 ----
int ncopies, j, nargs;
call_expr_arg_iterator iter;
+ if (!STMT_VINFO_RELEVANT_P (stmt_info))
+ return false;
+
+ if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
+ return false;
+
+ /* FORNOW: not yet supported. */
+ if (STMT_VINFO_LIVE_P (stmt_info))
+ {
+ if (vect_print_dump_info (REPORT_DETAILS))
+ fprintf (vect_dump, "value used after loop.");
+ return false;
+ }
+
/* Is STMT a vectorizable call? */
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
return false;
*************** vectorizable_conversion (tree stmt, bloc
*** 1994,2000 ****
if (!STMT_VINFO_RELEVANT_P (stmt_info))
return false;
! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
if (STMT_VINFO_LIVE_P (stmt_info))
{
--- 2008,2015 ----
if (!STMT_VINFO_RELEVANT_P (stmt_info))
return false;
! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
! return false;
if (STMT_VINFO_LIVE_P (stmt_info))
{
*************** vectorizable_assignment (tree stmt, bloc
*** 2134,2145 ****
if (ncopies > 1)
return false; /* FORNOW */
- /* Is vectorizable assignment? */
if (!STMT_VINFO_RELEVANT_P (stmt_info))
return false;
! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
return false;
--- 2149,2169 ----
if (ncopies > 1)
return false; /* FORNOW */
if (!STMT_VINFO_RELEVANT_P (stmt_info))
return false;
! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
! return false;
!
! /* FORNOW: not yet supported. */
! if (STMT_VINFO_LIVE_P (stmt_info))
! {
! if (vect_print_dump_info (REPORT_DETAILS))
! fprintf (vect_dump, "value used after loop.");
! return false;
! }
+ /* Is vectorizable assignment? */
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
return false;
*************** vectorizable_operation (tree stmt, block
*** 2246,2265 ****
gcc_assert (ncopies >= 1);
- /* Is STMT a vectorizable binary/unary operation? */
if (!STMT_VINFO_RELEVANT_P (stmt_info))
return false;
! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
if (STMT_VINFO_LIVE_P (stmt_info))
{
- /* FORNOW: not yet supported. */
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "value used after loop.");
return false;
}
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
return false;
--- 2270,2290 ----
gcc_assert (ncopies >= 1);
if (!STMT_VINFO_RELEVANT_P (stmt_info))
return false;
! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
! return false;
+ /* FORNOW: not yet supported. */
if (STMT_VINFO_LIVE_P (stmt_info))
{
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "value used after loop.");
return false;
}
+ /* Is STMT a vectorizable binary/unary operation? */
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
return false;
*************** vectorizable_type_demotion (tree stmt, b
*** 2514,2534 ****
optab optab;
enum machine_mode vec_mode;
- /* Is STMT a vectorizable type-demotion operation? */
-
if (!STMT_VINFO_RELEVANT_P (stmt_info))
return false;
!
! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
!
if (STMT_VINFO_LIVE_P (stmt_info))
{
- /* FORNOW: not yet supported. */
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "value used after loop.");
return false;
}
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
return false;
--- 2539,2559 ----
optab optab;
enum machine_mode vec_mode;
if (!STMT_VINFO_RELEVANT_P (stmt_info))
return false;
!
! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
! return false;
!
! /* FORNOW: not yet supported. */
if (STMT_VINFO_LIVE_P (stmt_info))
{
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "value used after loop.");
return false;
}
+ /* Is STMT a vectorizable type-demotion operation? */
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
return false;
*************** vectorizable_type_promotion (tree stmt,
*** 2723,2743 ****
int j;
tree vectype_in;
- /* Is STMT a vectorizable type-promotion operation? */
-
if (!STMT_VINFO_RELEVANT_P (stmt_info))
return false;
! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
if (STMT_VINFO_LIVE_P (stmt_info))
{
- /* FORNOW: not yet supported. */
if (vect_print_dump_info (REPORT_DETAILS))
! fprintf (vect_dump, "value used after loop.");
return false;
}
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
return false;
--- 2748,2768 ----
int j;
tree vectype_in;
if (!STMT_VINFO_RELEVANT_P (stmt_info))
return false;
! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
! return false;
+ /* FORNOW: not yet supported. */
if (STMT_VINFO_LIVE_P (stmt_info))
{
if (vect_print_dump_info (REPORT_DETAILS))
! fprintf (vect_dump, "value used after loop.");
return false;
}
+ /* Is STMT a vectorizable type-promotion operation? */
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
return false;
*************** vectorizable_store (tree stmt, block_stm
*** 3064,3069 ****
--- 3089,3107 ----
VEC(tree,heap) *dr_chain = NULL, *oprnds = NULL, *result_chain = NULL;
gcc_assert (ncopies >= 1);
+ if (!STMT_VINFO_RELEVANT_P (stmt_info))
+ return false;
+
+ if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
+ return false;
+
+ if (STMT_VINFO_LIVE_P (stmt_info))
+ {
+ if (vect_print_dump_info (REPORT_DETAILS))
+ fprintf (vect_dump, "value used after loop.");
+ return false;
+ }
+
/* Is vectorizable store? */
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
*************** vectorizable_load (tree stmt, block_stmt
*** 3710,3729 ****
bool strided_load = false;
tree first_stmt;
- /* Is vectorizable load? */
if (!STMT_VINFO_RELEVANT_P (stmt_info))
return false;
! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
if (STMT_VINFO_LIVE_P (stmt_info))
{
- /* FORNOW: not yet supported. */
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "value used after loop.");
return false;
}
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
return false;
--- 3748,3768 ----
bool strided_load = false;
tree first_stmt;
if (!STMT_VINFO_RELEVANT_P (stmt_info))
return false;
! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
! return false;
+ /* FORNOW: not yet supported. */
if (STMT_VINFO_LIVE_P (stmt_info))
{
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "value used after loop.");
return false;
}
+ /* Is vectorizable load? */
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
return false;
*************** vectorizable_live_operation (tree stmt,
*** 4010,4016 ****
tree def, def_stmt;
enum vect_def_type dt;
! if (!STMT_VINFO_LIVE_P (stmt_info))
return false;
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
--- 4049,4057 ----
tree def, def_stmt;
enum vect_def_type dt;
! gcc_assert (STMT_VINFO_LIVE_P (stmt_info));
!
! if (STMT_VINFO_DEF_TYPE (stmt_info) == vect_reduction_def)
return false;
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
*************** vectorizable_condition (tree stmt, block
*** 4123,4138 ****
if (!STMT_VINFO_RELEVANT_P (stmt_info))
return false;
! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
if (STMT_VINFO_LIVE_P (stmt_info))
{
- /* FORNOW: not yet supported. */
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "value used after loop.");
return false;
}
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
return false;
--- 4164,4181 ----
if (!STMT_VINFO_RELEVANT_P (stmt_info))
return false;
! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
! return false;
+ /* FORNOW: not yet supported. */
if (STMT_VINFO_LIVE_P (stmt_info))
{
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "value used after loop.");
return false;
}
+ /* Is vectorizable conditional operation? */
if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
return false;
*************** vect_transform_stmt (tree stmt, block_st
*** 4225,4336 ****
tree orig_stmt_in_pattern;
bool done;
! if (STMT_VINFO_RELEVANT_P (stmt_info))
{
! switch (STMT_VINFO_TYPE (stmt_info))
! {
! case type_demotion_vec_info_type:
! done = vectorizable_type_demotion (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
!
! case type_promotion_vec_info_type:
! done = vectorizable_type_promotion (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
!
! case type_conversion_vec_info_type:
! done = vectorizable_conversion (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
!
! case op_vec_info_type:
! done = vectorizable_operation (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
!
! case assignment_vec_info_type:
! done = vectorizable_assignment (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
!
! case load_vec_info_type:
! done = vectorizable_load (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
!
! case store_vec_info_type:
! done = vectorizable_store (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! if (DR_GROUP_FIRST_DR (stmt_info))
! {
! /* In case of interleaving, the whole chain is vectorized when the
! last store in the chain is reached. Store stmts before the last
! one are skipped, and there vec_stmt_info shouldn't be freed
! meanwhile. */
! *strided_store = true;
! if (STMT_VINFO_VEC_STMT (stmt_info))
! is_store = true;
}
! else
! is_store = true;
! break;
! case condition_vec_info_type:
! done = vectorizable_condition (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
! case call_vec_info_type:
! done = vectorizable_call (stmt, bsi, &vec_stmt);
! break;
! default:
! if (vect_print_dump_info (REPORT_DETAILS))
! fprintf (vect_dump, "stmt not supported.");
! gcc_unreachable ();
! }
! gcc_assert (vec_stmt || *strided_store);
! if (vec_stmt)
{
! STMT_VINFO_VEC_STMT (stmt_info) = vec_stmt;
! orig_stmt_in_pattern = STMT_VINFO_RELATED_STMT (stmt_info);
! if (orig_stmt_in_pattern)
{
! stmt_vec_info stmt_vinfo = vinfo_for_stmt (orig_stmt_in_pattern);
! if (STMT_VINFO_IN_PATTERN_P (stmt_vinfo))
! {
! gcc_assert (STMT_VINFO_RELATED_STMT (stmt_vinfo) == stmt);
!
! /* STMT was inserted by the vectorizer to replace a
! computation idiom. ORIG_STMT_IN_PATTERN is a stmt in the
! original sequence that computed this idiom. We need to
! record a pointer to VEC_STMT in the stmt_info of
! ORIG_STMT_IN_PATTERN. See more details in the
! documentation of vect_pattern_recog. */
!
! STMT_VINFO_VEC_STMT (stmt_vinfo) = vec_stmt;
! }
}
}
}
- if (STMT_VINFO_LIVE_P (stmt_info))
- {
- switch (STMT_VINFO_TYPE (stmt_info))
- {
- case reduc_vec_info_type:
- done = vectorizable_reduction (stmt, bsi, &vec_stmt);
- gcc_assert (done);
- break;
-
- default:
- done = vectorizable_live_operation (stmt, bsi, &vec_stmt);
- gcc_assert (done);
- }
- }
-
return is_store;
}
--- 4268,4372 ----
tree orig_stmt_in_pattern;
bool done;
! switch (STMT_VINFO_TYPE (stmt_info))
{
! case type_demotion_vec_info_type:
! done = vectorizable_type_demotion (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
!
! case type_promotion_vec_info_type:
! done = vectorizable_type_promotion (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
!
! case type_conversion_vec_info_type:
! done = vectorizable_conversion (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
!
! case op_vec_info_type:
! done = vectorizable_operation (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
!
! case assignment_vec_info_type:
! done = vectorizable_assignment (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
!
! case load_vec_info_type:
! done = vectorizable_load (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
!
! case store_vec_info_type:
! done = vectorizable_store (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! if (DR_GROUP_FIRST_DR (stmt_info))
! {
! /* In case of interleaving, the whole chain is vectorized when the
! last store in the chain is reached. Store stmts before the last
! one are skipped, and there vec_stmt_info shouldn't be freed
! meanwhile. */
! *strided_store = true;
! if (STMT_VINFO_VEC_STMT (stmt_info))
! is_store = true;
}
! else
! is_store = true;
! break;
! case condition_vec_info_type:
! done = vectorizable_condition (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
!
! case call_vec_info_type:
! done = vectorizable_call (stmt, bsi, &vec_stmt);
! break;
!
! case reduc_vec_info_type:
! done = vectorizable_reduction (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! break;
! default:
! if (!STMT_VINFO_LIVE_P (stmt_info))
! {
! if (vect_print_dump_info (REPORT_DETAILS))
! fprintf (vect_dump, "stmt not supported.");
! gcc_unreachable ();
! }
! }
! if (STMT_VINFO_LIVE_P (stmt_info)
! && STMT_VINFO_TYPE (stmt_info) != reduc_vec_info_type)
! {
! done = vectorizable_live_operation (stmt, bsi, &vec_stmt);
! gcc_assert (done);
! }
! if (vec_stmt)
! {
! STMT_VINFO_VEC_STMT (stmt_info) = vec_stmt;
! orig_stmt_in_pattern = STMT_VINFO_RELATED_STMT (stmt_info);
! if (orig_stmt_in_pattern)
{
! stmt_vec_info stmt_vinfo = vinfo_for_stmt (orig_stmt_in_pattern);
! /* STMT was inserted by the vectorizer to replace a computation idiom.
! ORIG_STMT_IN_PATTERN is a stmt in the original sequence that
! computed this idiom. We need to record a pointer to VEC_STMT in
! the stmt_info of ORIG_STMT_IN_PATTERN. See more details in the
! documentation of vect_pattern_recog. */
! if (STMT_VINFO_IN_PATTERN_P (stmt_vinfo))
{
! gcc_assert (STMT_VINFO_RELATED_STMT (stmt_vinfo) == stmt);
! STMT_VINFO_VEC_STMT (stmt_vinfo) = vec_stmt;
}
}
}
return is_store;
}
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 4/5
2007-04-02 9:48 ` Zdenek Dvorak
` (3 preceding siblings ...)
2007-04-11 11:51 ` [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 1/5 Dorit Nuzman
@ 2007-04-11 11:51 ` Dorit Nuzman
2007-04-12 12:17 ` Zdenek Dvorak
2007-04-11 11:51 ` [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 3/5 Dorit Nuzman
2007-04-11 11:51 ` [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 5/5 Dorit Nuzman
6 siblings, 1 reply; 18+ messages in thread
From: Dorit Nuzman @ 2007-04-11 11:51 UTC (permalink / raw)
To: Zdenek Dvorak; +Cc: gcc-patches
[-- Attachment #1: Type: text/plain, Size: 1240 bytes --]
Zdenek Dvorak <rakdver@atrey.karlin.mff.cuni.cz> wrote on 02/04/2007
12:48:12:
> Hello,
>
> I will look at the patch. However, would it be possible to split it
> into smaller parts? The changes seem to be quite independent,
>
> Zdenek
>
> > This is exactly the same patch as was committed to autovect last week:
> > http://gcc.gnu.org/ml/gcc-patches/2007-03/msg01904.html
> > (but relative to mainline, and after testig on mainline).
> >
> > These are a few cleanups (really technical things, no new
functionality),
> > some of which will be also useful for outer-loop vectorization:
> >
...
> >
> > o A bunch of other cleanups (that are not necessarily inspired by
> > outer-loop vectorization): ... Remove the '#ifdef
> > ADJUST_IN_EPILOG' and the code path that was disabled by it (which
removes
> > unused code and simplifies the function); ...
> >
Tested on the vectorizer testcases
Bootstrapped and passed full testing on powerpc-linux together with the
other parts of this patch.
thanks,
dorit
ChangeLog:
* tree-vect-transform.c (get_initial_def_for_reduction): Clean away
the unused code for reduction without adjust-in-epilog to simplify
the
function.
(See attached file: vect_cleanups.4.no_adjust.txt)
[-- Attachment #2: vect_cleanups.4.no_adjust.txt --]
[-- Type: text/plain, Size: 6271 bytes --]
*** tree-vect-transform.c.2 2007-04-10 21:54:08.000000000 +0300
--- tree-vect-transform.c.4 2007-04-10 22:27:49.000000000 +0300
*************** vect_finish_stmt_generation (tree stmt,
*** 944,951 ****
}
- #define ADJUST_IN_EPILOG 1
-
/* Function get_initial_def_for_reduction
Input:
--- 944,949 ----
*************** vect_finish_stmt_generation (tree stmt,
*** 953,970 ****
INIT_VAL - the initial value of the reduction variable
Output:
! SCALAR_DEF - a tree that holds a value to be added to the final result
! of the reduction (used for "ADJUST_IN_EPILOG" - see below).
Return a vector variable, initialized according to the operation that STMT
! performs. This vector will be used as the initial value of the
! vector of partial results.
! Option1 ("ADJUST_IN_EPILOG"): Initialize the vector as follows:
add: [0,0,...,0,0]
mult: [1,1,...,1,1]
min/max: [init_val,init_val,..,init_val,init_val]
bit and/or: [init_val,init_val,..,init_val,init_val]
! and when necessary (e.g. add/mult case) let the caller know
that it needs to adjust the result by init_val.
Option2: Initialize the vector as follows:
--- 951,968 ----
INIT_VAL - the initial value of the reduction variable
Output:
! ADJUSTMENT_DEF - a tree that holds a value to be added to the final result
! of the reduction (used for adjusting the epilog - see below).
Return a vector variable, initialized according to the operation that STMT
! performs. This vector will be used as the initial value of the
! vector of partial results.
! Option1 (adjust in epilog): Initialize the vector as follows:
add: [0,0,...,0,0]
mult: [1,1,...,1,1]
min/max: [init_val,init_val,..,init_val,init_val]
bit and/or: [init_val,init_val,..,init_val,init_val]
! and when necessary (e.g. add/mult case) let the caller know
that it needs to adjust the result by init_val.
Option2: Initialize the vector as follows:
*************** vect_finish_stmt_generation (tree stmt,
*** 985,1068 ****
or [0,0,0,0] and let the caller know that it needs to adjust
the result at the end by 'init_val'.
! FORNOW: We use the "ADJUST_IN_EPILOG" scheme.
! TODO: Use some cost-model to estimate which scheme is more profitable.
! */
static tree
! get_initial_def_for_reduction (tree stmt, tree init_val, tree *scalar_def)
{
stmt_vec_info stmt_vinfo = vinfo_for_stmt (stmt);
tree vectype = STMT_VINFO_VECTYPE (stmt_vinfo);
int nunits = TYPE_VECTOR_SUBPARTS (vectype);
- int nelements;
enum tree_code code = TREE_CODE (GIMPLE_STMT_OPERAND (stmt, 1));
tree type = TREE_TYPE (init_val);
! tree def;
! tree vec, t = NULL_TREE;
! bool need_epilog_adjust;
int i;
tree vector_type;
gcc_assert (INTEGRAL_TYPE_P (type) || SCALAR_FLOAT_TYPE_P (type));
switch (code)
{
case WIDEN_SUM_EXPR:
case DOT_PROD_EXPR:
case PLUS_EXPR:
if (INTEGRAL_TYPE_P (type))
! def = build_int_cst (type, 0);
else
! def = build_real (type, dconst0);
!
! #ifdef ADJUST_IN_EPILOG
! /* All the 'nunits' elements are set to 0. The final result will be
! adjusted by 'init_val' at the loop epilog. */
! nelements = nunits;
! need_epilog_adjust = true;
! #else
! /* 'nunits - 1' elements are set to 0; The last element is set to
! 'init_val'. No further adjustments at the epilog are needed. */
! nelements = nunits - 1;
! need_epilog_adjust = false;
! #endif
break;
case MIN_EXPR:
case MAX_EXPR:
! def = init_val;
! nelements = nunits;
! need_epilog_adjust = false;
break;
default:
gcc_unreachable ();
}
! for (i = nelements - 1; i >= 0; --i)
! t = tree_cons (NULL_TREE, def, t);
!
! if (nelements == nunits - 1)
! {
! /* Set the last element of the vector. */
! t = tree_cons (NULL_TREE, init_val, t);
! nelements += 1;
! }
! gcc_assert (nelements == nunits);
!
! vector_type = get_vectype_for_scalar_type (TREE_TYPE (def));
! if (TREE_CODE (init_val) == INTEGER_CST || TREE_CODE (init_val) == REAL_CST)
! vec = build_vector (vector_type, t);
! else
! vec = build_constructor_from_list (vector_type, t);
!
! if (!need_epilog_adjust)
! *scalar_def = NULL_TREE;
! else
! *scalar_def = init_val;
!
! return vect_init_vector (stmt, vec, vector_type);
}
--- 983,1038 ----
or [0,0,0,0] and let the caller know that it needs to adjust
the result at the end by 'init_val'.
! FORNOW, we are using the 'adjust in epilog' scheme, because this way the
! initialization vector is simpler (same element in all entries).
! A cost model should help decide between these two schemes. */
static tree
! get_initial_def_for_reduction (tree stmt, tree init_val, tree *adjustment_def)
{
stmt_vec_info stmt_vinfo = vinfo_for_stmt (stmt);
tree vectype = STMT_VINFO_VECTYPE (stmt_vinfo);
int nunits = TYPE_VECTOR_SUBPARTS (vectype);
enum tree_code code = TREE_CODE (GIMPLE_STMT_OPERAND (stmt, 1));
tree type = TREE_TYPE (init_val);
! tree vecdef;
! tree def_for_init;
! tree init_def;
! tree t = NULL_TREE;
int i;
tree vector_type;
gcc_assert (INTEGRAL_TYPE_P (type) || SCALAR_FLOAT_TYPE_P (type));
+ vecdef = vect_get_vec_def_for_operand (init_val, stmt, NULL);
switch (code)
{
case WIDEN_SUM_EXPR:
case DOT_PROD_EXPR:
case PLUS_EXPR:
+ *adjustment_def = init_val;
+ /* Create a vector of zeros for init_def. */
if (INTEGRAL_TYPE_P (type))
! def_for_init = build_int_cst (type, 0);
else
! def_for_init = build_real (type, dconst0);
! for (i = nunits - 1; i >= 0; --i)
! t = tree_cons (NULL_TREE, def_for_init, t);
! vector_type = get_vectype_for_scalar_type (TREE_TYPE (def_for_init));
! init_def = build_vector (vector_type, t);
break;
case MIN_EXPR:
case MAX_EXPR:
! *adjustment_def = NULL_TREE;
! init_def = vecdef;
break;
default:
gcc_unreachable ();
}
! return init_def;
}
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 2/5
2007-04-02 9:48 ` Zdenek Dvorak
2007-04-02 9:55 ` Zdenek Dvorak
@ 2007-04-11 11:51 ` Dorit Nuzman
2007-04-11 23:53 ` Zdenek Dvorak
2007-04-11 11:51 ` [patch] vectorizer cleanups (and prep for outer-loop vectorization) Dorit Nuzman
` (4 subsequent siblings)
6 siblings, 1 reply; 18+ messages in thread
From: Dorit Nuzman @ 2007-04-11 11:51 UTC (permalink / raw)
To: Zdenek Dvorak; +Cc: gcc-patches
[-- Attachment #1: Type: text/plain, Size: 2590 bytes --]
Zdenek Dvorak <rakdver@atrey.karlin.mff.cuni.cz> wrote on 02/04/2007
12:48:12:
> Hello,
>
> I will look at the patch. However, would it be possible to split it
> into smaller parts? The changes seem to be quite independent,
>
> Zdenek
>
> > This is exactly the same patch as was committed to autovect last week:
> > http://gcc.gnu.org/ml/gcc-patches/2007-03/msg01904.html
> > (but relative to mainline, and after testig on mainline).
> >
> > These are a few cleanups (really technical things, no new
functionality),
> > some of which will be also useful for outer-loop vectorization:
> >
...
> >
> > o Vectorization of induction is currently done upon encountering a use
of
> > the induction variable. Instead, we now vectorize the induction
earlier,
> > when we encounter the respective induction phi-node. This simplifies
things
> > a bit cause it's more in-line with how we vectorize other things (when
we
> > encounter a use of the induction we just get the vector def from the
> > STMT_VINFO_VEC_STMT field of the defining phi-node), and also we don't
need
> > to worry about vectorizing the same induction multiple times. When
looking
> > at outer-loops this will be even more useful cause when we vectorize a
> > phi-node we know exactly relative to which loop-nest is the induction,
and
> > don't need any extra logic to check for that.
> >
...
Tested on the vectorizer testcases
Bootstrapped and passed full testing on powerpc-linux together with the
other parts of this patch.
thanks,
dorit
ChangeLog:
* tree-vectorizer.h (stmt_vec_info_type): Add enum value
induc_vec_info_type.
(vectorizable_induction): New function declaration.
* tree-vect-transform.c (get_initial_def_for_induction): No need to
check if already vectorized. Find first place in BB where new stmts
can be inserted. Takes only one argument.
(vectorizable_induction): New function.
(vect_transform_stmt): Add case for induc_vec_info_type to call
vectorizable_induction.
(vect_transform_loop): Consider phis for vectorization.
* tree-vect-analyze.c (vect_determine_vectorization_factor):
Simplify
condition.
(analyze_operations): Call vectorizable_induction when analyzing
phis.
Fix comment.
(vect_mark_stmts_to_be_vectorized): Remove redundant checks.
(vect_mark_relevant): Include phis in relevance analysis.
(vect_mark_stmts_to_be_vectorize): Likewise.
* tree-vect-patterns.c (widened_name_p): Remove obsolete asserts.
(See attached file: vect_cleanups.2.induc-phis.txt)
[-- Attachment #2: vect_cleanups.2.induc-phis.txt --]
[-- Type: text/plain, Size: 14040 bytes --]
*** tree-vect-analyze.c.1 2007-04-10 21:22:46.000000000 +0300
--- tree-vect-analyze.c 2007-04-10 21:47:18.000000000 +0300
*************** vect_determine_vectorization_factor (loo
*** 122,134 ****
gcc_assert (stmt_info);
! /* Two cases of "relevant" phis: those that define an
! induction that is used in the loop, and those that
! define a reduction. */
! if ((STMT_VINFO_RELEVANT (stmt_info) == vect_used_in_loop
! && STMT_VINFO_DEF_TYPE (stmt_info) == vect_induction_def)
! || (STMT_VINFO_RELEVANT (stmt_info) == vect_used_by_reduction
! && STMT_VINFO_DEF_TYPE (stmt_info) == vect_reduction_def))
{
gcc_assert (!STMT_VINFO_VECTYPE (stmt_info));
scalar_type = TREE_TYPE (PHI_RESULT (phi));
--- 122,128 ----
gcc_assert (stmt_info);
! if (STMT_VINFO_RELEVANT_P (stmt_info))
{
gcc_assert (!STMT_VINFO_VECTYPE (stmt_info));
scalar_type = TREE_TYPE (PHI_RESULT (phi));
*************** vect_analyze_operations (loop_vec_info l
*** 311,316 ****
--- 305,312 ----
for (phi = phi_nodes (bb); phi; phi = PHI_CHAIN (phi))
{
+ ok = true;
+
stmt_info = vinfo_for_stmt (phi);
if (vect_print_dump_info (REPORT_DETAILS))
{
*************** vect_analyze_operations (loop_vec_info l
*** 331,345 ****
if (STMT_VINFO_RELEVANT (stmt_info) == vect_used_in_loop
&& STMT_VINFO_DEF_TYPE (stmt_info) != vect_induction_def)
{
! /* Most likely a reduction-like computation that is used
! in the loop. */
if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
! fprintf (vect_dump, "not vectorized: unsupported pattern.");
! return false;
}
if (STMT_VINFO_RELEVANT_P (stmt_info))
! need_to_vectorize = true;
}
for (si = bsi_start (bb); !bsi_end_p (si); bsi_next (&si))
--- 327,355 ----
if (STMT_VINFO_RELEVANT (stmt_info) == vect_used_in_loop
&& STMT_VINFO_DEF_TYPE (stmt_info) != vect_induction_def)
{
! /* A scalar-dependence cycle that we don't support. */
if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
! fprintf (vect_dump, "not vectorized: scalar dependence cycle.");
! return false;
}
if (STMT_VINFO_RELEVANT_P (stmt_info))
! {
! need_to_vectorize = true;
! if (STMT_VINFO_DEF_TYPE (stmt_info) == vect_induction_def)
! ok = vectorizable_induction (phi, NULL, NULL);
! }
!
! if (!ok)
! {
! if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
! {
! fprintf (vect_dump,
! "not vectorized: relevant phi not supported: ");
! print_generic_expr (vect_dump, phi, TDF_SLIM);
! }
! return false;
! }
}
for (si = bsi_start (bb); !bsi_end_p (si); bsi_next (&si))
*************** vect_mark_relevant (VEC(tree,heap) **wor
*** 2106,2116 ****
if (relevant > STMT_VINFO_RELEVANT (stmt_info))
STMT_VINFO_RELEVANT (stmt_info) = relevant;
- if (TREE_CODE (stmt) == PHI_NODE)
- /* Don't put phi-nodes in the worklist. Phis that are marked relevant
- or live will fail vectorization later on. */
- return;
-
if (STMT_VINFO_RELEVANT (stmt_info) == save_relevant
&& STMT_VINFO_LIVE_P (stmt_info) == save_live_p)
{
--- 2116,2121 ----
*************** vect_mark_stmts_to_be_vectorized (loop_v
*** 2228,2254 ****
worklist = VEC_alloc (tree, heap, 64);
/* 1. Init worklist. */
-
- bb = loop->header;
- for (phi = phi_nodes (bb); phi; phi = PHI_CHAIN (phi))
- {
- if (vect_print_dump_info (REPORT_DETAILS))
- {
- fprintf (vect_dump, "init: phi relevant? ");
- print_generic_expr (vect_dump, phi, TDF_SLIM);
- }
-
- if (vect_stmt_relevant_p (phi, loop_vinfo, &relevant, &live_p))
- vect_mark_relevant (&worklist, phi, relevant, live_p);
- }
-
for (i = 0; i < nbbs; i++)
{
bb = bbs[i];
for (si = bsi_start (bb); !bsi_end_p (si); bsi_next (&si))
{
stmt = bsi_stmt (si);
-
if (vect_print_dump_info (REPORT_DETAILS))
{
fprintf (vect_dump, "init: stmt relevant? ");
--- 2233,2255 ----
worklist = VEC_alloc (tree, heap, 64);
/* 1. Init worklist. */
for (i = 0; i < nbbs; i++)
{
bb = bbs[i];
+ for (phi = phi_nodes (bb); phi; phi = PHI_CHAIN (phi))
+ {
+ if (vect_print_dump_info (REPORT_DETAILS))
+ {
+ fprintf (vect_dump, "init: phi relevant? ");
+ print_generic_expr (vect_dump, phi, TDF_SLIM);
+ }
+
+ if (vect_stmt_relevant_p (phi, loop_vinfo, &relevant, &live_p))
+ vect_mark_relevant (&worklist, phi, relevant, live_p);
+ }
for (si = bsi_start (bb); !bsi_end_p (si); bsi_next (&si))
{
stmt = bsi_stmt (si);
if (vect_print_dump_info (REPORT_DETAILS))
{
fprintf (vect_dump, "init: stmt relevant? ");
*************** vect_mark_stmts_to_be_vectorized (loop_v
*** 2279,2286 ****
relevance properties of STMT.
*/
- gcc_assert (TREE_CODE (stmt) != PHI_NODE);
-
ann = stmt_ann (stmt);
stmt_vinfo = vinfo_for_stmt (stmt);
--- 2280,2285 ----
*************** vect_mark_stmts_to_be_vectorized (loop_v
*** 2318,2324 ****
/* case 2.2: */
if (STMT_VINFO_DEF_TYPE (stmt_vinfo) == vect_reduction_def)
{
- gcc_assert (relevant == vect_unused_in_loop && live_p);
relevant = vect_used_by_reduction;
live_p = false;
}
--- 2317,2322 ----
*** tree-vect-transform.c.1 2007-04-10 21:22:55.000000000 +0300
--- tree-vect-transform.c 2007-04-10 21:35:51.000000000 +0300
*************** vect_init_vector (tree stmt, tree vector
*** 514,520 ****
/* Function get_initial_def_for_induction
Input:
- STMT - a stmt that performs an induction operation in the loop.
IV_PHI - the initial value of the induction variable
Output:
--- 514,519 ----
*************** vect_init_vector (tree stmt, tree vector
*** 524,532 ****
[X, X + S, X + 2*S, X + 3*S]. */
static tree
! get_initial_def_for_induction (tree stmt, tree iv_phi)
{
! stmt_vec_info stmt_vinfo = vinfo_for_stmt (stmt);
loop_vec_info loop_vinfo = STMT_VINFO_LOOP_VINFO (stmt_vinfo);
struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
tree scalar_type = TREE_TYPE (iv_phi);
--- 523,531 ----
[X, X + S, X + 2*S, X + 3*S]. */
static tree
! get_initial_def_for_induction (tree iv_phi)
{
! stmt_vec_info stmt_vinfo = vinfo_for_stmt (iv_phi);
loop_vec_info loop_vinfo = STMT_VINFO_LOOP_VINFO (stmt_vinfo);
struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
tree scalar_type = TREE_TYPE (iv_phi);
*************** get_initial_def_for_induction (tree stmt
*** 549,575 ****
tree expr;
stmt_vec_info phi_info = vinfo_for_stmt (iv_phi);
tree stmts;
gcc_assert (phi_info);
! if (STMT_VINFO_VEC_STMT (phi_info))
{
! induction_phi = STMT_VINFO_VEC_STMT (phi_info);
! gcc_assert (TREE_CODE (induction_phi) == PHI_NODE);
!
! if (vect_print_dump_info (REPORT_DETAILS))
! {
! fprintf (vect_dump, "induction already vectorized:");
! print_generic_expr (vect_dump, iv_phi, TDF_SLIM);
! fprintf (vect_dump, "\n");
! print_generic_expr (vect_dump, induction_phi, TDF_SLIM);
! }
!
! return PHI_RESULT (induction_phi);
}
- gcc_assert (ncopies >= 1);
-
access_fn = analyze_scalar_evolution (loop, PHI_RESULT (iv_phi));
gcc_assert (access_fn);
ok = vect_is_simple_iv_evolution (loop->num, access_fn, &init_expr, &step_expr);
--- 548,568 ----
tree expr;
stmt_vec_info phi_info = vinfo_for_stmt (iv_phi);
tree stmts;
+ tree stmt = NULL_TREE;
+ block_stmt_iterator si;
+ basic_block bb = bb_for_stmt (iv_phi);
gcc_assert (phi_info);
+ gcc_assert (ncopies >= 1);
! /* Find the first insertion point in the BB. */
! for (si = bsi_start (bb); !bsi_end_p (si); bsi_next (&si))
{
! stmt = bsi_stmt (si);
! if (TREE_CODE (stmt) != LABEL_EXPR)
! break;
}
access_fn = analyze_scalar_evolution (loop, PHI_RESULT (iv_phi));
gcc_assert (access_fn);
ok = vect_is_simple_iv_evolution (loop->num, access_fn, &init_expr, &step_expr);
*************** vect_get_vec_def_for_operand (tree op, t
*** 833,839 ****
gcc_assert (TREE_CODE (def_stmt) == PHI_NODE);
/* Get the def before the loop */
! return get_initial_def_for_induction (stmt, def_stmt);
}
default:
--- 826,832 ----
gcc_assert (TREE_CODE (def_stmt) == PHI_NODE);
/* Get the def before the loop */
! return get_initial_def_for_induction (def_stmt);
}
default:
*************** vect_min_worthwhile_factor (enum tree_co
*** 2233,2238 ****
--- 2226,2284 ----
}
+ /* Function vectorizable_induction
+
+ Check if PHI performs an induction computation that can be vectorized.
+ If VEC_STMT is also passed, vectorize the induction PHI: create a vectorized
+ phi to replace it, put it in VEC_STMT, and add it to the same basic block.
+ Return FALSE if not a vectorizable STMT, TRUE otherwise. */
+
+ bool
+ vectorizable_induction (tree phi, block_stmt_iterator *bsi ATTRIBUTE_UNUSED,
+ tree *vec_stmt)
+ {
+ stmt_vec_info stmt_info = vinfo_for_stmt (phi);
+ tree vectype = STMT_VINFO_VECTYPE (stmt_info);
+ loop_vec_info loop_vinfo = STMT_VINFO_LOOP_VINFO (stmt_info);
+ int nunits = TYPE_VECTOR_SUBPARTS (vectype);
+ int ncopies = LOOP_VINFO_VECT_FACTOR (loop_vinfo) / nunits;
+ tree vec_def;
+
+ gcc_assert (ncopies >= 1);
+
+ if (!STMT_VINFO_RELEVANT_P (stmt_info))
+ return false;
+
+ gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_induction_def);
+
+ if (STMT_VINFO_LIVE_P (stmt_info))
+ {
+ /* FORNOW: not yet supported. */
+ if (vect_print_dump_info (REPORT_DETAILS))
+ fprintf (vect_dump, "value used after loop.");
+ return false;
+ }
+
+ if (TREE_CODE (phi) != PHI_NODE)
+ return false;
+
+ if (!vec_stmt) /* transformation not required. */
+ {
+ STMT_VINFO_TYPE (stmt_info) = induc_vec_info_type;
+ return true;
+ }
+
+ /** Transform. **/
+
+ if (vect_print_dump_info (REPORT_DETAILS))
+ fprintf (vect_dump, "transform induction phi.");
+
+ vec_def = get_initial_def_for_induction (phi);
+ *vec_stmt = SSA_NAME_DEF_STMT (vec_def);
+ return true;
+ }
+
+
/* Function vectorizable_operation.
Check if STMT performs a binary or unary operation that can be vectorized.
*************** vect_transform_stmt (tree stmt, block_st
*** 4285,4290 ****
--- 4331,4341 ----
gcc_assert (done);
break;
+ case induc_vec_info_type:
+ done = vectorizable_induction (stmt, bsi, &vec_stmt);
+ gcc_assert (done);
+ break;
+
case op_vec_info_type:
done = vectorizable_operation (stmt, bsi, &vec_stmt);
gcc_assert (done);
*************** vect_transform_loop (loop_vec_info loop_
*** 5192,5202 ****
for (i = 0; i < nbbs; i++)
{
basic_block bb = bbs[i];
for (si = bsi_start (bb); !bsi_end_p (si);)
{
tree stmt = bsi_stmt (si);
- stmt_vec_info stmt_info;
bool is_store;
if (vect_print_dump_info (REPORT_DETAILS))
--- 5243,5281 ----
for (i = 0; i < nbbs; i++)
{
basic_block bb = bbs[i];
+ stmt_vec_info stmt_info;
+ tree phi;
+
+ for (phi = phi_nodes (bb); phi; phi = PHI_CHAIN (phi))
+ {
+ if (vect_print_dump_info (REPORT_DETAILS))
+ {
+ fprintf (vect_dump, "------>vectorizing phi: ");
+ print_generic_expr (vect_dump, phi, TDF_SLIM);
+ }
+ stmt_info = vinfo_for_stmt (phi);
+ if (!stmt_info)
+ continue;
+ if (!STMT_VINFO_RELEVANT_P (stmt_info)
+ && !STMT_VINFO_LIVE_P (stmt_info))
+ continue;
+
+ if ((TYPE_VECTOR_SUBPARTS (STMT_VINFO_VECTYPE (stmt_info))
+ != (unsigned HOST_WIDE_INT) vectorization_factor)
+ && vect_print_dump_info (REPORT_DETAILS))
+ fprintf (vect_dump, "multiple-types.");
+
+ if (STMT_VINFO_DEF_TYPE (stmt_info) == vect_induction_def)
+ {
+ if (vect_print_dump_info (REPORT_DETAILS))
+ fprintf (vect_dump, "transform phi.");
+ vect_transform_stmt (phi, NULL, NULL);
+ }
+ }
for (si = bsi_start (bb); !bsi_end_p (si);)
{
tree stmt = bsi_stmt (si);
bool is_store;
if (vect_print_dump_info (REPORT_DETAILS))
Index: tree-vectorizer.h
===================================================================
*** tree-vectorizer.h (revision 123694)
--- tree-vectorizer.h (working copy)
*************** enum stmt_vec_info_type {
*** 167,172 ****
--- 167,173 ----
assignment_vec_info_type,
condition_vec_info_type,
reduc_vec_info_type,
+ induc_vec_info_type,
type_promotion_vec_info_type,
type_demotion_vec_info_type,
type_conversion_vec_info_type
*************** extern bool vectorizable_call (tree, blo
*** 428,433 ****
--- 429,435 ----
extern bool vectorizable_condition (tree, block_stmt_iterator *, tree *);
extern bool vectorizable_live_operation (tree, block_stmt_iterator *, tree *);
extern bool vectorizable_reduction (tree, block_stmt_iterator *, tree *);
+ extern bool vectorizable_induction (tree, block_stmt_iterator *, tree *);
/* Driver for transformation stage. */
extern void vect_transform_loop (loop_vec_info);
Index: tree-vect-patterns.c
===================================================================
*** tree-vect-patterns.c (revision 123694)
--- tree-vect-patterns.c (working copy)
*************** widened_name_p (tree name, tree use_stmt
*** 109,118 ****
if (!vect_is_simple_use (oprnd0, loop_vinfo, &dummy, &dummy, &dt))
return false;
- if (dt != vect_invariant_def && dt != vect_constant_def
- && dt != vect_loop_def)
- return false;
-
return true;
}
--- 109,114 ----
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 5/5
2007-04-02 9:48 ` Zdenek Dvorak
` (5 preceding siblings ...)
2007-04-11 11:51 ` [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 3/5 Dorit Nuzman
@ 2007-04-11 11:51 ` Dorit Nuzman
2007-04-12 12:22 ` Zdenek Dvorak
6 siblings, 1 reply; 18+ messages in thread
From: Dorit Nuzman @ 2007-04-11 11:51 UTC (permalink / raw)
To: Zdenek Dvorak; +Cc: gcc-patches
[-- Attachment #1: Type: text/plain, Size: 2489 bytes --]
Zdenek Dvorak <rakdver@atrey.karlin.mff.cuni.cz> wrote on 02/04/2007
12:48:12:
> Hello,
>
> I will look at the patch. However, would it be possible to split it
> into smaller parts? The changes seem to be quite independent,
>
> Zdenek
>
> > This is exactly the same patch as was committed to autovect last week:
> > http://gcc.gnu.org/ml/gcc-patches/2007-03/msg01904.html
> > (but relative to mainline, and after testig on mainline).
> >
> > These are a few cleanups (really technical things, no new
functionality),
> > some of which will be also useful for outer-loop vectorization:
> >
> > o Remove the field exit_cond from loop_vec_info, which used to hold the
> > loop-exit stmt. Instead, a stmt that is identified as a loop-exit
control
> > stmt is marked as a 'loop_exit_ctrl_vec_info_type' (new enum value for
> > stmt_vec_info_type). This will be useful when looking at outer-loops
> > because we have more than one stmt like that in the loop (there's also
one
> > in the inner-loop).
...
> > o A bunch of other cleanups (that are not necessarily inspired by
> > outer-loop vectorization): ... Add a flag to allow disabling
> > tree-reassociation, to help test reductions that are otherwise
transformed
> > by reassociation to something we currently don't recognize; Fix
> > indentation; A few other small miscellaneous stuff.
> >
Tested on the vectorizer testcases
Bootstrapped and passed full testing on powerpc-linux together with the
other parts of this patch.
thanks,
dorit
ChangeLog:
* tree-vectorizer.c (destroy_loop_vec_info): Set loop->aux to NULL.
* tree-vect-analyze.c (vect_analyze_loop_form): Set loop->aux.
* tree-vectorizer.h (NITERS_KNOWN_P): New.
* tree-vect-analyze.c (vect_analyze_loop_form): Call NITERS_KNOWN_P
instead of LOOP_VINFO_INT_NITERS to avoid having to geneate
loop_info.
* tree-vect-analyze.c (vect_determine_vectorization_factor): Add
dump print.
(vect_analyze_operations): Fix indenetation. Fix a comment. Fix a
print message.
(vect_analyze_scalar_cycles): Fix indentation.
(vect_enhance_data_refs_alignment): Fix check in case of peeling.
(vect_mark_relevant): Include phis in relevance analysis.
* tree-vect-transform.c (vect_transform_loop): Add an assert.
* tree-ssa-reassoc.c (gate_reassoc): New function.
* common.opt (tree-reassoc): New flag.
(pass_reassoc): Add gate.
(See attached file: vect_cleanups.5.misc.txt)
[-- Attachment #2: vect_cleanups.5.misc.txt --]
[-- Type: text/plain, Size: 6721 bytes --]
*** tree-vect-transform.c.4 2007-04-10 22:27:49.000000000 +0300
--- tree-vect-transform.c.5 2007-04-10 22:35:42.000000000 +0300
*************** vect_transform_loop (loop_vec_info loop_
*** 5262,5267 ****
--- 5262,5268 ----
continue;
}
+ gcc_assert (STMT_VINFO_VECTYPE (stmt_info));
if ((TYPE_VECTOR_SUBPARTS (STMT_VINFO_VECTYPE (stmt_info))
!= (unsigned HOST_WIDE_INT) vectorization_factor)
&& vect_print_dump_info (REPORT_DETAILS))
*** tree-vectorizer.c.3 2007-04-10 22:18:57.000000000 +0300
--- tree-vectorizer.c.5 2007-04-10 22:36:00.000000000 +0300
*************** destroy_loop_vec_info (loop_vec_info loo
*** 1511,1516 ****
--- 1511,1517 ----
VEC_free (tree, heap, LOOP_VINFO_MAY_MISALIGN_STMTS (loop_vinfo));
free (loop_vinfo);
+ loop->aux = NULL;
}
*** tree-vectorizer.h.2 2007-04-10 21:55:20.000000000 +0300
--- tree-vectorizer.h.5 2007-04-10 22:36:08.000000000 +0300
*************** typedef struct _loop_vec_info {
*** 151,159 ****
#define LOOP_VINFO_MAY_MISALIGN_STMTS(L) (L)->may_misalign_stmts
#define LOOP_VINFO_LOC(L) (L)->loop_line_number
#define LOOP_VINFO_NITERS_KNOWN_P(L) \
! (host_integerp ((L)->num_iters,0) \
! && TREE_INT_CST_LOW ((L)->num_iters) > 0)
/*-----------------------------------------------------------------*/
/* Info on vectorized defs. */
--- 151,162 ----
#define LOOP_VINFO_MAY_MISALIGN_STMTS(L) (L)->may_misalign_stmts
#define LOOP_VINFO_LOC(L) (L)->loop_line_number
+ #define NITERS_KNOWN_P(n) \
+ (host_integerp ((n),0) \
+ && TREE_INT_CST_LOW ((n)) > 0)
+
#define LOOP_VINFO_NITERS_KNOWN_P(L) \
! NITERS_KNOWN_P((L)->num_iters)
/*-----------------------------------------------------------------*/
/* Info on vectorized defs. */
Index: common.opt
===================================================================
*** common.opt (revision 123694)
--- common.opt (working copy)
*************** funroll-all-loops
*** 1070,1075 ****
--- 1070,1079 ----
Common Report Var(flag_unroll_all_loops) Optimization
Perform loop unrolling for all loops
+ ftree-reassoc
+ Common Report Var(flag_tree_reassoc) Init(1) Optimization
+ Perform reassociation on trees
+
; Nonzero means that loop optimizer may assume that the induction variables
; that control loops do not overflow and that the loops with nontrivial
; exit condition are not infinite
Index: tree-ssa-reassoc.c
===================================================================
*** tree-ssa-reassoc.c (revision 123694)
--- tree-ssa-reassoc.c (working copy)
*************** execute_reassoc (void)
*** 1476,1485 ****
return 0;
}
struct tree_opt_pass pass_reassoc =
{
"reassoc", /* name */
! NULL, /* gate */
execute_reassoc, /* execute */
NULL, /* sub */
NULL, /* next */
--- 1476,1491 ----
return 0;
}
+ static bool
+ gate_reassoc (void)
+ {
+ return flag_tree_reassoc != 0;
+ }
+
struct tree_opt_pass pass_reassoc =
{
"reassoc", /* name */
! gate_reassoc, /* gate */
execute_reassoc, /* execute */
NULL, /* sub */
NULL, /* next */
*** tree-vect-analyze.c.3 2007-04-10 23:16:30.000000000 +0300
--- tree-vect-analyze.c.5 2007-04-10 23:15:40.000000000 +0300
*************** vect_determine_vectorization_factor (loo
*** 262,268 ****
}
/* TODO: Analyze cost. Decide if worth while to vectorize. */
!
if (vectorization_factor <= 1)
{
if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
--- 262,269 ----
}
/* TODO: Analyze cost. Decide if worth while to vectorize. */
! if (vect_print_dump_info (REPORT_DETAILS))
! fprintf (vect_dump, "vectorization factor = %d", vectorization_factor);
if (vectorization_factor <= 1)
{
if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
*************** static bool
*** 1434,1439 ****
--- 1435,1441 ----
vect_enhance_data_refs_alignment (loop_vec_info loop_vinfo)
{
VEC (data_reference_p, heap) *datarefs = LOOP_VINFO_DATAREFS (loop_vinfo);
+ struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
enum dr_alignment_support supportable_dr_alignment;
struct data_reference *dr0 = NULL;
struct data_reference *dr;
*************** vect_enhance_data_refs_alignment (loop_v
*** 1531,1537 ****
/* Often peeling for alignment will require peeling for loop-bound, which in
turn requires that we know how to adjust the loop ivs after the loop. */
! if (!vect_can_advance_ivs_p (loop_vinfo))
do_peeling = false;
if (do_peeling)
--- 1533,1540 ----
/* Often peeling for alignment will require peeling for loop-bound, which in
turn requires that we know how to adjust the loop ivs after the loop. */
! if (!vect_can_advance_ivs_p (loop_vinfo)
! || !slpeel_can_duplicate_loop_p (loop, single_exit (loop)))
do_peeling = false;
if (do_peeling)
*************** vect_analyze_loop_form (struct loop *loo
*** 2641,2650 ****
return false;
}
! loop_vinfo = new_loop_vec_info (loop);
! LOOP_VINFO_NITERS (loop_vinfo) = number_of_iterations;
!
! if (!LOOP_VINFO_NITERS_KNOWN_P (loop_vinfo))
{
if (vect_print_dump_info (REPORT_DETAILS))
{
--- 2644,2650 ----
return false;
}
! if (!NITERS_KNOWN_P (number_of_iterations))
{
if (vect_print_dump_info (REPORT_DETAILS))
{
*************** vect_analyze_loop_form (struct loop *loo
*** 2652,2667 ****
print_generic_expr (vect_dump, number_of_iterations, TDF_DETAILS);
}
}
! else
! if (LOOP_VINFO_INT_NITERS (loop_vinfo) == 0)
{
if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
fprintf (vect_dump, "not vectorized: number of iterations = 0.");
return NULL;
}
LOOP_VINFO_EXIT_COND (loop_vinfo) = loop_cond;
return loop_vinfo;
}
--- 2652,2670 ----
print_generic_expr (vect_dump, number_of_iterations, TDF_DETAILS);
}
}
! else if (TREE_INT_CST_LOW (number_of_iterations) == 0)
{
if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
fprintf (vect_dump, "not vectorized: number of iterations = 0.");
return NULL;
}
+ loop_vinfo = new_loop_vec_info (loop);
+ LOOP_VINFO_NITERS (loop_vinfo) = number_of_iterations;
LOOP_VINFO_EXIT_COND (loop_vinfo) = loop_cond;
+ gcc_assert (!loop->aux);
+ loop->aux = loop_vinfo;
return loop_vinfo;
}
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [patch] vectorizer cleanups (and prep for outer-loop vectorization)
2007-04-02 9:48 ` Zdenek Dvorak
2007-04-02 9:55 ` Zdenek Dvorak
2007-04-11 11:51 ` [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 2/5 Dorit Nuzman
@ 2007-04-11 11:51 ` Dorit Nuzman
2007-04-11 11:51 ` [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 1/5 Dorit Nuzman
` (3 subsequent siblings)
6 siblings, 0 replies; 18+ messages in thread
From: Dorit Nuzman @ 2007-04-11 11:51 UTC (permalink / raw)
To: Zdenek Dvorak; +Cc: gcc-patches
Zdenek Dvorak <rakdver@atrey.karlin.mff.cuni.cz> wrote on 02/04/2007
12:48:12:
> Hello,
>
> I will look at the patch. However, would it be possible to split it
> into smaller parts? The changes seem to be quite independent,
>
coming up...
thanks,
dorit
> Zdenek
>
> > This is exactly the same patch as was committed to autovect last week:
> > http://gcc.gnu.org/ml/gcc-patches/2007-03/msg01904.html
> > (but relative to mainline, and after testig on mainline).
> >
> > These are a few cleanups (really technical things, no new
functionality),
> > some of which will be also useful for outer-loop vectorization:
> >
> > o Remove the field exit_cond from loop_vec_info, which used to hold the
> > loop-exit stmt. Instead, a stmt that is identified as a loop-exit
control
> > stmt is marked as a 'loop_exit_ctrl_vec_info_type' (new enum value for
> > stmt_vec_info_type). This will be useful when looking at outer-loops
> > because we have more than one stmt like that in the loop (there's also
one
> > in the inner-loop).
> >
> > o Vectorization of induction is currently done upon encountering a use
of
> > the induction variable. Instead, we now vectorize the induction
earlier,
> > when we encounter the respective induction phi-node. This simplifies
things
> > a bit cause it's more in-line with how we vectorize other things (when
we
> > encounter a use of the induction we just get the vector def from the
> > STMT_VINFO_VEC_STMT field of the defining phi-node), and also we don't
need
> > to worry about vectorizing the same induction multiple times. When
looking
> > at outer-loops this will be even more useful cause when we vectorize a
> > phi-node we know exactly relative to which loop-nest is the induction,
and
> > don't need any extra logic to check for that.
> >
> > o Reorganize the calls to the vectorizable_* functions: In the drivers
that
> > call these functions (vect_analyze_operations and vect_transform_stmt),
we
> > currently call most of these functions only for RELEVANT_P stmts, and
then
> > call vectorizable_reduction only for LIVE_P stmts. Now, instead, each
of
> > the vectorizable_* functions checks that STMT_VINFO_RELEVANT/LIVE_P are
as
> > expected, and in the two driver functions we don't check for
> > STMT_VINFO_RELEVANT_P anymore, but apply the vectorizable_* functions
to
> > all stmts. When looking at outer-loops this is useful because a
reduction
> > (in the inner-loop) may now be marked RELEVANT (if it is used in the
> > outer-loop), and not only LIVE as it is currently.
> >
> > o A bunch of other cleanups (that are not necessarily inspired by
> > outer-loop vectorization): Factor out code from
> > vect_mark_stmts_to_be_vectorized to process_use; Remove the '#ifdef
> > ADJUST_IN_EPILOG' and the code path that was disabled by it (which
removes
> > unused code and simplifies the function); Add a flag to allow disabling
> > tree-reassociation, to help test reductions that are otherwise
transformed
> > by reassociation to something we currently don't recognize; Fix
> > indentation; A few ther small miscellaneous stuff.
> >
> > Bootstrapped and tested (full regression testing) on powerpc64-linux
and
> > i386-linux.
> > Also bootstrapped with vectorization enabled on powerpc64-linux.
> >
> > ok for mainline?
> >
> > :ADDPATCH vectorizer,non-algorithmic:
> >
> > ChangeLog:
> >
> > * tree-vectorizer.h (_loop_vec_info): Remove unused field
> > exit_cond.
> > (stmt_vec_info_type): Add enum value
loop_exit_ctrl_vec_info_type.
> > * tree-vectorizer.c (new_loop_vec_info): Remove setting of
> > LOOP_VINFO_EXIT_COND.
> > * tree-vect-analyze.c (vect_stmt_relevant_p): Replace use of
> > LOOP_VINFO_EXIT_COND with use of loop_exit_ctrl_vec_info_type.
> > (vect_analyze_loop_form): Likewise.
> >
> > * tree-vectorizer.h (stmt_vec_info_type): Add enum value
> > induc_vec_info_type.
> > (vectorizable_induction): New function declaration.
> > * tree-vect-transform.c (get_initial_def_for_induction): No
need to
> > check if already vectorized. Find first place in BB where nre
> > stmts
> > can be inserted.
> > (vectorizable_induction): New function.
> > (vect_transform_stmt): Add case for induc_vec_info_type to call
> > vectorizable_induction.
> > (vect_transform_loop): Consider phis for vectorization.
> > * tree-vect-analyze.c (analyze_operations): Call
> > vectorizable_induction when analyzing phis and when analyzing
> > stmts.
> >
> > * tree-vectorizer.c (vect_is_simple_use): Remove redundant
check.
> > (destroy_loop_vec_info): Set loop->aux to NULL.
> >
> > * tree-vect-analyze.c (vect_determine_vectorization_factor):
> > Simplify
> > condition. Add dump print.
> > (vect_analyze_operations): Fix indenetation. Fix a comment.
Fix a
> > print message.
> > (vect_analyze_scalar_cycles): Fix indentation.
> > (vect_enhance_data_refs_alignment): Fix check in case of
peeling.
> > (vect_mark_relevant): Include phis in relevance analysis.
> > (process_use): New function.
> > (vect_mark_stmts_to_be_vectorized): Factor out code to
process_use.
> > Check phis in all bbs.
> >
> > * tree-vect-patterns.c (widened_name_p): Remove redundant
check.
> >
> > * tree-vect-analyze.c (vect_analyze_operations): Reorganize
calls
> > to
> > vectorizable_* functions.
> > * tree-vect-transform.c (vectorizable_call): Add check for
> > STMT_VINFO_RELEVANT_P, STMT_VINFO_DEF_TYPE and
STMT_VINFO_LIVE_P.
> > (vectorizable_store): likewise.
> > (vectorizable_conversion): Add check for STMT_VINFO_DEF_TYPE.
> > Add comments.
> > (vectorizable_operation, vectorizable_type_demotion): Likewise.
> > (vectorizable_type_promotion, vectorizable_load): Likewise.
> > (vectorizable_live_operation, vectorizable_condition):
Likewise.
> > (vectorizable_assignment): Add check for STMT_VINFO_DEF_TYPE
and
> > STMT_VINFO_LIVE_P.
> > (vect_transform_stmt): Reorganize calls to vectorizable_*
> > functions.
> > (vect_analyze_loop_form): Use the new NITERS_KNOWN_P macro so
that
> > this check could be done without creating a loop_vinfo first.
> > Record
> > loop_vinfo in loop->aux.
> >
> > * tree-vect-transform.c (get_initial_def_for_reduction): Clean
away
> > the unused code for reduction without adjust-in-epilog to
simplify
> > the
> > function.
> >
> > * tree-vect-transform.c (vect_transform_loop): Add an assert.
> >
> > * tree-ssa-reassoc.c (gate_reassoc): New function.
> > * common.opt (tree-reassoc): New flag.
> > (pass_reassoc): Add gate.
> >
> >
> > (See attached file: mainline-vect-cleanups.mar27.txt)
> > Index: ChangeLog
> > ===================================================================
> > *** ChangeLog (revision 123103)
> > --- ChangeLog (working copy)
> > ***************
> > *** 1,3 ****
> > --- 1,70 ----
> > + 2007-03-20 Dorit Nuzman <dorit@il.ibm.com>
> > +
> > + * tree-vectorizer.h (_loop_vec_info): Remove unused field
exit_cond.
> > + (stmt_vec_info_type): Add enum value loop_exit_ctrl_vec_info_type.
> > + * tree-vectorizer.c (new_loop_vec_info): Remove setting of
> > + LOOP_VINFO_EXIT_COND.
> > + * tree-vect-analyze.c (vect_stmt_relevant_p): Replace use of
> > + LOOP_VINFO_EXIT_COND with use of loop_exit_ctrl_vec_info_type.
> > + (vect_analyze_loop_form): Likewise.
> > +
> > + * tree-vectorizer.h (stmt_vec_info_type): Add enum value
> > + induc_vec_info_type.
> > + (vectorizable_induction): New function declaration.
> > + * tree-vect-transform.c (get_initial_def_for_induction): No need
to
> > + check if already vectorized. Find first place in BB where nre
stmts
> > + can be inserted.
> > + (vectorizable_induction): New function.
> > + (vect_transform_stmt): Add case for induc_vec_info_type to call
> > + vectorizable_induction.
> > + (vect_transform_loop): Consider phis for vectorization.
> > + * tree-vect-analyze.c (analyze_operations): Call
> > + vectorizable_induction when analyzing phis and when analyzing
> > + stmts.
> > +
> > + * tree-vectorizer.c (vect_is_simple_use): Remove redundant
check.
> > + (destroy_loop_vec_info): Set loop->aux to NULL.
> > +
> > + * tree-vect-analyze.c (vect_determine_vectorization_factor):
Simplify
> > + condition. Add dump print.
> > + (vect_analyze_operations): Fix indenetation. Fix a comment. Fix
a
> > + print message.
> > + (vect_analyze_scalar_cycles): Fix indentation.
> > + (vect_enhance_data_refs_alignment): Fix check in case of peeling.
> > + (vect_mark_relevant): Include phis in relevance analysis.
> > + (process_use): New function.
> > + (vect_mark_stmts_to_be_vectorized): Factor out code to
process_use.
> > + Check phis in all bbs.
> > +
> > + * tree-vect-patterns.c (widened_name_p): Remove redundant
check.
> > +
> > + * tree-vect-analyze.c (vect_analyze_operations): Reorganize calls
to
> > + vectorizable_* functions.
> > + * tree-vect-transform.c (vectorizable_call): Add check for
> > + STMT_VINFO_RELEVANT_P, STMT_VINFO_DEF_TYPE and STMT_VINFO_LIVE_P.
> > + (vectorizable_store): likewise.
> > + (vectorizable_conversion): Add check for STMT_VINFO_DEF_TYPE.
> > + Add comments.
> > + (vectorizable_operation, vectorizable_type_demotion): Likewise.
> > + (vectorizable_type_promotion, vectorizable_load): Likewise.
> > + (vectorizable_live_operation, vectorizable_condition): Likewise.
> > + (vectorizable_assignment): Add check for STMT_VINFO_DEF_TYPE and
> > + STMT_VINFO_LIVE_P.
> > + (vect_transform_stmt): Reorganize calls to vectorizable_*
functions.
> > + (vect_analyze_loop_form): Use the new NITERS_KNOWN_P macro so that
> > + this check could be done without creating a loop_vinfo first.
Record
> > + loop_vinfo in loop->aux.
> > +
> > + * tree-vect-transform.c (get_initial_def_for_reduction): Clean
away
> > + the unused code for reduction without adjust-in-epilog to simplify
the
> > + function.
> > +
> > + * tree-vect-transform.c (vect_transform_loop): Add an assert.
> > +
> > + * tree-ssa-reassoc.c (gate_reassoc): New function.
> > + * common.opt (tree-reassoc): New flag.
> > + (pass_reassoc): Add gate.
> > +
> > 2007-03-20 Mark Mitchell <mark@codesourcery.com>
> >
> > * config/arm/elf.h (TARGET_ASM_DESTRUCTOR): Define.
> > Index: testsuite/gcc.dg/vect/vect-1.c
> > ===================================================================
> > *** testsuite/gcc.dg/vect/vect-1.c (revision 123103)
> > --- testsuite/gcc.dg/vect/vect-1.c (working copy)
> > *************** foo (int n)
> > *** 38,44 ****
> > inner-loop: cross iteration cycle; multi-dimensional arrays).
*/
> > diff = 0;
> > for (i = 0; i < N; i++) {
> > ! for (i = 0; i < N; i++) {
> > diff += (image[i][j] - block[i][j]);
> > }
> > }
> > --- 38,44 ----
> > inner-loop: cross iteration cycle; multi-dimensional arrays).
*/
> > diff = 0;
> > for (i = 0; i < N; i++) {
> > ! for (j = 0; j < N; j++) {
> > diff += (image[i][j] - block[i][j]);
> > }
> > }
> > Index: testsuite/gcc.dg/vect/vect.exp
> > ===================================================================
> > *** testsuite/gcc.dg/vect/vect.exp (revision 123103)
> > --- testsuite/gcc.dg/vect/vect.exp (working copy)
> > *************** dg-runtest [lsort [glob -nocomplain $src
> > *** 169,174 ****
> > --- 169,175 ----
> > "" $DEFAULT_VECTCFLAGS
> >
> > # With -Os
> > + set DEFAULT_VECTCFLAGS $SAVED_DEFAULT_VECTCFLAGS
> > lappend DEFAULT_VECTCFLAGS "-Os"
> > dg-runtest [lsort [glob -nocomplain
$srcdir/$subdir/Os-vect-*.\[cS\]]] \
> > "" $DEFAULT_VECTCFLAGS
> > Index: tree-vectorizer.c
> > ===================================================================
> > *** tree-vectorizer.c (revision 123103)
> > --- tree-vectorizer.c (working copy)
> > *************** new_loop_vec_info (struct loop *loop)
> > *** 1420,1426 ****
> >
> > LOOP_VINFO_LOOP (res) = loop;
> > LOOP_VINFO_BBS (res) = bbs;
> > - LOOP_VINFO_EXIT_COND (res) = NULL;
> > LOOP_VINFO_NITERS (res) = NULL;
> > LOOP_VINFO_VECTORIZABLE_P (res) = 0;
> > LOOP_PEELING_FOR_ALIGNMENT (res) = 0;
> > --- 1420,1425 ----
> > *************** destroy_loop_vec_info (loop_vec_info loo
> > *** 1511,1516 ****
> > --- 1510,1516 ----
> > VEC_free (tree, heap, LOOP_VINFO_MAY_MISALIGN_STMTS (loop_vinfo));
> >
> > free (loop_vinfo);
> > + loop->aux = NULL;
> > }
> >
> >
> > *************** vect_is_simple_use (tree operand, loop_v
> > *** 1714,1728 ****
> > return false;
> > }
> >
> > - /* stmts inside the loop that have been identified as performing
> > - a reduction operation cannot have uses in the loop. */
> > - if (*dt == vect_reduction_def && TREE_CODE (*def_stmt) != PHI_NODE)
> > - {
> > - if (vect_print_dump_info (REPORT_DETAILS))
> > - fprintf (vect_dump, "reduction used in loop.");
> > - return false;
> > - }
> > -
> > if (vect_print_dump_info (REPORT_DETAILS))
> > fprintf (vect_dump, "type of def: %d.",*dt);
> >
> > --- 1714,1719 ----
> > *************** vect_is_simple_use (tree operand, loop_v
> > *** 1730,1742 ****
> > {
> > case PHI_NODE:
> > *def = PHI_RESULT (*def_stmt);
> > - gcc_assert (*dt == vect_induction_def || *dt ==
vect_reduction_def
> > - || *dt == vect_invariant_def);
> > break;
> >
> > case GIMPLE_MODIFY_STMT:
> > *def = GIMPLE_STMT_OPERAND (*def_stmt, 0);
> > - gcc_assert (*dt == vect_loop_def || *dt == vect_invariant_def);
> > break;
> >
> > default:
> > --- 1721,1730 ----
> > Index: tree-vectorizer.h
> > ===================================================================
> > *** tree-vectorizer.h (revision 123103)
> > --- tree-vectorizer.h (working copy)
> > *************** typedef struct _loop_vec_info {
> > *** 93,101 ****
> > /* The loop basic blocks. */
> > basic_block *bbs;
> >
> > - /* The loop exit_condition. */
> > - tree exit_cond;
> > -
> > /* Number of iterations. */
> > tree num_iters;
> >
> > --- 93,98 ----
> > *************** typedef struct _loop_vec_info {
> > *** 138,144 ****
> > /* Access Functions. */
> > #define LOOP_VINFO_LOOP(L) (L)->loop
> > #define LOOP_VINFO_BBS(L) (L)->bbs
> > - #define LOOP_VINFO_EXIT_COND(L) (L)->exit_cond
> > #define LOOP_VINFO_NITERS(L) (L)->num_iters
> > #define LOOP_VINFO_VECTORIZABLE_P(L) (L)->vectorizable
> > #define LOOP_VINFO_VECT_FACTOR(L) (L)->vectorization_factor
> > --- 135,140 ----
> > *************** typedef struct _loop_vec_info {
> > *** 151,159 ****
> > #define LOOP_VINFO_MAY_MISALIGN_STMTS(L) (L)->may_misalign_stmts
> > #define LOOP_VINFO_LOC(L) (L)->loop_line_number
> >
> > #define LOOP_VINFO_NITERS_KNOWN_P(L) \
> > ! (host_integerp ((L)->num_iters,0) \
> > ! && TREE_INT_CST_LOW ((L)->num_iters) > 0)
> >
> > /*-----------------------------------------------------------------*/
> > /* Info on vectorized defs. */
> > --- 147,158 ----
> > #define LOOP_VINFO_MAY_MISALIGN_STMTS(L) (L)->may_misalign_stmts
> > #define LOOP_VINFO_LOC(L) (L)->loop_line_number
> >
> > + #define NITERS_KNOWN_P(n) \
> > + (host_integerp ((n),0) \
> > + && TREE_INT_CST_LOW ((n)) > 0)
> > +
> > #define LOOP_VINFO_NITERS_KNOWN_P(L) \
> > ! NITERS_KNOWN_P((L)->num_iters)
> >
> > /*-----------------------------------------------------------------*/
> > /* Info on vectorized defs. */
> > *************** enum stmt_vec_info_type {
> > *** 167,175 ****
> > assignment_vec_info_type,
> > condition_vec_info_type,
> > reduc_vec_info_type,
> > type_promotion_vec_info_type,
> > type_demotion_vec_info_type,
> > ! type_conversion_vec_info_type
> > };
> >
> > /* Indicates whether/how a variable is used in the loop. */
> > --- 166,176 ----
> > assignment_vec_info_type,
> > condition_vec_info_type,
> > reduc_vec_info_type,
> > + induc_vec_info_type,
> > type_promotion_vec_info_type,
> > type_demotion_vec_info_type,
> > ! type_conversion_vec_info_type,
> > ! loop_exit_ctrl_vec_info_type
> > };
> >
> > /* Indicates whether/how a variable is used in the loop. */
> > *************** extern bool vectorizable_call (tree, blo
> > *** 428,433 ****
> > --- 429,435 ----
> > extern bool vectorizable_condition (tree, block_stmt_iterator *, tree
*);
> > extern bool vectorizable_live_operation (tree,
> block_stmt_iterator *, tree *);
> > extern bool vectorizable_reduction (tree, block_stmt_iterator *, tree
*);
> > + extern bool vectorizable_induction (tree, block_stmt_iterator *, tree
*);
> > /* Driver for transformation stage. */
> > extern void vect_transform_loop (loop_vec_info);
> >
> > Index: tree-vect-analyze.c
> > ===================================================================
> > *** tree-vect-analyze.c (revision 123103)
> > --- tree-vect-analyze.c (working copy)
> > *************** vect_determine_vectorization_factor (loo
> > *** 122,134 ****
> >
> > gcc_assert (stmt_info);
> >
> > ! /* Two cases of "relevant" phis: those that define an
> > ! induction that is used in the loop, and those that
> > ! define a reduction. */
> > ! if ((STMT_VINFO_RELEVANT (stmt_info) == vect_used_in_loop
> > ! && STMT_VINFO_DEF_TYPE (stmt_info) == vect_induction_def)
> > ! || (STMT_VINFO_RELEVANT (stmt_info) ==
vect_used_by_reduction
> > ! && STMT_VINFO_DEF_TYPE (stmt_info) == vect_reduction_def))
> > {
> > gcc_assert (!STMT_VINFO_VECTYPE (stmt_info));
> > scalar_type = TREE_TYPE (PHI_RESULT (phi));
> > --- 122,128 ----
> >
> > gcc_assert (stmt_info);
> >
> > ! if (STMT_VINFO_RELEVANT_P (stmt_info))
> > {
> > gcc_assert (!STMT_VINFO_VECTYPE (stmt_info));
> > scalar_type = TREE_TYPE (PHI_RESULT (phi));
> > *************** vect_determine_vectorization_factor (loo
> > *** 268,274 ****
> > }
> >
> > /* TODO: Analyze cost. Decide if worth while to vectorize. */
> > !
> > if (vectorization_factor <= 1)
> > {
> > if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> > --- 262,269 ----
> > }
> >
> > /* TODO: Analyze cost. Decide if worth while to vectorize. */
> > ! if (vect_print_dump_info (REPORT_DETAILS))
> > ! fprintf (vect_dump, "vectorization factor = %d",
> vectorization_factor);
> > if (vectorization_factor <= 1)
> > {
> > if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> > *************** vect_analyze_operations (loop_vec_info l
> > *** 311,316 ****
> > --- 306,313 ----
> >
> > for (phi = phi_nodes (bb); phi; phi = PHI_CHAIN (phi))
> > {
> > + ok = true;
> > +
> > stmt_info = vinfo_for_stmt (phi);
> > if (vect_print_dump_info (REPORT_DETAILS))
> > {
> > *************** vect_analyze_operations (loop_vec_info l
> > *** 319,341 ****
> > }
> >
> > gcc_assert (stmt_info);
> > -
> > if (STMT_VINFO_LIVE_P (stmt_info))
> > {
> > /* FORNOW: not yet supported. */
> > if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> > fprintf (vect_dump, "not vectorized: value used after loop.");
> > ! return false;
> > }
> >
> > if (STMT_VINFO_RELEVANT (stmt_info) == vect_used_in_loop
> > && STMT_VINFO_DEF_TYPE (stmt_info) != vect_induction_def)
> > {
> > ! /* Most likely a reduction-like computation that is used
> > ! in the loop. */
> > if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> > ! fprintf (vect_dump, "not vectorized: unsupported
pattern.");
> > ! return false;
> > }
> > }
> >
> > --- 316,351 ----
> > }
> >
> > gcc_assert (stmt_info);
> > if (STMT_VINFO_LIVE_P (stmt_info))
> > {
> > /* FORNOW: not yet supported. */
> > if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> > fprintf (vect_dump, "not vectorized: value used after loop.");
> > ! return false;
> > }
> >
> > if (STMT_VINFO_RELEVANT (stmt_info) == vect_used_in_loop
> > && STMT_VINFO_DEF_TYPE (stmt_info) != vect_induction_def)
> > {
> > ! /* A scalar-dependence cycle that we don't support. */
> > ! if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> > ! fprintf (vect_dump, "not vectorized: scalar
> dependence cycle.");
> > ! return false;
> > ! }
> > !
> > ! if (STMT_VINFO_RELEVANT_P (stmt_info)
> > ! && STMT_VINFO_DEF_TYPE (stmt_info) == vect_induction_def)
> > ! ok = vectorizable_induction (phi, NULL, NULL);
> > !
> > ! if (!ok)
> > ! {
> > if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> > ! {
> > ! fprintf (vect_dump,
> > ! "not vectorized: relevant phi not supported: ");
> > ! print_generic_expr (vect_dump, phi, TDF_SLIM);
> > ! }
> > ! return false;
> > }
> > }
> >
> > *************** vect_analyze_operations (loop_vec_info l
> > *** 343,348 ****
> > --- 353,359 ----
> > {
> > tree stmt = bsi_stmt (si);
> > stmt_vec_info stmt_info = vinfo_for_stmt (stmt);
> > + enum vect_def_type relevance = STMT_VINFO_RELEVANT (stmt_info);
> >
> > if (vect_print_dump_info (REPORT_DETAILS))
> > {
> > *************** vect_analyze_operations (loop_vec_info l
> > *** 367,421 ****
> > continue;
> > }
> >
> > if (STMT_VINFO_RELEVANT_P (stmt_info))
> > {
> > gcc_assert (GIMPLE_STMT_P (stmt)
> > || !VECTOR_MODE_P (TYPE_MODE (TREE_TYPE (stmt))));
> > gcc_assert (STMT_VINFO_VECTYPE (stmt_info));
> > -
> > - ok = (vectorizable_type_promotion (stmt, NULL, NULL)
> > - || vectorizable_type_demotion (stmt, NULL, NULL)
> > - || vectorizable_conversion (stmt, NULL, NULL)
> > - || vectorizable_operation (stmt, NULL, NULL)
> > - || vectorizable_assignment (stmt, NULL, NULL)
> > - || vectorizable_load (stmt, NULL, NULL)
> > - || vectorizable_call (stmt, NULL, NULL)
> > - || vectorizable_store (stmt, NULL, NULL)
> > - || vectorizable_condition (stmt, NULL, NULL));
> > -
> > - if (!ok)
> > - {
> > - if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> > - {
> > - fprintf (vect_dump,
> > - "not vectorized: relevant stmt not supported: ");
> > - print_generic_expr (vect_dump, stmt, TDF_SLIM);
> > - }
> > - return false;
> > - }
> > need_to_vectorize = true;
> > ! }
> > !
> > ! if (STMT_VINFO_LIVE_P (stmt_info))
> > ! {
> > ! ok = vectorizable_reduction (stmt, NULL, NULL);
> >
> > ! if (ok)
> > ! need_to_vectorize = true;
> > ! else
> > ! ok = vectorizable_live_operation (stmt, NULL, NULL);
> >
> > ! if (!ok)
> > {
> > ! if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> > ! {
> > ! fprintf (vect_dump,
> > ! "not vectorized: live stmt not supported: ");
> > ! print_generic_expr (vect_dump, stmt, TDF_SLIM);
> > ! }
> > ! return false;
> > }
> > ! }
> > } /* stmts in bb */
> > } /* bbs */
> >
> > --- 378,434 ----
> > continue;
> > }
> >
> > + switch (STMT_VINFO_DEF_TYPE (stmt_info))
> > + {
> > + case vect_loop_def:
> > + break;
> > +
> > + case vect_reduction_def:
> > + gcc_assert (relevance == vect_unused_in_loop);
> > + break;
> > +
> > + case vect_induction_def:
> > + case vect_constant_def:
> > + case vect_invariant_def:
> > + case vect_unknown_def_type:
> > + default:
> > + gcc_unreachable ();
> > + }
> > +
> > if (STMT_VINFO_RELEVANT_P (stmt_info))
> > {
> > gcc_assert (GIMPLE_STMT_P (stmt)
> > || !VECTOR_MODE_P (TYPE_MODE (TREE_TYPE (stmt))));
> > gcc_assert (STMT_VINFO_VECTYPE (stmt_info));
> > need_to_vectorize = true;
> > ! }
> >
> > ! ok = (vectorizable_type_promotion (stmt, NULL, NULL)
> > ! || vectorizable_type_demotion (stmt, NULL, NULL)
> > ! || vectorizable_conversion (stmt, NULL, NULL)
> > ! || vectorizable_operation (stmt, NULL, NULL)
> > ! || vectorizable_assignment (stmt, NULL, NULL)
> > ! || vectorizable_load (stmt, NULL, NULL)
> > ! || vectorizable_call (stmt, NULL, NULL)
> > ! || vectorizable_store (stmt, NULL, NULL)
> > ! || vectorizable_condition (stmt, NULL, NULL)
> > ! || vectorizable_reduction (stmt, NULL, NULL));
> > !
> > ! /* Stmts that are (also) "live" (i.e. - that are used out
> of the loop)
> > ! need extra handling, except for vectorizable reductions. */
> > ! if (STMT_VINFO_LIVE_P (stmt_info)
> > ! && STMT_VINFO_TYPE (stmt_info) != reduc_vec_info_type)
> > ! ok |= vectorizable_live_operation (stmt, NULL, NULL);
> >
> > ! if (!ok)
> > ! {
> > ! if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> > {
> > ! fprintf (vect_dump, "not vectorized: stmt not supported: ");
> > ! print_generic_expr (vect_dump, stmt, TDF_SLIM);
> > }
> > ! return false;
> > ! }
> > } /* stmts in bb */
> > } /* bbs */
> >
> > *************** vect_analyze_scalar_cycles (loop_vec_inf
> > *** 605,618 ****
> > }
> >
> > if (!access_fn
> > ! || !vect_is_simple_iv_evolution (loop->num, access_fn,
> &dumy, &dumy))
> > {
> > ! VEC_safe_push (tree, heap, worklist, phi);
> > continue;
> > }
> >
> > if (vect_print_dump_info (REPORT_DETAILS))
> > ! fprintf (vect_dump, "Detected induction.");
> > STMT_VINFO_DEF_TYPE (stmt_vinfo) = vect_induction_def;
> > }
> >
> > --- 618,631 ----
> > }
> >
> > if (!access_fn
> > ! || !vect_is_simple_iv_evolution (loop->num, access_fn, &dumy,
&dumy))
> > {
> > ! VEC_safe_push (tree, heap, worklist, phi);
> > continue;
> > }
> >
> > if (vect_print_dump_info (REPORT_DETAILS))
> > ! fprintf (vect_dump, "Detected induction.");
> > STMT_VINFO_DEF_TYPE (stmt_vinfo) = vect_induction_def;
> > }
> >
> > *************** static bool
> > *** 1418,1423 ****
> > --- 1431,1437 ----
> > vect_enhance_data_refs_alignment (loop_vec_info loop_vinfo)
> > {
> > VEC (data_reference_p, heap) *datarefs = LOOP_VINFO_DATAREFS
> (loop_vinfo);
> > + struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
> > enum dr_alignment_support supportable_dr_alignment;
> > struct data_reference *dr0 = NULL;
> > struct data_reference *dr;
> > *************** vect_enhance_data_refs_alignment (loop_v
> > *** 1515,1521 ****
> >
> > /* Often peeling for alignment will require peeling for loop-
> bound, which in
> > turn requires that we know how to adjust the loop ivs after
> the loop. */
> > ! if (!vect_can_advance_ivs_p (loop_vinfo))
> > do_peeling = false;
> >
> > if (do_peeling)
> > --- 1529,1536 ----
> >
> > /* Often peeling for alignment will require peeling for loop-
> bound, which in
> > turn requires that we know how to adjust the loop ivs after
> the loop. */
> > ! if (!vect_can_advance_ivs_p (loop_vinfo)
> > ! || !slpeel_can_duplicate_loop_p (loop, single_exit (loop)))
> > do_peeling = false;
> >
> > if (do_peeling)
> > *************** vect_mark_relevant (VEC(tree,heap) **wor
> > *** 2100,2110 ****
> > if (relevant > STMT_VINFO_RELEVANT (stmt_info))
> > STMT_VINFO_RELEVANT (stmt_info) = relevant;
> >
> > - if (TREE_CODE (stmt) == PHI_NODE)
> > - /* Don't put phi-nodes in the worklist. Phis that are marked
relevant
> > - or live will fail vectorization later on. */
> > - return;
> > -
> > if (STMT_VINFO_RELEVANT (stmt_info) == save_relevant
> > && STMT_VINFO_LIVE_P (stmt_info) == save_live_p)
> > {
> > --- 2115,2120 ----
> > *************** vect_stmt_relevant_p (tree stmt, loop_ve
> > *** 2143,2149 ****
> > *live_p = false;
> >
> > /* cond stmt other than loop exit cond. */
> > ! if (is_ctrl_stmt (stmt) && (stmt != LOOP_VINFO_EXIT_COND
(loop_vinfo)))
> > *relevant = vect_used_in_loop;
> >
> > /* changing memory. */
> > --- 2153,2160 ----
> > *live_p = false;
> >
> > /* cond stmt other than loop exit cond. */
> > ! if (is_ctrl_stmt (stmt)
> > ! && STMT_VINFO_TYPE (vinfo_for_stmt (stmt)) !=
> loop_exit_ctrl_vec_info_type)
> > *relevant = vect_used_in_loop;
> >
> > /* changing memory. */
> > *************** vect_stmt_relevant_p (tree stmt, loop_ve
> > *** 2180,2185 ****
> > --- 2191,2271 ----
> > }
> >
> >
> > + /*
> > + Function process_use.
> > +
> > + Inputs:
> > + - a USE in STMT in a loop represented by LOOP_VINFO
> > + - LIVE_P, RELEVANT - enum values to be set in the STMT_VINFO
> of the stmt
> > + that defined USE. This is dont by calling mark_relevant
andpassing it
> > + the WORKLIST (to add DEF_STMT to the WORKlist in case itis
> relevant).
> > +
> > + Outputs:
> > + Generally, LIVE_P and RELEVANT are used to define the liveness and
> > + relevance info of the DEF_STMT of this USE:
> > + STMT_VINFO_LIVE_P (DEF_STMT_info) <-- live_p
> > + STMT_VINFO_RELEVANT (DEF_STMT_info) <-- relevant
> > + Exceptions:
> > + - case 1: If USE is used only for address computations (e.g.
> array indexing),
> > + which does not need to be directly vectorized, then the
> liveness/relevance
> > + of the respective DEF_STMT is left unchanged.
> > + - case 2: If STMT is a reduction phi and DEF_STMT is a
> reduction stmt, we
> > + skip DEF_STMT cause it had already been processed.
> > +
> > + Return true if everyting is as expected. Return false otherwise.
*/
> > +
> > + static bool
> > + process_use (tree stmt, tree use, loop_vec_info loop_vinfo, bool
live_p,
> > + enum vect_relevant relevant, VEC(tree,heap) **worklist)
> > + {
> > + struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
> > + stmt_vec_info stmt_vinfo = vinfo_for_stmt (stmt);
> > + stmt_vec_info dstmt_vinfo;
> > + basic_block def_bb;
> > + tree def, def_stmt;
> > + enum vect_def_type dt;
> > +
> > + /* case 1: we are only interested in uses that need to be
> vectorized. Uses
> > + that are used for address computation are not considered
> relevant. */
> > + if (!exist_non_indexing_operands_for_use_p (use, stmt))
> > + return true;
> > +
> > + if (!vect_is_simple_use (use, loop_vinfo, &def_stmt, &def, &dt))
> > + {
> > + if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> > + fprintf (vect_dump, "not vectorized: unsupported use in
stmt.");
> > + return false;
> > + }
> > +
> > + if (!def_stmt || IS_EMPTY_STMT (def_stmt))
> > + return true;
> > +
> > + def_bb = bb_for_stmt (def_stmt);
> > + if (!flow_bb_inside_loop_p (loop, def_bb))
> > + return true;
> > +
> > + /* case 2: A reduction phi defining a reduction stmt
> (DEF_STMT). DEF_STMT
> > + must have already been processed, so we just check that
> everything is as
> > + expected, and we are done. */
> > + dstmt_vinfo = vinfo_for_stmt (def_stmt);
> > + if (TREE_CODE (stmt) == PHI_NODE
> > + && STMT_VINFO_DEF_TYPE (stmt_vinfo) == vect_reduction_def
> > + && TREE_CODE (def_stmt) != PHI_NODE
> > + && STMT_VINFO_DEF_TYPE (dstmt_vinfo) == vect_reduction_def)
> > + {
> > + if (STMT_VINFO_IN_PATTERN_P (dstmt_vinfo))
> > + dstmt_vinfo = vinfo_for_stmt (STMT_VINFO_RELATED_STMT
(dstmt_vinfo));
> > + gcc_assert (STMT_VINFO_RELEVANT (dstmt_vinfo) <
> vect_used_by_reduction);
> > + gcc_assert (STMT_VINFO_LIVE_P (dstmt_vinfo)
> > + || STMT_VINFO_RELEVANT (dstmt_vinfo) > vect_unused_in_loop);
> > + return true;
> > + }
> > +
> > + vect_mark_relevant (worklist, def_stmt, relevant, live_p);
> > + return true;
> > + }
> > +
> > +
> > /* Function vect_mark_stmts_to_be_vectorized.
> >
> > Not all stmts in the loop need to be vectorized. For example:
> > *************** vect_mark_stmts_to_be_vectorized (loop_v
> > *** 2204,2220 ****
> > basic_block *bbs = LOOP_VINFO_BBS (loop_vinfo);
> > unsigned int nbbs = loop->num_nodes;
> > block_stmt_iterator si;
> > ! tree stmt, use;
> > stmt_ann_t ann;
> > - ssa_op_iter iter;
> > unsigned int i;
> > stmt_vec_info stmt_vinfo;
> > basic_block bb;
> > tree phi;
> > bool live_p;
> > enum vect_relevant relevant;
> > - tree def, def_stmt;
> > - enum vect_def_type dt;
> >
> > if (vect_print_dump_info (REPORT_DETAILS))
> > fprintf (vect_dump, "=== vect_mark_stmts_to_be_vectorized ===");
> > --- 2290,2303 ----
> > basic_block *bbs = LOOP_VINFO_BBS (loop_vinfo);
> > unsigned int nbbs = loop->num_nodes;
> > block_stmt_iterator si;
> > ! tree stmt;
> > stmt_ann_t ann;
> > unsigned int i;
> > stmt_vec_info stmt_vinfo;
> > basic_block bb;
> > tree phi;
> > bool live_p;
> > enum vect_relevant relevant;
> >
> > if (vect_print_dump_info (REPORT_DETAILS))
> > fprintf (vect_dump, "=== vect_mark_stmts_to_be_vectorized ===");
> > *************** vect_mark_stmts_to_be_vectorized (loop_v
> > *** 2222,2248 ****
> > worklist = VEC_alloc (tree, heap, 64);
> >
> > /* 1. Init worklist. */
> > -
> > - bb = loop->header;
> > - for (phi = phi_nodes (bb); phi; phi = PHI_CHAIN (phi))
> > - {
> > - if (vect_print_dump_info (REPORT_DETAILS))
> > - {
> > - fprintf (vect_dump, "init: phi relevant? ");
> > - print_generic_expr (vect_dump, phi, TDF_SLIM);
> > - }
> > -
> > - if (vect_stmt_relevant_p (phi, loop_vinfo, &relevant, &live_p))
> > - vect_mark_relevant (&worklist, phi, relevant, live_p);
> > - }
> > -
> > for (i = 0; i < nbbs; i++)
> > {
> > bb = bbs[i];
> > for (si = bsi_start (bb); !bsi_end_p (si); bsi_next (&si))
> > {
> > stmt = bsi_stmt (si);
> > -
> > if (vect_print_dump_info (REPORT_DETAILS))
> > {
> > fprintf (vect_dump, "init: stmt relevant? ");
> > --- 2305,2327 ----
> > worklist = VEC_alloc (tree, heap, 64);
> >
> > /* 1. Init worklist. */
> > for (i = 0; i < nbbs; i++)
> > {
> > bb = bbs[i];
> > + for (phi = phi_nodes (bb); phi; phi = PHI_CHAIN (phi))
> > + {
> > + if (vect_print_dump_info (REPORT_DETAILS))
> > + {
> > + fprintf (vect_dump, "init: phi relevant? ");
> > + print_generic_expr (vect_dump, phi, TDF_SLIM);
> > + }
> > +
> > + if (vect_stmt_relevant_p (phi, loop_vinfo, &relevant, &live_p))
> > + vect_mark_relevant (&worklist, phi, relevant, live_p);
> > + }
> > for (si = bsi_start (bb); !bsi_end_p (si); bsi_next (&si))
> > {
> > stmt = bsi_stmt (si);
> > if (vect_print_dump_info (REPORT_DETAILS))
> > {
> > fprintf (vect_dump, "init: stmt relevant? ");
> > *************** vect_mark_stmts_to_be_vectorized (loop_v
> > *** 2254,2355 ****
> > }
> > }
> >
> > -
> > /* 2. Process_worklist */
> > -
> > while (VEC_length (tree, worklist) > 0)
> > {
> > ! stmt = VEC_pop (tree, worklist);
> >
> > if (vect_print_dump_info (REPORT_DETAILS))
> > {
> > fprintf (vect_dump, "worklist: examine stmt: ");
> > print_generic_expr (vect_dump, stmt, TDF_SLIM);
> > }
> >
> > ! /* Examine the USEs of STMT. For each ssa-name USE that is
defined
> > ! in the loop, mark the stmt that defines it (DEF_STMT) as
> > ! relevant/irrelevant and live/dead according to the liveness
and
> > ! relevance properties of STMT.
> > ! */
> > !
> > ! gcc_assert (TREE_CODE (stmt) != PHI_NODE);
> > !
> > ann = stmt_ann (stmt);
> > stmt_vinfo = vinfo_for_stmt (stmt);
> > -
> > relevant = STMT_VINFO_RELEVANT (stmt_vinfo);
> > live_p = STMT_VINFO_LIVE_P (stmt_vinfo);
> >
> > /* Generally, the liveness and relevance properties of STMT are
> > ! propagated to the DEF_STMTs of its USEs:
> > ! STMT_VINFO_LIVE_P (DEF_STMT_info) <-- live_p
> > ! STMT_VINFO_RELEVANT (DEF_STMT_info) <-- relevant
> > !
> > ! Exceptions:
> > !
> > ! (case 1)
> > ! If USE is used only for address computations (e.g.
> array indexing),
> > ! which does not need to be directly vectorized, then the
> > ! liveness/relevance of the respective DEF_STMT is
leftunchanged.
> > !
> > ! (case 2)
> > ! If STMT has been identified as defining a reduction
> variable, then
> > ! we want to set liveness/relevance as follows:
> > ! STMT_VINFO_LIVE_P (DEF_STMT_info) <-- false
> > ! STMT_VINFO_RELEVANT (DEF_STMT_info) <--
> vect_used_by_reduction
> > ! because even though STMT is classified as live
> (since it defines a
> > ! value that is used across loop iterations) and
> irrelevant (since it
> > ! is not used inside the loop), it will be
> vectorized, and therefore
> > ! the corresponding DEF_STMTs need to marked as relevant.
> > ! We distinguish between two kinds of relevant stmts -
> those that are
> > ! used by a reduction computation, and those that are (also)
used by
> > ! a regular computation. This allows us later on to identify
stmts
> > ! that are used solely by a reduction, and therefore the order
of
> > ! the results that they produce does not have to be kept.
> > ! */
> >
> > - /* case 2.2: */
> > if (STMT_VINFO_DEF_TYPE (stmt_vinfo) == vect_reduction_def)
> > ! {
> > ! gcc_assert (relevant == vect_unused_in_loop && live_p);
> > relevant = vect_used_by_reduction;
> > ! live_p = false;
> > }
> >
> > ! i = 0;
> > ! FOR_EACH_SSA_TREE_OPERAND (use, stmt, iter, SSA_OP_USE)
> > {
> > ! if (vect_print_dump_info (REPORT_DETAILS))
> > {
> > - fprintf (vect_dump, "worklist: examine use %d: ", i++);
> > - print_generic_expr (vect_dump, use, TDF_SLIM);
> > - }
> > -
> > - /* case 1: we are only interested in uses that need to be
> vectorized.
> > - Uses that are used for address computation are not considered
> > - relevant.
> > - */
> > - if (!exist_non_indexing_operands_for_use_p (use, stmt))
> > - continue;
> > -
> > - if (!vect_is_simple_use (use, loop_vinfo, &def_stmt, &def, &dt))
> > - {
> > - if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> > - fprintf (vect_dump, "not vectorized: unsupported use in
stmt.");
> > VEC_free (tree, heap, worklist);
> > return false;
> > ! }
> > !
> > ! if (!def_stmt || IS_EMPTY_STMT (def_stmt))
> > ! continue;
> > !
> > ! bb = bb_for_stmt (def_stmt);
> > ! if (!flow_bb_inside_loop_p (loop, bb))
> > ! continue;
> > ! vect_mark_relevant (&worklist, def_stmt, relevant, live_p);
> > }
> > ! } /* while worklist */
> >
> > VEC_free (tree, heap, worklist);
> > return true;
> > --- 2333,2414 ----
> > }
> > }
> >
> > /* 2. Process_worklist */
> > while (VEC_length (tree, worklist) > 0)
> > {
> > ! use_operand_p use_p;
> > ! ssa_op_iter iter;
> >
> > + stmt = VEC_pop (tree, worklist);
> > if (vect_print_dump_info (REPORT_DETAILS))
> > {
> > fprintf (vect_dump, "worklist: examine stmt: ");
> > print_generic_expr (vect_dump, stmt, TDF_SLIM);
> > }
> >
> > ! /* Examine the USEs of STMT. For each USE, mark the stmt
> that defines it
> > ! (DEF_STMT) as relevant/irrelevant and live/dead according to the
> > ! liveness and relevance properties of STMT. */
> > ann = stmt_ann (stmt);
> > stmt_vinfo = vinfo_for_stmt (stmt);
> > relevant = STMT_VINFO_RELEVANT (stmt_vinfo);
> > live_p = STMT_VINFO_LIVE_P (stmt_vinfo);
> >
> > /* Generally, the liveness and relevance properties of STMT are
> > ! propagated as is to the DEF_STMTs of its USEs:
> > ! live_p <-- STMT_VINFO_LIVE_P (STMT_VINFO)
> > ! relevant <-- STMT_VINFO_RELEVANT (STMT_VINFO)
> > !
> > ! One exception is when STMT has been identified as defining
areduction
> > ! variable; in this case we set the liveness/relevance as follows:
> > ! live_p = false
> > ! relevant = vect_used_by_reduction
> > ! This is because we distinguish between two kinds of relevant
stmts -
> > ! those that are used by a reduction computation, and those that
are
> > ! (also) used by a regular computation. This allows us later on to
> > ! identify stmts that are used solely by a reduction, and therefore
the
> > ! order of the results that they produce does not have to be kept.
> > !
> > ! Reduction phis are expected to be used by a reduction
> stmt; Other
> > ! reduction stmts are expected to be unused in the loop. These are
the
> > ! expected values of "relevant" for reduction phis/stmts in the
loop:
> > !
> > ! relevance: phi stmt
> > ! vect_unused_in_loop ok
> > ! vect_used_by_reduction ok
> > ! vect_used_in_loop */
> >
> > if (STMT_VINFO_DEF_TYPE (stmt_vinfo) == vect_reduction_def)
> > ! {
> > ! switch (relevant)
> > ! {
> > ! case vect_unused_in_loop:
> > ! gcc_assert (TREE_CODE (stmt) != PHI_NODE);
> > ! break;
> > ! case vect_used_by_reduction:
> > ! if (TREE_CODE (stmt) == PHI_NODE)
> > ! break;
> > ! case vect_used_in_loop:
> > ! default:
> > ! if (vect_print_dump_info (REPORT_DETAILS))
> > ! fprintf (vect_dump, "unsupported use of reduction.");
> > ! VEC_free (tree, heap, worklist);
> > ! return false;
> > ! }
> > relevant = vect_used_by_reduction;
> > ! live_p = false;
> > }
> >
> > ! FOR_EACH_PHI_OR_STMT_USE (use_p, stmt, iter, SSA_OP_USE)
> > {
> > ! tree op = USE_FROM_PTR (use_p);
> > ! if (!process_use (stmt, op, loop_vinfo, live_p, relevant,
&worklist))
> > {
> > VEC_free (tree, heap, worklist);
> > return false;
> > ! }
> > }
> > ! } /* while worklist */
> >
> > VEC_free (tree, heap, worklist);
> > return true;
> > *************** vect_analyze_loop_form (struct loop *loo
> > *** 2582,2591 ****
> > return false;
> > }
> >
> > ! loop_vinfo = new_loop_vec_info (loop);
> > ! LOOP_VINFO_NITERS (loop_vinfo) = number_of_iterations;
> > !
> > ! if (!LOOP_VINFO_NITERS_KNOWN_P (loop_vinfo))
> > {
> > if (vect_print_dump_info (REPORT_DETAILS))
> > {
> > --- 2641,2647 ----
> > return false;
> > }
> >
> > ! if (!NITERS_KNOWN_P (number_of_iterations))
> > {
> > if (vect_print_dump_info (REPORT_DETAILS))
> > {
> > *************** vect_analyze_loop_form (struct loop *loo
> > *** 2594,2608 ****
> > }
> > }
> > else
> > ! if (LOOP_VINFO_INT_NITERS (loop_vinfo) == 0)
> > {
> > if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> > fprintf (vect_dump, "not vectorized: number of iterations =
0.");
> > return NULL;
> > }
> >
> > ! LOOP_VINFO_EXIT_COND (loop_vinfo) = loop_cond;
> >
> > return loop_vinfo;
> > }
> >
> > --- 2650,2669 ----
> > }
> > }
> > else
> > ! if (TREE_INT_CST_LOW (number_of_iterations) == 0)
> > {
> > if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> > fprintf (vect_dump, "not vectorized: number of iterations =
0.");
> > return NULL;
> > }
> >
> > ! loop_vinfo = new_loop_vec_info (loop);
> > ! LOOP_VINFO_NITERS (loop_vinfo) = number_of_iterations;
> > !
> > ! STMT_VINFO_TYPE (vinfo_for_stmt (loop_cond)) =
> loop_exit_ctrl_vec_info_type;
> >
> > + gcc_assert (!loop->aux);
> > + loop->aux = loop_vinfo;
> > return loop_vinfo;
> > }
> >
> > Index: common.opt
> > ===================================================================
> > *** common.opt (revision 123103)
> > --- common.opt (working copy)
> > *************** funroll-all-loops
> > *** 1070,1075 ****
> > --- 1070,1079 ----
> > Common Report Var(flag_unroll_all_loops) Optimization
> > Perform loop unrolling for all loops
> >
> > + ftree-reassoc
> > + Common Report Var(flag_tree_reassoc) Init(1) Optimization
> > + Perform reassociation on trees
> > +
> > ; Nonzero means that loop optimizer may assume that the
> induction variables
> > ; that control loops do not overflow and that the loops with
nontrivial
> > ; exit condition are not infinite
> > Index: tree-vect-patterns.c
> > ===================================================================
> > *** tree-vect-patterns.c (revision 123103)
> > --- tree-vect-patterns.c (working copy)
> > *************** widened_name_p (tree name, tree use_stmt
> > *** 109,118 ****
> > if (!vect_is_simple_use (oprnd0, loop_vinfo, &dummy, &dummy, &dt))
> > return false;
> >
> > - if (dt != vect_invariant_def && dt != vect_constant_def
> > - && dt != vect_loop_def)
> > - return false;
> > -
> > return true;
> > }
> >
> > --- 109,114 ----
> > Index: tree-vect-transform.c
> > ===================================================================
> > *** tree-vect-transform.c (revision 123103)
> > --- tree-vect-transform.c (working copy)
> > *************** vect_init_vector (tree stmt, tree vector
> > *** 514,520 ****
> > /* Function get_initial_def_for_induction
> >
> > Input:
> > - STMT - a stmt that performs an induction operation in the loop.
> > IV_PHI - the initial value of the induction variable
> >
> > Output:
> > --- 514,519 ----
> > *************** vect_init_vector (tree stmt, tree vector
> > *** 524,532 ****
> > [X, X + S, X + 2*S, X + 3*S]. */
> >
> > static tree
> > ! get_initial_def_for_induction (tree stmt, tree iv_phi)
> > {
> > ! stmt_vec_info stmt_vinfo = vinfo_for_stmt (stmt);
> > loop_vec_info loop_vinfo = STMT_VINFO_LOOP_VINFO (stmt_vinfo);
> > struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
> > tree scalar_type = TREE_TYPE (iv_phi);
> > --- 523,531 ----
> > [X, X + S, X + 2*S, X + 3*S]. */
> >
> > static tree
> > ! get_initial_def_for_induction (tree iv_phi)
> > {
> > ! stmt_vec_info stmt_vinfo = vinfo_for_stmt (iv_phi);
> > loop_vec_info loop_vinfo = STMT_VINFO_LOOP_VINFO (stmt_vinfo);
> > struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
> > tree scalar_type = TREE_TYPE (iv_phi);
> > *************** get_initial_def_for_induction (tree stmt
> > *** 549,575 ****
> > tree expr;
> > stmt_vec_info phi_info = vinfo_for_stmt (iv_phi);
> > tree stmts;
> >
> > gcc_assert (phi_info);
> >
> > ! if (STMT_VINFO_VEC_STMT (phi_info))
> > {
> > ! induction_phi = STMT_VINFO_VEC_STMT (phi_info);
> > ! gcc_assert (TREE_CODE (induction_phi) == PHI_NODE);
> > !
> > ! if (vect_print_dump_info (REPORT_DETAILS))
> > ! {
> > ! fprintf (vect_dump, "induction already vectorized:");
> > ! print_generic_expr (vect_dump, iv_phi, TDF_SLIM);
> > ! fprintf (vect_dump, "\n");
> > ! print_generic_expr (vect_dump, induction_phi, TDF_SLIM);
> > ! }
> > !
> > ! return PHI_RESULT (induction_phi);
> > }
> >
> > - gcc_assert (ncopies >= 1);
> > -
> > access_fn = analyze_scalar_evolution (loop, PHI_RESULT (iv_phi));
> > gcc_assert (access_fn);
> > ok = vect_is_simple_iv_evolution (loop->num, access_fn,
> &init_expr, &step_expr);
> > --- 548,568 ----
> > tree expr;
> > stmt_vec_info phi_info = vinfo_for_stmt (iv_phi);
> > tree stmts;
> > + tree stmt = NULL_TREE;
> > + block_stmt_iterator si;
> > + basic_block bb = bb_for_stmt (iv_phi);
> >
> > gcc_assert (phi_info);
> > + gcc_assert (ncopies >= 1);
> >
> > ! /* Find the first insertion point in the BB. */
> > ! for (si = bsi_start (bb); !bsi_end_p (si); bsi_next (&si))
> > {
> > ! stmt = bsi_stmt (si);
> > ! if (TREE_CODE (stmt) != LABEL_EXPR)
> > ! break;
> > }
> >
> > access_fn = analyze_scalar_evolution (loop, PHI_RESULT (iv_phi));
> > gcc_assert (access_fn);
> > ok = vect_is_simple_iv_evolution (loop->num, access_fn,
> &init_expr, &step_expr);
> > *************** vect_get_vec_def_for_operand (tree op, t
> > *** 833,839 ****
> > gcc_assert (TREE_CODE (def_stmt) == PHI_NODE);
> >
> > /* Get the def before the loop */
> > ! return get_initial_def_for_induction (stmt, def_stmt);
> > }
> >
> > default:
> > --- 826,832 ----
> > gcc_assert (TREE_CODE (def_stmt) == PHI_NODE);
> >
> > /* Get the def before the loop */
> > ! return get_initial_def_for_induction (def_stmt);
> > }
> >
> > default:
> > *************** vect_finish_stmt_generation (tree stmt,
> > *** 951,958 ****
> > }
> >
> >
> > - #define ADJUST_IN_EPILOG 1
> > -
> > /* Function get_initial_def_for_reduction
> >
> > Input:
> > --- 944,949 ----
> > *************** vect_finish_stmt_generation (tree stmt,
> > *** 960,977 ****
> > INIT_VAL - the initial value of the reduction variable
> >
> > Output:
> > ! SCALAR_DEF - a tree that holds a value to be added to the final
result
> > ! of the reduction (used for "ADJUST_IN_EPILOG" - see below).
> > Return a vector variable, initialized according to the
> operation that STMT
> > ! performs. This vector will be used as the initial value of the
> > ! vector of partial results.
> >
> > ! Option1 ("ADJUST_IN_EPILOG"): Initialize the vector as follows:
> > add: [0,0,...,0,0]
> > mult: [1,1,...,1,1]
> > min/max: [init_val,init_val,..,init_val,init_val]
> > bit and/or: [init_val,init_val,..,init_val,init_val]
> > ! and when necessary (e.g. add/mult case) let the caller know
> > that it needs to adjust the result by init_val.
> >
> > Option2: Initialize the vector as follows:
> > --- 951,968 ----
> > INIT_VAL - the initial value of the reduction variable
> >
> > Output:
> > ! ADJUSTMENT_DEF - a tree that holds a value to be added to the
> final result
> > ! of the reduction (used for adjusting the epilog - see below).
> > Return a vector variable, initialized according to the
> operation that STMT
> > ! performs. This vector will be used as the initial value of
the
> > ! vector of partial results.
> >
> > ! Option1 (adjust in epilog): Initialize the vector as follows:
> > add: [0,0,...,0,0]
> > mult: [1,1,...,1,1]
> > min/max: [init_val,init_val,..,init_val,init_val]
> > bit and/or: [init_val,init_val,..,init_val,init_val]
> > ! and when necessary (e.g. add/mult case) let the caller know
> > that it needs to adjust the result by init_val.
> >
> > Option2: Initialize the vector as follows:
> > *************** vect_finish_stmt_generation (tree stmt,
> > *** 992,1075 ****
> > or [0,0,0,0] and let the caller know that it needs to adjust
> > the result at the end by 'init_val'.
> >
> > ! FORNOW: We use the "ADJUST_IN_EPILOG" scheme.
> > ! TODO: Use some cost-model to estimate which scheme is more
profitable.
> > ! */
> >
> > static tree
> > ! get_initial_def_for_reduction (tree stmt, tree init_val, tree
*scalar_def)
> > {
> > stmt_vec_info stmt_vinfo = vinfo_for_stmt (stmt);
> > tree vectype = STMT_VINFO_VECTYPE (stmt_vinfo);
> > int nunits = TYPE_VECTOR_SUBPARTS (vectype);
> > - int nelements;
> > enum tree_code code = TREE_CODE (GIMPLE_STMT_OPERAND (stmt, 1));
> > tree type = TREE_TYPE (init_val);
> > ! tree def;
> > ! tree vec, t = NULL_TREE;
> > ! bool need_epilog_adjust;
> > int i;
> > tree vector_type;
> >
> > gcc_assert (INTEGRAL_TYPE_P (type) || SCALAR_FLOAT_TYPE_P (type));
> >
> > switch (code)
> > {
> > case WIDEN_SUM_EXPR:
> > case DOT_PROD_EXPR:
> > case PLUS_EXPR:
> > if (INTEGRAL_TYPE_P (type))
> > ! def = build_int_cst (type, 0);
> > else
> > ! def = build_real (type, dconst0);
> > !
> > ! #ifdef ADJUST_IN_EPILOG
> > ! /* All the 'nunits' elements are set to 0. The final result will
be
> > ! adjusted by 'init_val' at the loop epilog. */
> > ! nelements = nunits;
> > ! need_epilog_adjust = true;
> > ! #else
> > ! /* 'nunits - 1' elements are set to 0; The last element is set to
> > ! 'init_val'. No further adjustments at the epilog are needed.
*/
> > ! nelements = nunits - 1;
> > ! need_epilog_adjust = false;
> > ! #endif
> > break;
> >
> > case MIN_EXPR:
> > case MAX_EXPR:
> > ! def = init_val;
> > ! nelements = nunits;
> > ! need_epilog_adjust = false;
> > break;
> >
> > default:
> > gcc_unreachable ();
> > }
> >
> > ! for (i = nelements - 1; i >= 0; --i)
> > ! t = tree_cons (NULL_TREE, def, t);
> > !
> > ! if (nelements == nunits - 1)
> > ! {
> > ! /* Set the last element of the vector. */
> > ! t = tree_cons (NULL_TREE, init_val, t);
> > ! nelements += 1;
> > ! }
> > ! gcc_assert (nelements == nunits);
> > !
> > ! vector_type = get_vectype_for_scalar_type (TREE_TYPE (def));
> > ! if (TREE_CODE (init_val) == INTEGER_CST || TREE_CODE
> (init_val) == REAL_CST)
> > ! vec = build_vector (vector_type, t);
> > ! else
> > ! vec = build_constructor_from_list (vector_type, t);
> > !
> > ! if (!need_epilog_adjust)
> > ! *scalar_def = NULL_TREE;
> > ! else
> > ! *scalar_def = init_val;
> > !
> > ! return vect_init_vector (stmt, vec, vector_type);
> > }
> >
> >
> > --- 983,1038 ----
> > or [0,0,0,0] and let the caller know that it needs to adjust
> > the result at the end by 'init_val'.
> >
> > ! FORNOW, we are using the 'adjust in epilog' scheme, because
> this way the
> > ! initialization vector is simpler (same element in all entries).
> > ! A cost model should help decide between these two schemes. */
> >
> > static tree
> > ! get_initial_def_for_reduction (tree stmt, tree init_val, tree
> *adjustment_def)
> > {
> > stmt_vec_info stmt_vinfo = vinfo_for_stmt (stmt);
> > tree vectype = STMT_VINFO_VECTYPE (stmt_vinfo);
> > int nunits = TYPE_VECTOR_SUBPARTS (vectype);
> > enum tree_code code = TREE_CODE (GIMPLE_STMT_OPERAND (stmt, 1));
> > tree type = TREE_TYPE (init_val);
> > ! tree vecdef;
> > ! tree def_for_init;
> > ! tree init_def;
> > ! tree t = NULL_TREE;
> > int i;
> > tree vector_type;
> >
> > gcc_assert (INTEGRAL_TYPE_P (type) || SCALAR_FLOAT_TYPE_P (type));
> > + vecdef = vect_get_vec_def_for_operand (init_val, stmt, NULL);
> >
> > switch (code)
> > {
> > case WIDEN_SUM_EXPR:
> > case DOT_PROD_EXPR:
> > case PLUS_EXPR:
> > + *adjustment_def = init_val;
> > + /* Create a vector of zeros for init_def. */
> > if (INTEGRAL_TYPE_P (type))
> > ! def_for_init = build_int_cst (type, 0);
> > else
> > ! def_for_init = build_real (type, dconst0);
> > ! for (i = nunits - 1; i >= 0; --i)
> > ! t = tree_cons (NULL_TREE, def_for_init, t);
> > ! vector_type = get_vectype_for_scalar_type (TREE_TYPE
(def_for_init));
> > ! init_def = build_vector (vector_type, t);
> > break;
> >
> > case MIN_EXPR:
> > case MAX_EXPR:
> > ! *adjustment_def = NULL_TREE;
> > ! init_def = vecdef;
> > break;
> >
> > default:
> > gcc_unreachable ();
> > }
> >
> > ! return init_def;
> > }
> >
> >
> > *************** vectorizable_call (tree stmt, block_stmt
> > *** 1816,1821 ****
> > --- 1779,1798 ----
> > int ncopies, j, nargs;
> > call_expr_arg_iterator iter;
> >
> > + if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > + return false;
> > +
> > + if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
> > + return false;
> > +
> > + /* FORNOW: not yet supported. */
> > + if (STMT_VINFO_LIVE_P (stmt_info))
> > + {
> > + if (vect_print_dump_info (REPORT_DETAILS))
> > + fprintf (vect_dump, "value used after loop.");
> > + return false;
> > + }
> > +
> > /* Is STMT a vectorizable call? */
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > return false;
> > *************** vectorizable_conversion (tree stmt, bloc
> > *** 1994,2000 ****
> > if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > return false;
> >
> > ! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
> >
> > if (STMT_VINFO_LIVE_P (stmt_info))
> > {
> > --- 1971,1978 ----
> > if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > return false;
> >
> > ! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
> > ! return false;
> >
> > if (STMT_VINFO_LIVE_P (stmt_info))
> > {
> > *************** vectorizable_assignment (tree stmt, bloc
> > *** 2134,2145 ****
> > if (ncopies > 1)
> > return false; /* FORNOW */
> >
> > - /* Is vectorizable assignment? */
> > if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > return false;
> >
> > ! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
> >
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > return false;
> >
> > --- 2112,2132 ----
> > if (ncopies > 1)
> > return false; /* FORNOW */
> >
> > if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > return false;
> >
> > ! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
> > ! return false;
> > !
> > ! /* FORNOW: not yet supported. */
> > ! if (STMT_VINFO_LIVE_P (stmt_info))
> > ! {
> > ! if (vect_print_dump_info (REPORT_DETAILS))
> > ! fprintf (vect_dump, "value used after loop.");
> > ! return false;
> > ! }
> >
> > + /* Is vectorizable assignment? */
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > return false;
> >
> > *************** vect_min_worthwhile_factor (enum tree_co
> > *** 2209,2214 ****
> > --- 2196,2254 ----
> > }
> >
> >
> > + /* Function vectorizable_induction
> > +
> > + Check if PHI performs an induction computation that can be
vectorized.
> > + If VEC_STMT is also passed, vectorize the induction PHI:
> create a vectorized
> > + phi to replace it, put it in VEC_STMT, and add it to the same
> basic block.
> > + Return FALSE if not a vectorizable STMT, TRUE otherwise. */
> > +
> > + bool
> > + vectorizable_induction (tree phi, block_stmt_iterator *bsi
> ATTRIBUTE_UNUSED,
> > + tree *vec_stmt)
> > + {
> > + stmt_vec_info stmt_info = vinfo_for_stmt (phi);
> > + tree vectype = STMT_VINFO_VECTYPE (stmt_info);
> > + loop_vec_info loop_vinfo = STMT_VINFO_LOOP_VINFO (stmt_info);
> > + int nunits = TYPE_VECTOR_SUBPARTS (vectype);
> > + int ncopies = LOOP_VINFO_VECT_FACTOR (loop_vinfo) / nunits;
> > + tree vec_def;
> > +
> > + gcc_assert (ncopies >= 1);
> > +
> > + if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > + return false;
> > +
> > + gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_induction_def);
> > +
> > + if (STMT_VINFO_LIVE_P (stmt_info))
> > + {
> > + /* FORNOW: not yet supported. */
> > + if (vect_print_dump_info (REPORT_DETAILS))
> > + fprintf (vect_dump, "value used after loop.");
> > + return false;
> > + }
> > +
> > + if (TREE_CODE (phi) != PHI_NODE)
> > + return false;
> > +
> > + if (!vec_stmt) /* transformation not required. */
> > + {
> > + STMT_VINFO_TYPE (stmt_info) = induc_vec_info_type;
> > + return true;
> > + }
> > +
> > + /** Transform. **/
> > +
> > + if (vect_print_dump_info (REPORT_DETAILS))
> > + fprintf (vect_dump, "transform induction phi.");
> > +
> > + vec_def = get_initial_def_for_induction (phi);
> > + *vec_stmt = SSA_NAME_DEF_STMT (vec_def);
> > + return true;
> > + }
> > +
> > +
> > /* Function vectorizable_operation.
> >
> > Check if STMT performs a binary or unary operation that can
> be vectorized.
> > *************** vectorizable_operation (tree stmt, block
> > *** 2246,2265 ****
> >
> > gcc_assert (ncopies >= 1);
> >
> > - /* Is STMT a vectorizable binary/unary operation? */
> > if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > return false;
> >
> > ! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
> >
> > if (STMT_VINFO_LIVE_P (stmt_info))
> > {
> > - /* FORNOW: not yet supported. */
> > if (vect_print_dump_info (REPORT_DETAILS))
> > fprintf (vect_dump, "value used after loop.");
> > return false;
> > }
> >
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > return false;
> >
> > --- 2286,2306 ----
> >
> > gcc_assert (ncopies >= 1);
> >
> > if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > return false;
> >
> > ! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
> > ! return false;
> >
> > + /* FORNOW: not yet supported. */
> > if (STMT_VINFO_LIVE_P (stmt_info))
> > {
> > if (vect_print_dump_info (REPORT_DETAILS))
> > fprintf (vect_dump, "value used after loop.");
> > return false;
> > }
> >
> > + /* Is STMT a vectorizable binary/unary operation? */
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > return false;
> >
> > *************** vectorizable_type_demotion (tree stmt, b
> > *** 2514,2534 ****
> > optab optab;
> > enum machine_mode vec_mode;
> >
> > - /* Is STMT a vectorizable type-demotion operation? */
> > -
> > if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > return false;
> > !
> > ! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
> > !
> > if (STMT_VINFO_LIVE_P (stmt_info))
> > {
> > - /* FORNOW: not yet supported. */
> > if (vect_print_dump_info (REPORT_DETAILS))
> > fprintf (vect_dump, "value used after loop.");
> > return false;
> > }
> >
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > return false;
> >
> > --- 2555,2575 ----
> > optab optab;
> > enum machine_mode vec_mode;
> >
> > if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > return false;
> > !
> > ! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
> > ! return false;
> > !
> > ! /* FORNOW: not yet supported. */
> > if (STMT_VINFO_LIVE_P (stmt_info))
> > {
> > if (vect_print_dump_info (REPORT_DETAILS))
> > fprintf (vect_dump, "value used after loop.");
> > return false;
> > }
> >
> > + /* Is STMT a vectorizable type-demotion operation? */
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > return false;
> >
> > *************** vectorizable_type_promotion (tree stmt,
> > *** 2723,2743 ****
> > int j;
> > tree vectype_in;
> >
> > - /* Is STMT a vectorizable type-promotion operation? */
> > -
> > if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > return false;
> >
> > ! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
> >
> > if (STMT_VINFO_LIVE_P (stmt_info))
> > {
> > - /* FORNOW: not yet supported. */
> > if (vect_print_dump_info (REPORT_DETAILS))
> > ! fprintf (vect_dump, "value used after loop.");
> > return false;
> > }
> >
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > return false;
> >
> > --- 2764,2784 ----
> > int j;
> > tree vectype_in;
> >
> > if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > return false;
> >
> > ! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
> > ! return false;
> >
> > + /* FORNOW: not yet supported. */
> > if (STMT_VINFO_LIVE_P (stmt_info))
> > {
> > if (vect_print_dump_info (REPORT_DETAILS))
> > ! fprintf (vect_dump, "value used after loop.");
> > return false;
> > }
> >
> > + /* Is STMT a vectorizable type-promotion operation? */
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > return false;
> >
> > *************** vectorizable_store (tree stmt, block_stm
> > *** 3064,3069 ****
> > --- 3105,3123 ----
> > VEC(tree,heap) *dr_chain = NULL, *oprnds = NULL, *result_chain =
NULL;
> > gcc_assert (ncopies >= 1);
> >
> > + if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > + return false;
> > +
> > + if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
> > + return false;
> > +
> > + if (STMT_VINFO_LIVE_P (stmt_info))
> > + {
> > + if (vect_print_dump_info (REPORT_DETAILS))
> > + fprintf (vect_dump, "value used after loop.");
> > + return false;
> > + }
> > +
> > /* Is vectorizable store? */
> >
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > *************** vectorizable_load (tree stmt, block_stmt
> > *** 3710,3729 ****
> > bool strided_load = false;
> > tree first_stmt;
> >
> > - /* Is vectorizable load? */
> > if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > return false;
> >
> > ! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
> >
> > if (STMT_VINFO_LIVE_P (stmt_info))
> > {
> > - /* FORNOW: not yet supported. */
> > if (vect_print_dump_info (REPORT_DETAILS))
> > fprintf (vect_dump, "value used after loop.");
> > return false;
> > }
> >
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > return false;
> >
> > --- 3764,3784 ----
> > bool strided_load = false;
> > tree first_stmt;
> >
> > if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > return false;
> >
> > ! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
> > ! return false;
> >
> > + /* FORNOW: not yet supported. */
> > if (STMT_VINFO_LIVE_P (stmt_info))
> > {
> > if (vect_print_dump_info (REPORT_DETAILS))
> > fprintf (vect_dump, "value used after loop.");
> > return false;
> > }
> >
> > + /* Is vectorizable load? */
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > return false;
> >
> > *************** vectorizable_live_operation (tree stmt,
> > *** 4010,4016 ****
> > tree def, def_stmt;
> > enum vect_def_type dt;
> >
> > ! if (!STMT_VINFO_LIVE_P (stmt_info))
> > return false;
> >
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > --- 4065,4073 ----
> > tree def, def_stmt;
> > enum vect_def_type dt;
> >
> > ! gcc_assert (STMT_VINFO_LIVE_P (stmt_info));
> > !
> > ! if (STMT_VINFO_DEF_TYPE (stmt_info) == vect_reduction_def)
> > return false;
> >
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > *************** vectorizable_condition (tree stmt, block
> > *** 4123,4138 ****
> > if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > return false;
> >
> > ! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
> >
> > if (STMT_VINFO_LIVE_P (stmt_info))
> > {
> > - /* FORNOW: not yet supported. */
> > if (vect_print_dump_info (REPORT_DETAILS))
> > fprintf (vect_dump, "value used after loop.");
> > return false;
> > }
> >
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > return false;
> >
> > --- 4180,4197 ----
> > if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > return false;
> >
> > ! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
> > ! return false;
> >
> > + /* FORNOW: not yet supported. */
> > if (STMT_VINFO_LIVE_P (stmt_info))
> > {
> > if (vect_print_dump_info (REPORT_DETAILS))
> > fprintf (vect_dump, "value used after loop.");
> > return false;
> > }
> >
> > + /* Is vectorizable conditional operation? */
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > return false;
> >
> > *************** vect_transform_stmt (tree stmt, block_st
> > *** 4225,4334 ****
> > tree orig_stmt_in_pattern;
> > bool done;
> >
> > ! if (STMT_VINFO_RELEVANT_P (stmt_info))
> > {
> > ! switch (STMT_VINFO_TYPE (stmt_info))
> > ! {
> > ! case type_demotion_vec_info_type:
> > ! done = vectorizable_type_demotion (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> >
> > ! case type_promotion_vec_info_type:
> > ! done = vectorizable_type_promotion (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> > !
> > ! case type_conversion_vec_info_type:
> > ! done = vectorizable_conversion (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> > !
> > ! case op_vec_info_type:
> > ! done = vectorizable_operation (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> > !
> > ! case assignment_vec_info_type:
> > ! done = vectorizable_assignment (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> > !
> > ! case load_vec_info_type:
> > ! done = vectorizable_load (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> > !
> > ! case store_vec_info_type:
> > ! done = vectorizable_store (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! if (DR_GROUP_FIRST_DR (stmt_info))
> > ! {
> > ! /* In case of interleaving, the whole chain is vectorized when
the
> > ! last store in the chain is reached. Store stmts before the
last
> > ! one are skipped, and there vec_stmt_info shouldn't be freed
> > ! meanwhile. */
> > ! *strided_store = true;
> > ! if (STMT_VINFO_VEC_STMT (stmt_info))
> > ! is_store = true;
> > }
> > ! else
> > ! is_store = true;
> > ! break;
> > !
> > ! case condition_vec_info_type:
> > ! done = vectorizable_condition (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> >
> > ! case call_vec_info_type:
> > ! done = vectorizable_call (stmt, bsi, &vec_stmt);
> > ! break;
> >
> > ! default:
> > ! if (vect_print_dump_info (REPORT_DETAILS))
> > ! fprintf (vect_dump, "stmt not supported.");
> > ! gcc_unreachable ();
> > ! }
> > !
> > ! gcc_assert (vec_stmt || *strided_store);
> > ! if (vec_stmt)
> > {
> > ! STMT_VINFO_VEC_STMT (stmt_info) = vec_stmt;
> > ! orig_stmt_in_pattern = STMT_VINFO_RELATED_STMT (stmt_info);
> > ! if (orig_stmt_in_pattern)
> > ! {
> > ! stmt_vec_info stmt_vinfo = vinfo_for_stmt
(orig_stmt_in_pattern);
> > ! if (STMT_VINFO_IN_PATTERN_P (stmt_vinfo))
> > ! {
> > ! gcc_assert (STMT_VINFO_RELATED_STMT (stmt_vinfo) == stmt);
> > !
> > ! /* STMT was inserted by the vectorizer to replace a
> > ! computation idiom. ORIG_STMT_IN_PATTERN is a stmt in the
> > ! original sequence that computed this idiom. We need to
> > ! record a pointer to VEC_STMT in the stmt_info of
> > ! ORIG_STMT_IN_PATTERN. See more details in the
> > ! documentation of vect_pattern_recog. */
> > !
> > ! STMT_VINFO_VEC_STMT (stmt_vinfo) = vec_stmt;
> > ! }
> > ! }
> > }
> > }
> >
> > ! if (STMT_VINFO_LIVE_P (stmt_info))
> > {
> > ! switch (STMT_VINFO_TYPE (stmt_info))
> > ! {
> > ! case reduc_vec_info_type:
> > ! done = vectorizable_reduction (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> >
> > ! default:
> > ! done = vectorizable_live_operation (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! }
> > }
> >
> > return is_store;
> > --- 4284,4391 ----
> > tree orig_stmt_in_pattern;
> > bool done;
> >
> > ! switch (STMT_VINFO_TYPE (stmt_info))
> > {
> > ! case type_demotion_vec_info_type:
> > ! done = vectorizable_type_demotion (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> >
> > ! case type_promotion_vec_info_type:
> > ! done = vectorizable_type_promotion (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> > !
> > ! case type_conversion_vec_info_type:
> > ! done = vectorizable_conversion (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> > !
> > ! case induc_vec_info_type:
> > ! done = vectorizable_induction (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> > !
> > ! case op_vec_info_type:
> > ! done = vectorizable_operation (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> > !
> > ! case assignment_vec_info_type:
> > ! done = vectorizable_assignment (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> > !
> > ! case load_vec_info_type:
> > ! done = vectorizable_load (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> > !
> > ! case store_vec_info_type:
> > ! done = vectorizable_store (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! if (DR_GROUP_FIRST_DR (stmt_info))
> > ! {
> > ! /* In case of interleaving, the whole chain is vectorized when
the
> > ! last store in the chain is reached. Store stmts before the
last
> > ! one are skipped, and there vec_stmt_info shouldn't be freed
> > ! meanwhile. */
> > ! *strided_store = true;
> > ! if (STMT_VINFO_VEC_STMT (stmt_info))
> > ! is_store = true;
> > }
> > ! else
> > ! is_store = true;
> > ! break;
> >
> > ! case condition_vec_info_type:
> > ! done = vectorizable_condition (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> > !
> > ! case call_vec_info_type:
> > ! done = vectorizable_call (stmt, bsi, &vec_stmt);
> > ! break;
> > !
> > ! case reduc_vec_info_type:
> > ! done = vectorizable_reduction (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> >
> > ! default:
> > ! if (!STMT_VINFO_LIVE_P (stmt_info))
> > {
> > ! if (vect_print_dump_info (REPORT_DETAILS))
> > ! fprintf (vect_dump, "stmt not supported.");
> > ! gcc_unreachable ();
> > }
> > }
> >
> > ! if (STMT_VINFO_LIVE_P (stmt_info)
> > ! && STMT_VINFO_TYPE (stmt_info) != reduc_vec_info_type)
> > {
> > ! done = vectorizable_live_operation (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! }
> >
> > ! if (vec_stmt)
> > ! {
> > ! STMT_VINFO_VEC_STMT (stmt_info) = vec_stmt;
> > ! orig_stmt_in_pattern = STMT_VINFO_RELATED_STMT (stmt_info);
> > ! if (orig_stmt_in_pattern)
> > ! {
> > ! stmt_vec_info stmt_vinfo = vinfo_for_stmt
(orig_stmt_in_pattern);
> > ! /* STMT was inserted by the vectorizer to replace a
> computation idiom.
> > ! ORIG_STMT_IN_PATTERN is a stmt in the original sequence that
> > ! computed this idiom. We need to record a pointer to VEC_STMT
in
> > ! the stmt_info of ORIG_STMT_IN_PATTERN. See more details in
the
> > ! documentation of vect_pattern_recog. */
> > ! if (STMT_VINFO_IN_PATTERN_P (stmt_vinfo))
> > ! {
> > ! gcc_assert (STMT_VINFO_RELATED_STMT (stmt_vinfo) == stmt);
> > ! STMT_VINFO_VEC_STMT (stmt_vinfo) = vec_stmt;
> > ! }
> > ! }
> > }
> >
> > return is_store;
> > *************** vect_transform_loop (loop_vec_info loop_
> > *** 5156,5166 ****
> > for (i = 0; i < nbbs; i++)
> > {
> > basic_block bb = bbs[i];
> >
> > for (si = bsi_start (bb); !bsi_end_p (si);)
> > {
> > tree stmt = bsi_stmt (si);
> > - stmt_vec_info stmt_info;
> > bool is_store;
> >
> > if (vect_print_dump_info (REPORT_DETAILS))
> > --- 5213,5251 ----
> > for (i = 0; i < nbbs; i++)
> > {
> > basic_block bb = bbs[i];
> > + stmt_vec_info stmt_info;
> > + tree phi;
> > +
> > + for (phi = phi_nodes (bb); phi; phi = PHI_CHAIN (phi))
> > + {
> > + if (vect_print_dump_info (REPORT_DETAILS))
> > + {
> > + fprintf (vect_dump, "------>vectorizing phi: ");
> > + print_generic_expr (vect_dump, phi, TDF_SLIM);
> > + }
> > + stmt_info = vinfo_for_stmt (phi);
> > + if (!stmt_info)
> > + continue;
> > + if (!STMT_VINFO_RELEVANT_P (stmt_info)
> > + && !STMT_VINFO_LIVE_P (stmt_info))
> > + continue;
> > +
> > + if ((TYPE_VECTOR_SUBPARTS (STMT_VINFO_VECTYPE (stmt_info))
> > + != (unsigned HOST_WIDE_INT) vectorization_factor)
> > + && vect_print_dump_info (REPORT_DETAILS))
> > + fprintf (vect_dump, "multiple-types.");
> > +
> > + if (STMT_VINFO_DEF_TYPE (stmt_info) == vect_induction_def)
> > + {
> > + if (vect_print_dump_info (REPORT_DETAILS))
> > + fprintf (vect_dump, "transform phi.");
> > + vect_transform_stmt (phi, NULL, NULL);
> > + }
> > + }
> >
> > for (si = bsi_start (bb); !bsi_end_p (si);)
> > {
> > tree stmt = bsi_stmt (si);
> > bool is_store;
> >
> > if (vect_print_dump_info (REPORT_DETAILS))
> > *************** vect_transform_loop (loop_vec_info loop_
> > *** 5177,5182 ****
> > --- 5262,5268 ----
> > continue;
> > }
> >
> > + gcc_assert (STMT_VINFO_VECTYPE (stmt_info));
> > if ((TYPE_VECTOR_SUBPARTS (STMT_VINFO_VECTYPE (stmt_info))
> > != (unsigned HOST_WIDE_INT) vectorization_factor)
> > && vect_print_dump_info (REPORT_DETAILS))
> > Index: tree-ssa-reassoc.c
> > ===================================================================
> > *** tree-ssa-reassoc.c (revision 123103)
> > --- tree-ssa-reassoc.c (working copy)
> > *************** execute_reassoc (void)
> > *** 1476,1485 ****
> > return 0;
> > }
> >
> > struct tree_opt_pass pass_reassoc =
> > {
> > "reassoc", /* name */
> > ! NULL, /* gate */
> > execute_reassoc, /* execute */
> > NULL, /* sub */
> > NULL, /* next */
> > --- 1476,1491 ----
> > return 0;
> > }
> >
> > + static bool
> > + gate_reassoc (void)
> > + {
> > + return flag_tree_reassoc != 0;
> > + }
> > +
> > struct tree_opt_pass pass_reassoc =
> > {
> > "reassoc", /* name */
> > ! gate_reassoc, /* gate */
> > execute_reassoc, /* execute */
> > NULL, /* sub */
> > NULL, /* next */
>
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 3/5
2007-04-02 9:48 ` Zdenek Dvorak
` (4 preceding siblings ...)
2007-04-11 11:51 ` [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 4/5 Dorit Nuzman
@ 2007-04-11 11:51 ` Dorit Nuzman
2007-04-12 12:11 ` Zdenek Dvorak
2007-04-11 11:51 ` [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 5/5 Dorit Nuzman
6 siblings, 1 reply; 18+ messages in thread
From: Dorit Nuzman @ 2007-04-11 11:51 UTC (permalink / raw)
To: Zdenek Dvorak; +Cc: gcc-patches
[-- Attachment #1: Type: text/plain, Size: 1223 bytes --]
Zdenek Dvorak <rakdver@atrey.karlin.mff.cuni.cz> wrote on 02/04/2007
12:48:12:
> Hello,
>
> I will look at the patch. However, would it be possible to split it
> into smaller parts? The changes seem to be quite independent,
>
> Zdenek
>
> > This is exactly the same patch as was committed to autovect last week:
> > http://gcc.gnu.org/ml/gcc-patches/2007-03/msg01904.html
> > (but relative to mainline, and after testig on mainline).
> >
> > These are a few cleanups (really technical things, no new
functionality),
> > some of which will be also useful for outer-loop vectorization:
> >
...
> >
> > o A bunch of other cleanups (that are not necessarily inspired by
> > outer-loop vectorization): Factor out code from
> > vect_mark_stmts_to_be_vectorized to process_use;
Tested on the vectorizer testcases
Bootstrapped and passed full testing on powerpc-linux together with the
other parts of this patch.
thanks,
dorit
ChangeLog:
* tree-vect-analyze.c (process_use): New function.
(vect_mark_stmts_to_be_vectorized): Factor out code to process_use.
Check phis in all bbs.
* tree-vectorizer.c (vect_is_simple_use): Remove redundant checks.
(See attached file: vect_cleanups.3.process_use.txt)
[-- Attachment #2: vect_cleanups.3.process_use.txt --]
[-- Type: text/plain, Size: 12163 bytes --]
*** tree-vect-analyze.c.2 2007-04-10 21:54:14.000000000 +0300
--- tree-vect-analyze.c.3 2007-04-10 22:19:13.000000000 +0300
*************** vect_stmt_relevant_p (tree stmt, loop_ve
*** 2191,2196 ****
--- 2191,2271 ----
}
+ /*
+ Function process_use.
+
+ Inputs:
+ - a USE in STMT in a loop represented by LOOP_VINFO
+ - LIVE_P, RELEVANT - enum values to be set in the STMT_VINFO of the stmt
+ that defined USE. This is dont by calling mark_relevant and passing it
+ the WORKLIST (to add DEF_STMT to the WORKlist in case itis relevant).
+
+ Outputs:
+ Generally, LIVE_P and RELEVANT are used to define the liveness and
+ relevance info of the DEF_STMT of this USE:
+ STMT_VINFO_LIVE_P (DEF_STMT_info) <-- live_p
+ STMT_VINFO_RELEVANT (DEF_STMT_info) <-- relevant
+ Exceptions:
+ - case 1: If USE is used only for address computations (e.g. array indexing),
+ which does not need to be directly vectorized, then the liveness/relevance
+ of the respective DEF_STMT is left unchanged.
+ - case 2: If STMT is a reduction phi and DEF_STMT is a reduction stmt, we
+ skip DEF_STMT cause it had already been processed.
+
+ Return true if everyting is as expected. Return false otherwise. */
+
+ static bool
+ process_use (tree stmt, tree use, loop_vec_info loop_vinfo, bool live_p,
+ enum vect_relevant relevant, VEC(tree,heap) **worklist)
+ {
+ struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
+ stmt_vec_info stmt_vinfo = vinfo_for_stmt (stmt);
+ stmt_vec_info dstmt_vinfo;
+ basic_block def_bb;
+ tree def, def_stmt;
+ enum vect_def_type dt;
+
+ /* case 1: we are only interested in uses that need to be vectorized. Uses
+ that are used for address computation are not considered relevant. */
+ if (!exist_non_indexing_operands_for_use_p (use, stmt))
+ return true;
+
+ if (!vect_is_simple_use (use, loop_vinfo, &def_stmt, &def, &dt))
+ {
+ if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
+ fprintf (vect_dump, "not vectorized: unsupported use in stmt.");
+ return false;
+ }
+
+ if (!def_stmt || IS_EMPTY_STMT (def_stmt))
+ return true;
+
+ def_bb = bb_for_stmt (def_stmt);
+ if (!flow_bb_inside_loop_p (loop, def_bb))
+ return true;
+
+ /* case 2: A reduction phi defining a reduction stmt (DEF_STMT). DEF_STMT
+ must have already been processed, so we just check that everything is as
+ expected, and we are done. */
+ dstmt_vinfo = vinfo_for_stmt (def_stmt);
+ if (TREE_CODE (stmt) == PHI_NODE
+ && STMT_VINFO_DEF_TYPE (stmt_vinfo) == vect_reduction_def
+ && TREE_CODE (def_stmt) != PHI_NODE
+ && STMT_VINFO_DEF_TYPE (dstmt_vinfo) == vect_reduction_def)
+ {
+ if (STMT_VINFO_IN_PATTERN_P (dstmt_vinfo))
+ dstmt_vinfo = vinfo_for_stmt (STMT_VINFO_RELATED_STMT (dstmt_vinfo));
+ gcc_assert (STMT_VINFO_RELEVANT (dstmt_vinfo) < vect_used_by_reduction);
+ gcc_assert (STMT_VINFO_LIVE_P (dstmt_vinfo)
+ || STMT_VINFO_RELEVANT (dstmt_vinfo) > vect_unused_in_loop);
+ return true;
+ }
+
+ vect_mark_relevant (worklist, def_stmt, relevant, live_p);
+ return true;
+ }
+
+
/* Function vect_mark_stmts_to_be_vectorized.
Not all stmts in the loop need to be vectorized. For example:
*************** vect_mark_stmts_to_be_vectorized (loop_v
*** 2204,2220 ****
basic_block *bbs = LOOP_VINFO_BBS (loop_vinfo);
unsigned int nbbs = loop->num_nodes;
block_stmt_iterator si;
! tree stmt, use;
stmt_ann_t ann;
- ssa_op_iter iter;
unsigned int i;
stmt_vec_info stmt_vinfo;
basic_block bb;
tree phi;
bool live_p;
enum vect_relevant relevant;
- tree def, def_stmt;
- enum vect_def_type dt;
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "=== vect_mark_stmts_to_be_vectorized ===");
--- 2293,2306 ----
basic_block *bbs = LOOP_VINFO_BBS (loop_vinfo);
unsigned int nbbs = loop->num_nodes;
block_stmt_iterator si;
! tree stmt;
stmt_ann_t ann;
unsigned int i;
stmt_vec_info stmt_vinfo;
basic_block bb;
tree phi;
bool live_p;
enum vect_relevant relevant;
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "=== vect_mark_stmts_to_be_vectorized ===");
*************** vect_mark_stmts_to_be_vectorized (loop_v
*** 2261,2359 ****
}
}
-
/* 2. Process_worklist */
-
while (VEC_length (tree, worklist) > 0)
{
! stmt = VEC_pop (tree, worklist);
if (vect_print_dump_info (REPORT_DETAILS))
{
fprintf (vect_dump, "worklist: examine stmt: ");
print_generic_expr (vect_dump, stmt, TDF_SLIM);
}
! /* Examine the USEs of STMT. For each ssa-name USE that is defined
! in the loop, mark the stmt that defines it (DEF_STMT) as
! relevant/irrelevant and live/dead according to the liveness and
! relevance properties of STMT.
! */
!
ann = stmt_ann (stmt);
stmt_vinfo = vinfo_for_stmt (stmt);
-
relevant = STMT_VINFO_RELEVANT (stmt_vinfo);
live_p = STMT_VINFO_LIVE_P (stmt_vinfo);
/* Generally, the liveness and relevance properties of STMT are
! propagated to the DEF_STMTs of its USEs:
! STMT_VINFO_LIVE_P (DEF_STMT_info) <-- live_p
! STMT_VINFO_RELEVANT (DEF_STMT_info) <-- relevant
!
! Exceptions:
!
! (case 1)
! If USE is used only for address computations (e.g. array indexing),
! which does not need to be directly vectorized, then the
! liveness/relevance of the respective DEF_STMT is left unchanged.
!
! (case 2)
! If STMT has been identified as defining a reduction variable, then
! we want to set liveness/relevance as follows:
! STMT_VINFO_LIVE_P (DEF_STMT_info) <-- false
! STMT_VINFO_RELEVANT (DEF_STMT_info) <-- vect_used_by_reduction
! because even though STMT is classified as live (since it defines a
! value that is used across loop iterations) and irrelevant (since it
! is not used inside the loop), it will be vectorized, and therefore
! the corresponding DEF_STMTs need to marked as relevant.
! We distinguish between two kinds of relevant stmts - those that are
! used by a reduction computation, and those that are (also) used by
! a regular computation. This allows us later on to identify stmts
! that are used solely by a reduction, and therefore the order of
! the results that they produce does not have to be kept.
! */
- /* case 2.2: */
if (STMT_VINFO_DEF_TYPE (stmt_vinfo) == vect_reduction_def)
! {
relevant = vect_used_by_reduction;
! live_p = false;
}
! i = 0;
! FOR_EACH_SSA_TREE_OPERAND (use, stmt, iter, SSA_OP_USE)
{
! if (vect_print_dump_info (REPORT_DETAILS))
! {
! fprintf (vect_dump, "worklist: examine use %d: ", i++);
! print_generic_expr (vect_dump, use, TDF_SLIM);
! }
!
! /* case 1: we are only interested in uses that need to be vectorized.
! Uses that are used for address computation are not considered
! relevant.
! */
! if (!exist_non_indexing_operands_for_use_p (use, stmt))
! continue;
!
! if (!vect_is_simple_use (use, loop_vinfo, &def_stmt, &def, &dt))
{
- if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
- fprintf (vect_dump, "not vectorized: unsupported use in stmt.");
VEC_free (tree, heap, worklist);
return false;
! }
!
! if (!def_stmt || IS_EMPTY_STMT (def_stmt))
! continue;
!
! bb = bb_for_stmt (def_stmt);
! if (!flow_bb_inside_loop_p (loop, bb))
! continue;
! vect_mark_relevant (&worklist, def_stmt, relevant, live_p);
}
! } /* while worklist */
VEC_free (tree, heap, worklist);
return true;
--- 2336,2417 ----
}
}
/* 2. Process_worklist */
while (VEC_length (tree, worklist) > 0)
{
! use_operand_p use_p;
! ssa_op_iter iter;
+ stmt = VEC_pop (tree, worklist);
if (vect_print_dump_info (REPORT_DETAILS))
{
fprintf (vect_dump, "worklist: examine stmt: ");
print_generic_expr (vect_dump, stmt, TDF_SLIM);
}
! /* Examine the USEs of STMT. For each USE, mark the stmt that defines it
! (DEF_STMT) as relevant/irrelevant and live/dead according to the
! liveness and relevance properties of STMT. */
ann = stmt_ann (stmt);
stmt_vinfo = vinfo_for_stmt (stmt);
relevant = STMT_VINFO_RELEVANT (stmt_vinfo);
live_p = STMT_VINFO_LIVE_P (stmt_vinfo);
/* Generally, the liveness and relevance properties of STMT are
! propagated as is to the DEF_STMTs of its USEs:
! live_p <-- STMT_VINFO_LIVE_P (STMT_VINFO)
! relevant <-- STMT_VINFO_RELEVANT (STMT_VINFO)
!
! One exception is when STMT has been identified as defining a reduction
! variable; in this case we set the liveness/relevance as follows:
! live_p = false
! relevant = vect_used_by_reduction
! This is because we distinguish between two kinds of relevant stmts -
! those that are used by a reduction computation, and those that are
! (also) used by a regular computation. This allows us later on to
! identify stmts that are used solely by a reduction, and therefore the
! order of the results that they produce does not have to be kept.
!
! Reduction phis are expected to be used by a reduction stmt; Other
! reduction stmts are expected to be unused in the loop. These are the
! expected values of "relevant" for reduction phis/stmts in the loop:
!
! relevance: phi stmt
! vect_unused_in_loop ok
! vect_used_by_reduction ok
! vect_used_in_loop */
if (STMT_VINFO_DEF_TYPE (stmt_vinfo) == vect_reduction_def)
! {
! switch (relevant)
! {
! case vect_unused_in_loop:
! gcc_assert (TREE_CODE (stmt) != PHI_NODE);
! break;
! case vect_used_by_reduction:
! if (TREE_CODE (stmt) == PHI_NODE)
! break;
! case vect_used_in_loop:
! default:
! if (vect_print_dump_info (REPORT_DETAILS))
! fprintf (vect_dump, "unsupported use of reduction.");
! VEC_free (tree, heap, worklist);
! return false;
! }
relevant = vect_used_by_reduction;
! live_p = false;
}
! FOR_EACH_PHI_OR_STMT_USE (use_p, stmt, iter, SSA_OP_USE)
{
! tree op = USE_FROM_PTR (use_p);
! if (!process_use (stmt, op, loop_vinfo, live_p, relevant, &worklist))
{
VEC_free (tree, heap, worklist);
return false;
! }
}
! } /* while worklist */
VEC_free (tree, heap, worklist);
return true;
Index: tree-vectorizer.c
===================================================================
*** tree-vectorizer.c (revision 123694)
--- tree-vectorizer.c (working copy)
*************** vect_is_simple_use (tree operand, loop_v
*** 1714,1728 ****
return false;
}
- /* stmts inside the loop that have been identified as performing
- a reduction operation cannot have uses in the loop. */
- if (*dt == vect_reduction_def && TREE_CODE (*def_stmt) != PHI_NODE)
- {
- if (vect_print_dump_info (REPORT_DETAILS))
- fprintf (vect_dump, "reduction used in loop.");
- return false;
- }
-
if (vect_print_dump_info (REPORT_DETAILS))
fprintf (vect_dump, "type of def: %d.",*dt);
--- 1714,1719 ----
*************** vect_is_simple_use (tree operand, loop_v
*** 1730,1742 ****
{
case PHI_NODE:
*def = PHI_RESULT (*def_stmt);
- gcc_assert (*dt == vect_induction_def || *dt == vect_reduction_def
- || *dt == vect_invariant_def);
break;
case GIMPLE_MODIFY_STMT:
*def = GIMPLE_STMT_OPERAND (*def_stmt, 0);
- gcc_assert (*dt == vect_loop_def || *dt == vect_invariant_def);
break;
default:
--- 1721,1730 ----
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 2/5
2007-04-11 11:51 ` [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 2/5 Dorit Nuzman
@ 2007-04-11 23:53 ` Zdenek Dvorak
2007-04-14 18:29 ` Dorit Nuzman
0 siblings, 1 reply; 18+ messages in thread
From: Zdenek Dvorak @ 2007-04-11 23:53 UTC (permalink / raw)
To: Dorit Nuzman; +Cc: gcc-patches
Hello,
> * tree-vectorizer.h (stmt_vec_info_type): Add enum value
> induc_vec_info_type.
> (vectorizable_induction): New function declaration.
> * tree-vect-transform.c (get_initial_def_for_induction): No need to
> check if already vectorized. Find first place in BB where new stmts
> can be inserted. Takes only one argument.
> (vectorizable_induction): New function.
> (vect_transform_stmt): Add case for induc_vec_info_type to call
> vectorizable_induction.
> (vect_transform_loop): Consider phis for vectorization.
> * tree-vect-analyze.c (vect_determine_vectorization_factor):
> Simplify
> condition.
> (analyze_operations): Call vectorizable_induction when analyzing
> phis.
> Fix comment.
> (vect_mark_stmts_to_be_vectorized): Remove redundant checks.
> (vect_mark_relevant): Include phis in relevance analysis.
> (vect_mark_stmts_to_be_vectorize): Likewise.
> ! /* Find the first insertion point in the BB. */
> ! for (si = bsi_start (bb); !bsi_end_p (si); bsi_next (&si))
> {
> ! stmt = bsi_stmt (si);
> ! if (TREE_CODE (stmt) != LABEL_EXPR)
> ! break;
> }
>
use bsi_after_labels. Otherwise OK.
Zdenek
> * tree-vect-patterns.c (widened_name_p): Remove obsolete asserts.
It would be better to write this as "Remove duplicate asserts." (I
actually spent a few minutes trying to figure out how the patch makes
the checks obsolete, before I noticed that :-)
> Index: tree-vect-patterns.c
> ===================================================================
> *** tree-vect-patterns.c (revision 123694)
> --- tree-vect-patterns.c (working copy)
> *************** widened_name_p (tree name, tree use_stmt
> *** 109,118 ****
> if (!vect_is_simple_use (oprnd0, loop_vinfo, &dummy, &dummy, &dt))
> return false;
>
> - if (dt != vect_invariant_def && dt != vect_constant_def
> - && dt != vect_loop_def)
> - return false;
> -
> return true;
> }
>
> --- 109,114 ----
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 3/5
2007-04-11 11:51 ` [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 3/5 Dorit Nuzman
@ 2007-04-12 12:11 ` Zdenek Dvorak
2007-04-14 18:54 ` Dorit Nuzman
0 siblings, 1 reply; 18+ messages in thread
From: Zdenek Dvorak @ 2007-04-12 12:11 UTC (permalink / raw)
To: Dorit Nuzman; +Cc: gcc-patches
Hello,
> > > o A bunch of other cleanups (that are not necessarily inspired by
> > > outer-loop vectorization): Factor out code from
> > > vect_mark_stmts_to_be_vectorized to process_use;
>
> Tested on the vectorizer testcases
> Bootstrapped and passed full testing on powerpc-linux together with the
> other parts of this patch.
>
> thanks,
> dorit
>
> ChangeLog:
>
> * tree-vect-analyze.c (process_use): New function.
> (vect_mark_stmts_to_be_vectorized): Factor out code to process_use.
> Check phis in all bbs.
> * tree-vectorizer.c (vect_is_simple_use): Remove redundant checks.
> *************** vect_is_simple_use (tree operand, loop_v
> *** 1730,1742 ****
> {
> case PHI_NODE:
> *def = PHI_RESULT (*def_stmt);
> - gcc_assert (*dt == vect_induction_def || *dt == vect_reduction_def
> - || *dt == vect_invariant_def);
> break;
>
> case GIMPLE_MODIFY_STMT:
> *def = GIMPLE_STMT_OPERAND (*def_stmt, 0);
> - gcc_assert (*dt == vect_loop_def || *dt == vect_invariant_def);
> break;
>
> default:
> --- 1721,1730 ----
I would prefer to keep these asserts, unless there is some trivial
reason that makes them redundant (that I do not see). Otherwise OK.
Zdenek
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 4/5
2007-04-11 11:51 ` [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 4/5 Dorit Nuzman
@ 2007-04-12 12:17 ` Zdenek Dvorak
0 siblings, 0 replies; 18+ messages in thread
From: Zdenek Dvorak @ 2007-04-12 12:17 UTC (permalink / raw)
To: Dorit Nuzman; +Cc: gcc-patches
Hello,
> * tree-vect-transform.c (get_initial_def_for_reduction): Clean away
> the unused code for reduction without adjust-in-epilog to simplify
> the
> function.
OK.
Zdenek
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 5/5
2007-04-11 11:51 ` [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 5/5 Dorit Nuzman
@ 2007-04-12 12:22 ` Zdenek Dvorak
2007-04-12 14:23 ` Daniel Berlin
0 siblings, 1 reply; 18+ messages in thread
From: Zdenek Dvorak @ 2007-04-12 12:22 UTC (permalink / raw)
To: Dorit Nuzman; +Cc: gcc-patches
Hello,
> * tree-vectorizer.c (destroy_loop_vec_info): Set loop->aux to NULL.
> * tree-vect-analyze.c (vect_analyze_loop_form): Set loop->aux.
>
> * tree-vectorizer.h (NITERS_KNOWN_P): New.
> * tree-vect-analyze.c (vect_analyze_loop_form): Call NITERS_KNOWN_P
> instead of LOOP_VINFO_INT_NITERS to avoid having to geneate
> loop_info.
>
> * tree-vect-analyze.c (vect_determine_vectorization_factor): Add
> dump print.
> (vect_analyze_operations): Fix indenetation. Fix a comment. Fix a
indentation
> print message.
> (vect_analyze_scalar_cycles): Fix indentation.
> (vect_enhance_data_refs_alignment): Fix check in case of peeling.
> (vect_mark_relevant): Include phis in relevance analysis.
>
> * tree-vect-transform.c (vect_transform_loop): Add an assert.
>
> * tree-ssa-reassoc.c (gate_reassoc): New function.
> * common.opt (tree-reassoc): New flag.
> (pass_reassoc): Add gate.
OK. I cannot approve adding the -ftree-reassoc flag, though, as it is
not part of loop optimizer.
Zdenek
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 5/5
2007-04-12 12:22 ` Zdenek Dvorak
@ 2007-04-12 14:23 ` Daniel Berlin
0 siblings, 0 replies; 18+ messages in thread
From: Daniel Berlin @ 2007-04-12 14:23 UTC (permalink / raw)
To: Zdenek Dvorak; +Cc: Dorit Nuzman, gcc-patches
On 4/12/07, Zdenek Dvorak <rakdver@atrey.karlin.mff.cuni.cz> wrote:
> Hello,
>
> > * tree-vectorizer.c (destroy_loop_vec_info): Set loop->aux to NULL.
> > * tree-vect-analyze.c (vect_analyze_loop_form): Set loop->aux.
> >
> > * tree-vectorizer.h (NITERS_KNOWN_P): New.
> > * tree-vect-analyze.c (vect_analyze_loop_form): Call NITERS_KNOWN_P
> > instead of LOOP_VINFO_INT_NITERS to avoid having to geneate
> > loop_info.
> >
> > * tree-vect-analyze.c (vect_determine_vectorization_factor): Add
> > dump print.
> > (vect_analyze_operations): Fix indenetation. Fix a comment. Fix a
>
> indentation
>
> > print message.
> > (vect_analyze_scalar_cycles): Fix indentation.
> > (vect_enhance_data_refs_alignment): Fix check in case of peeling.
> > (vect_mark_relevant): Include phis in relevance analysis.
> >
> > * tree-vect-transform.c (vect_transform_loop): Add an assert.
> >
> > * tree-ssa-reassoc.c (gate_reassoc): New function.
> > * common.opt (tree-reassoc): New flag.
> > (pass_reassoc): Add gate.
>
> OK. I cannot approve adding the -ftree-reassoc flag, though, as it is
> not part of loop optimizer.
>
> Zdenek
>
Right. You should submit that separately. Also, you need to update
invoke.texi to document the new option.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 2/5
2007-04-11 23:53 ` Zdenek Dvorak
@ 2007-04-14 18:29 ` Dorit Nuzman
2007-04-14 18:56 ` Zdenek Dvorak
0 siblings, 1 reply; 18+ messages in thread
From: Dorit Nuzman @ 2007-04-14 18:29 UTC (permalink / raw)
To: Zdenek Dvorak; +Cc: gcc-patches
Zdenek Dvorak <rakdver@atrey.karlin.mff.cuni.cz> wrote on 12/04/2007
02:53:16:
...
>
> > ! /* Find the first insertion point in the BB. */
> > ! for (si = bsi_start (bb); !bsi_end_p (si); bsi_next (&si))
> > {
> > ! stmt = bsi_stmt (si);
> > ! if (TREE_CODE (stmt) != LABEL_EXPR)
> > ! break;
> > }
> >
>
> use bsi_after_labels. Otherwise OK.
>
I knew there had to be something like that... :-)
> Zdenek
>
> > * tree-vect-patterns.c (widened_name_p): Remove obsolete
asserts.
>
> It would be better to write this as "Remove duplicate asserts." (I
> actually spent a few minutes trying to figure out how the patch makes
> the checks obsolete, before I noticed that :-)
>
about:
- if (dt != vect_invariant_def && dt != vect_constant_def
- && dt != vect_loop_def)
- return false;
-
this check is not really duplicate; the def type can also be an induction
def, which is be fine, it's just something we didn't expect to have before.
So the intent was really that there's no need to be too strict here, and
that the checks could be relaxed (i.e. removed). Here is, for example, a
testcase that is affected by removing this part of the patch - with the
patch, a widening summation pattern is detected on powerpc, and without it,
it is vectorized using vector unpacking instead:
unsigned short i;
unsigned int intsum = 0;
/* widenning sum: sum shorts into int. */
for (i = 0; i < N; i++)
{
intsum += i;
}
(I guess I should go ahead and add this testcase with the patch?)
thanks,
dorit
> > Index: tree-vect-patterns.c
> > ===================================================================
> > *** tree-vect-patterns.c (revision 123694)
> > --- tree-vect-patterns.c (working copy)
> > *************** widened_name_p (tree name, tree use_stmt
> > *** 109,118 ****
> > if (!vect_is_simple_use (oprnd0, loop_vinfo, &dummy, &dummy, &dt))
> > return false;
> >
> > - if (dt != vect_invariant_def && dt != vect_constant_def
> > - && dt != vect_loop_def)
> > - return false;
> > -
> > return true;
> > }
> >
> > --- 109,114 ----
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 3/5
2007-04-12 12:11 ` Zdenek Dvorak
@ 2007-04-14 18:54 ` Dorit Nuzman
2007-04-14 19:32 ` Zdenek Dvorak
0 siblings, 1 reply; 18+ messages in thread
From: Dorit Nuzman @ 2007-04-14 18:54 UTC (permalink / raw)
To: Zdenek Dvorak; +Cc: gcc-patches
Zdenek Dvorak <rakdver@atrey.karlin.mff.cuni.cz> wrote on 12/04/2007
15:11:22:
> > *************** vect_is_simple_use (tree operand, loop_v
> > *** 1730,1742 ****
> > {
> > case PHI_NODE:
> > *def = PHI_RESULT (*def_stmt);
> > - gcc_assert (*dt == vect_induction_def || *dt ==
vect_reduction_def
> > - || *dt == vect_invariant_def);
> > break;
> >
> > case GIMPLE_MODIFY_STMT:
> > *def = GIMPLE_STMT_OPERAND (*def_stmt, 0);
> > - gcc_assert (*dt == vect_loop_def || *dt == vect_invariant_def);
> > break;
> >
> > default:
> > --- 1721,1730 ----
>
> I would prefer to keep these asserts, unless there is some trivial
> reason that makes them redundant (that I do not see). Otherwise OK.
>
The first assert is redundant/duplicate because currently we have phis only
in the loop header, and they can define only reductions/inductions, or
something invariant if it's some weird case that hasn't been cleaned away
for some reason. It is not made redundant by this patch, and there's no
problem in leaving this assert in for now, but when we look at outer-loops
we'll have phis in the loop that aren't loop-header phis (the loop-closed
phis of the inner-loop), and these can define other kinds of computations,
so at that point this assert will be wrong.
The second assert is now wrong, because with this patch we put phis in the
worklist during vect_mark_stmts_to_be_vectorized, so we get to call
vect_is_simple_use on their arguments, where we'd fail on this assert,
cause the argument of a phi-node can be defined by a reduction/induction
stmt (and adding these possibilities to the assert would just make us check
that the def in one of all possible types, which would make it redundant,
and that's why I described it as a "Remove redundant checks", which is not
the most accurate description...)
So I'll leave the first assert in, remove the second assert, and fix the
ChangeLog entry to "Remove a no longer relavant assert"?
thanks,
dorit
> Zdenek
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 2/5
2007-04-14 18:29 ` Dorit Nuzman
@ 2007-04-14 18:56 ` Zdenek Dvorak
0 siblings, 0 replies; 18+ messages in thread
From: Zdenek Dvorak @ 2007-04-14 18:56 UTC (permalink / raw)
To: Dorit Nuzman; +Cc: gcc-patches
Hello,
> > It would be better to write this as "Remove duplicate asserts." (I
> > actually spent a few minutes trying to figure out how the patch makes
> > the checks obsolete, before I noticed that :-)
> >
>
> about:
> - if (dt != vect_invariant_def && dt != vect_constant_def
> - && dt != vect_loop_def)
> - return false;
> -
>
> this check is not really duplicate; the def type can also be an induction
> def, which is be fine, it's just something we didn't expect to have before.
> So the intent was really that there's no need to be too strict here, and
> that the checks could be relaxed (i.e. removed). Here is, for example, a
> testcase that is affected by removing this part of the patch - with the
> patch, a widening summation pattern is detected on powerpc, and without it,
> it is vectorized using vector unpacking instead:
>
> unsigned short i;
> unsigned int intsum = 0;
> /* widenning sum: sum shorts into int. */
> for (i = 0; i < N; i++)
> {
> intsum += i;
> }
>
> (I guess I should go ahead and add this testcase with the patch?)
it would not harm, definitely.
Zdenek
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 3/5
2007-04-14 18:54 ` Dorit Nuzman
@ 2007-04-14 19:32 ` Zdenek Dvorak
0 siblings, 0 replies; 18+ messages in thread
From: Zdenek Dvorak @ 2007-04-14 19:32 UTC (permalink / raw)
To: Dorit Nuzman; +Cc: gcc-patches
Hello,
> So I'll leave the first assert in, remove the second assert, and fix the
> ChangeLog entry to "Remove a no longer relavant assert"?
OK.
Zdenek
^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2007-04-14 18:56 UTC | newest]
Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2007-04-02 8:43 [patch] vectorizer cleanups (and prep for outer-loop vectorization) Dorit Nuzman
2007-04-02 9:48 ` Zdenek Dvorak
2007-04-02 9:55 ` Zdenek Dvorak
2007-04-11 11:51 ` [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 2/5 Dorit Nuzman
2007-04-11 23:53 ` Zdenek Dvorak
2007-04-14 18:29 ` Dorit Nuzman
2007-04-14 18:56 ` Zdenek Dvorak
2007-04-11 11:51 ` [patch] vectorizer cleanups (and prep for outer-loop vectorization) Dorit Nuzman
2007-04-11 11:51 ` [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 1/5 Dorit Nuzman
2007-04-11 11:51 ` [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 4/5 Dorit Nuzman
2007-04-12 12:17 ` Zdenek Dvorak
2007-04-11 11:51 ` [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 3/5 Dorit Nuzman
2007-04-12 12:11 ` Zdenek Dvorak
2007-04-14 18:54 ` Dorit Nuzman
2007-04-14 19:32 ` Zdenek Dvorak
2007-04-11 11:51 ` [patch] vectorizer cleanups (and prep for outer-loop vectorization) - part 5/5 Dorit Nuzman
2007-04-12 12:22 ` Zdenek Dvorak
2007-04-12 14:23 ` Daniel Berlin
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).