From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 48697 invoked by alias); 28 Aug 2018 11:25:35 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Received: (qmail 48681 invoked by uid 89); 28 Aug 2018 11:25:34 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-11.1 required=5.0 tests=BAYES_00,GIT_PATCH_2,GIT_PATCH_3,KAM_ASCII_DIVIDERS,SPF_PASS autolearn=ham version=3.3.2 spammy= X-HELO: foss.arm.com Received: from usa-sjc-mx-foss1.foss.arm.com (HELO foss.arm.com) (217.140.101.70) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Tue, 28 Aug 2018 11:25:30 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 438FC80D for ; Tue, 28 Aug 2018 04:25:21 -0700 (PDT) Received: from localhost (unknown [10.32.98.51]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 9B0803F557 for ; Tue, 28 Aug 2018 04:25:20 -0700 (PDT) From: Richard Sandiford To: gcc-patches@gcc.gnu.org Mail-Followup-To: gcc-patches@gcc.gnu.org, richard.sandiford@arm.com Subject: [6/6] Link imm uses for pattern stmts References: <87tvnerb5m.fsf@arm.com> Date: Tue, 28 Aug 2018 11:25:00 -0000 In-Reply-To: <87tvnerb5m.fsf@arm.com> (Richard Sandiford's message of "Tue, 28 Aug 2018 12:19:33 +0100") Message-ID: <8736uyraw0.fsf@arm.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-SW-Source: 2018-08/txt/msg01756.txt.bz2 One of the warts of the vectoriser IR is that it doesn't link SSA name uses for pattern statements, leading to complicated special cases in vect_mark_stmts_to_be_vectorized and (especially) vect_detect_hybrid_slp. It also makes it harder to check how an SSA name is used after pattern replacement (something I need for a later patch). This patch adds a mode in which tree-ssa-operands.c can update statements in the same non-invasive way as for debug statements. It then uses this mode to update pattern statements when adding them to a vec_basic_block, so that pattern statements become even more like statements that existed from the outset. 2018-08-28 Richard Sandiford gcc/ * tree-ssa-operands.h (update_stmt_operands): Add a transparent_p argument. * tree-ssa-operands.c (opf_transparent, opf_sticky): New macros. (get_mem_ref_operands, get_tmr_operands): Preserve opf_sticky rather than just opf_no_vops. (get_expr_operands): Preserve opf_sticky bits in the use flags. Assert that opf_no_vops and opf_transparent are already set for the debug statements. Use opf_transparent rather than is_gimple_debug when deciding whether to mark something as having its address taken. (parse_ssa_operands): Add a transparent_p argument. Set the opf_no_vops and opf_transparent flags when the argument is true, or when dealing with debug statements. Check opf_no_vops before adding vuses and vdefs. (build_ssa_operands): Add a transparent_p argument and pass it to parse_ssa_operands. (verify_ssa_operands): Update call to parse_ssa_operands. (update_stmt_operands): Add a transparent_p argument and pass it to build_ssa_operands. * gimple-ssa.h (update_stmt, update_stmt_if_modified) (update_stmt_fn): Add an optional transparent_p parameter and update call to update_stmt_operands. * tree-vect-slp.c (vect_detect_hybrid_slp_1): Delete. (vect_detect_hybrid_slp_2): Likewise. (vect_detect_hybrid_slp): Don't treat pattern statements specially. * tree-vect-stmts.c (vect_mark_stmts_to_be_vectorized): Likewise. (vect_remove_dead_scalar_stmts): Remove pattern statements from the containing vec_info. * tree-vectorizer.h (vec_info::add_pattern_stmt_to_block): Declare. * tree-vectorizer.c (vec_basic_block::add_to_end) (vec_basic_block::add_before): Call add_pattern_stmt_to_block. (vec_basic_block::remove, vec_info::remove_stmt): Call remove_pattern_stmt_from_block. (vec_basic_block::add_pattern_stmt_to_block): New function. (remove_pattern_stmt_from_block): Likewise. (vec_info::free_stmt_vec_info): Handle pattern statements. (vec_info::lookup_single_use): Accept pattern statements as well as original statements. Ignore uses in statements that have been replaced by a pattern statement. * tree-vect-patterns.c (vect_init_pattern_stmt): Don't call gimple_set_bb. (vect_look_through_possible_promotion): Use vinfo->lookup_single_use instead of has_single_use. Track single uses for pattern statements too. Index: gcc/tree-ssa-operands.h =================================================================== --- gcc/tree-ssa-operands.h 2018-05-02 08:37:32.405761509 +0100 +++ gcc/tree-ssa-operands.h 2018-08-28 12:05:19.262917177 +0100 @@ -94,7 +94,7 @@ extern void init_ssa_operands (struct fu extern void fini_ssa_operands (struct function *); extern bool verify_ssa_operands (struct function *, gimple *stmt); extern void free_stmt_operands (struct function *, gimple *); -extern void update_stmt_operands (struct function *, gimple *); +extern void update_stmt_operands (struct function *, gimple *, bool); extern void swap_ssa_operands (gimple *, tree *, tree *); extern bool verify_imm_links (FILE *f, tree var); Index: gcc/tree-ssa-operands.c =================================================================== --- gcc/tree-ssa-operands.c 2018-08-28 11:25:46.242879876 +0100 +++ gcc/tree-ssa-operands.c 2018-08-28 12:05:19.262917177 +0100 @@ -99,6 +99,14 @@ #define opf_not_non_addressable (1 << 4) /* Operand is having its address taken. */ #define opf_address_taken (1 << 5) +/* Operand must have no effect on code generation. This is used for + debug statements, and also for statements that a pass has no intention + of adding to the block in their current form. */ +#define opf_transparent (1 << 6) + +/* Flags that must never be dropped. */ +#define opf_sticky (opf_no_vops | opf_transparent) + /* Array for building all the use operands. */ static vec build_uses; @@ -590,7 +598,7 @@ get_mem_ref_operands (struct function *f /* If requested, add a USE operand for the base pointer. */ get_expr_operands (fn, stmt, pptr, opf_non_addressable | opf_use - | (flags & (opf_no_vops|opf_not_non_addressable))); + | (flags & (opf_sticky | opf_not_non_addressable))); } @@ -605,11 +613,11 @@ get_tmr_operands (struct function *fn, g /* First record the real operands. */ get_expr_operands (fn, stmt, - &TMR_BASE (expr), opf_use | (flags & opf_no_vops)); + &TMR_BASE (expr), opf_use | (flags & opf_sticky)); get_expr_operands (fn, stmt, - &TMR_INDEX (expr), opf_use | (flags & opf_no_vops)); + &TMR_INDEX (expr), opf_use | (flags & opf_sticky)); get_expr_operands (fn, stmt, - &TMR_INDEX2 (expr), opf_use | (flags & opf_no_vops)); + &TMR_INDEX2 (expr), opf_use | (flags & opf_sticky)); add_virtual_operand (fn, stmt, flags); } @@ -703,14 +711,14 @@ get_expr_operands (struct function *fn, enum tree_code code; enum tree_code_class codeclass; tree expr = *expr_p; - int uflags = opf_use; + int uflags = opf_use | (flags & opf_sticky); + gcc_checking_assert (!is_gimple_debug (stmt) + || ((flags & opf_no_vops) + && (flags & opf_transparent))); if (expr == NULL) return; - if (is_gimple_debug (stmt)) - uflags |= (flags & opf_no_vops); - code = TREE_CODE (expr); codeclass = TREE_CODE_CLASS (code); @@ -723,7 +731,7 @@ get_expr_operands (struct function *fn, resolution). */ if ((!(flags & opf_non_addressable) || (flags & opf_not_non_addressable)) - && !is_gimple_debug (stmt)) + && !(flags & opf_transparent)) mark_address_taken (TREE_OPERAND (expr, 0)); /* Otherwise, there may be variables referenced inside but there @@ -885,43 +893,50 @@ get_expr_operands (struct function *fn, /* Parse STMT looking for operands. When finished, the various - build_* operand vectors will have potential operands in them. */ + build_* operand vectors will have potential operands in them. + TRANSPARENT_P as for update_stmt_operands. */ static void -parse_ssa_operands (struct function *fn, gimple *stmt) +parse_ssa_operands (struct function *fn, gimple *stmt, bool transparent_p) { enum gimple_code code = gimple_code (stmt); size_t i, n, start = 0; + int flags = (transparent_p || code == GIMPLE_DEBUG + ? opf_no_vops | opf_transparent : 0); switch (code) { case GIMPLE_ASM: + /* Not supported yet (but could be if needed). */ + gcc_assert (!transparent_p); get_asm_stmt_operands (fn, as_a (stmt)); break; case GIMPLE_TRANSACTION: /* The start of a transaction is a memory barrier. */ - add_virtual_operand (fn, stmt, opf_def | opf_use); + add_virtual_operand (fn, stmt, opf_def | opf_use | flags); break; case GIMPLE_DEBUG: if (gimple_debug_bind_p (stmt) && gimple_debug_bind_has_value_p (stmt)) get_expr_operands (fn, stmt, gimple_debug_bind_get_value_ptr (stmt), - opf_use | opf_no_vops); + opf_use | flags); break; case GIMPLE_RETURN: - append_vuse (gimple_vop (fn)); + if (!(flags & opf_no_vops)) + append_vuse (gimple_vop (fn)); goto do_default; case GIMPLE_CALL: /* Add call-clobbered operands, if needed. */ - maybe_add_call_vops (fn, as_a (stmt)); + if (!(flags & opf_no_vops)) + maybe_add_call_vops (fn, as_a (stmt)); /* FALLTHRU */ case GIMPLE_ASSIGN: - get_expr_operands (fn, stmt, gimple_op_ptr (stmt, 0), opf_def); + get_expr_operands (fn, stmt, gimple_op_ptr (stmt, 0), opf_def | flags); start = 1; /* FALLTHRU */ @@ -929,22 +944,23 @@ parse_ssa_operands (struct function *fn, do_default: n = gimple_num_ops (stmt); for (i = start; i < n; i++) - get_expr_operands (fn, stmt, gimple_op_ptr (stmt, i), opf_use); + get_expr_operands (fn, stmt, gimple_op_ptr (stmt, i), opf_use | flags); break; } } -/* Create an operands cache for STMT. */ +/* Create an operands cache for STMT. TRANSPARENT_P as for + update_stmt_operands. */ static void -build_ssa_operands (struct function *fn, gimple *stmt) +build_ssa_operands (struct function *fn, gimple *stmt, bool transparent_p) { /* Initially assume that the statement has no volatile operands. */ gimple_set_has_volatile_ops (stmt, false); start_ssa_stmt_operands (); - parse_ssa_operands (fn, stmt); + parse_ssa_operands (fn, stmt, transparent_p); finalize_ssa_stmt_operands (fn, stmt); } @@ -963,7 +979,7 @@ verify_ssa_operands (struct function *fn /* build_ssa_operands w/o finalizing them. */ gimple_set_has_volatile_ops (stmt, false); start_ssa_stmt_operands (); - parse_ssa_operands (fn, stmt); + parse_ssa_operands (fn, stmt, false); /* Now verify the built operands are the same as present in STMT. */ def = gimple_vdef (stmt); @@ -1065,10 +1081,11 @@ free_stmt_operands (struct function *fn, } -/* Get the operands of statement STMT. */ +/* Get the operands of statement STMT. TRANSPARENT_P says that opf_transparent + semantics should be used whatever form STMT happens to have. */ void -update_stmt_operands (struct function *fn, gimple *stmt) +update_stmt_operands (struct function *fn, gimple *stmt, bool transparent_p) { /* If update_stmt_operands is called before SSA is initialized, do nothing. */ @@ -1078,7 +1095,7 @@ update_stmt_operands (struct function *f timevar_push (TV_TREE_OPS); gcc_assert (gimple_modified_p (stmt)); - build_ssa_operands (fn, stmt); + build_ssa_operands (fn, stmt, transparent_p); gimple_set_modified (stmt, false); timevar_pop (TV_TREE_OPS); Index: gcc/gimple-ssa.h =================================================================== --- gcc/gimple-ssa.h 2018-05-02 08:37:33.501751141 +0100 +++ gcc/gimple-ssa.h 2018-08-28 12:05:19.262917177 +0100 @@ -164,36 +164,42 @@ gimple_vdef_op (gimple *g) return NULL_DEF_OPERAND_P; } -/* Mark statement S as modified, and update it. */ +/* Mark statement S as modified, and update it. TRANSPARENT_P is true + if the update must have no effect on code generation, in much the + same way as for debug statements. This means in particular that the + statement should not cause things to be marked addressable and should + not use virtual operands. */ static inline void -update_stmt (gimple *s) +update_stmt (gimple *s, bool transparent_p = false) { if (gimple_has_ops (s)) { gimple_set_modified (s, true); - update_stmt_operands (cfun, s); + update_stmt_operands (cfun, s, transparent_p); } } -/* Update statement S if it has been optimized. */ +/* Update statement S if it has been optimized. TRANSPARENT_P is as for + update_stmt. */ static inline void -update_stmt_if_modified (gimple *s) +update_stmt_if_modified (gimple *s, bool transparent_p = false) { if (gimple_modified_p (s)) - update_stmt_operands (cfun, s); + update_stmt_operands (cfun, s, transparent_p); } -/* Mark statement S as modified, and update it. */ +/* Mark statement S as modified, and update it. TRANSPARENT_P is as for + update_stmt. */ static inline void -update_stmt_fn (struct function *fn, gimple *s) +update_stmt_fn (struct function *fn, gimple *s, bool transparent_p = false) { if (gimple_has_ops (s)) { gimple_set_modified (s, true); - update_stmt_operands (fn, s); + update_stmt_operands (fn, s, transparent_p); } } Index: gcc/tree-vect-slp.c =================================================================== --- gcc/tree-vect-slp.c 2018-08-28 12:05:16.522940287 +0100 +++ gcc/tree-vect-slp.c 2018-08-28 12:05:19.262917177 +0100 @@ -2302,50 +2302,6 @@ vect_detect_hybrid_slp_stmts (slp_tree n vect_detect_hybrid_slp_stmts (child, i, stype); } -/* Helpers for vect_detect_hybrid_slp walking pattern stmt uses. */ - -static tree -vect_detect_hybrid_slp_1 (tree *tp, int *, void *data) -{ - walk_stmt_info *wi = (walk_stmt_info *)data; - loop_vec_info loop_vinfo = (loop_vec_info) wi->info; - - if (wi->is_lhs) - return NULL_TREE; - - stmt_vec_info def_stmt_info = loop_vinfo->lookup_def (*tp); - if (def_stmt_info && PURE_SLP_STMT (def_stmt_info)) - { - if (dump_enabled_p ()) - { - dump_printf_loc (MSG_NOTE, vect_location, "marking hybrid: "); - dump_gimple_stmt (MSG_NOTE, TDF_SLIM, def_stmt_info->stmt, 0); - } - STMT_SLP_TYPE (def_stmt_info) = hybrid; - } - - return NULL_TREE; -} - -static tree -vect_detect_hybrid_slp_2 (gimple_stmt_iterator *gsi, bool *handled, - walk_stmt_info *wi) -{ - loop_vec_info loop_vinfo = (loop_vec_info) wi->info; - stmt_vec_info use_vinfo = loop_vinfo->lookup_stmt (gsi_stmt (*gsi)); - /* If the stmt is in a SLP instance then this isn't a reason - to mark use definitions in other SLP instances as hybrid. */ - if (! STMT_SLP_TYPE (use_vinfo) - && (STMT_VINFO_RELEVANT (use_vinfo) - || VECTORIZABLE_CYCLE_DEF (STMT_VINFO_DEF_TYPE (use_vinfo))) - && ! (gimple_code (gsi_stmt (*gsi)) == GIMPLE_PHI - && STMT_VINFO_DEF_TYPE (use_vinfo) == vect_reduction_def)) - ; - else - *handled = true; - return NULL_TREE; -} - /* Find stmts that must be both vectorized and SLPed. */ void @@ -2357,22 +2313,7 @@ vect_detect_hybrid_slp (loop_vec_info lo DUMP_VECT_SCOPE ("vect_detect_hybrid_slp"); - /* First walk all pattern stmt in the loop and mark defs of uses as - hybrid because immediate uses in them are not recorded. */ - vec_basic_block *vec_bb; - FOR_EACH_VEC_ELT (loop_vinfo->blocks, i, vec_bb) - FOR_EACH_VEC_BB_STMT (vec_bb, stmt_info) - if (is_pattern_stmt_p (stmt_info)) - { - walk_stmt_info wi; - memset (&wi, 0, sizeof (wi)); - wi.info = loop_vinfo; - gimple_stmt_iterator gsi = gsi_for_stmt (stmt_info->stmt); - walk_gimple_stmt (&gsi, vect_detect_hybrid_slp_2, - vect_detect_hybrid_slp_1, &wi); - } - - /* Then walk the SLP instance trees marking stmts with uses in + /* Walk the SLP instance trees marking stmts with uses in non-SLP stmts as hybrid, also propagating hybrid down the SLP tree, collecting the above info on-the-fly. */ FOR_EACH_VEC_ELT (slp_instances, i, instance) Index: gcc/tree-vect-stmts.c =================================================================== --- gcc/tree-vect-stmts.c 2018-08-28 12:05:16.522940287 +0100 +++ gcc/tree-vect-stmts.c 2018-08-28 12:05:19.266917143 +0100 @@ -703,54 +703,13 @@ vect_mark_stmts_to_be_vectorized (loop_v break; } - if (is_pattern_stmt_p (stmt_vinfo)) - { - /* Pattern statements are not inserted into the code, so - FOR_EACH_PHI_OR_STMT_USE optimizes their operands out, and we - have to scan the RHS or function arguments instead. */ - if (gassign *assign = dyn_cast (stmt_vinfo->stmt)) - { - enum tree_code rhs_code = gimple_assign_rhs_code (assign); - tree op = gimple_assign_rhs1 (assign); - - i = 1; - if (rhs_code == COND_EXPR && COMPARISON_CLASS_P (op)) - { - if (!process_use (stmt_vinfo, TREE_OPERAND (op, 0), - loop_vinfo, relevant, &worklist, false) - || !process_use (stmt_vinfo, TREE_OPERAND (op, 1), - loop_vinfo, relevant, &worklist, false)) - return false; - i = 2; - } - for (; i < gimple_num_ops (assign); i++) - { - op = gimple_op (assign, i); - if (TREE_CODE (op) == SSA_NAME - && !process_use (stmt_vinfo, op, loop_vinfo, relevant, - &worklist, false)) - return false; - } - } - else if (gcall *call = dyn_cast (stmt_vinfo->stmt)) - { - for (i = 0; i < gimple_call_num_args (call); i++) - { - tree arg = gimple_call_arg (call, i); - if (!process_use (stmt_vinfo, arg, loop_vinfo, relevant, - &worklist, false)) - return false; - } - } - } - else - FOR_EACH_PHI_OR_STMT_USE (use_p, stmt_vinfo->stmt, iter, SSA_OP_USE) - { - tree op = USE_FROM_PTR (use_p); - if (!process_use (stmt_vinfo, op, loop_vinfo, relevant, - &worklist, false)) - return false; - } + FOR_EACH_PHI_OR_STMT_USE (use_p, stmt_vinfo->stmt, iter, SSA_OP_USE) + { + tree op = USE_FROM_PTR (use_p); + if (!process_use (stmt_vinfo, op, loop_vinfo, relevant, + &worklist, false)) + return false; + } if (STMT_VINFO_GATHER_SCATTER_P (stmt_vinfo)) { @@ -10849,7 +10808,9 @@ vect_remove_dead_scalar_stmts (vec_info stmt_info = prev_stmt_info) { prev_stmt_info = stmt_info->prev; - if (!is_pattern_stmt_p (stmt_info)) + if (is_pattern_stmt_p (stmt_info)) + vinfo->remove_stmt (stmt_info); + else vect_maybe_remove_scalar_stmt (stmt_info); } } Index: gcc/tree-vectorizer.h =================================================================== --- gcc/tree-vectorizer.h 2018-08-28 12:05:16.522940287 +0100 +++ gcc/tree-vectorizer.h 2018-08-28 12:05:19.266917143 +0100 @@ -193,6 +193,8 @@ #define SLP_TREE_DEF_TYPE(S) (S)->def stmt_vec_info last () const { return m_last; } private: + void add_pattern_stmt_to_block (stmt_vec_info); + /* The block itself. */ basic_block m_bb; Index: gcc/tree-vectorizer.c =================================================================== --- gcc/tree-vectorizer.c 2018-08-28 12:05:14.014961439 +0100 +++ gcc/tree-vectorizer.c 2018-08-28 12:05:19.266917143 +0100 @@ -459,6 +459,9 @@ vec_basic_block::add_to_end (stmt_vec_in stmt_info->block = this; stmt_info->prev = m_last; m_last = stmt_info; + + if (is_pattern_stmt_p (stmt_info)) + add_pattern_stmt_to_block (stmt_info); } /* Add STMT_INFO to the block, inserting it before NEXT_STMT_INFO. */ @@ -479,6 +482,34 @@ vec_basic_block::add_before (stmt_vec_in stmt_info->prev = next_stmt_info->prev; stmt_info->next = next_stmt_info; next_stmt_info->prev = stmt_info; + + if (is_pattern_stmt_p (stmt_info)) + add_pattern_stmt_to_block (stmt_info); +} + +/* Record that pattern statement STMT_INFO has just been added to the + vec_basic_block. Adding it to the underlying basic_block would be + problematic because we need to be able to duplicate the original + scalar code in the middle of vectorization. Instead we just set the + statement's gimple_bb and link its SSA uses. */ + +void +vec_basic_block::add_pattern_stmt_to_block (stmt_vec_info stmt_info) +{ + gimple_set_bb (stmt_info->stmt, m_bb); + update_stmt (stmt_info->stmt, true); + gcc_assert (!gimple_vdef (stmt_info->stmt)); + gcc_assert (!gimple_vuse (stmt_info->stmt)); +} + +/* Record that STMT_INFO has been removed from its vec_basic_block. + Undo the effect of add_pattern_stmt_to_block. */ + +static void +remove_pattern_stmt_from_block (stmt_vec_info stmt_info) +{ + gimple_set_bb (stmt_info->stmt, NULL); + delink_stmt_imm_use (stmt_info->stmt); } /* Remove STMT_INFO from the block. */ @@ -497,6 +528,9 @@ vec_basic_block::remove (stmt_vec_info s m_last = stmt_info->prev; stmt_info->block = NULL; stmt_info->prev = stmt_info->next = NULL; + + if (is_pattern_stmt_p (stmt_info)) + remove_pattern_stmt_from_block (stmt_info); } /* Initialize the vec_info with kind KIND_IN and target cost data @@ -610,12 +644,37 @@ vec_info::lookup_def (tree name) stmt_vec_info vec_info::lookup_single_use (stmt_vec_info stmt_info) { - tree lhs = gimple_get_lhs (stmt_info->stmt); - use_operand_p dummy; - gimple *use_stmt; - if (single_imm_use (lhs, &dummy, &use_stmt)) - return lookup_stmt (use_stmt); - return NULL; + gimple *stmts[2] = { stmt_info->stmt, NULL }; + unsigned int num_stmts = 1; + /* If STMT_INFO replaces one of the original scalar statements, + check for uses of that statement's results too, to simulate the + effect of vect_stmt_to_be_vectorized. */ + if (is_main_pattern_stmt_p (stmt_info)) + stmts[num_stmts++] = STMT_VINFO_RELATED_STMT (stmt_info)->stmt; + stmt_vec_info result = NULL; + for (unsigned int i = 0; i < num_stmts; ++i) + { + use_operand_p use_p; + imm_use_iterator imm_iter; + tree lhs = gimple_get_lhs (stmts[i]); + FOR_EACH_IMM_USE_FAST (use_p, imm_iter, lhs) + { + gimple *use_stmt = USE_STMT (use_p); + if (is_gimple_debug (use_stmt)) + continue; + stmt_vec_info use_stmt_info = lookup_stmt (use_stmt); + if (!use_stmt_info) + return NULL; + /* Ignore statements that have been replaced by a pattern + and consider only the replacement statement. */ + if (STMT_VINFO_IN_PATTERN_P (use_stmt_info)) + continue; + if (result) + return NULL; + result = use_stmt_info; + } + } + return result; } /* Return vectorization information about DR. */ @@ -650,16 +709,18 @@ vec_info::move_dr (stmt_vec_info new_stm void vec_info::remove_stmt (stmt_vec_info stmt_info) { - gcc_assert (!stmt_info->pattern_stmt_p); set_vinfo_for_stmt (stmt_info->stmt, NULL); gimple_stmt_iterator si = gsi_for_stmt (stmt_info->stmt); unlink_stmt_vdef (stmt_info->stmt); - if (is_a (stmt_info->stmt)) - remove_phi_node (&si, true); - else + if (!is_pattern_stmt_p (stmt_info)) { - gsi_remove (&si, true); - release_defs (stmt_info->stmt); + if (is_a (stmt_info->stmt)) + remove_phi_node (&si, true); + else + { + gsi_remove (&si, true); + release_defs (stmt_info->stmt); + } } stmt_info->block->remove (stmt_info); free_stmt_vec_info (stmt_info); @@ -749,12 +810,13 @@ vec_info::free_stmt_vec_infos (void) void vec_info::free_stmt_vec_info (stmt_vec_info stmt_info) { - if (stmt_info->pattern_stmt_p) + if (is_pattern_stmt_p (stmt_info)) { - gimple_set_bb (stmt_info->stmt, NULL); tree lhs = gimple_get_lhs (stmt_info->stmt); if (lhs && TREE_CODE (lhs) == SSA_NAME) release_ssa_name (lhs); + if (stmt_info->block) + remove_pattern_stmt_from_block (stmt_info); } STMT_VINFO_SAME_ALIGN_REFS (stmt_info).release (); Index: gcc/tree-vect-patterns.c =================================================================== --- gcc/tree-vect-patterns.c 2018-08-28 12:05:16.518940320 +0100 +++ gcc/tree-vect-patterns.c 2018-08-28 12:05:19.262917177 +0100 @@ -106,7 +106,6 @@ vect_init_pattern_stmt (gimple *pattern_ stmt_vec_info pattern_stmt_info = vinfo->lookup_stmt (pattern_stmt); if (pattern_stmt_info == NULL) pattern_stmt_info = orig_stmt_info->vinfo->add_stmt (pattern_stmt); - gimple_set_bb (pattern_stmt, gimple_bb (orig_stmt_info->stmt)); pattern_stmt_info->pattern_stmt_p = true; STMT_VINFO_RELATED_STMT (pattern_stmt_info) = orig_stmt_info; @@ -412,11 +411,9 @@ vect_look_through_possible_promotion (ve break; caster = def_stmt_info; - /* Ignore pattern statements, since we don't link uses for them. */ if (caster && single_use_p - && !STMT_VINFO_RELATED_STMT (caster) - && !has_single_use (res)) + && !vinfo->lookup_single_use (caster)) *single_use_p = false; gassign *assign = dyn_cast (def_stmt);