[6/6] Link imm uses for pattern stmts

[Richard Sandiford <richard.sandiford@arm.com>]

One of the warts of the vectoriser IR is that it doesn't link SSA name
uses for pattern statements, leading to complicated special cases in
vect_mark_stmts_to_be_vectorized and (especially) vect_detect_hybrid_slp.
It also makes it harder to check how an SSA name is used after pattern
replacement (something I need for a later patch).

This patch adds a mode in which tree-ssa-operands.c can update
statements in the same non-invasive way as for debug statements.
It then uses this mode to update pattern statements when adding
them to a vec_basic_block, so that pattern statements become
even more like statements that existed from the outset.


2018-08-28  Richard Sandiford  <richard.sandiford@arm.com>

gcc/
	* tree-ssa-operands.h (update_stmt_operands): Add a transparent_p
	argument.
	* tree-ssa-operands.c (opf_transparent, opf_sticky): New macros.
	(get_mem_ref_operands, get_tmr_operands): Preserve opf_sticky
	rather than just opf_no_vops.
	(get_expr_operands): Preserve opf_sticky bits in the use flags.
	Assert that opf_no_vops and opf_transparent are already set
	for the debug statements.  Use opf_transparent rather than
	is_gimple_debug when deciding whether to mark something as
	having its address taken.
	(parse_ssa_operands): Add a transparent_p argument.  Set the
	opf_no_vops and opf_transparent flags when the argument is true,
	or when dealing with debug statements.  Check opf_no_vops before
	adding vuses and vdefs.
	(build_ssa_operands): Add a transparent_p argument and pass it to
	parse_ssa_operands.
	(verify_ssa_operands): Update call to parse_ssa_operands.
	(update_stmt_operands): Add a transparent_p argument and pass it to
	build_ssa_operands.
	* gimple-ssa.h (update_stmt, update_stmt_if_modified)
	(update_stmt_fn): Add an optional transparent_p parameter and
	update call to update_stmt_operands.
	* tree-vect-slp.c (vect_detect_hybrid_slp_1): Delete.
	(vect_detect_hybrid_slp_2): Likewise.
	(vect_detect_hybrid_slp): Don't treat pattern statements specially.
	* tree-vect-stmts.c (vect_mark_stmts_to_be_vectorized): Likewise.
	(vect_remove_dead_scalar_stmts): Remove pattern statements from
	the containing vec_info.
	* tree-vectorizer.h (vec_info::add_pattern_stmt_to_block): Declare.
	* tree-vectorizer.c (vec_basic_block::add_to_end)
	(vec_basic_block::add_before): Call add_pattern_stmt_to_block.
	(vec_basic_block::remove, vec_info::remove_stmt): Call
	remove_pattern_stmt_from_block.
	(vec_basic_block::add_pattern_stmt_to_block): New function.
	(remove_pattern_stmt_from_block): Likewise.
	(vec_info::free_stmt_vec_info): Handle pattern statements.
	(vec_info::lookup_single_use): Accept pattern statements
	as well as original statements.  Ignore uses in statements
	that have been replaced by a pattern statement.
	* tree-vect-patterns.c (vect_init_pattern_stmt): Don't call
	gimple_set_bb.
	(vect_look_through_possible_promotion): Use vinfo->lookup_single_use
	instead of has_single_use.  Track single uses for pattern statements
	too.

Index: gcc/tree-ssa-operands.h
===================================================================
--- gcc/tree-ssa-operands.h	2018-05-02 08:37:32.405761509 +0100
+++ gcc/tree-ssa-operands.h	2018-08-28 12:05:19.262917177 +0100
@@ -94,7 +94,7 @@ extern void init_ssa_operands (struct fu
 extern void fini_ssa_operands (struct function *);
 extern bool verify_ssa_operands (struct function *, gimple *stmt);
 extern void free_stmt_operands (struct function *, gimple *);
-extern void update_stmt_operands (struct function *, gimple *);
+extern void update_stmt_operands (struct function *, gimple *, bool);
 extern void swap_ssa_operands (gimple *, tree *, tree *);
 extern bool verify_imm_links (FILE *f, tree var);
 
Index: gcc/tree-ssa-operands.c
===================================================================
--- gcc/tree-ssa-operands.c	2018-08-28 11:25:46.242879876 +0100
+++ gcc/tree-ssa-operands.c	2018-08-28 12:05:19.262917177 +0100
@@ -99,6 +99,14 @@ #define opf_not_non_addressable (1 << 4)
 /* Operand is having its address taken.  */
 #define opf_address_taken (1 << 5)
 
+/* Operand must have no effect on code generation.  This is used for
+   debug statements, and also for statements that a pass has no intention
+   of adding to the block in their current form.  */
+#define opf_transparent (1 << 6)
+
+/* Flags that must never be dropped.  */
+#define opf_sticky (opf_no_vops | opf_transparent)
+
 /* Array for building all the use operands.  */
 static vec<tree *> build_uses;
 
@@ -590,7 +598,7 @@ get_mem_ref_operands (struct function *f
   /* If requested, add a USE operand for the base pointer.  */
   get_expr_operands (fn, stmt, pptr,
 		     opf_non_addressable | opf_use
-		     | (flags & (opf_no_vops|opf_not_non_addressable)));
+		     | (flags & (opf_sticky | opf_not_non_addressable)));
 }
 
 
@@ -605,11 +613,11 @@ get_tmr_operands (struct function *fn, g
 
   /* First record the real operands.  */
   get_expr_operands (fn, stmt,
-		     &TMR_BASE (expr), opf_use | (flags & opf_no_vops));
+		     &TMR_BASE (expr), opf_use | (flags & opf_sticky));
   get_expr_operands (fn, stmt,
-		     &TMR_INDEX (expr), opf_use | (flags & opf_no_vops));
+		     &TMR_INDEX (expr), opf_use | (flags & opf_sticky));
   get_expr_operands (fn, stmt,
-		     &TMR_INDEX2 (expr), opf_use | (flags & opf_no_vops));
+		     &TMR_INDEX2 (expr), opf_use | (flags & opf_sticky));
 
   add_virtual_operand (fn, stmt, flags);
 }
@@ -703,14 +711,14 @@ get_expr_operands (struct function *fn,
   enum tree_code code;
   enum tree_code_class codeclass;
   tree expr = *expr_p;
-  int uflags = opf_use;
+  int uflags = opf_use | (flags & opf_sticky);
+  gcc_checking_assert (!is_gimple_debug (stmt)
+		       || ((flags & opf_no_vops)
+			   && (flags & opf_transparent)));
 
   if (expr == NULL)
     return;
 
-  if (is_gimple_debug (stmt))
-    uflags |= (flags & opf_no_vops);
-
   code = TREE_CODE (expr);
   codeclass = TREE_CODE_CLASS (code);
 
@@ -723,7 +731,7 @@ get_expr_operands (struct function *fn,
 	 resolution).  */
       if ((!(flags & opf_non_addressable)
 	   || (flags & opf_not_non_addressable))
-	  && !is_gimple_debug (stmt))
+	  && !(flags & opf_transparent))
 	mark_address_taken (TREE_OPERAND (expr, 0));
 
       /* Otherwise, there may be variables referenced inside but there
@@ -885,43 +893,50 @@ get_expr_operands (struct function *fn,
 
 
 /* Parse STMT looking for operands.  When finished, the various
-   build_* operand vectors will have potential operands in them.  */
+   build_* operand vectors will have potential operands in them.
+   TRANSPARENT_P as for update_stmt_operands.  */
 
 static void
-parse_ssa_operands (struct function *fn, gimple *stmt)
+parse_ssa_operands (struct function *fn, gimple *stmt, bool transparent_p)
 {
   enum gimple_code code = gimple_code (stmt);
   size_t i, n, start = 0;
+  int flags = (transparent_p || code == GIMPLE_DEBUG
+	       ? opf_no_vops | opf_transparent : 0);
 
   switch (code)
     {
     case GIMPLE_ASM:
+      /* Not supported yet (but could be if needed).  */
+      gcc_assert (!transparent_p);
       get_asm_stmt_operands (fn, as_a <gasm *> (stmt));
       break;
 
     case GIMPLE_TRANSACTION:
       /* The start of a transaction is a memory barrier.  */
-      add_virtual_operand (fn, stmt, opf_def | opf_use);
+      add_virtual_operand (fn, stmt, opf_def | opf_use | flags);
       break;
 
     case GIMPLE_DEBUG:
       if (gimple_debug_bind_p (stmt)
 	  && gimple_debug_bind_has_value_p (stmt))
 	get_expr_operands (fn, stmt, gimple_debug_bind_get_value_ptr (stmt),
-			   opf_use | opf_no_vops);
+			   opf_use | flags);
       break;
 
     case GIMPLE_RETURN:
-      append_vuse (gimple_vop (fn));
+      if (!(flags & opf_no_vops))
+	append_vuse (gimple_vop (fn));
       goto do_default;
 
     case GIMPLE_CALL:
       /* Add call-clobbered operands, if needed.  */
-      maybe_add_call_vops (fn, as_a <gcall *> (stmt));
+      if (!(flags & opf_no_vops))
+	maybe_add_call_vops (fn, as_a <gcall *> (stmt));
       /* FALLTHRU */
 
     case GIMPLE_ASSIGN:
-      get_expr_operands (fn, stmt, gimple_op_ptr (stmt, 0), opf_def);
+      get_expr_operands (fn, stmt, gimple_op_ptr (stmt, 0), opf_def | flags);
       start = 1;
       /* FALLTHRU */
 
@@ -929,22 +944,23 @@ parse_ssa_operands (struct function *fn,
     do_default:
       n = gimple_num_ops (stmt);
       for (i = start; i < n; i++)
-	get_expr_operands (fn, stmt, gimple_op_ptr (stmt, i), opf_use);
+	get_expr_operands (fn, stmt, gimple_op_ptr (stmt, i), opf_use | flags);
       break;
     }
 }
 
 
-/* Create an operands cache for STMT.  */
+/* Create an operands cache for STMT.  TRANSPARENT_P as for
+   update_stmt_operands.  */
 
 static void
-build_ssa_operands (struct function *fn, gimple *stmt)
+build_ssa_operands (struct function *fn, gimple *stmt, bool transparent_p)
 {
   /* Initially assume that the statement has no volatile operands.  */
   gimple_set_has_volatile_ops (stmt, false);
 
   start_ssa_stmt_operands ();
-  parse_ssa_operands (fn, stmt);
+  parse_ssa_operands (fn, stmt, transparent_p);
   finalize_ssa_stmt_operands (fn, stmt);
 }
 
@@ -963,7 +979,7 @@ verify_ssa_operands (struct function *fn
   /* build_ssa_operands w/o finalizing them.  */
   gimple_set_has_volatile_ops (stmt, false);
   start_ssa_stmt_operands ();
-  parse_ssa_operands (fn, stmt);
+  parse_ssa_operands (fn, stmt, false);
 
   /* Now verify the built operands are the same as present in STMT.  */
   def = gimple_vdef (stmt);
@@ -1065,10 +1081,11 @@ free_stmt_operands (struct function *fn,
 }
 
 
-/* Get the operands of statement STMT.  */
+/* Get the operands of statement STMT.  TRANSPARENT_P says that opf_transparent
+   semantics should be used whatever form STMT happens to have.  */
 
 void
-update_stmt_operands (struct function *fn, gimple *stmt)
+update_stmt_operands (struct function *fn, gimple *stmt, bool transparent_p)
 {
   /* If update_stmt_operands is called before SSA is initialized, do
      nothing.  */
@@ -1078,7 +1095,7 @@ update_stmt_operands (struct function *f
   timevar_push (TV_TREE_OPS);
 
   gcc_assert (gimple_modified_p (stmt));
-  build_ssa_operands (fn, stmt);
+  build_ssa_operands (fn, stmt, transparent_p);
   gimple_set_modified (stmt, false);
 
   timevar_pop (TV_TREE_OPS);
Index: gcc/gimple-ssa.h
===================================================================
--- gcc/gimple-ssa.h	2018-05-02 08:37:33.501751141 +0100
+++ gcc/gimple-ssa.h	2018-08-28 12:05:19.262917177 +0100
@@ -164,36 +164,42 @@ gimple_vdef_op (gimple *g)
   return NULL_DEF_OPERAND_P;
 }
 
-/* Mark statement S as modified, and update it.  */
+/* Mark statement S as modified, and update it.  TRANSPARENT_P is true
+   if the update must have no effect on code generation, in much the
+   same way as for debug statements.  This means in particular that the
+   statement should not cause things to be marked addressable and should
+   not use virtual operands.  */
 
 static inline void
-update_stmt (gimple *s)
+update_stmt (gimple *s, bool transparent_p = false)
 {
   if (gimple_has_ops (s))
     {
       gimple_set_modified (s, true);
-      update_stmt_operands (cfun, s);
+      update_stmt_operands (cfun, s, transparent_p);
     }
 }
 
-/* Update statement S if it has been optimized.  */
+/* Update statement S if it has been optimized.  TRANSPARENT_P is as for
+   update_stmt.  */
 
 static inline void
-update_stmt_if_modified (gimple *s)
+update_stmt_if_modified (gimple *s, bool transparent_p = false)
 {
   if (gimple_modified_p (s))
-    update_stmt_operands (cfun, s);
+    update_stmt_operands (cfun, s, transparent_p);
 }
 
-/* Mark statement S as modified, and update it.  */
+/* Mark statement S as modified, and update it.  TRANSPARENT_P is as for
+   update_stmt.  */
 
 static inline void
-update_stmt_fn (struct function *fn, gimple *s)
+update_stmt_fn (struct function *fn, gimple *s, bool transparent_p = false)
 {
   if (gimple_has_ops (s))
     {
       gimple_set_modified (s, true);
-      update_stmt_operands (fn, s);
+      update_stmt_operands (fn, s, transparent_p);
     }
 }
 
Index: gcc/tree-vect-slp.c
===================================================================
--- gcc/tree-vect-slp.c	2018-08-28 12:05:16.522940287 +0100
+++ gcc/tree-vect-slp.c	2018-08-28 12:05:19.262917177 +0100
@@ -2302,50 +2302,6 @@ vect_detect_hybrid_slp_stmts (slp_tree n
       vect_detect_hybrid_slp_stmts (child, i, stype);
 }
 
-/* Helpers for vect_detect_hybrid_slp walking pattern stmt uses.  */
-
-static tree
-vect_detect_hybrid_slp_1 (tree *tp, int *, void *data)
-{
-  walk_stmt_info *wi = (walk_stmt_info *)data;
-  loop_vec_info loop_vinfo = (loop_vec_info) wi->info;
-
-  if (wi->is_lhs)
-    return NULL_TREE;
-
-  stmt_vec_info def_stmt_info = loop_vinfo->lookup_def (*tp);
-  if (def_stmt_info && PURE_SLP_STMT (def_stmt_info))
-    {
-      if (dump_enabled_p ())
-	{
-	  dump_printf_loc (MSG_NOTE, vect_location, "marking hybrid: ");
-	  dump_gimple_stmt (MSG_NOTE, TDF_SLIM, def_stmt_info->stmt, 0);
-	}
-      STMT_SLP_TYPE (def_stmt_info) = hybrid;
-    }
-
-  return NULL_TREE;
-}
-
-static tree
-vect_detect_hybrid_slp_2 (gimple_stmt_iterator *gsi, bool *handled,
-			  walk_stmt_info *wi)
-{
-  loop_vec_info loop_vinfo = (loop_vec_info) wi->info;
-  stmt_vec_info use_vinfo = loop_vinfo->lookup_stmt (gsi_stmt (*gsi));
-  /* If the stmt is in a SLP instance then this isn't a reason
-     to mark use definitions in other SLP instances as hybrid.  */
-  if (! STMT_SLP_TYPE (use_vinfo)
-      && (STMT_VINFO_RELEVANT (use_vinfo)
-	  || VECTORIZABLE_CYCLE_DEF (STMT_VINFO_DEF_TYPE (use_vinfo)))
-      && ! (gimple_code (gsi_stmt (*gsi)) == GIMPLE_PHI
-	    && STMT_VINFO_DEF_TYPE (use_vinfo) == vect_reduction_def))
-    ;
-  else
-    *handled = true;
-  return NULL_TREE;
-}
-
 /* Find stmts that must be both vectorized and SLPed.  */
 
 void
@@ -2357,22 +2313,7 @@ vect_detect_hybrid_slp (loop_vec_info lo
 
   DUMP_VECT_SCOPE ("vect_detect_hybrid_slp");
 
-  /* First walk all pattern stmt in the loop and mark defs of uses as
-     hybrid because immediate uses in them are not recorded.  */
-  vec_basic_block *vec_bb;
-  FOR_EACH_VEC_ELT (loop_vinfo->blocks, i, vec_bb)
-    FOR_EACH_VEC_BB_STMT (vec_bb, stmt_info)
-      if (is_pattern_stmt_p (stmt_info))
-	{
-	  walk_stmt_info wi;
-	  memset (&wi, 0, sizeof (wi));
-	  wi.info = loop_vinfo;
-	  gimple_stmt_iterator gsi = gsi_for_stmt (stmt_info->stmt);
-	  walk_gimple_stmt (&gsi, vect_detect_hybrid_slp_2,
-			    vect_detect_hybrid_slp_1, &wi);
-	}
-
-  /* Then walk the SLP instance trees marking stmts with uses in
+  /* Walk the SLP instance trees marking stmts with uses in
      non-SLP stmts as hybrid, also propagating hybrid down the
      SLP tree, collecting the above info on-the-fly.  */
   FOR_EACH_VEC_ELT (slp_instances, i, instance)
Index: gcc/tree-vect-stmts.c
===================================================================
--- gcc/tree-vect-stmts.c	2018-08-28 12:05:16.522940287 +0100
+++ gcc/tree-vect-stmts.c	2018-08-28 12:05:19.266917143 +0100
@@ -703,54 +703,13 @@ vect_mark_stmts_to_be_vectorized (loop_v
             break;
         }
 
-      if (is_pattern_stmt_p (stmt_vinfo))
-        {
-          /* Pattern statements are not inserted into the code, so
-             FOR_EACH_PHI_OR_STMT_USE optimizes their operands out, and we
-             have to scan the RHS or function arguments instead.  */
-	  if (gassign *assign = dyn_cast <gassign *> (stmt_vinfo->stmt))
-	    {
-	      enum tree_code rhs_code = gimple_assign_rhs_code (assign);
-	      tree op = gimple_assign_rhs1 (assign);
-
-	      i = 1;
-	      if (rhs_code == COND_EXPR && COMPARISON_CLASS_P (op))
-		{
-		  if (!process_use (stmt_vinfo, TREE_OPERAND (op, 0),
-				    loop_vinfo, relevant, &worklist, false)
-		      || !process_use (stmt_vinfo, TREE_OPERAND (op, 1),
-				       loop_vinfo, relevant, &worklist, false))
-		    return false;
-		  i = 2;
-		}
-	      for (; i < gimple_num_ops (assign); i++)
-		{
-		  op = gimple_op (assign, i);
-                  if (TREE_CODE (op) == SSA_NAME
-		      && !process_use (stmt_vinfo, op, loop_vinfo, relevant,
-				       &worklist, false))
-                    return false;
-                 }
-            }
-	  else if (gcall *call = dyn_cast <gcall *> (stmt_vinfo->stmt))
-	    {
-	      for (i = 0; i < gimple_call_num_args (call); i++)
-		{
-		  tree arg = gimple_call_arg (call, i);
-		  if (!process_use (stmt_vinfo, arg, loop_vinfo, relevant,
-				    &worklist, false))
-                    return false;
-		}
-	    }
-        }
-      else
-	FOR_EACH_PHI_OR_STMT_USE (use_p, stmt_vinfo->stmt, iter, SSA_OP_USE)
-          {
-            tree op = USE_FROM_PTR (use_p);
-	    if (!process_use (stmt_vinfo, op, loop_vinfo, relevant,
-			      &worklist, false))
-              return false;
-          }
+      FOR_EACH_PHI_OR_STMT_USE (use_p, stmt_vinfo->stmt, iter, SSA_OP_USE)
+	{
+	  tree op = USE_FROM_PTR (use_p);
+	  if (!process_use (stmt_vinfo, op, loop_vinfo, relevant,
+			    &worklist, false))
+	    return false;
+	}
 
       if (STMT_VINFO_GATHER_SCATTER_P (stmt_vinfo))
 	{
@@ -10849,7 +10808,9 @@ vect_remove_dead_scalar_stmts (vec_info
 	   stmt_info = prev_stmt_info)
 	{
 	  prev_stmt_info = stmt_info->prev;
-	  if (!is_pattern_stmt_p (stmt_info))
+	  if (is_pattern_stmt_p (stmt_info))
+	    vinfo->remove_stmt (stmt_info);
+	  else
 	    vect_maybe_remove_scalar_stmt (stmt_info);
 	}
     }
Index: gcc/tree-vectorizer.h
===================================================================
--- gcc/tree-vectorizer.h	2018-08-28 12:05:16.522940287 +0100
+++ gcc/tree-vectorizer.h	2018-08-28 12:05:19.266917143 +0100
@@ -193,6 +193,8 @@ #define SLP_TREE_DEF_TYPE(S)			 (S)->def
   stmt_vec_info last () const { return m_last; }
 
 private:
+  void add_pattern_stmt_to_block (stmt_vec_info);
+
   /* The block itself.  */
   basic_block m_bb;
 
Index: gcc/tree-vectorizer.c
===================================================================
--- gcc/tree-vectorizer.c	2018-08-28 12:05:14.014961439 +0100
+++ gcc/tree-vectorizer.c	2018-08-28 12:05:19.266917143 +0100
@@ -459,6 +459,9 @@ vec_basic_block::add_to_end (stmt_vec_in
   stmt_info->block = this;
   stmt_info->prev = m_last;
   m_last = stmt_info;
+
+  if (is_pattern_stmt_p (stmt_info))
+    add_pattern_stmt_to_block (stmt_info);
 }
 
 /* Add STMT_INFO to the block, inserting it before NEXT_STMT_INFO.  */
@@ -479,6 +482,34 @@ vec_basic_block::add_before (stmt_vec_in
   stmt_info->prev = next_stmt_info->prev;
   stmt_info->next = next_stmt_info;
   next_stmt_info->prev = stmt_info;
+
+  if (is_pattern_stmt_p (stmt_info))
+    add_pattern_stmt_to_block (stmt_info);
+}
+
+/* Record that pattern statement STMT_INFO has just been added to the
+   vec_basic_block.  Adding it to the underlying basic_block would be
+   problematic because we need to be able to duplicate the original
+   scalar code in the middle of vectorization.  Instead we just set the
+   statement's gimple_bb and link its SSA uses.  */
+
+void
+vec_basic_block::add_pattern_stmt_to_block (stmt_vec_info stmt_info)
+{
+  gimple_set_bb (stmt_info->stmt, m_bb);
+  update_stmt (stmt_info->stmt, true);
+  gcc_assert (!gimple_vdef (stmt_info->stmt));
+  gcc_assert (!gimple_vuse (stmt_info->stmt));
+}
+
+/* Record that STMT_INFO has been removed from its vec_basic_block.
+   Undo the effect of add_pattern_stmt_to_block.  */
+
+static void
+remove_pattern_stmt_from_block (stmt_vec_info stmt_info)
+{
+  gimple_set_bb (stmt_info->stmt, NULL);
+  delink_stmt_imm_use (stmt_info->stmt);
 }
 
 /* Remove STMT_INFO from the block.  */
@@ -497,6 +528,9 @@ vec_basic_block::remove (stmt_vec_info s
     m_last = stmt_info->prev;
   stmt_info->block = NULL;
   stmt_info->prev = stmt_info->next = NULL;
+
+  if (is_pattern_stmt_p (stmt_info))
+    remove_pattern_stmt_from_block (stmt_info);
 }
 \f
 /* Initialize the vec_info with kind KIND_IN and target cost data
@@ -610,12 +644,37 @@ vec_info::lookup_def (tree name)
 stmt_vec_info
 vec_info::lookup_single_use (stmt_vec_info stmt_info)
 {
-  tree lhs = gimple_get_lhs (stmt_info->stmt);
-  use_operand_p dummy;
-  gimple *use_stmt;
-  if (single_imm_use (lhs, &dummy, &use_stmt))
-    return lookup_stmt (use_stmt);
-  return NULL;
+  gimple *stmts[2] = { stmt_info->stmt, NULL };
+  unsigned int num_stmts = 1;
+  /* If STMT_INFO replaces one of the original scalar statements,
+     check for uses of that statement's results too, to simulate the
+     effect of vect_stmt_to_be_vectorized.  */
+  if (is_main_pattern_stmt_p (stmt_info))
+    stmts[num_stmts++] = STMT_VINFO_RELATED_STMT (stmt_info)->stmt;
+  stmt_vec_info result = NULL;
+  for (unsigned int i = 0; i < num_stmts; ++i)
+    {
+      use_operand_p use_p;
+      imm_use_iterator imm_iter;
+      tree lhs = gimple_get_lhs (stmts[i]);
+      FOR_EACH_IMM_USE_FAST (use_p, imm_iter, lhs)
+	{
+	  gimple *use_stmt = USE_STMT (use_p);
+	  if (is_gimple_debug (use_stmt))
+	    continue;
+	  stmt_vec_info use_stmt_info = lookup_stmt (use_stmt);
+	  if (!use_stmt_info)
+	    return NULL;
+	  /* Ignore statements that have been replaced by a pattern
+	     and consider only the replacement statement.  */
+	  if (STMT_VINFO_IN_PATTERN_P (use_stmt_info))
+	    continue;
+	  if (result)
+	    return NULL;
+	  result = use_stmt_info;
+	}
+    }
+  return result;
 }
 
 /* Return vectorization information about DR.  */
@@ -650,16 +709,18 @@ vec_info::move_dr (stmt_vec_info new_stm
 void
 vec_info::remove_stmt (stmt_vec_info stmt_info)
 {
-  gcc_assert (!stmt_info->pattern_stmt_p);
   set_vinfo_for_stmt (stmt_info->stmt, NULL);
   gimple_stmt_iterator si = gsi_for_stmt (stmt_info->stmt);
   unlink_stmt_vdef (stmt_info->stmt);
-  if (is_a <gphi *> (stmt_info->stmt))
-    remove_phi_node (&si, true);
-  else
+  if (!is_pattern_stmt_p (stmt_info))
     {
-      gsi_remove (&si, true);
-      release_defs (stmt_info->stmt);
+      if (is_a <gphi *> (stmt_info->stmt))
+	remove_phi_node (&si, true);
+      else
+	{
+	  gsi_remove (&si, true);
+	  release_defs (stmt_info->stmt);
+	}
     }
   stmt_info->block->remove (stmt_info);
   free_stmt_vec_info (stmt_info);
@@ -749,12 +810,13 @@ vec_info::free_stmt_vec_infos (void)
 void
 vec_info::free_stmt_vec_info (stmt_vec_info stmt_info)
 {
-  if (stmt_info->pattern_stmt_p)
+  if (is_pattern_stmt_p (stmt_info))
     {
-      gimple_set_bb (stmt_info->stmt, NULL);
       tree lhs = gimple_get_lhs (stmt_info->stmt);
       if (lhs && TREE_CODE (lhs) == SSA_NAME)
 	release_ssa_name (lhs);
+      if (stmt_info->block)
+	remove_pattern_stmt_from_block (stmt_info);
     }
 
   STMT_VINFO_SAME_ALIGN_REFS (stmt_info).release ();
Index: gcc/tree-vect-patterns.c
===================================================================
--- gcc/tree-vect-patterns.c	2018-08-28 12:05:16.518940320 +0100
+++ gcc/tree-vect-patterns.c	2018-08-28 12:05:19.262917177 +0100
@@ -106,7 +106,6 @@ vect_init_pattern_stmt (gimple *pattern_
   stmt_vec_info pattern_stmt_info = vinfo->lookup_stmt (pattern_stmt);
   if (pattern_stmt_info == NULL)
     pattern_stmt_info = orig_stmt_info->vinfo->add_stmt (pattern_stmt);
-  gimple_set_bb (pattern_stmt, gimple_bb (orig_stmt_info->stmt));
 
   pattern_stmt_info->pattern_stmt_p = true;
   STMT_VINFO_RELATED_STMT (pattern_stmt_info) = orig_stmt_info;
@@ -412,11 +411,9 @@ vect_look_through_possible_promotion (ve
 	break;
       caster = def_stmt_info;
 
-      /* Ignore pattern statements, since we don't link uses for them.  */
       if (caster
 	  && single_use_p
-	  && !STMT_VINFO_RELATED_STMT (caster)
-	  && !has_single_use (res))
+	  && !vinfo->lookup_single_use (caster))
 	*single_use_p = false;
 
       gassign *assign = dyn_cast <gassign *> (def_stmt);