[RFC] Don't move cold code out of loop by checking bb count

[RFC] Don't move cold code out of loop by checking bb count

[Xiong Hu Luo]

There was a patch trying to avoid move cold block out of loop:

https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html

Richard suggested to "never hoist anything from a bb with lower execution
frequency to a bb with higher one in LIM invariantness_dom_walker
before_dom_children".

This patch does this profile count check in both gimple LIM
move_computations_worker and RTL loop-invariant.c find_invariants_bb,
if the loop bb is colder than loop preheader, don't hoist it out of
loop.

Also, the profile count in loop split pass should be corrected to avoid
lim2 and lim4 mismatch behavior, currently, the new loop preheader generated
by loop_version is set to "[count: 0]:", then lim4 after lsplt pass will
move statement out of loop unexpectely when lim2 didn't move it.  This
change could fix regression on 544.nab_r from -1.55% to +0.46%.

SPEC2017 performance evaluation shows 1% performance improvement for
intrate GEOMEAN and no obvious regression for others.  Especially,
500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
on P8LE.

Regression and bootstrap tested pass on P8LE, any comments?  Thanks.

gcc/ChangeLog:

	* loop-invariant.c (find_invariants_bb): Check profile count
	before motion.
	(find_invariants_body): Add argument.
	* tree-ssa-loop-im.c (move_computations_worker): Check profile
	count before motion.
	(execute_sm): Likewise.
	(execute_sm_exit): Check pointer validness.
	* tree-ssa-loop-split.c (split_loop): Correct probability.
	(do_split_loop_on_cond): Likewise.

gcc/testsuite/ChangeLog:

	* gcc.dg/tree-ssa/recip-3.c: Adjust.
---
 gcc/loop-invariant.c                    |  10 +-
 gcc/testsuite/gcc.dg/tree-ssa/recip-3.c |   2 +-
 gcc/tree-ssa-loop-im.c                  | 164 +++++++++++++++++++++++-
 gcc/tree-ssa-loop-split.c               |  14 +-
 4 files changed, 177 insertions(+), 13 deletions(-)

diff --git a/gcc/loop-invariant.c b/gcc/loop-invariant.c
index bdc7b59dd5f..7b5d64d11f9 100644
--- a/gcc/loop-invariant.c
+++ b/gcc/loop-invariant.c
@@ -1183,9 +1183,14 @@ find_invariants_insn (rtx_insn *insn, bool always_reached, bool always_executed)
    call.  */
 
 static void
-find_invariants_bb (basic_block bb, bool always_reached, bool always_executed)
+find_invariants_bb (class loop *loop, basic_block bb, bool always_reached,
+		    bool always_executed)
 {
   rtx_insn *insn;
+  basic_block preheader = loop_preheader_edge (loop)->src;
+
+  if (preheader->count > bb->count)
+    return;
 
   FOR_BB_INSNS (bb, insn)
     {
@@ -1214,8 +1219,7 @@ find_invariants_body (class loop *loop, basic_block *body,
   unsigned i;
 
   for (i = 0; i < loop->num_nodes; i++)
-    find_invariants_bb (body[i],
-			bitmap_bit_p (always_reached, i),
+    find_invariants_bb (loop, body[i], bitmap_bit_p (always_reached, i),
 			bitmap_bit_p (always_executed, i));
 }
 
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
index 638bf38db8c..641c91e719e 100644
--- a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
+++ b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
@@ -23,4 +23,4 @@ float h ()
 	F[0] += E / d;
 }
 
-/* { dg-final { scan-tree-dump-times " / " 1 "recip" } } */
+/* { dg-final { scan-tree-dump-times " / " 5 "recip" } } */
diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
index 7de47edbcb3..2bfb5e8ec15 100644
--- a/gcc/tree-ssa-loop-im.c
+++ b/gcc/tree-ssa-loop-im.c
@@ -1147,6 +1147,61 @@ move_computations_worker (basic_block bb)
 	  continue;
 	}
 
+      edge e = loop_preheader_edge (level);
+      if (e->src->count > bb->count)
+	{
+	  if (dump_file && (dump_flags & TDF_DETAILS))
+	    {
+	      fprintf (dump_file, "PHI node NOT moved to %d from %d:\n",
+		       e->src->index, bb->index);
+	      print_gimple_stmt (dump_file, stmt, 0);
+	      fprintf (dump_file, "(cost %u) out of loop %d.\n\n", cost,
+		       level->num);
+	    }
+	  gsi_next (&bsi);
+	  continue;
+	}
+      else
+	{
+	  unsigned i;
+	  bool skip_phi_move = false;
+	  for (i = 0; i < gimple_phi_num_args (stmt); i++)
+	    {
+	      tree def = PHI_ARG_DEF (stmt, i);
+
+	      if (TREE_CODE (def) != SSA_NAME)
+		continue;
+
+	      gimple *def_stmt = SSA_NAME_DEF_STMT (def);
+
+	      if (!gimple_bb (def_stmt))
+		continue;
+
+	      if (!dominated_by_p (CDI_DOMINATORS, e->src,
+				   gimple_bb (def_stmt)))
+		{
+		  if (dump_file && (dump_flags & TDF_DETAILS))
+		    {
+		      fprintf (dump_file,
+			       "PHI node NOT moved to %d [local count:%d] from "
+			       "%d [local count:%d]:\n",
+			       e->src->index, e->src->count.value (), bb->index,
+			       bb->count.value ());
+		      print_gimple_stmt (dump_file, stmt, 0);
+		      fprintf (dump_file, "(cost %u) out of loop %d.\n\n", cost,
+			       level->num);
+		    }
+		  skip_phi_move = true;
+		  break;
+		}
+	    }
+	  if (skip_phi_move)
+	    {
+	      gsi_next (&bsi);
+	      continue;
+	    }
+	}
+
       if (dump_file && (dump_flags & TDF_DETAILS))
 	{
 	  fprintf (dump_file, "Moving PHI node\n");
@@ -1184,14 +1239,13 @@ move_computations_worker (basic_block bb)
 	  tree lhs = gimple_assign_lhs (new_stmt);
 	  SSA_NAME_RANGE_INFO (lhs) = NULL;
 	}
-      gsi_insert_on_edge (loop_preheader_edge (level), new_stmt);
+      gsi_insert_on_edge (e, new_stmt);
       remove_phi_node (&bsi, false);
     }
 
   for (gimple_stmt_iterator bsi = gsi_start_bb (bb); !gsi_end_p (bsi); )
     {
       edge e;
-
       gimple *stmt = gsi_stmt (bsi);
 
       lim_data = get_lim_data (stmt);
@@ -1214,7 +1268,90 @@ move_computations_worker (basic_block bb)
       /* We do not really want to move conditionals out of the loop; we just
 	 placed it here to force its operands to be moved if necessary.  */
       if (gimple_code (stmt) == GIMPLE_COND)
-	continue;
+	{
+	  gsi_next (&bsi);
+	  continue;
+	}
+
+      e = loop_preheader_edge (level);
+      if (e->src->count > bb->count)
+	{
+	  if (dump_file && (dump_flags & TDF_DETAILS))
+	    {
+	      fprintf (dump_file,
+		       "stmt: Statement NOT moved to %d [local count:%d] from "
+		       "%d [local count:%d]:\n",
+		       e->src->index, e->src->count.value (), bb->index,
+		       bb->count.value ());
+	      print_gimple_stmt (dump_file, stmt, 0);
+	      fprintf (dump_file, "(cost %u) out of loop %d.\n\n", cost,
+		       level->num);
+	    }
+	  gsi_next (&bsi);
+	  continue;
+	}
+      else
+	{
+	  if (is_gimple_assign (stmt))
+	    {
+	      tree rhs1 = gimple_assign_rhs1 (stmt);
+	      tree rhs2 = gimple_assign_rhs2 (stmt);
+	      if (TREE_CODE (rhs1) == MEM_REF)
+		{
+		  rhs2 = TREE_OPERAND (rhs1, 1);
+		  rhs1 = TREE_OPERAND (rhs1, 0);
+		}
+	      gimple *stmt1 = NULL, *stmt2 = NULL;
+	      basic_block def_bb;
+	      if (rhs1 && TREE_CODE (rhs1) == SSA_NAME)
+		{
+		  stmt1 = SSA_NAME_DEF_STMT (rhs1);
+		  def_bb = gimple_bb (stmt1);
+		  if (stmt1
+		      && def_bb
+		      && (def_bb == bb
+			  || !dominated_by_p (CDI_DOMINATORS, e->src, def_bb)))
+		    {
+		      if (dump_file && (dump_flags & TDF_DETAILS))
+			{
+			  fprintf (dump_file,
+				   "stmt1: Statement NOT moved to %d [local "
+				   "count:%d] from %d [local count:%d]:\n",
+				   e->src->index, e->src->count.value (),
+				   bb->index, bb->count.value ());
+			  print_gimple_stmt (dump_file, stmt, 0);
+			  fprintf (dump_file, "(cost %u) out of loop %d.\n\n",
+				   cost, level->num);
+			}
+		      gsi_next (&bsi);
+		      continue;
+		    }
+		}
+	      if (rhs2 && TREE_CODE (rhs2) == SSA_NAME)
+		{
+		  stmt2 = SSA_NAME_DEF_STMT (rhs2);
+		  def_bb = gimple_bb (stmt2);
+		  if (stmt2 && def_bb
+		      && (def_bb == bb
+			  || !dominated_by_p (CDI_DOMINATORS, e->src, def_bb)))
+		    {
+		      if (dump_file && (dump_flags & TDF_DETAILS))
+			{
+			  fprintf (dump_file,
+				   "stmt2: Statement NOT moved to %d [local "
+				   "count:%d] from %d [local count:%d]:\n",
+				   e->src->index, e->src->count.value (),
+				   bb->index, bb->count.value ());
+			  print_gimple_stmt (dump_file, stmt, 0);
+			  fprintf (dump_file, "(cost %u) out of loop %d.\n\n",
+				   cost, level->num);
+			}
+		      gsi_next (&bsi);
+		      continue;
+		    }
+		}
+	    }
+	}
 
       if (dump_file && (dump_flags & TDF_DETAILS))
 	{
@@ -1224,7 +1361,6 @@ move_computations_worker (basic_block bb)
 		   cost, level->num);
 	}
 
-      e = loop_preheader_edge (level);
       gcc_assert (!gimple_vdef (stmt));
       if (gimple_vuse (stmt))
 	{
@@ -2094,6 +2230,19 @@ execute_sm (class loop *loop, im_mem_ref *ref,
   bool multi_threaded_model_p = false;
   gimple_stmt_iterator gsi;
   sm_aux *aux = new sm_aux;
+  basic_block bb = gimple_bb (first_mem_ref_loc (loop, ref)->stmt);
+
+  edge e = loop_preheader_edge (loop);
+  if (e->src->count > bb->count)
+    {
+      if (dump_file && (dump_flags & TDF_DETAILS))
+	{
+	  fprintf (dump_file, "Don't execute store motion of ");
+	  print_generic_expr (dump_file, ref->mem.ref);
+	  fprintf (dump_file, " from loop %d\n", loop->num);
+	}
+      return;
+    }
 
   if (dump_file && (dump_flags & TDF_DETAILS))
     {
@@ -2202,7 +2351,12 @@ execute_sm_exit (class loop *loop, edge ex, vec<seq_entry> &seq,
 	}
       else
 	{
-	  sm_aux *aux = *aux_map.get (ref);
+	  sm_aux **paux = aux_map.get (ref);
+	  sm_aux *aux;
+	  if (paux)
+	    aux = *paux;
+	  else
+	    continue;
 	  if (!aux->store_flag || kind == sm_ord)
 	    {
 	      gassign *store;
diff --git a/gcc/tree-ssa-loop-split.c b/gcc/tree-ssa-loop-split.c
index 3a09bbc39e5..4cae82936b9 100644
--- a/gcc/tree-ssa-loop-split.c
+++ b/gcc/tree-ssa-loop-split.c
@@ -577,14 +577,17 @@ split_loop (class loop *loop1)
 	if (!initial_true)
 	  cond = fold_build1 (TRUTH_NOT_EXPR, boolean_type_node, cond); 
 
+	edge true_edge = EDGE_SUCC (bbs[i], 0)->flags & EDGE_TRUE_VALUE
+			   ? EDGE_SUCC (bbs[i], 0)
+			   : EDGE_SUCC (bbs[i], 1);
 	/* Now version the loop, placing loop2 after loop1 connecting
 	   them, and fix up SSA form for that.  */
 	initialize_original_copy_tables ();
 	basic_block cond_bb;
 
 	class loop *loop2 = loop_version (loop1, cond, &cond_bb,
-					   profile_probability::always (),
-					   profile_probability::always (),
+					   true_edge->probability,
+					   true_edge->probability.invert (),
 					   profile_probability::always (),
 					   profile_probability::always (),
 					   true);
@@ -1486,8 +1489,8 @@ do_split_loop_on_cond (struct loop *loop1, edge invar_branch)
   initialize_original_copy_tables ();
 
   struct loop *loop2 = loop_version (loop1, boolean_true_node, NULL,
-				     profile_probability::always (),
-				     profile_probability::never (),
+				     invar_branch->probability,
+				     invar_branch->probability.invert (),
 				     profile_probability::always (),
 				     profile_probability::always (),
 				     true);
@@ -1530,6 +1533,9 @@ do_split_loop_on_cond (struct loop *loop1, edge invar_branch)
   to_loop1->flags |= true_invar ? EDGE_FALSE_VALUE : EDGE_TRUE_VALUE;
   to_loop2->flags |= true_invar ? EDGE_TRUE_VALUE : EDGE_FALSE_VALUE;
 
+  to_loop1->probability = invar_branch->probability.invert ();
+  to_loop2->probability = invar_branch->probability;
+
   /* Due to introduction of a control flow edge from loop1 latch to loop2
      pre-header, we should update PHIs in loop2 to reflect this connection
      between loop1 and loop2.  */
-- 
2.27.0.90.geebb51ba8c

Re: [RFC] Don't move cold code out of loop by checking bb count

[Richard Biener]

On Mon, Aug 2, 2021 at 7:05 AM Xiong Hu Luo <luoxhu@linux.ibm.com> wrote:
>
> There was a patch trying to avoid move cold block out of loop:
>
> https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html
>
> Richard suggested to "never hoist anything from a bb with lower execution
> frequency to a bb with higher one in LIM invariantness_dom_walker
> before_dom_children".
>
> This patch does this profile count check in both gimple LIM
> move_computations_worker and RTL loop-invariant.c find_invariants_bb,
> if the loop bb is colder than loop preheader, don't hoist it out of
> loop.
>
> Also, the profile count in loop split pass should be corrected to avoid
> lim2 and lim4 mismatch behavior, currently, the new loop preheader generated
> by loop_version is set to "[count: 0]:", then lim4 after lsplt pass will
> move statement out of loop unexpectely when lim2 didn't move it.  This
> change could fix regression on 544.nab_r from -1.55% to +0.46%.
>
> SPEC2017 performance evaluation shows 1% performance improvement for
> intrate GEOMEAN and no obvious regression for others.  Especially,
> 500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
> largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
> on P8LE.
>
> Regression and bootstrap tested pass on P8LE, any comments?  Thanks.

While I'm not familiar with the RTL invariant motion pass the patch there
looks reasonable.  Note that we should assess the profile quality
somehow - I'm not sure how to do that, CCed Honza for that.

For the GIMPLE part the patch looks quite complicated - but note it
probably has to be since LIM performs kind of a "CSE" on loads
(and stores for store-motion), so when there are multiple stmts
affected by a hoisting decision the biggest block count has to be
accounted.  Likewise when there are dependent stmts involved
that might include conditional stmts (a "PHI"), but the overall
cost should be looked at.

Now - GIMPLE LIM "costing" is somewhat backward right now
and it isn't set up to consider those multiple involved stmts.  Plus
the store-motion part does not have any cost part (but it depends
on previously decided invariant motions).

I think the way you implemented the check will cause no hoisting
to be performed instead of, say, hoisting to a different loop level
only.  Possibly shown when you consider a loop nest like

  for (;;)
    if (unlikely_cond)
      for (;;)
         invariant;

we want to hoist 'invariant' but only from the inner loop even if it
is invariant also in the outer loop.  But for example if there is
a store motion opportunity like

  for (;;)
     {
        if (unlikely_cond)
          for (;;)
            a = ...;
        a = ...;
     }

we'd still want to perform the store motion on the outer loop.

Note that store-motion already performs part of the transform
before dependent code is moved in move_computations (that
you patched).

IIRC your main concern were the COND_EXPRs we insert
for hoisted conditional stmts?

Thanks,
Richard.

> gcc/ChangeLog:
>
>         * loop-invariant.c (find_invariants_bb): Check profile count
>         before motion.
>         (find_invariants_body): Add argument.
>         * tree-ssa-loop-im.c (move_computations_worker): Check profile
>         count before motion.
>         (execute_sm): Likewise.
>         (execute_sm_exit): Check pointer validness.
>         * tree-ssa-loop-split.c (split_loop): Correct probability.
>         (do_split_loop_on_cond): Likewise.
>
> gcc/testsuite/ChangeLog:
>
>         * gcc.dg/tree-ssa/recip-3.c: Adjust.
> ---
>  gcc/loop-invariant.c                    |  10 +-
>  gcc/testsuite/gcc.dg/tree-ssa/recip-3.c |   2 +-
>  gcc/tree-ssa-loop-im.c                  | 164 +++++++++++++++++++++++-
>  gcc/tree-ssa-loop-split.c               |  14 +-
>  4 files changed, 177 insertions(+), 13 deletions(-)
>
> diff --git a/gcc/loop-invariant.c b/gcc/loop-invariant.c
> index bdc7b59dd5f..7b5d64d11f9 100644
> --- a/gcc/loop-invariant.c
> +++ b/gcc/loop-invariant.c
> @@ -1183,9 +1183,14 @@ find_invariants_insn (rtx_insn *insn, bool always_reached, bool always_executed)
>     call.  */
>
>  static void
> -find_invariants_bb (basic_block bb, bool always_reached, bool always_executed)
> +find_invariants_bb (class loop *loop, basic_block bb, bool always_reached,
> +                   bool always_executed)
>  {
>    rtx_insn *insn;
> +  basic_block preheader = loop_preheader_edge (loop)->src;
> +
> +  if (preheader->count > bb->count)
> +    return;
>
>    FOR_BB_INSNS (bb, insn)
>      {
> @@ -1214,8 +1219,7 @@ find_invariants_body (class loop *loop, basic_block *body,
>    unsigned i;
>
>    for (i = 0; i < loop->num_nodes; i++)
> -    find_invariants_bb (body[i],
> -                       bitmap_bit_p (always_reached, i),
> +    find_invariants_bb (loop, body[i], bitmap_bit_p (always_reached, i),
>                         bitmap_bit_p (always_executed, i));
>  }
>
> diff --git a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
> index 638bf38db8c..641c91e719e 100644
> --- a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
> +++ b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
> @@ -23,4 +23,4 @@ float h ()
>         F[0] += E / d;
>  }
>
> -/* { dg-final { scan-tree-dump-times " / " 1 "recip" } } */
> +/* { dg-final { scan-tree-dump-times " / " 5 "recip" } } */
> diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
> index 7de47edbcb3..2bfb5e8ec15 100644
> --- a/gcc/tree-ssa-loop-im.c
> +++ b/gcc/tree-ssa-loop-im.c
> @@ -1147,6 +1147,61 @@ move_computations_worker (basic_block bb)
>           continue;
>         }
>
> +      edge e = loop_preheader_edge (level);
> +      if (e->src->count > bb->count)
> +       {
> +         if (dump_file && (dump_flags & TDF_DETAILS))
> +           {
> +             fprintf (dump_file, "PHI node NOT moved to %d from %d:\n",
> +                      e->src->index, bb->index);
> +             print_gimple_stmt (dump_file, stmt, 0);
> +             fprintf (dump_file, "(cost %u) out of loop %d.\n\n", cost,
> +                      level->num);
> +           }
> +         gsi_next (&bsi);
> +         continue;
> +       }
> +      else
> +       {
> +         unsigned i;
> +         bool skip_phi_move = false;
> +         for (i = 0; i < gimple_phi_num_args (stmt); i++)
> +           {
> +             tree def = PHI_ARG_DEF (stmt, i);
> +
> +             if (TREE_CODE (def) != SSA_NAME)
> +               continue;
> +
> +             gimple *def_stmt = SSA_NAME_DEF_STMT (def);
> +
> +             if (!gimple_bb (def_stmt))
> +               continue;
> +
> +             if (!dominated_by_p (CDI_DOMINATORS, e->src,
> +                                  gimple_bb (def_stmt)))
> +               {
> +                 if (dump_file && (dump_flags & TDF_DETAILS))
> +                   {
> +                     fprintf (dump_file,
> +                              "PHI node NOT moved to %d [local count:%d] from "
> +                              "%d [local count:%d]:\n",
> +                              e->src->index, e->src->count.value (), bb->index,
> +                              bb->count.value ());
> +                     print_gimple_stmt (dump_file, stmt, 0);
> +                     fprintf (dump_file, "(cost %u) out of loop %d.\n\n", cost,
> +                              level->num);
> +                   }
> +                 skip_phi_move = true;
> +                 break;
> +               }
> +           }
> +         if (skip_phi_move)
> +           {
> +             gsi_next (&bsi);
> +             continue;
> +           }
> +       }
> +
>        if (dump_file && (dump_flags & TDF_DETAILS))
>         {
>           fprintf (dump_file, "Moving PHI node\n");
> @@ -1184,14 +1239,13 @@ move_computations_worker (basic_block bb)
>           tree lhs = gimple_assign_lhs (new_stmt);
>           SSA_NAME_RANGE_INFO (lhs) = NULL;
>         }
> -      gsi_insert_on_edge (loop_preheader_edge (level), new_stmt);
> +      gsi_insert_on_edge (e, new_stmt);
>        remove_phi_node (&bsi, false);
>      }
>
>    for (gimple_stmt_iterator bsi = gsi_start_bb (bb); !gsi_end_p (bsi); )
>      {
>        edge e;
> -
>        gimple *stmt = gsi_stmt (bsi);
>
>        lim_data = get_lim_data (stmt);
> @@ -1214,7 +1268,90 @@ move_computations_worker (basic_block bb)
>        /* We do not really want to move conditionals out of the loop; we just
>          placed it here to force its operands to be moved if necessary.  */
>        if (gimple_code (stmt) == GIMPLE_COND)
> -       continue;
> +       {
> +         gsi_next (&bsi);
> +         continue;
> +       }
> +
> +      e = loop_preheader_edge (level);
> +      if (e->src->count > bb->count)
> +       {
> +         if (dump_file && (dump_flags & TDF_DETAILS))
> +           {
> +             fprintf (dump_file,
> +                      "stmt: Statement NOT moved to %d [local count:%d] from "
> +                      "%d [local count:%d]:\n",
> +                      e->src->index, e->src->count.value (), bb->index,
> +                      bb->count.value ());
> +             print_gimple_stmt (dump_file, stmt, 0);
> +             fprintf (dump_file, "(cost %u) out of loop %d.\n\n", cost,
> +                      level->num);
> +           }
> +         gsi_next (&bsi);
> +         continue;
> +       }
> +      else
> +       {
> +         if (is_gimple_assign (stmt))
> +           {
> +             tree rhs1 = gimple_assign_rhs1 (stmt);
> +             tree rhs2 = gimple_assign_rhs2 (stmt);
> +             if (TREE_CODE (rhs1) == MEM_REF)
> +               {
> +                 rhs2 = TREE_OPERAND (rhs1, 1);
> +                 rhs1 = TREE_OPERAND (rhs1, 0);
> +               }
> +             gimple *stmt1 = NULL, *stmt2 = NULL;
> +             basic_block def_bb;
> +             if (rhs1 && TREE_CODE (rhs1) == SSA_NAME)
> +               {
> +                 stmt1 = SSA_NAME_DEF_STMT (rhs1);
> +                 def_bb = gimple_bb (stmt1);
> +                 if (stmt1
> +                     && def_bb
> +                     && (def_bb == bb
> +                         || !dominated_by_p (CDI_DOMINATORS, e->src, def_bb)))
> +                   {
> +                     if (dump_file && (dump_flags & TDF_DETAILS))
> +                       {
> +                         fprintf (dump_file,
> +                                  "stmt1: Statement NOT moved to %d [local "
> +                                  "count:%d] from %d [local count:%d]:\n",
> +                                  e->src->index, e->src->count.value (),
> +                                  bb->index, bb->count.value ());
> +                         print_gimple_stmt (dump_file, stmt, 0);
> +                         fprintf (dump_file, "(cost %u) out of loop %d.\n\n",
> +                                  cost, level->num);
> +                       }
> +                     gsi_next (&bsi);
> +                     continue;
> +                   }
> +               }
> +             if (rhs2 && TREE_CODE (rhs2) == SSA_NAME)
> +               {
> +                 stmt2 = SSA_NAME_DEF_STMT (rhs2);
> +                 def_bb = gimple_bb (stmt2);
> +                 if (stmt2 && def_bb
> +                     && (def_bb == bb
> +                         || !dominated_by_p (CDI_DOMINATORS, e->src, def_bb)))
> +                   {
> +                     if (dump_file && (dump_flags & TDF_DETAILS))
> +                       {
> +                         fprintf (dump_file,
> +                                  "stmt2: Statement NOT moved to %d [local "
> +                                  "count:%d] from %d [local count:%d]:\n",
> +                                  e->src->index, e->src->count.value (),
> +                                  bb->index, bb->count.value ());
> +                         print_gimple_stmt (dump_file, stmt, 0);
> +                         fprintf (dump_file, "(cost %u) out of loop %d.\n\n",
> +                                  cost, level->num);
> +                       }
> +                     gsi_next (&bsi);
> +                     continue;
> +                   }
> +               }
> +           }
> +       }
>
>        if (dump_file && (dump_flags & TDF_DETAILS))
>         {
> @@ -1224,7 +1361,6 @@ move_computations_worker (basic_block bb)
>                    cost, level->num);
>         }
>
> -      e = loop_preheader_edge (level);
>        gcc_assert (!gimple_vdef (stmt));
>        if (gimple_vuse (stmt))
>         {
> @@ -2094,6 +2230,19 @@ execute_sm (class loop *loop, im_mem_ref *ref,
>    bool multi_threaded_model_p = false;
>    gimple_stmt_iterator gsi;
>    sm_aux *aux = new sm_aux;
> +  basic_block bb = gimple_bb (first_mem_ref_loc (loop, ref)->stmt);
> +
> +  edge e = loop_preheader_edge (loop);
> +  if (e->src->count > bb->count)
> +    {
> +      if (dump_file && (dump_flags & TDF_DETAILS))
> +       {
> +         fprintf (dump_file, "Don't execute store motion of ");
> +         print_generic_expr (dump_file, ref->mem.ref);
> +         fprintf (dump_file, " from loop %d\n", loop->num);
> +       }
> +      return;
> +    }
>
>    if (dump_file && (dump_flags & TDF_DETAILS))
>      {
> @@ -2202,7 +2351,12 @@ execute_sm_exit (class loop *loop, edge ex, vec<seq_entry> &seq,
>         }
>        else
>         {
> -         sm_aux *aux = *aux_map.get (ref);
> +         sm_aux **paux = aux_map.get (ref);
> +         sm_aux *aux;
> +         if (paux)
> +           aux = *paux;
> +         else
> +           continue;
>           if (!aux->store_flag || kind == sm_ord)
>             {
>               gassign *store;
> diff --git a/gcc/tree-ssa-loop-split.c b/gcc/tree-ssa-loop-split.c
> index 3a09bbc39e5..4cae82936b9 100644
> --- a/gcc/tree-ssa-loop-split.c
> +++ b/gcc/tree-ssa-loop-split.c
> @@ -577,14 +577,17 @@ split_loop (class loop *loop1)
>         if (!initial_true)
>           cond = fold_build1 (TRUTH_NOT_EXPR, boolean_type_node, cond);
>
> +       edge true_edge = EDGE_SUCC (bbs[i], 0)->flags & EDGE_TRUE_VALUE
> +                          ? EDGE_SUCC (bbs[i], 0)
> +                          : EDGE_SUCC (bbs[i], 1);
>         /* Now version the loop, placing loop2 after loop1 connecting
>            them, and fix up SSA form for that.  */
>         initialize_original_copy_tables ();
>         basic_block cond_bb;
>
>         class loop *loop2 = loop_version (loop1, cond, &cond_bb,
> -                                          profile_probability::always (),
> -                                          profile_probability::always (),
> +                                          true_edge->probability,
> +                                          true_edge->probability.invert (),
>                                            profile_probability::always (),
>                                            profile_probability::always (),
>                                            true);
> @@ -1486,8 +1489,8 @@ do_split_loop_on_cond (struct loop *loop1, edge invar_branch)
>    initialize_original_copy_tables ();
>
>    struct loop *loop2 = loop_version (loop1, boolean_true_node, NULL,
> -                                    profile_probability::always (),
> -                                    profile_probability::never (),
> +                                    invar_branch->probability,
> +                                    invar_branch->probability.invert (),
>                                      profile_probability::always (),
>                                      profile_probability::always (),
>                                      true);
> @@ -1530,6 +1533,9 @@ do_split_loop_on_cond (struct loop *loop1, edge invar_branch)
>    to_loop1->flags |= true_invar ? EDGE_FALSE_VALUE : EDGE_TRUE_VALUE;
>    to_loop2->flags |= true_invar ? EDGE_TRUE_VALUE : EDGE_FALSE_VALUE;
>
> +  to_loop1->probability = invar_branch->probability.invert ();
> +  to_loop2->probability = invar_branch->probability;
> +
>    /* Due to introduction of a control flow edge from loop1 latch to loop2
>       pre-header, we should update PHIs in loop2 to reflect this connection
>       between loop1 and loop2.  */
> --
> 2.27.0.90.geebb51ba8c
>

Re: [RFC] Don't move cold code out of loop by checking bb count

[Xionghu Luo]

Hi,

On 2021/8/6 20:15, Richard Biener wrote:
> On Mon, Aug 2, 2021 at 7:05 AM Xiong Hu Luo <luoxhu@linux.ibm.com> wrote:
>>
>> There was a patch trying to avoid move cold block out of loop:
>>
>> https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html
>>
>> Richard suggested to "never hoist anything from a bb with lower execution
>> frequency to a bb with higher one in LIM invariantness_dom_walker
>> before_dom_children".
>>
>> This patch does this profile count check in both gimple LIM
>> move_computations_worker and RTL loop-invariant.c find_invariants_bb,
>> if the loop bb is colder than loop preheader, don't hoist it out of
>> loop.
>>
>> Also, the profile count in loop split pass should be corrected to avoid
>> lim2 and lim4 mismatch behavior, currently, the new loop preheader generated
>> by loop_version is set to "[count: 0]:", then lim4 after lsplt pass will
>> move statement out of loop unexpectely when lim2 didn't move it.  This
>> change could fix regression on 544.nab_r from -1.55% to +0.46%.
>>
>> SPEC2017 performance evaluation shows 1% performance improvement for
>> intrate GEOMEAN and no obvious regression for others.  Especially,
>> 500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
>> largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
>> on P8LE.
>>
>> Regression and bootstrap tested pass on P8LE, any comments?  Thanks.
> 
> While I'm not familiar with the RTL invariant motion pass the patch there
> looks reasonable.  Note that we should assess the profile quality
> somehow - I'm not sure how to do that, CCed Honza for that.

Thanks.

> 
> For the GIMPLE part the patch looks quite complicated - but note it
> probably has to be since LIM performs kind of a "CSE" on loads
> (and stores for store-motion), so when there are multiple stmts
> affected by a hoisting decision the biggest block count has to be
> accounted.  Likewise when there are dependent stmts involved
> that might include conditional stmts (a "PHI"), but the overall
> cost should be looked at.

Currently, The gimple code check two situations with the patch:
1) The statement or PHI‘s BB is *colder* then preheader, don't move it out
of loop;
2) The statement or PHI's BB is *hotter* then preheader, but any of it's rhs
couldn't be moved out of loop, also don't move it out of loop to avoid definition
not dominates use error.

May be I could collect the number of instructions not hoisted with the patch
on regression tests and SPEC2017 to do a estimation for "multiple stmts affected"
and "overall cost" need to be considered?  But it seems move_computations_worker
couldn't rollback if we still want to hoist multiple stmts out during the iterations?

> 
> Now - GIMPLE LIM "costing" is somewhat backward right now
> and it isn't set up to consider those multiple involved stmts.  Plus
> the store-motion part does not have any cost part (but it depends
> on previously decided invariant motions).
> 
> I think the way you implemented the check will cause no hoisting
> to be performed instead of, say, hoisting to a different loop level
> only.  Possibly shown when you consider a loop nest like
> 
>    for (;;)
>      if (unlikely_cond)
>        for (;;)
>           invariant;
> 
> we want to hoist 'invariant' but only from the inner loop even if it
> is invariant also in the outer loop.


For this case, theorotically I think the master GCC will optimize it to:

  invariant;
  for (;;)
    if (unlikely_cond)
      for (;;)
         ;

'invariant' is moved out of outer loop, but with the patch, it will get:

  for (;;)
    if (unlikely_cond)
      {
        invariant;
        for (;;)
           ;
      }

'invariant' is *cold* for outer loop, but it is still *hot* for inner loop,
so hoist it out of inner loop, this is exactly what we want, right?


>  But for example if there is
> a store motion opportunity like
> 
>    for (;;)
>       {
>          if (unlikely_cond)
>            for (;;)
>              a = ...;
>          a = ...;
>       }
> 
> we'd still want to perform the store motion on the outer loop.
> 
> Note that store-motion already performs part of the transform
> before dependent code is moved in move_computations (that
> you patched).

Yes.  do_store_motion is running before move_computations_worker, store
motion happens earlier in execute_sm, I also added the check in execute_sm
to stop cold store moved out of loop.  So for your case, I think my patch
will similarly optimize it to:

  for (;;)
     {
        if (unlikely_cond)
          {
            for (;;)
              ;
            a = ...;
          }
     }
    a = ...;

Whether this is better?  Will construct cases to verify it.

> 
> IIRC your main concern were the COND_EXPRs we insert
> for hoisted conditional stmts?

Not sure what you mean here of COND_EXPRs?


Thanks,
Xionghu

> 
> Thanks,
> Richard.
> 
>> gcc/ChangeLog:
>>
>>          * loop-invariant.c (find_invariants_bb): Check profile count
>>          before motion.
>>          (find_invariants_body): Add argument.
>>          * tree-ssa-loop-im.c (move_computations_worker): Check profile
>>          count before motion.
>>          (execute_sm): Likewise.
>>          (execute_sm_exit): Check pointer validness.
>>          * tree-ssa-loop-split.c (split_loop): Correct probability.
>>          (do_split_loop_on_cond): Likewise.
>>
>> gcc/testsuite/ChangeLog:
>>
>>          * gcc.dg/tree-ssa/recip-3.c: Adjust.
>> ---
>>   gcc/loop-invariant.c                    |  10 +-
>>   gcc/testsuite/gcc.dg/tree-ssa/recip-3.c |   2 +-
>>   gcc/tree-ssa-loop-im.c                  | 164 +++++++++++++++++++++++-
>>   gcc/tree-ssa-loop-split.c               |  14 +-
>>   4 files changed, 177 insertions(+), 13 deletions(-)
>>
>> diff --git a/gcc/loop-invariant.c b/gcc/loop-invariant.c
>> index bdc7b59dd5f..7b5d64d11f9 100644
>> --- a/gcc/loop-invariant.c
>> +++ b/gcc/loop-invariant.c
>> @@ -1183,9 +1183,14 @@ find_invariants_insn (rtx_insn *insn, bool always_reached, bool always_executed)
>>      call.  */
>>
>>   static void
>> -find_invariants_bb (basic_block bb, bool always_reached, bool always_executed)
>> +find_invariants_bb (class loop *loop, basic_block bb, bool always_reached,
>> +                   bool always_executed)
>>   {
>>     rtx_insn *insn;
>> +  basic_block preheader = loop_preheader_edge (loop)->src;
>> +
>> +  if (preheader->count > bb->count)
>> +    return;
>>
>>     FOR_BB_INSNS (bb, insn)
>>       {
>> @@ -1214,8 +1219,7 @@ find_invariants_body (class loop *loop, basic_block *body,
>>     unsigned i;
>>
>>     for (i = 0; i < loop->num_nodes; i++)
>> -    find_invariants_bb (body[i],
>> -                       bitmap_bit_p (always_reached, i),
>> +    find_invariants_bb (loop, body[i], bitmap_bit_p (always_reached, i),
>>                          bitmap_bit_p (always_executed, i));
>>   }
>>
>> diff --git a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
>> index 638bf38db8c..641c91e719e 100644
>> --- a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
>> +++ b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
>> @@ -23,4 +23,4 @@ float h ()
>>          F[0] += E / d;
>>   }
>>
>> -/* { dg-final { scan-tree-dump-times " / " 1 "recip" } } */
>> +/* { dg-final { scan-tree-dump-times " / " 5 "recip" } } */
>> diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
>> index 7de47edbcb3..2bfb5e8ec15 100644
>> --- a/gcc/tree-ssa-loop-im.c
>> +++ b/gcc/tree-ssa-loop-im.c
>> @@ -1147,6 +1147,61 @@ move_computations_worker (basic_block bb)
>>            continue;
>>          }
>>
>> +      edge e = loop_preheader_edge (level);
>> +      if (e->src->count > bb->count)
>> +       {
>> +         if (dump_file && (dump_flags & TDF_DETAILS))
>> +           {
>> +             fprintf (dump_file, "PHI node NOT moved to %d from %d:\n",
>> +                      e->src->index, bb->index);
>> +             print_gimple_stmt (dump_file, stmt, 0);
>> +             fprintf (dump_file, "(cost %u) out of loop %d.\n\n", cost,
>> +                      level->num);
>> +           }
>> +         gsi_next (&bsi);
>> +         continue;
>> +       }
>> +      else
>> +       {
>> +         unsigned i;
>> +         bool skip_phi_move = false;
>> +         for (i = 0; i < gimple_phi_num_args (stmt); i++)
>> +           {
>> +             tree def = PHI_ARG_DEF (stmt, i);
>> +
>> +             if (TREE_CODE (def) != SSA_NAME)
>> +               continue;
>> +
>> +             gimple *def_stmt = SSA_NAME_DEF_STMT (def);
>> +
>> +             if (!gimple_bb (def_stmt))
>> +               continue;
>> +
>> +             if (!dominated_by_p (CDI_DOMINATORS, e->src,
>> +                                  gimple_bb (def_stmt)))
>> +               {
>> +                 if (dump_file && (dump_flags & TDF_DETAILS))
>> +                   {
>> +                     fprintf (dump_file,
>> +                              "PHI node NOT moved to %d [local count:%d] from "
>> +                              "%d [local count:%d]:\n",
>> +                              e->src->index, e->src->count.value (), bb->index,
>> +                              bb->count.value ());
>> +                     print_gimple_stmt (dump_file, stmt, 0);
>> +                     fprintf (dump_file, "(cost %u) out of loop %d.\n\n", cost,
>> +                              level->num);
>> +                   }
>> +                 skip_phi_move = true;
>> +                 break;
>> +               }
>> +           }
>> +         if (skip_phi_move)
>> +           {
>> +             gsi_next (&bsi);
>> +             continue;
>> +           }
>> +       }
>> +
>>         if (dump_file && (dump_flags & TDF_DETAILS))
>>          {
>>            fprintf (dump_file, "Moving PHI node\n");
>> @@ -1184,14 +1239,13 @@ move_computations_worker (basic_block bb)
>>            tree lhs = gimple_assign_lhs (new_stmt);
>>            SSA_NAME_RANGE_INFO (lhs) = NULL;
>>          }
>> -      gsi_insert_on_edge (loop_preheader_edge (level), new_stmt);
>> +      gsi_insert_on_edge (e, new_stmt);
>>         remove_phi_node (&bsi, false);
>>       }
>>
>>     for (gimple_stmt_iterator bsi = gsi_start_bb (bb); !gsi_end_p (bsi); )
>>       {
>>         edge e;
>> -
>>         gimple *stmt = gsi_stmt (bsi);
>>
>>         lim_data = get_lim_data (stmt);
>> @@ -1214,7 +1268,90 @@ move_computations_worker (basic_block bb)
>>         /* We do not really want to move conditionals out of the loop; we just
>>           placed it here to force its operands to be moved if necessary.  */
>>         if (gimple_code (stmt) == GIMPLE_COND)
>> -       continue;
>> +       {
>> +         gsi_next (&bsi);
>> +         continue;
>> +       }
>> +
>> +      e = loop_preheader_edge (level);
>> +      if (e->src->count > bb->count)
>> +       {
>> +         if (dump_file && (dump_flags & TDF_DETAILS))
>> +           {
>> +             fprintf (dump_file,
>> +                      "stmt: Statement NOT moved to %d [local count:%d] from "
>> +                      "%d [local count:%d]:\n",
>> +                      e->src->index, e->src->count.value (), bb->index,
>> +                      bb->count.value ());
>> +             print_gimple_stmt (dump_file, stmt, 0);
>> +             fprintf (dump_file, "(cost %u) out of loop %d.\n\n", cost,
>> +                      level->num);
>> +           }
>> +         gsi_next (&bsi);
>> +         continue;
>> +       }
>> +      else
>> +       {
>> +         if (is_gimple_assign (stmt))
>> +           {
>> +             tree rhs1 = gimple_assign_rhs1 (stmt);
>> +             tree rhs2 = gimple_assign_rhs2 (stmt);
>> +             if (TREE_CODE (rhs1) == MEM_REF)
>> +               {
>> +                 rhs2 = TREE_OPERAND (rhs1, 1);
>> +                 rhs1 = TREE_OPERAND (rhs1, 0);
>> +               }
>> +             gimple *stmt1 = NULL, *stmt2 = NULL;
>> +             basic_block def_bb;
>> +             if (rhs1 && TREE_CODE (rhs1) == SSA_NAME)
>> +               {
>> +                 stmt1 = SSA_NAME_DEF_STMT (rhs1);
>> +                 def_bb = gimple_bb (stmt1);
>> +                 if (stmt1
>> +                     && def_bb
>> +                     && (def_bb == bb
>> +                         || !dominated_by_p (CDI_DOMINATORS, e->src, def_bb)))
>> +                   {
>> +                     if (dump_file && (dump_flags & TDF_DETAILS))
>> +                       {
>> +                         fprintf (dump_file,
>> +                                  "stmt1: Statement NOT moved to %d [local "
>> +                                  "count:%d] from %d [local count:%d]:\n",
>> +                                  e->src->index, e->src->count.value (),
>> +                                  bb->index, bb->count.value ());
>> +                         print_gimple_stmt (dump_file, stmt, 0);
>> +                         fprintf (dump_file, "(cost %u) out of loop %d.\n\n",
>> +                                  cost, level->num);
>> +                       }
>> +                     gsi_next (&bsi);
>> +                     continue;
>> +                   }
>> +               }
>> +             if (rhs2 && TREE_CODE (rhs2) == SSA_NAME)
>> +               {
>> +                 stmt2 = SSA_NAME_DEF_STMT (rhs2);
>> +                 def_bb = gimple_bb (stmt2);
>> +                 if (stmt2 && def_bb
>> +                     && (def_bb == bb
>> +                         || !dominated_by_p (CDI_DOMINATORS, e->src, def_bb)))
>> +                   {
>> +                     if (dump_file && (dump_flags & TDF_DETAILS))
>> +                       {
>> +                         fprintf (dump_file,
>> +                                  "stmt2: Statement NOT moved to %d [local "
>> +                                  "count:%d] from %d [local count:%d]:\n",
>> +                                  e->src->index, e->src->count.value (),
>> +                                  bb->index, bb->count.value ());
>> +                         print_gimple_stmt (dump_file, stmt, 0);
>> +                         fprintf (dump_file, "(cost %u) out of loop %d.\n\n",
>> +                                  cost, level->num);
>> +                       }
>> +                     gsi_next (&bsi);
>> +                     continue;
>> +                   }
>> +               }
>> +           }
>> +       }
>>
>>         if (dump_file && (dump_flags & TDF_DETAILS))
>>          {
>> @@ -1224,7 +1361,6 @@ move_computations_worker (basic_block bb)
>>                     cost, level->num);
>>          }
>>
>> -      e = loop_preheader_edge (level);
>>         gcc_assert (!gimple_vdef (stmt));
>>         if (gimple_vuse (stmt))
>>          {
>> @@ -2094,6 +2230,19 @@ execute_sm (class loop *loop, im_mem_ref *ref,
>>     bool multi_threaded_model_p = false;
>>     gimple_stmt_iterator gsi;
>>     sm_aux *aux = new sm_aux;
>> +  basic_block bb = gimple_bb (first_mem_ref_loc (loop, ref)->stmt);
>> +
>> +  edge e = loop_preheader_edge (loop);
>> +  if (e->src->count > bb->count)
>> +    {
>> +      if (dump_file && (dump_flags & TDF_DETAILS))
>> +       {
>> +         fprintf (dump_file, "Don't execute store motion of ");
>> +         print_generic_expr (dump_file, ref->mem.ref);
>> +         fprintf (dump_file, " from loop %d\n", loop->num);
>> +       }
>> +      return;
>> +    }
>>
>>     if (dump_file && (dump_flags & TDF_DETAILS))
>>       {
>> @@ -2202,7 +2351,12 @@ execute_sm_exit (class loop *loop, edge ex, vec<seq_entry> &seq,
>>          }
>>         else
>>          {
>> -         sm_aux *aux = *aux_map.get (ref);
>> +         sm_aux **paux = aux_map.get (ref);
>> +         sm_aux *aux;
>> +         if (paux)
>> +           aux = *paux;
>> +         else
>> +           continue;
>>            if (!aux->store_flag || kind == sm_ord)
>>              {
>>                gassign *store;
>> diff --git a/gcc/tree-ssa-loop-split.c b/gcc/tree-ssa-loop-split.c
>> index 3a09bbc39e5..4cae82936b9 100644
>> --- a/gcc/tree-ssa-loop-split.c
>> +++ b/gcc/tree-ssa-loop-split.c
>> @@ -577,14 +577,17 @@ split_loop (class loop *loop1)
>>          if (!initial_true)
>>            cond = fold_build1 (TRUTH_NOT_EXPR, boolean_type_node, cond);
>>
>> +       edge true_edge = EDGE_SUCC (bbs[i], 0)->flags & EDGE_TRUE_VALUE
>> +                          ? EDGE_SUCC (bbs[i], 0)
>> +                          : EDGE_SUCC (bbs[i], 1);
>>          /* Now version the loop, placing loop2 after loop1 connecting
>>             them, and fix up SSA form for that.  */
>>          initialize_original_copy_tables ();
>>          basic_block cond_bb;
>>
>>          class loop *loop2 = loop_version (loop1, cond, &cond_bb,
>> -                                          profile_probability::always (),
>> -                                          profile_probability::always (),
>> +                                          true_edge->probability,
>> +                                          true_edge->probability.invert (),
>>                                             profile_probability::always (),
>>                                             profile_probability::always (),
>>                                             true);
>> @@ -1486,8 +1489,8 @@ do_split_loop_on_cond (struct loop *loop1, edge invar_branch)
>>     initialize_original_copy_tables ();
>>
>>     struct loop *loop2 = loop_version (loop1, boolean_true_node, NULL,
>> -                                    profile_probability::always (),
>> -                                    profile_probability::never (),
>> +                                    invar_branch->probability,
>> +                                    invar_branch->probability.invert (),
>>                                       profile_probability::always (),
>>                                       profile_probability::always (),
>>                                       true);
>> @@ -1530,6 +1533,9 @@ do_split_loop_on_cond (struct loop *loop1, edge invar_branch)
>>     to_loop1->flags |= true_invar ? EDGE_FALSE_VALUE : EDGE_TRUE_VALUE;
>>     to_loop2->flags |= true_invar ? EDGE_TRUE_VALUE : EDGE_FALSE_VALUE;
>>
>> +  to_loop1->probability = invar_branch->probability.invert ();
>> +  to_loop2->probability = invar_branch->probability;
>> +
>>     /* Due to introduction of a control flow edge from loop1 latch to loop2
>>        pre-header, we should update PHIs in loop2 to reflect this connection
>>        between loop1 and loop2.  */
>> --
>> 2.27.0.90.geebb51ba8c
>>

Re: [RFC] Don't move cold code out of loop by checking bb count

[Ulrich Drepper]

On Tue, Aug 10, 2021 at 4:03 AM Xionghu Luo via Gcc-patches
<gcc-patches@gcc.gnu.org> wrote:
> For this case, theorotically I think the master GCC will optimize it to:
>
>   invariant;
>   for (;;)
>     if (unlikely_cond)
>       for (;;)
>          ;
>
> 'invariant' is moved out of outer loop, but with the patch, it will get:
>
>   for (;;)
>     if (unlikely_cond)
>       {
>         invariant;
>         for (;;)
>            ;
>       }
>
> 'invariant' is *cold* for outer loop, but it is still *hot* for inner loop,
> so hoist it out of inner loop, this is exactly what we want, right?

Is relying on absolute numbers really what you want?  If the
'unlikely_cond' condition depends on the iteration count of the outer
loop the probability of it being true in each individual iteration can
be low (at least that's how I use unlikely) but the overall
probability of needing the code is higher 1 - (1 - p)^n  if 'p' is the
probability of 'unlikely_cond' and 'n' is the number of iterations.
Assuming complete independence of the loop iterations, otherwise it's
rather an upper limit.

At the very least I'd generate code like this:

  first = true;
  for (;;)
    if (unlikely_cond)
      {
        if (first)
          {
            invariant;
            first = false;
          }
        for (;;)
           ;
      }

If it's worth hoisting the code the the extra test and flag should be
small in cost in comparison.

If 'unlikely_cond' does not in any way depend on the loop iteration
then I think your code generation is fine.

[PATCH v2] Don't move cold code out of loop by checking bb count

[Xionghu Luo]

On 2021/8/10 12:25, Ulrich Drepper wrote:
> On Tue, Aug 10, 2021 at 4:03 AM Xionghu Luo via Gcc-patches
> <gcc-patches@gcc.gnu.org> wrote:
>> For this case, theorotically I think the master GCC will optimize it to:
>>
>>    invariant;
>>    for (;;)
>>      if (unlikely_cond)
>>        for (;;)
>>           ;
>>
>> 'invariant' is moved out of outer loop, but with the patch, it will get:
>>
>>    for (;;)
>>      if (unlikely_cond)
>>        {
>>          invariant;
>>          for (;;)
>>             ;
>>        }
>>
>> 'invariant' is *cold* for outer loop, but it is still *hot* for inner loop,
>> so hoist it out of inner loop, this is exactly what we want, right?
> 
> Is relying on absolute numbers really what you want?  If the
> 'unlikely_cond' condition depends on the iteration count of the outer
> loop the probability of it being true in each individual iteration can
> be low (at least that's how I use unlikely) but the overall
> probability of needing the code is higher 1 - (1 - p)^n  if 'p' is the
> probability of 'unlikely_cond' and 'n' is the number of iterations.
> Assuming complete independence of the loop iterations, otherwise it's
> rather an upper limit.
> 
> At the very least I'd generate code like this:
> 
>    first = true;
>    for (;;)
>      if (unlikely_cond)
>        {
>          if (first)
>            {
>              invariant;
>              first = false;
>            }
>          for (;;)
>             ;
>        }
> 
> If it's worth hoisting the code the the extra test and flag should be
> small in cost in comparison.
> 
> If 'unlikely_cond' does not in any way depend on the loop iteration
> then I think your code generation is fine.


Thanks for your good suggestion, I am also not sure whether it is
necessary to do it this way:)  But I found that even the first step
of 


  for (;;)
    if (unlikely_cond)
      {
        invariant;
        for (;;)
           ;
      }

is not supported yet.  So I added a new function *find_coldest_out_loop*
to search the coldest function between outermost invariant loop and own
loop in compute_invariantness to move invariant out to cold loop first:



[PATCH v2] Don't move cold code out of loop by checking bb count


There was a patch trying to avoid move cold block out of loop:

https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html

Richard suggested to "never hoist anything from a bb with lower execution
frequency to a bb with higher one in LIM invariantness_dom_walker
before_dom_children".

In gimple LIM analysis,  add find_coldest_out_loop to move invariants to
expected target loop, then in both gimple LIM move_computations_worker
and RTL loop-invariant.c find_invariants_bb, if profile count check find
 the loop bb is colder than target loop preheader, don't hoist it out of
loop.

SPEC2017 performance evaluation shows 1% performance improvement for
intrate GEOMEAN and no obvious regression for others.  Especially,
500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
on P8LE.

Regression and bootstrap tested pass on P8LE, any comments?  Thanks.

gcc/ChangeLog:

	* loop-invariant.c (find_invariants_bb): Check profile count
	before motion.
	(find_invariants_body): Add argument.
	* tree-ssa-loop-im.c (find_coldest_out_loop): New function.
	(outermost_invariant_loop): Use find_coldest_out_loop.
	(determine_max_movement): Likewise.
	(move_computations_worker): Check profile count before motion.
	(execute_sm): Likewise.
	(execute_sm_exit): Check pointer validness.

gcc/testsuite/ChangeLog:

	* gcc.dg/tree-ssa/recip-3.c: Adjust.
	* gcc.dg/tree-ssa/ssa-lim-16.c: New test.
	* gcc.dg/tree-ssa/ssa-lim-17.c: New test.
---
 gcc/loop-invariant.c                       |  10 +-
 gcc/tree-ssa-loop-im.c                     | 186 +++++++++++++++++++--
 gcc/testsuite/gcc.dg/tree-ssa/recip-3.c    |   2 +-
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-16.c |  21 +++
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-17.c |  26 +++
 5 files changed, 231 insertions(+), 14 deletions(-)
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-16.c
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-17.c

diff --git a/gcc/loop-invariant.c b/gcc/loop-invariant.c
index fca0c2b24be..5c3be7bf0eb 100644
--- a/gcc/loop-invariant.c
+++ b/gcc/loop-invariant.c
@@ -1183,9 +1183,14 @@ find_invariants_insn (rtx_insn *insn, bool always_reached, bool always_executed)
    call.  */
 
 static void
-find_invariants_bb (basic_block bb, bool always_reached, bool always_executed)
+find_invariants_bb (class loop *loop, basic_block bb, bool always_reached,
+		    bool always_executed)
 {
   rtx_insn *insn;
+  basic_block preheader = loop_preheader_edge (loop)->src;
+
+  if (preheader->count > bb->count)
+    return;
 
   FOR_BB_INSNS (bb, insn)
     {
@@ -1214,8 +1219,7 @@ find_invariants_body (class loop *loop, basic_block *body,
   unsigned i;
 
   for (i = 0; i < loop->num_nodes; i++)
-    find_invariants_bb (body[i],
-			bitmap_bit_p (always_reached, i),
+    find_invariants_bb (loop, body[i], bitmap_bit_p (always_reached, i),
 			bitmap_bit_p (always_executed, i));
 }
 
diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
index d9f75d5025e..2b90caa90fc 100644
--- a/gcc/tree-ssa-loop-im.c
+++ b/gcc/tree-ssa-loop-im.c
@@ -417,6 +417,23 @@ movement_possibility (gimple *stmt)
   return ret;
 }
 
+/* Find coldest loop between min_loop and loop by comapring profile count.  */
+
+static class loop *
+find_coldest_out_loop (class loop *min_loop, class loop *loop)
+{
+  class loop *cold_loop = min_loop;
+  profile_count min_count = loop_preheader_edge (min_loop)->src->count;
+
+  while (min_loop != loop)
+    {
+      min_loop = superloop_at_depth (loop, loop_depth (min_loop) + 1);
+      if (loop_preheader_edge (min_loop)->src->count < min_count)
+	cold_loop = min_loop;
+    }
+  return cold_loop;
+}
+
 /* Suppose that operand DEF is used inside the LOOP.  Returns the outermost
    loop to that we could move the expression using DEF if it did not have
    other operands, i.e. the outermost loop enclosing LOOP in that the value
@@ -431,18 +448,18 @@ outermost_invariant_loop (tree def, class loop *loop)
   struct lim_aux_data *lim_data;
 
   if (!def)
-    return superloop_at_depth (loop, 1);
+    return find_coldest_out_loop (superloop_at_depth (loop, 1), loop);
 
   if (TREE_CODE (def) != SSA_NAME)
     {
       gcc_assert (is_gimple_min_invariant (def));
-      return superloop_at_depth (loop, 1);
+      return find_coldest_out_loop (superloop_at_depth (loop, 1), loop);
     }
 
   def_stmt = SSA_NAME_DEF_STMT (def);
   def_bb = gimple_bb (def_stmt);
   if (!def_bb)
-    return superloop_at_depth (loop, 1);
+    return find_coldest_out_loop (superloop_at_depth (loop, 1), loop);
 
   max_loop = find_common_loop (loop, def_bb->loop_father);
 
@@ -452,7 +469,11 @@ outermost_invariant_loop (tree def, class loop *loop)
 				 loop_outer (lim_data->max_loop));
   if (max_loop == loop)
     return NULL;
-  max_loop = superloop_at_depth (loop, loop_depth (max_loop) + 1);
+  max_loop = find_coldest_out_loop (max_loop, loop);
+  if (max_loop == loop)
+    return max_loop;
+  else
+    max_loop = superloop_at_depth (loop, loop_depth (max_loop) + 1);
 
   return max_loop;
 }
@@ -684,7 +705,7 @@ determine_max_movement (gimple *stmt, bool must_preserve_exec)
   if (must_preserve_exec)
     level = ALWAYS_EXECUTED_IN (bb);
   else
-    level = superloop_at_depth (loop, 1);
+    level = find_coldest_out_loop (superloop_at_depth (loop, 1), loop);
   lim_data->max_loop = level;
 
   if (gphi *phi = dyn_cast <gphi *> (stmt))
@@ -787,6 +808,7 @@ determine_max_movement (gimple *stmt, bool must_preserve_exec)
 						     loop, ref);
 	  if (!lim_data->max_loop)
 	    return false;
+	  lim_data->max_loop = find_coldest_out_loop (lim_data->max_loop, loop);
 	}
       else if (! add_dependency (gimple_vuse (stmt), lim_data, loop, false))
 	return false;
@@ -1154,6 +1176,58 @@ move_computations_worker (basic_block bb)
 	  continue;
 	}
 
+      edge e = loop_preheader_edge (level);
+      if (e->src->count > bb->count)
+	{
+	  if (dump_file && (dump_flags & TDF_DETAILS))
+	    {
+	      fprintf (dump_file, "PHI node NOT moved to %d from %d:\n",
+		       e->src->index, bb->index);
+	      print_gimple_stmt (dump_file, stmt, 0);
+	      fprintf (dump_file, "(cost %u) out of loop %d.\n\n", cost,
+		       level->num);
+	    }
+	  gsi_next (&bsi);
+	  continue;
+	}
+      else
+	{
+	  unsigned i;
+	  bool skip_phi_move = false;
+	  for (i = 0; i < gimple_phi_num_args (stmt); i++)
+	    {
+	      tree def = PHI_ARG_DEF (stmt, i);
+
+	      if (TREE_CODE (def) != SSA_NAME)
+		continue;
+
+	      gimple *def_stmt = SSA_NAME_DEF_STMT (def);
+
+	      if (!gimple_bb (def_stmt))
+		continue;
+
+	      if (!dominated_by_p (CDI_DOMINATORS, e->src,
+				   gimple_bb (def_stmt)))
+		{
+		  if (dump_file && (dump_flags & TDF_DETAILS))
+		    {
+		      fprintf (dump_file, "PHI node NOT moved to %d from %d\n",
+			       e->src->index, bb->index);
+		      print_gimple_stmt (dump_file, stmt, 0);
+		      fprintf (dump_file, "(cost %u) out of loop %d.\n\n", cost,
+			       level->num);
+		    }
+		  skip_phi_move = true;
+		  break;
+		}
+	    }
+	  if (skip_phi_move)
+	    {
+	      gsi_next (&bsi);
+	      continue;
+	    }
+	}
+
       if (dump_file && (dump_flags & TDF_DETAILS))
 	{
 	  fprintf (dump_file, "Moving PHI node\n");
@@ -1191,14 +1265,13 @@ move_computations_worker (basic_block bb)
 	  tree lhs = gimple_assign_lhs (new_stmt);
 	  SSA_NAME_RANGE_INFO (lhs) = NULL;
 	}
-      gsi_insert_on_edge (loop_preheader_edge (level), new_stmt);
+      gsi_insert_on_edge (e, new_stmt);
       remove_phi_node (&bsi, false);
     }
 
   for (gimple_stmt_iterator bsi = gsi_start_bb (bb); !gsi_end_p (bsi); )
     {
       edge e;
-
       gimple *stmt = gsi_stmt (bsi);
 
       lim_data = get_lim_data (stmt);
@@ -1221,7 +1294,83 @@ move_computations_worker (basic_block bb)
       /* We do not really want to move conditionals out of the loop; we just
 	 placed it here to force its operands to be moved if necessary.  */
       if (gimple_code (stmt) == GIMPLE_COND)
-	continue;
+	{
+	  gsi_next (&bsi);
+	  continue;
+	}
+
+      e = loop_preheader_edge (level);
+      if (e->src->count > bb->count)
+	{
+	  if (dump_file && (dump_flags & TDF_DETAILS))
+	    {
+	      fprintf (dump_file, "stmt: Statement NOT moved to %d from %d \n",
+		       e->src->index, bb->index);
+	      print_gimple_stmt (dump_file, stmt, 0);
+	      fprintf (dump_file, "(cost %u) out of loop %d.\n\n", cost,
+		       level->num);
+	    }
+	  gsi_next (&bsi);
+	  continue;
+	}
+      else
+	{
+	  if (is_gimple_assign (stmt))
+	    {
+	      tree rhs1 = gimple_assign_rhs1 (stmt);
+	      tree rhs2 = gimple_assign_rhs2 (stmt);
+	      if (TREE_CODE (rhs1) == MEM_REF)
+		{
+		  rhs2 = TREE_OPERAND (rhs1, 1);
+		  rhs1 = TREE_OPERAND (rhs1, 0);
+		}
+	      gimple *stmt1 = NULL, *stmt2 = NULL;
+	      basic_block def_bb;
+	      if (rhs1 && TREE_CODE (rhs1) == SSA_NAME)
+		{
+		  stmt1 = SSA_NAME_DEF_STMT (rhs1);
+		  def_bb = gimple_bb (stmt1);
+		  if (stmt1
+		      && def_bb
+		      && (def_bb == bb
+			  || !dominated_by_p (CDI_DOMINATORS, e->src, def_bb)))
+		    {
+		      if (dump_file && (dump_flags & TDF_DETAILS))
+			{
+			  fprintf (dump_file,
+				   "stmt1: Statement NOT moved to %d from %d\n",
+				   e->src->index, bb->index);
+			  print_gimple_stmt (dump_file, stmt, 0);
+			  fprintf (dump_file, "(cost %u) out of loop %d.\n\n",
+				   cost, level->num);
+			}
+		      gsi_next (&bsi);
+		      continue;
+		    }
+		}
+	      if (rhs2 && TREE_CODE (rhs2) == SSA_NAME)
+		{
+		  stmt2 = SSA_NAME_DEF_STMT (rhs2);
+		  def_bb = gimple_bb (stmt2);
+		  if (stmt2 && def_bb
+		      && (def_bb == bb
+			  || !dominated_by_p (CDI_DOMINATORS, e->src, def_bb)))
+		    {
+		      if (dump_file && (dump_flags & TDF_DETAILS))
+			{
+			  fprintf (dump_file,
+				   "stmt2: Statement NOT moved to %d from %d\n",
+				   e->src->index, bb->index);
+			  print_gimple_stmt (dump_file, stmt, 0);
+			  fprintf (dump_file, "(cost %u) out of loop %d.\n\n",
+				   cost, level->num);
+			}
+		      gsi_next (&bsi);
+		      continue;
+		    }
+		}
+	    }
+	}
 
       if (dump_file && (dump_flags & TDF_DETAILS))
 	{
@@ -1231,7 +1380,6 @@ move_computations_worker (basic_block bb)
 		   cost, level->num);
 	}
 
-      e = loop_preheader_edge (level);
       gcc_assert (!gimple_vdef (stmt));
       if (gimple_vuse (stmt))
 	{
@@ -2133,6 +2281,19 @@ execute_sm (class loop *loop, im_mem_ref *ref,
   bool multi_threaded_model_p = false;
   gimple_stmt_iterator gsi;
   sm_aux *aux = new sm_aux;
+  basic_block bb = gimple_bb (first_mem_ref_loc (loop, ref)->stmt);
+
+  edge e = loop_preheader_edge (loop);
+  if (e->src->count > bb->count)
+    {
+      if (dump_file && (dump_flags & TDF_DETAILS))
+	{
+	  fprintf (dump_file, "Don't execute store motion of ");
+	  print_generic_expr (dump_file, ref->mem.ref);
+	  fprintf (dump_file, " from loop %d\n", loop->num);
+	}
+      return;
+    }
 
   if (dump_file && (dump_flags & TDF_DETAILS))
     {
@@ -2241,7 +2402,12 @@ execute_sm_exit (class loop *loop, edge ex, vec<seq_entry> &seq,
 	}
       else
 	{
-	  sm_aux *aux = *aux_map.get (ref);
+	  sm_aux **paux = aux_map.get (ref);
+	  sm_aux *aux;
+	  if (paux)
+	    aux = *paux;
+	  else
+	    continue;
 	  if (!aux->store_flag || kind == sm_ord)
 	    {
 	      gassign *store;
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
index 638bf38db8c..641c91e719e 100644
--- a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
+++ b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
@@ -23,4 +23,4 @@ float h ()
 	F[0] += E / d;
 }
 
-/* { dg-final { scan-tree-dump-times " / " 1 "recip" } } */
+/* { dg-final { scan-tree-dump-times " / " 5 "recip" } } */
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-16.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-16.c
new file mode 100644
index 00000000000..49b5979dc7d
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-16.c
@@ -0,0 +1,21 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+volatile int x;
+void
+assert_fail (int, char *, char *);
+void
+foo (int *a, int n, int k)
+{
+  int i;
+
+  for (i = 0; i < n; i++)
+    {
+      if (__builtin_expect (x, 0))
+	assert_fail (k / 5, "one", "two");
+      a[i] = k;
+    }
+}
+
+/* { dg-final { scan-tree-dump-times "stmt: Statement NOT moved to" 1 "lim2" } } */
+/* { dg-final { scan-tree-dump-times "stmt: Statement NOT moved to" 1 "lim4" } } */
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-17.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-17.c
new file mode 100644
index 00000000000..3b1c7c0cb3e
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-17.c
@@ -0,0 +1,26 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+volatile int x;
+void
+assert_fail (int, char *, char *);
+void
+foo (int *a, int n, int m, int k, int s)
+{
+  int i;
+  int j;
+
+  for (i = 0; i < m; i++)
+    {
+      if (__builtin_expect (x, 0))
+	for (j = 0; j < n; j++)
+	  {
+	    assert_fail (k / 5, "one", "two");
+	  a[s] = k;
+	}
+      a[s] = s;
+    }
+}
+
+/* { dg-final { scan-tree-dump-times "out of loop 2" 4 "lim2" } } */
+/* { dg-final { scan-tree-dump-times "out of loop 1" 3 "lim2" } } */
-- 
2.27.0.90.geebb51ba8c

Re: [RFC] Don't move cold code out of loop by checking bb count

[Richard Biener]

On Tue, Aug 10, 2021 at 4:03 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>
> Hi,
>
> On 2021/8/6 20:15, Richard Biener wrote:
> > On Mon, Aug 2, 2021 at 7:05 AM Xiong Hu Luo <luoxhu@linux.ibm.com> wrote:
> >>
> >> There was a patch trying to avoid move cold block out of loop:
> >>
> >> https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html
> >>
> >> Richard suggested to "never hoist anything from a bb with lower execution
> >> frequency to a bb with higher one in LIM invariantness_dom_walker
> >> before_dom_children".
> >>
> >> This patch does this profile count check in both gimple LIM
> >> move_computations_worker and RTL loop-invariant.c find_invariants_bb,
> >> if the loop bb is colder than loop preheader, don't hoist it out of
> >> loop.
> >>
> >> Also, the profile count in loop split pass should be corrected to avoid
> >> lim2 and lim4 mismatch behavior, currently, the new loop preheader generated
> >> by loop_version is set to "[count: 0]:", then lim4 after lsplt pass will
> >> move statement out of loop unexpectely when lim2 didn't move it.  This
> >> change could fix regression on 544.nab_r from -1.55% to +0.46%.
> >>
> >> SPEC2017 performance evaluation shows 1% performance improvement for
> >> intrate GEOMEAN and no obvious regression for others.  Especially,
> >> 500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
> >> largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
> >> on P8LE.
> >>
> >> Regression and bootstrap tested pass on P8LE, any comments?  Thanks.
> >
> > While I'm not familiar with the RTL invariant motion pass the patch there
> > looks reasonable.  Note that we should assess the profile quality
> > somehow - I'm not sure how to do that, CCed Honza for that.
>
> Thanks.
>
> >
> > For the GIMPLE part the patch looks quite complicated - but note it
> > probably has to be since LIM performs kind of a "CSE" on loads
> > (and stores for store-motion), so when there are multiple stmts
> > affected by a hoisting decision the biggest block count has to be
> > accounted.  Likewise when there are dependent stmts involved
> > that might include conditional stmts (a "PHI"), but the overall
> > cost should be looked at.
>
> Currently, The gimple code check two situations with the patch:
> 1) The statement or PHI‘s BB is *colder* then preheader, don't move it out
> of loop;
> 2) The statement or PHI's BB is *hotter* then preheader, but any of it's rhs
> couldn't be moved out of loop, also don't move it out of loop to avoid definition
> not dominates use error.

But part 2) is obviously already done.  What I tried to say is your heuristic
doesn't integrate nicely with the pass but I admitted that it might be a bit
difficult to find a place to add this heuristic.

There is lim_data->cost which we could bias negatively but then this is
a cost that is independent on the hoisting distance.  But doing this would
work at least for the case where the immediately enclosing loop preheader
is hotter than the stmt and with this it would be a patch that's similarly
simple as the RTL one.

Another possibility is to simply only adjust PHI processing in
compute_invariantness, capping movement according to the hotness
heuristic.  The same could be done for regular stmts there but I'm
not sure that will do good in the end since this function is supposed
to compute "correctness" (well, it also has the cost stuff), and it's
not the place to do overall cost considerations.

> May be I could collect the number of instructions not hoisted with the patch
> on regression tests and SPEC2017 to do a estimation for "multiple stmts affected"
> and "overall cost" need to be considered?  But it seems move_computations_worker
> couldn't rollback if we still want to hoist multiple stmts out during the iterations?
>
> >
> > Now - GIMPLE LIM "costing" is somewhat backward right now
> > and it isn't set up to consider those multiple involved stmts.  Plus
> > the store-motion part does not have any cost part (but it depends
> > on previously decided invariant motions).
> >
> > I think the way you implemented the check will cause no hoisting
> > to be performed instead of, say, hoisting to a different loop level
> > only.  Possibly shown when you consider a loop nest like
> >
> >    for (;;)
> >      if (unlikely_cond)
> >        for (;;)
> >           invariant;
> >
> > we want to hoist 'invariant' but only from the inner loop even if it
> > is invariant also in the outer loop.
>
>
> For this case, theorotically I think the master GCC will optimize it to:
>
>   invariant;
>   for (;;)
>     if (unlikely_cond)
>       for (;;)
>          ;
>
> 'invariant' is moved out of outer loop, but with the patch, it will get:
>
>   for (;;)
>     if (unlikely_cond)
>       {
>         invariant;
>         for (;;)
>            ;
>       }
>
> 'invariant' is *cold* for outer loop, but it is still *hot* for inner loop,
> so hoist it out of inner loop, this is exactly what we want, right?

Yes.  I had doubts your patch would achieve that.

>
> >  But for example if there is
> > a store motion opportunity like
> >
> >    for (;;)
> >       {
> >          if (unlikely_cond)
> >            for (;;)
> >              a = ...;
> >          a = ...;
> >       }
> >
> > we'd still want to perform the store motion on the outer loop.
> >
> > Note that store-motion already performs part of the transform
> > before dependent code is moved in move_computations (that
> > you patched).
>
> Yes.  do_store_motion is running before move_computations_worker, store
> motion happens earlier in execute_sm, I also added the check in execute_sm
> to stop cold store moved out of loop.  So for your case, I think my patch
> will similarly optimize it to:
>
>   for (;;)
>      {
>         if (unlikely_cond)
>           {
>             for (;;)
>               ;
>             a = ...;
>           }
>      }
>     a = ...;
>
> Whether this is better?  Will construct cases to verify it.
>
> >
> > IIRC your main concern were the COND_EXPRs we insert
> > for hoisted conditional stmts?
>
> Not sure what you mean here of COND_EXPRs?

The PHIs we hoist and for which we insert COND_EXPRs.  IIRC that was
your original complaint.  So my question is whether we can fix the really
bad cases moving PHIs with sth local to compute_invariantness.

Richard.

>
> Thanks,
> Xionghu
>
> >
> > Thanks,
> > Richard.
> >
> >> gcc/ChangeLog:
> >>
> >>          * loop-invariant.c (find_invariants_bb): Check profile count
> >>          before motion.
> >>          (find_invariants_body): Add argument.
> >>          * tree-ssa-loop-im.c (move_computations_worker): Check profile
> >>          count before motion.
> >>          (execute_sm): Likewise.
> >>          (execute_sm_exit): Check pointer validness.
> >>          * tree-ssa-loop-split.c (split_loop): Correct probability.
> >>          (do_split_loop_on_cond): Likewise.
> >>
> >> gcc/testsuite/ChangeLog:
> >>
> >>          * gcc.dg/tree-ssa/recip-3.c: Adjust.
> >> ---
> >>   gcc/loop-invariant.c                    |  10 +-
> >>   gcc/testsuite/gcc.dg/tree-ssa/recip-3.c |   2 +-
> >>   gcc/tree-ssa-loop-im.c                  | 164 +++++++++++++++++++++++-
> >>   gcc/tree-ssa-loop-split.c               |  14 +-
> >>   4 files changed, 177 insertions(+), 13 deletions(-)
> >>
> >> diff --git a/gcc/loop-invariant.c b/gcc/loop-invariant.c
> >> index bdc7b59dd5f..7b5d64d11f9 100644
> >> --- a/gcc/loop-invariant.c
> >> +++ b/gcc/loop-invariant.c
> >> @@ -1183,9 +1183,14 @@ find_invariants_insn (rtx_insn *insn, bool always_reached, bool always_executed)
> >>      call.  */
> >>
> >>   static void
> >> -find_invariants_bb (basic_block bb, bool always_reached, bool always_executed)
> >> +find_invariants_bb (class loop *loop, basic_block bb, bool always_reached,
> >> +                   bool always_executed)
> >>   {
> >>     rtx_insn *insn;
> >> +  basic_block preheader = loop_preheader_edge (loop)->src;
> >> +
> >> +  if (preheader->count > bb->count)
> >> +    return;
> >>
> >>     FOR_BB_INSNS (bb, insn)
> >>       {
> >> @@ -1214,8 +1219,7 @@ find_invariants_body (class loop *loop, basic_block *body,
> >>     unsigned i;
> >>
> >>     for (i = 0; i < loop->num_nodes; i++)
> >> -    find_invariants_bb (body[i],
> >> -                       bitmap_bit_p (always_reached, i),
> >> +    find_invariants_bb (loop, body[i], bitmap_bit_p (always_reached, i),
> >>                          bitmap_bit_p (always_executed, i));
> >>   }
> >>
> >> diff --git a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
> >> index 638bf38db8c..641c91e719e 100644
> >> --- a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
> >> +++ b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
> >> @@ -23,4 +23,4 @@ float h ()
> >>          F[0] += E / d;
> >>   }
> >>
> >> -/* { dg-final { scan-tree-dump-times " / " 1 "recip" } } */
> >> +/* { dg-final { scan-tree-dump-times " / " 5 "recip" } } */
> >> diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
> >> index 7de47edbcb3..2bfb5e8ec15 100644
> >> --- a/gcc/tree-ssa-loop-im.c
> >> +++ b/gcc/tree-ssa-loop-im.c
> >> @@ -1147,6 +1147,61 @@ move_computations_worker (basic_block bb)
> >>            continue;
> >>          }
> >>
> >> +      edge e = loop_preheader_edge (level);
> >> +      if (e->src->count > bb->count)
> >> +       {
> >> +         if (dump_file && (dump_flags & TDF_DETAILS))
> >> +           {
> >> +             fprintf (dump_file, "PHI node NOT moved to %d from %d:\n",
> >> +                      e->src->index, bb->index);
> >> +             print_gimple_stmt (dump_file, stmt, 0);
> >> +             fprintf (dump_file, "(cost %u) out of loop %d.\n\n", cost,
> >> +                      level->num);
> >> +           }
> >> +         gsi_next (&bsi);
> >> +         continue;
> >> +       }
> >> +      else
> >> +       {
> >> +         unsigned i;
> >> +         bool skip_phi_move = false;
> >> +         for (i = 0; i < gimple_phi_num_args (stmt); i++)
> >> +           {
> >> +             tree def = PHI_ARG_DEF (stmt, i);
> >> +
> >> +             if (TREE_CODE (def) != SSA_NAME)
> >> +               continue;
> >> +
> >> +             gimple *def_stmt = SSA_NAME_DEF_STMT (def);
> >> +
> >> +             if (!gimple_bb (def_stmt))
> >> +               continue;
> >> +
> >> +             if (!dominated_by_p (CDI_DOMINATORS, e->src,
> >> +                                  gimple_bb (def_stmt)))
> >> +               {
> >> +                 if (dump_file && (dump_flags & TDF_DETAILS))
> >> +                   {
> >> +                     fprintf (dump_file,
> >> +                              "PHI node NOT moved to %d [local count:%d] from "
> >> +                              "%d [local count:%d]:\n",
> >> +                              e->src->index, e->src->count.value (), bb->index,
> >> +                              bb->count.value ());
> >> +                     print_gimple_stmt (dump_file, stmt, 0);
> >> +                     fprintf (dump_file, "(cost %u) out of loop %d.\n\n", cost,
> >> +                              level->num);
> >> +                   }
> >> +                 skip_phi_move = true;
> >> +                 break;
> >> +               }
> >> +           }
> >> +         if (skip_phi_move)
> >> +           {
> >> +             gsi_next (&bsi);
> >> +             continue;
> >> +           }
> >> +       }
> >> +
> >>         if (dump_file && (dump_flags & TDF_DETAILS))
> >>          {
> >>            fprintf (dump_file, "Moving PHI node\n");
> >> @@ -1184,14 +1239,13 @@ move_computations_worker (basic_block bb)
> >>            tree lhs = gimple_assign_lhs (new_stmt);
> >>            SSA_NAME_RANGE_INFO (lhs) = NULL;
> >>          }
> >> -      gsi_insert_on_edge (loop_preheader_edge (level), new_stmt);
> >> +      gsi_insert_on_edge (e, new_stmt);
> >>         remove_phi_node (&bsi, false);
> >>       }
> >>
> >>     for (gimple_stmt_iterator bsi = gsi_start_bb (bb); !gsi_end_p (bsi); )
> >>       {
> >>         edge e;
> >> -
> >>         gimple *stmt = gsi_stmt (bsi);
> >>
> >>         lim_data = get_lim_data (stmt);
> >> @@ -1214,7 +1268,90 @@ move_computations_worker (basic_block bb)
> >>         /* We do not really want to move conditionals out of the loop; we just
> >>           placed it here to force its operands to be moved if necessary.  */
> >>         if (gimple_code (stmt) == GIMPLE_COND)
> >> -       continue;
> >> +       {
> >> +         gsi_next (&bsi);
> >> +         continue;
> >> +       }
> >> +
> >> +      e = loop_preheader_edge (level);
> >> +      if (e->src->count > bb->count)
> >> +       {
> >> +         if (dump_file && (dump_flags & TDF_DETAILS))
> >> +           {
> >> +             fprintf (dump_file,
> >> +                      "stmt: Statement NOT moved to %d [local count:%d] from "
> >> +                      "%d [local count:%d]:\n",
> >> +                      e->src->index, e->src->count.value (), bb->index,
> >> +                      bb->count.value ());
> >> +             print_gimple_stmt (dump_file, stmt, 0);
> >> +             fprintf (dump_file, "(cost %u) out of loop %d.\n\n", cost,
> >> +                      level->num);
> >> +           }
> >> +         gsi_next (&bsi);
> >> +         continue;
> >> +       }
> >> +      else
> >> +       {
> >> +         if (is_gimple_assign (stmt))
> >> +           {
> >> +             tree rhs1 = gimple_assign_rhs1 (stmt);
> >> +             tree rhs2 = gimple_assign_rhs2 (stmt);
> >> +             if (TREE_CODE (rhs1) == MEM_REF)
> >> +               {
> >> +                 rhs2 = TREE_OPERAND (rhs1, 1);
> >> +                 rhs1 = TREE_OPERAND (rhs1, 0);
> >> +               }
> >> +             gimple *stmt1 = NULL, *stmt2 = NULL;
> >> +             basic_block def_bb;
> >> +             if (rhs1 && TREE_CODE (rhs1) == SSA_NAME)
> >> +               {
> >> +                 stmt1 = SSA_NAME_DEF_STMT (rhs1);
> >> +                 def_bb = gimple_bb (stmt1);
> >> +                 if (stmt1
> >> +                     && def_bb
> >> +                     && (def_bb == bb
> >> +                         || !dominated_by_p (CDI_DOMINATORS, e->src, def_bb)))
> >> +                   {
> >> +                     if (dump_file && (dump_flags & TDF_DETAILS))
> >> +                       {
> >> +                         fprintf (dump_file,
> >> +                                  "stmt1: Statement NOT moved to %d [local "
> >> +                                  "count:%d] from %d [local count:%d]:\n",
> >> +                                  e->src->index, e->src->count.value (),
> >> +                                  bb->index, bb->count.value ());
> >> +                         print_gimple_stmt (dump_file, stmt, 0);
> >> +                         fprintf (dump_file, "(cost %u) out of loop %d.\n\n",
> >> +                                  cost, level->num);
> >> +                       }
> >> +                     gsi_next (&bsi);
> >> +                     continue;
> >> +                   }
> >> +               }
> >> +             if (rhs2 && TREE_CODE (rhs2) == SSA_NAME)
> >> +               {
> >> +                 stmt2 = SSA_NAME_DEF_STMT (rhs2);
> >> +                 def_bb = gimple_bb (stmt2);
> >> +                 if (stmt2 && def_bb
> >> +                     && (def_bb == bb
> >> +                         || !dominated_by_p (CDI_DOMINATORS, e->src, def_bb)))
> >> +                   {
> >> +                     if (dump_file && (dump_flags & TDF_DETAILS))
> >> +                       {
> >> +                         fprintf (dump_file,
> >> +                                  "stmt2: Statement NOT moved to %d [local "
> >> +                                  "count:%d] from %d [local count:%d]:\n",
> >> +                                  e->src->index, e->src->count.value (),
> >> +                                  bb->index, bb->count.value ());
> >> +                         print_gimple_stmt (dump_file, stmt, 0);
> >> +                         fprintf (dump_file, "(cost %u) out of loop %d.\n\n",
> >> +                                  cost, level->num);
> >> +                       }
> >> +                     gsi_next (&bsi);
> >> +                     continue;
> >> +                   }
> >> +               }
> >> +           }
> >> +       }
> >>
> >>         if (dump_file && (dump_flags & TDF_DETAILS))
> >>          {
> >> @@ -1224,7 +1361,6 @@ move_computations_worker (basic_block bb)
> >>                     cost, level->num);
> >>          }
> >>
> >> -      e = loop_preheader_edge (level);
> >>         gcc_assert (!gimple_vdef (stmt));
> >>         if (gimple_vuse (stmt))
> >>          {
> >> @@ -2094,6 +2230,19 @@ execute_sm (class loop *loop, im_mem_ref *ref,
> >>     bool multi_threaded_model_p = false;
> >>     gimple_stmt_iterator gsi;
> >>     sm_aux *aux = new sm_aux;
> >> +  basic_block bb = gimple_bb (first_mem_ref_loc (loop, ref)->stmt);
> >> +
> >> +  edge e = loop_preheader_edge (loop);
> >> +  if (e->src->count > bb->count)
> >> +    {
> >> +      if (dump_file && (dump_flags & TDF_DETAILS))
> >> +       {
> >> +         fprintf (dump_file, "Don't execute store motion of ");
> >> +         print_generic_expr (dump_file, ref->mem.ref);
> >> +         fprintf (dump_file, " from loop %d\n", loop->num);
> >> +       }
> >> +      return;
> >> +    }
> >>
> >>     if (dump_file && (dump_flags & TDF_DETAILS))
> >>       {
> >> @@ -2202,7 +2351,12 @@ execute_sm_exit (class loop *loop, edge ex, vec<seq_entry> &seq,
> >>          }
> >>         else
> >>          {
> >> -         sm_aux *aux = *aux_map.get (ref);
> >> +         sm_aux **paux = aux_map.get (ref);
> >> +         sm_aux *aux;
> >> +         if (paux)
> >> +           aux = *paux;
> >> +         else
> >> +           continue;
> >>            if (!aux->store_flag || kind == sm_ord)
> >>              {
> >>                gassign *store;
> >> diff --git a/gcc/tree-ssa-loop-split.c b/gcc/tree-ssa-loop-split.c
> >> index 3a09bbc39e5..4cae82936b9 100644
> >> --- a/gcc/tree-ssa-loop-split.c
> >> +++ b/gcc/tree-ssa-loop-split.c
> >> @@ -577,14 +577,17 @@ split_loop (class loop *loop1)
> >>          if (!initial_true)
> >>            cond = fold_build1 (TRUTH_NOT_EXPR, boolean_type_node, cond);
> >>
> >> +       edge true_edge = EDGE_SUCC (bbs[i], 0)->flags & EDGE_TRUE_VALUE
> >> +                          ? EDGE_SUCC (bbs[i], 0)
> >> +                          : EDGE_SUCC (bbs[i], 1);
> >>          /* Now version the loop, placing loop2 after loop1 connecting
> >>             them, and fix up SSA form for that.  */
> >>          initialize_original_copy_tables ();
> >>          basic_block cond_bb;
> >>
> >>          class loop *loop2 = loop_version (loop1, cond, &cond_bb,
> >> -                                          profile_probability::always (),
> >> -                                          profile_probability::always (),
> >> +                                          true_edge->probability,
> >> +                                          true_edge->probability.invert (),
> >>                                             profile_probability::always (),
> >>                                             profile_probability::always (),
> >>                                             true);
> >> @@ -1486,8 +1489,8 @@ do_split_loop_on_cond (struct loop *loop1, edge invar_branch)
> >>     initialize_original_copy_tables ();
> >>
> >>     struct loop *loop2 = loop_version (loop1, boolean_true_node, NULL,
> >> -                                    profile_probability::always (),
> >> -                                    profile_probability::never (),
> >> +                                    invar_branch->probability,
> >> +                                    invar_branch->probability.invert (),
> >>                                       profile_probability::always (),
> >>                                       profile_probability::always (),
> >>                                       true);
> >> @@ -1530,6 +1533,9 @@ do_split_loop_on_cond (struct loop *loop1, edge invar_branch)
> >>     to_loop1->flags |= true_invar ? EDGE_FALSE_VALUE : EDGE_TRUE_VALUE;
> >>     to_loop2->flags |= true_invar ? EDGE_TRUE_VALUE : EDGE_FALSE_VALUE;
> >>
> >> +  to_loop1->probability = invar_branch->probability.invert ();
> >> +  to_loop2->probability = invar_branch->probability;
> >> +
> >>     /* Due to introduction of a control flow edge from loop1 latch to loop2
> >>        pre-header, we should update PHIs in loop2 to reflect this connection
> >>        between loop1 and loop2.  */
> >> --
> >> 2.27.0.90.geebb51ba8c
> >>

Re: [RFC] Don't move cold code out of loop by checking bb count

[Xionghu Luo]

On 2021/8/26 19:33, Richard Biener wrote:
> On Tue, Aug 10, 2021 at 4:03 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>>
>> Hi,
>>
>> On 2021/8/6 20:15, Richard Biener wrote:
>>> On Mon, Aug 2, 2021 at 7:05 AM Xiong Hu Luo <luoxhu@linux.ibm.com> wrote:
>>>>
>>>> There was a patch trying to avoid move cold block out of loop:
>>>>
>>>> https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html
>>>>
>>>> Richard suggested to "never hoist anything from a bb with lower execution
>>>> frequency to a bb with higher one in LIM invariantness_dom_walker
>>>> before_dom_children".
>>>>
>>>> This patch does this profile count check in both gimple LIM
>>>> move_computations_worker and RTL loop-invariant.c find_invariants_bb,
>>>> if the loop bb is colder than loop preheader, don't hoist it out of
>>>> loop.
>>>>
>>>> Also, the profile count in loop split pass should be corrected to avoid
>>>> lim2 and lim4 mismatch behavior, currently, the new loop preheader generated
>>>> by loop_version is set to "[count: 0]:", then lim4 after lsplt pass will
>>>> move statement out of loop unexpectely when lim2 didn't move it.  This
>>>> change could fix regression on 544.nab_r from -1.55% to +0.46%.
>>>>
>>>> SPEC2017 performance evaluation shows 1% performance improvement for
>>>> intrate GEOMEAN and no obvious regression for others.  Especially,
>>>> 500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
>>>> largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
>>>> on P8LE.
>>>>
>>>> Regression and bootstrap tested pass on P8LE, any comments?  Thanks.
>>>
>>> While I'm not familiar with the RTL invariant motion pass the patch there
>>> looks reasonable.  Note that we should assess the profile quality
>>> somehow - I'm not sure how to do that, CCed Honza for that.
>>
>> Thanks.
>>
>>>
>>> For the GIMPLE part the patch looks quite complicated - but note it
>>> probably has to be since LIM performs kind of a "CSE" on loads
>>> (and stores for store-motion), so when there are multiple stmts
>>> affected by a hoisting decision the biggest block count has to be
>>> accounted.  Likewise when there are dependent stmts involved
>>> that might include conditional stmts (a "PHI"), but the overall
>>> cost should be looked at.
>>
>> Currently, The gimple code check two situations with the patch:
>> 1) The statement or PHI‘s BB is *colder* then preheader, don't move it out
>> of loop;
>> 2) The statement or PHI's BB is *hotter* then preheader, but any of it's rhs
>> couldn't be moved out of loop, also don't move it out of loop to avoid definition
>> not dominates use error.
> 
> But part 2) is obviously already done.  What I tried to say is your heuristic
> doesn't integrate nicely with the pass but I admitted that it might be a bit
> difficult to find a place to add this heuristic.
> 
> There is lim_data->cost which we could bias negatively but then this is
> a cost that is independent on the hoisting distance.  But doing this would
> work at least for the case where the immediately enclosing loop preheader
> is hotter than the stmt and with this it would be a patch that's similarly
> simple as the RTL one.
> 
> Another possibility is to simply only adjust PHI processing in
> compute_invariantness, capping movement according to the hotness
> heuristic.  The same could be done for regular stmts there but I'm
> not sure that will do good in the end since this function is supposed
> to compute "correctness" (well, it also has the cost stuff), and it's
> not the place to do overall cost considerations.

Thanks.  I found that adding a function find_coldest_out_loop and check it in
outermost_invariant_loop to find the coldest invariant loop between outermost
loop and itself could also reach the purpose.  Then the gimple code check is
redundant and could be removed.

> 
>> May be I could collect the number of instructions not hoisted with the patch
>> on regression tests and SPEC2017 to do a estimation for "multiple stmts affected"
>> and "overall cost" need to be considered?  But it seems move_computations_worker
>> couldn't rollback if we still want to hoist multiple stmts out during the iterations?
>>
>>>
>>> Now - GIMPLE LIM "costing" is somewhat backward right now
>>> and it isn't set up to consider those multiple involved stmts.  Plus
>>> the store-motion part does not have any cost part (but it depends
>>> on previously decided invariant motions).
>>>
>>> I think the way you implemented the check will cause no hoisting
>>> to be performed instead of, say, hoisting to a different loop level
>>> only.  Possibly shown when you consider a loop nest like
>>>
>>>     for (;;)
>>>       if (unlikely_cond)
>>>         for (;;)
>>>            invariant;
>>>
>>> we want to hoist 'invariant' but only from the inner loop even if it
>>> is invariant also in the outer loop.
>>
>>
>> For this case, theorotically I think the master GCC will optimize it to:
>>
>>    invariant;
>>    for (;;)
>>      if (unlikely_cond)
>>        for (;;)
>>           ;
>>
>> 'invariant' is moved out of outer loop, but with the patch, it will get:
>>
>>    for (;;)
>>      if (unlikely_cond)
>>        {
>>          invariant;
>>          for (;;)
>>             ;
>>        }
>>
>> 'invariant' is *cold* for outer loop, but it is still *hot* for inner loop,
>> so hoist it out of inner loop, this is exactly what we want, right?
> 
> Yes.  I had doubts your patch would achieve that.
> 


The below updated patch could achieve it:


There was a patch trying to avoid move cold block out of loop:

https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html

Richard suggested to "never hoist anything from a bb with lower execution
frequency to a bb with higher one in LIM invariantness_dom_walker
before_dom_children".

In gimple LIM analysis,  add find_coldest_out_loop to move invariants to
expected target loop, then  if profile count of the loop bb is colder
than target loop preheader, it won't be hoisted out of loop.

SPEC2017 performance evaluation shows 1% performance improvement for
intrate GEOMEAN and no obvious regression for others.  Especially,
500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
on P8LE.

Regression and bootstrap tested pass on P8LE, any comments?  Thanks.

gcc/ChangeLog:

	* loop-invariant.c (find_invariants_bb): Check profile count
	before motion.
	(find_invariants_body): Add argument.
	* tree-ssa-loop-im.c (find_coldest_out_loop): New function.
	(outermost_invariant_loop): Use find_coldest_out_loop.
	(determine_max_movement): Likewise.
	(move_computations_worker): Adjust and fix iteration udpate.
	(execute_sm): Likewise.
	(execute_sm_exit): Check pointer validness.

gcc/testsuite/ChangeLog:

	* gcc.dg/tree-ssa/recip-3.c: Adjust.
	* gcc.dg/tree-ssa/ssa-lim-16.c: New test.
	* gcc.dg/tree-ssa/ssa-lim-17.c: New test.
---
  gcc/loop-invariant.c                       | 10 ++-
  gcc/tree-ssa-loop-im.c                     | 79 ++++++++++++++++++----
  gcc/testsuite/gcc.dg/tree-ssa/recip-3.c    |  2 +-
  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-16.c | 20 ++++++
  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-17.c | 26 +++++++
  5 files changed, 121 insertions(+), 16 deletions(-)
  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-16.c
  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-17.c

diff --git a/gcc/loop-invariant.c b/gcc/loop-invariant.c
index fca0c2b24be..5c3be7bf0eb 100644
--- a/gcc/loop-invariant.c
+++ b/gcc/loop-invariant.c
@@ -1183,9 +1183,14 @@ find_invariants_insn (rtx_insn *insn, bool always_reached, bool always_executed)
     call.  */
  
  static void
-find_invariants_bb (basic_block bb, bool always_reached, bool always_executed)
+find_invariants_bb (class loop *loop, basic_block bb, bool always_reached,
+		    bool always_executed)
  {
    rtx_insn *insn;
+  basic_block preheader = loop_preheader_edge (loop)->src;
+
+  if (preheader->count > bb->count)
+    return;
  
    FOR_BB_INSNS (bb, insn)
      {
@@ -1214,8 +1219,7 @@ find_invariants_body (class loop *loop, basic_block *body,
    unsigned i;
  
    for (i = 0; i < loop->num_nodes; i++)
-    find_invariants_bb (body[i],
-			bitmap_bit_p (always_reached, i),
+    find_invariants_bb (loop, body[i], bitmap_bit_p (always_reached, i),
  			bitmap_bit_p (always_executed, i));
  }
  
diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
index d9f75d5025e..f5ab6a734e7 100644
--- a/gcc/tree-ssa-loop-im.c
+++ b/gcc/tree-ssa-loop-im.c
@@ -417,6 +417,28 @@ movement_possibility (gimple *stmt)
    return ret;
  }
  
+/* Find coldest loop between outmost_loop and loop by comapring profile count.  */
+
+static class loop *
+find_coldest_out_loop (class loop *outmost_loop, class loop *loop,
+		       basic_block def_bb = NULL)
+{
+  class loop *cold_loop, *min_loop;
+  cold_loop = min_loop = outmost_loop;
+  profile_count min_count = loop_preheader_edge (min_loop)->src->count;
+
+  if (def_bb && def_bb->count < loop_preheader_edge (loop)->src->count)
+    return NULL;
+
+  while (min_loop != loop)
+    {
+      min_loop = superloop_at_depth (loop, loop_depth (min_loop) + 1);
+      if (loop_preheader_edge (min_loop)->src->count < min_count)
+	cold_loop = min_loop;
+    }
+  return cold_loop;
+}
+
  /* Suppose that operand DEF is used inside the LOOP.  Returns the outermost
     loop to that we could move the expression using DEF if it did not have
     other operands, i.e. the outermost loop enclosing LOOP in that the value
@@ -431,18 +453,18 @@ outermost_invariant_loop (tree def, class loop *loop)
    struct lim_aux_data *lim_data;
  
    if (!def)
-    return superloop_at_depth (loop, 1);
+    return find_coldest_out_loop (superloop_at_depth (loop, 1), loop);
  
    if (TREE_CODE (def) != SSA_NAME)
      {
        gcc_assert (is_gimple_min_invariant (def));
-      return superloop_at_depth (loop, 1);
+      return find_coldest_out_loop (superloop_at_depth (loop, 1), loop);
      }
  
    def_stmt = SSA_NAME_DEF_STMT (def);
    def_bb = gimple_bb (def_stmt);
    if (!def_bb)
-    return superloop_at_depth (loop, 1);
+    return find_coldest_out_loop (superloop_at_depth (loop, 1), loop, def_bb);
  
    max_loop = find_common_loop (loop, def_bb->loop_father);
  
@@ -452,7 +474,13 @@ outermost_invariant_loop (tree def, class loop *loop)
  				 loop_outer (lim_data->max_loop));
    if (max_loop == loop)
      return NULL;
-  max_loop = superloop_at_depth (loop, loop_depth (max_loop) + 1);
+  max_loop = find_coldest_out_loop (max_loop, loop, def_bb);
+  if (!max_loop)
+    return NULL;
+  if (max_loop == loop)
+    return max_loop;
+  else
+    max_loop = superloop_at_depth (loop, loop_depth (max_loop) + 1);
  
    return max_loop;
  }
@@ -684,7 +712,11 @@ determine_max_movement (gimple *stmt, bool must_preserve_exec)
    if (must_preserve_exec)
      level = ALWAYS_EXECUTED_IN (bb);
    else
-    level = superloop_at_depth (loop, 1);
+    level = find_coldest_out_loop (superloop_at_depth (loop, 1), loop, bb);
+
+  if (!level)
+    return false;
+
    lim_data->max_loop = level;
  
    if (gphi *phi = dyn_cast <gphi *> (stmt))
@@ -783,8 +815,10 @@ determine_max_movement (gimple *stmt, bool must_preserve_exec)
        if (ref
  	  && MEM_ANALYZABLE (ref))
  	{
-	  lim_data->max_loop = outermost_indep_loop (lim_data->max_loop,
-						     loop, ref);
+	  level = outermost_indep_loop (lim_data->max_loop, loop, ref);
+	  if (!level)
+	    return false;
+	  lim_data->max_loop = find_coldest_out_loop (level, loop, bb);
  	  if (!lim_data->max_loop)
  	    return false;
  	}
@@ -1154,6 +1188,7 @@ move_computations_worker (basic_block bb)
  	  continue;
  	}
  
+      edge e = loop_preheader_edge (level);
        if (dump_file && (dump_flags & TDF_DETAILS))
  	{
  	  fprintf (dump_file, "Moving PHI node\n");
@@ -1191,14 +1226,13 @@ move_computations_worker (basic_block bb)
  	  tree lhs = gimple_assign_lhs (new_stmt);
  	  SSA_NAME_RANGE_INFO (lhs) = NULL;
  	}
-      gsi_insert_on_edge (loop_preheader_edge (level), new_stmt);
+      gsi_insert_on_edge (e, new_stmt);
        remove_phi_node (&bsi, false);
      }
  
    for (gimple_stmt_iterator bsi = gsi_start_bb (bb); !gsi_end_p (bsi); )
      {
        edge e;
-
        gimple *stmt = gsi_stmt (bsi);
  
        lim_data = get_lim_data (stmt);
@@ -1221,8 +1255,12 @@ move_computations_worker (basic_block bb)
        /* We do not really want to move conditionals out of the loop; we just
  	 placed it here to force its operands to be moved if necessary.  */
        if (gimple_code (stmt) == GIMPLE_COND)
-	continue;
+	{
+	  gsi_next (&bsi);
+	  continue;
+	}
  
+      e = loop_preheader_edge (level);
        if (dump_file && (dump_flags & TDF_DETAILS))
  	{
  	  fprintf (dump_file, "Moving statement\n");
@@ -1231,7 +1269,6 @@ move_computations_worker (basic_block bb)
  		   cost, level->num);
  	}
  
-      e = loop_preheader_edge (level);
        gcc_assert (!gimple_vdef (stmt));
        if (gimple_vuse (stmt))
  	{
@@ -2133,6 +2170,19 @@ execute_sm (class loop *loop, im_mem_ref *ref,
    bool multi_threaded_model_p = false;
    gimple_stmt_iterator gsi;
    sm_aux *aux = new sm_aux;
+  basic_block bb = gimple_bb (first_mem_ref_loc (loop, ref)->stmt);
+
+  edge e = loop_preheader_edge (loop);
+  if (e->src->count > bb->count)
+    {
+      if (dump_file && (dump_flags & TDF_DETAILS))
+	{
+	  fprintf (dump_file, "Don't execute store motion of ");
+	  print_generic_expr (dump_file, ref->mem.ref);
+	  fprintf (dump_file, " from loop %d\n", loop->num);
+	}
+      return;
+    }
  
    if (dump_file && (dump_flags & TDF_DETAILS))
      {
@@ -2241,7 +2291,12 @@ execute_sm_exit (class loop *loop, edge ex, vec<seq_entry> &seq,
  	}
        else
  	{
-	  sm_aux *aux = *aux_map.get (ref);
+	  sm_aux **paux = aux_map.get (ref);
+	  sm_aux *aux;
+	  if (paux)
+	    aux = *paux;
+	  else
+	    continue;
  	  if (!aux->store_flag || kind == sm_ord)
  	    {
  	      gassign *store;
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
index 638bf38db8c..641c91e719e 100644
--- a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
+++ b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
@@ -23,4 +23,4 @@ float h ()
  	F[0] += E / d;
  }
  
-/* { dg-final { scan-tree-dump-times " / " 1 "recip" } } */
+/* { dg-final { scan-tree-dump-times " / " 5 "recip" } } */
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-16.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-16.c
new file mode 100644
index 00000000000..2303f3d5d86
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-16.c
@@ -0,0 +1,20 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+volatile int x;
+void
+assert_fail (int, char *, char *);
+void
+foo (int *a, int n, int k)
+{
+  int i;
+
+  for (i = 0; i < n; i++)
+    {
+      if (__builtin_expect (x, 0))
+	assert_fail (k / 5, "one", "two");
+      a[i] = k;
+    }
+}
+
+/* { dg-final { scan-tree-dump-not "out of loop 1" "lim2" } } */
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-17.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-17.c
new file mode 100644
index 00000000000..3b1c7c0cb3e
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-17.c
@@ -0,0 +1,26 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+volatile int x;
+void
+assert_fail (int, char *, char *);
+void
+foo (int *a, int n, int m, int k, int s)
+{
+  int i;
+  int j;
+
+  for (i = 0; i < m; i++)
+    {
+      if (__builtin_expect (x, 0))
+	for (j = 0; j < n; j++)
+	  {
+	    assert_fail (k / 5, "one", "two");
+	  a[s] = k;
+	}
+      a[s] = s;
+    }
+}
+
+/* { dg-final { scan-tree-dump-times "out of loop 2" 4 "lim2" } } */
+/* { dg-final { scan-tree-dump-times "out of loop 1" 3 "lim2" } } */
-- 
2.27.0.90.geebb51ba8c

Re: [RFC] Don't move cold code out of loop by checking bb count

[Richard Biener]

On Thu, Sep 9, 2021 at 3:56 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>
>
>
> On 2021/8/26 19:33, Richard Biener wrote:
> > On Tue, Aug 10, 2021 at 4:03 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
> >>
> >> Hi,
> >>
> >> On 2021/8/6 20:15, Richard Biener wrote:
> >>> On Mon, Aug 2, 2021 at 7:05 AM Xiong Hu Luo <luoxhu@linux.ibm.com> wrote:
> >>>>
> >>>> There was a patch trying to avoid move cold block out of loop:
> >>>>
> >>>> https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html
> >>>>
> >>>> Richard suggested to "never hoist anything from a bb with lower execution
> >>>> frequency to a bb with higher one in LIM invariantness_dom_walker
> >>>> before_dom_children".
> >>>>
> >>>> This patch does this profile count check in both gimple LIM
> >>>> move_computations_worker and RTL loop-invariant.c find_invariants_bb,
> >>>> if the loop bb is colder than loop preheader, don't hoist it out of
> >>>> loop.
> >>>>
> >>>> Also, the profile count in loop split pass should be corrected to avoid
> >>>> lim2 and lim4 mismatch behavior, currently, the new loop preheader generated
> >>>> by loop_version is set to "[count: 0]:", then lim4 after lsplt pass will
> >>>> move statement out of loop unexpectely when lim2 didn't move it.  This
> >>>> change could fix regression on 544.nab_r from -1.55% to +0.46%.
> >>>>
> >>>> SPEC2017 performance evaluation shows 1% performance improvement for
> >>>> intrate GEOMEAN and no obvious regression for others.  Especially,
> >>>> 500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
> >>>> largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
> >>>> on P8LE.
> >>>>
> >>>> Regression and bootstrap tested pass on P8LE, any comments?  Thanks.
> >>>
> >>> While I'm not familiar with the RTL invariant motion pass the patch there
> >>> looks reasonable.  Note that we should assess the profile quality
> >>> somehow - I'm not sure how to do that, CCed Honza for that.
> >>
> >> Thanks.
> >>
> >>>
> >>> For the GIMPLE part the patch looks quite complicated - but note it
> >>> probably has to be since LIM performs kind of a "CSE" on loads
> >>> (and stores for store-motion), so when there are multiple stmts
> >>> affected by a hoisting decision the biggest block count has to be
> >>> accounted.  Likewise when there are dependent stmts involved
> >>> that might include conditional stmts (a "PHI"), but the overall
> >>> cost should be looked at.
> >>
> >> Currently, The gimple code check two situations with the patch:
> >> 1) The statement or PHI‘s BB is *colder* then preheader, don't move it out
> >> of loop;
> >> 2) The statement or PHI's BB is *hotter* then preheader, but any of it's rhs
> >> couldn't be moved out of loop, also don't move it out of loop to avoid definition
> >> not dominates use error.
> >
> > But part 2) is obviously already done.  What I tried to say is your heuristic
> > doesn't integrate nicely with the pass but I admitted that it might be a bit
> > difficult to find a place to add this heuristic.
> >
> > There is lim_data->cost which we could bias negatively but then this is
> > a cost that is independent on the hoisting distance.  But doing this would
> > work at least for the case where the immediately enclosing loop preheader
> > is hotter than the stmt and with this it would be a patch that's similarly
> > simple as the RTL one.
> >
> > Another possibility is to simply only adjust PHI processing in
> > compute_invariantness, capping movement according to the hotness
> > heuristic.  The same could be done for regular stmts there but I'm
> > not sure that will do good in the end since this function is supposed
> > to compute "correctness" (well, it also has the cost stuff), and it's
> > not the place to do overall cost considerations.
>
> Thanks.  I found that adding a function find_coldest_out_loop and check it in
> outermost_invariant_loop to find the coldest invariant loop between outermost
> loop and itself could also reach the purpose.  Then the gimple code check is
> redundant and could be removed.
>
> >
> >> May be I could collect the number of instructions not hoisted with the patch
> >> on regression tests and SPEC2017 to do a estimation for "multiple stmts affected"
> >> and "overall cost" need to be considered?  But it seems move_computations_worker
> >> couldn't rollback if we still want to hoist multiple stmts out during the iterations?
> >>
> >>>
> >>> Now - GIMPLE LIM "costing" is somewhat backward right now
> >>> and it isn't set up to consider those multiple involved stmts.  Plus
> >>> the store-motion part does not have any cost part (but it depends
> >>> on previously decided invariant motions).
> >>>
> >>> I think the way you implemented the check will cause no hoisting
> >>> to be performed instead of, say, hoisting to a different loop level
> >>> only.  Possibly shown when you consider a loop nest like
> >>>
> >>>     for (;;)
> >>>       if (unlikely_cond)
> >>>         for (;;)
> >>>            invariant;
> >>>
> >>> we want to hoist 'invariant' but only from the inner loop even if it
> >>> is invariant also in the outer loop.
> >>
> >>
> >> For this case, theorotically I think the master GCC will optimize it to:
> >>
> >>    invariant;
> >>    for (;;)
> >>      if (unlikely_cond)
> >>        for (;;)
> >>           ;
> >>
> >> 'invariant' is moved out of outer loop, but with the patch, it will get:
> >>
> >>    for (;;)
> >>      if (unlikely_cond)
> >>        {
> >>          invariant;
> >>          for (;;)
> >>             ;
> >>        }
> >>
> >> 'invariant' is *cold* for outer loop, but it is still *hot* for inner loop,
> >> so hoist it out of inner loop, this is exactly what we want, right?
> >
> > Yes.  I had doubts your patch would achieve that.
> >
>
>
> The below updated patch could achieve it:
>
>
> There was a patch trying to avoid move cold block out of loop:
>
> https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html
>
> Richard suggested to "never hoist anything from a bb with lower execution
> frequency to a bb with higher one in LIM invariantness_dom_walker
> before_dom_children".
>
> In gimple LIM analysis,  add find_coldest_out_loop to move invariants to
> expected target loop, then  if profile count of the loop bb is colder
> than target loop preheader, it won't be hoisted out of loop.
>
> SPEC2017 performance evaluation shows 1% performance improvement for
> intrate GEOMEAN and no obvious regression for others.  Especially,
> 500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
> largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
> on P8LE.
>
> Regression and bootstrap tested pass on P8LE, any comments?  Thanks.

Can you split the RTL and GIMPLE changes and measure them separately
please?

> gcc/ChangeLog:
>
>         * loop-invariant.c (find_invariants_bb): Check profile count
>         before motion.
>         (find_invariants_body): Add argument.
>         * tree-ssa-loop-im.c (find_coldest_out_loop): New function.
>         (outermost_invariant_loop): Use find_coldest_out_loop.
>         (determine_max_movement): Likewise.
>         (move_computations_worker): Adjust and fix iteration udpate.
>         (execute_sm): Likewise.
>         (execute_sm_exit): Check pointer validness.
>
> gcc/testsuite/ChangeLog:
>
>         * gcc.dg/tree-ssa/recip-3.c: Adjust.
>         * gcc.dg/tree-ssa/ssa-lim-16.c: New test.
>         * gcc.dg/tree-ssa/ssa-lim-17.c: New test.
> ---
>   gcc/loop-invariant.c                       | 10 ++-
>   gcc/tree-ssa-loop-im.c                     | 79 ++++++++++++++++++----
>   gcc/testsuite/gcc.dg/tree-ssa/recip-3.c    |  2 +-
>   gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-16.c | 20 ++++++
>   gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-17.c | 26 +++++++
>   5 files changed, 121 insertions(+), 16 deletions(-)
>   create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-16.c
>   create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-17.c
>
> diff --git a/gcc/loop-invariant.c b/gcc/loop-invariant.c
> index fca0c2b24be..5c3be7bf0eb 100644
> --- a/gcc/loop-invariant.c
> +++ b/gcc/loop-invariant.c
> @@ -1183,9 +1183,14 @@ find_invariants_insn (rtx_insn *insn, bool always_reached, bool always_executed)
>      call.  */
>
>   static void
> -find_invariants_bb (basic_block bb, bool always_reached, bool always_executed)
> +find_invariants_bb (class loop *loop, basic_block bb, bool always_reached,
> +                   bool always_executed)
>   {
>     rtx_insn *insn;
> +  basic_block preheader = loop_preheader_edge (loop)->src;
> +
> +  if (preheader->count > bb->count)
> +    return;
>
>     FOR_BB_INSNS (bb, insn)
>       {
> @@ -1214,8 +1219,7 @@ find_invariants_body (class loop *loop, basic_block *body,
>     unsigned i;
>
>     for (i = 0; i < loop->num_nodes; i++)
> -    find_invariants_bb (body[i],
> -                       bitmap_bit_p (always_reached, i),
> +    find_invariants_bb (loop, body[i], bitmap_bit_p (always_reached, i),
>                         bitmap_bit_p (always_executed, i));
>   }
>
> diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
> index d9f75d5025e..f5ab6a734e7 100644
> --- a/gcc/tree-ssa-loop-im.c
> +++ b/gcc/tree-ssa-loop-im.c
> @@ -417,6 +417,28 @@ movement_possibility (gimple *stmt)
>     return ret;
>   }
>
> +/* Find coldest loop between outmost_loop and loop by comapring profile count.  */
> +
> +static class loop *
> +find_coldest_out_loop (class loop *outmost_loop, class loop *loop,
> +                      basic_block def_bb = NULL)
> +{
> +  class loop *cold_loop, *min_loop;
> +  cold_loop = min_loop = outmost_loop;
> +  profile_count min_count = loop_preheader_edge (min_loop)->src->count;
> +
> +  if (def_bb && def_bb->count < loop_preheader_edge (loop)->src->count)
> +    return NULL;
> +
> +  while (min_loop != loop)
> +    {
> +      min_loop = superloop_at_depth (loop, loop_depth (min_loop) + 1);
> +      if (loop_preheader_edge (min_loop)->src->count < min_count)
> +       cold_loop = min_loop;
> +    }
> +  return cold_loop;
> +}
> +
>   /* Suppose that operand DEF is used inside the LOOP.  Returns the outermost
>      loop to that we could move the expression using DEF if it did not have
>      other operands, i.e. the outermost loop enclosing LOOP in that the value
> @@ -431,18 +453,18 @@ outermost_invariant_loop (tree def, class loop *loop)
>     struct lim_aux_data *lim_data;
>
>     if (!def)
> -    return superloop_at_depth (loop, 1);
> +    return find_coldest_out_loop (superloop_at_depth (loop, 1), loop);
>
>     if (TREE_CODE (def) != SSA_NAME)
>       {
>         gcc_assert (is_gimple_min_invariant (def));
> -      return superloop_at_depth (loop, 1);
> +      return find_coldest_out_loop (superloop_at_depth (loop, 1), loop);
>       }
>
>     def_stmt = SSA_NAME_DEF_STMT (def);
>     def_bb = gimple_bb (def_stmt);
>     if (!def_bb)
> -    return superloop_at_depth (loop, 1);
> +    return find_coldest_out_loop (superloop_at_depth (loop, 1), loop, def_bb);
>
>     max_loop = find_common_loop (loop, def_bb->loop_father);
>
> @@ -452,7 +474,13 @@ outermost_invariant_loop (tree def, class loop *loop)
>                                  loop_outer (lim_data->max_loop));
>     if (max_loop == loop)
>       return NULL;
> -  max_loop = superloop_at_depth (loop, loop_depth (max_loop) + 1);
> +  max_loop = find_coldest_out_loop (max_loop, loop, def_bb);
> +  if (!max_loop)
> +    return NULL;
> +  if (max_loop == loop)
> +    return max_loop;
> +  else
> +    max_loop = superloop_at_depth (loop, loop_depth (max_loop) + 1);
>
>     return max_loop;
>   }

As said 'outermost_invariant_loop' is the "correctness" part and I
don't like changing
it this way.  Instead determine_max_movement is what should be adjusted ...

> @@ -684,7 +712,11 @@ determine_max_movement (gimple *stmt, bool must_preserve_exec)
>     if (must_preserve_exec)
>       level = ALWAYS_EXECUTED_IN (bb);
>     else
> -    level = superloop_at_depth (loop, 1);
> +    level = find_coldest_out_loop (superloop_at_depth (loop, 1), loop, bb);

... which you do here (but you should apply that also to the must_preserve_exec
result).

> +
> +  if (!level)
> +    return false;
> +
>     lim_data->max_loop = level;
>
>     if (gphi *phi = dyn_cast <gphi *> (stmt))
> @@ -783,8 +815,10 @@ determine_max_movement (gimple *stmt, bool must_preserve_exec)
>         if (ref
>           && MEM_ANALYZABLE (ref))
>         {
> -         lim_data->max_loop = outermost_indep_loop (lim_data->max_loop,
> -                                                    loop, ref);
> +         level = outermost_indep_loop (lim_data->max_loop, loop, ref);
> +         if (!level)
> +           return false;
> +         lim_data->max_loop = find_coldest_out_loop (level, loop, bb);

... why again here?  outermost_indep_loop honors the passed max_loop.

>           if (!lim_data->max_loop)
>             return false;
>         }
> @@ -1154,6 +1188,7 @@ move_computations_worker (basic_block bb)
>           continue;
>         }
>
> +      edge e = loop_preheader_edge (level);

unncecessary change

>         if (dump_file && (dump_flags & TDF_DETAILS))
>         {
>           fprintf (dump_file, "Moving PHI node\n");
> @@ -1191,14 +1226,13 @@ move_computations_worker (basic_block bb)
>           tree lhs = gimple_assign_lhs (new_stmt);
>           SSA_NAME_RANGE_INFO (lhs) = NULL;
>         }
> -      gsi_insert_on_edge (loop_preheader_edge (level), new_stmt);
> +      gsi_insert_on_edge (e, new_stmt);
>         remove_phi_node (&bsi, false);
>       }
>
>     for (gimple_stmt_iterator bsi = gsi_start_bb (bb); !gsi_end_p (bsi); )
>       {
>         edge e;
> -
>         gimple *stmt = gsi_stmt (bsi);
>
>         lim_data = get_lim_data (stmt);
> @@ -1221,8 +1255,12 @@ move_computations_worker (basic_block bb)
>         /* We do not really want to move conditionals out of the loop; we just
>          placed it here to force its operands to be moved if necessary.  */
>         if (gimple_code (stmt) == GIMPLE_COND)
> -       continue;
> +       {
> +         gsi_next (&bsi);
> +         continue;
> +       }

looks like an omission - do you now run into this?

>
> +      e = loop_preheader_edge (level);

unnecessary change

>         if (dump_file && (dump_flags & TDF_DETAILS))
>         {
>           fprintf (dump_file, "Moving statement\n");
> @@ -1231,7 +1269,6 @@ move_computations_worker (basic_block bb)
>                    cost, level->num);
>         }
>
> -      e = loop_preheader_edge (level);
>         gcc_assert (!gimple_vdef (stmt));
>         if (gimple_vuse (stmt))
>         {
> @@ -2133,6 +2170,19 @@ execute_sm (class loop *loop, im_mem_ref *ref,
>     bool multi_threaded_model_p = false;
>     gimple_stmt_iterator gsi;
>     sm_aux *aux = new sm_aux;
> +  basic_block bb = gimple_bb (first_mem_ref_loc (loop, ref)->stmt);
> +
> +  edge e = loop_preheader_edge (loop);
> +  if (e->src->count > bb->count)
> +    {
> +      if (dump_file && (dump_flags & TDF_DETAILS))
> +       {
> +         fprintf (dump_file, "Don't execute store motion of ");
> +         print_generic_expr (dump_file, ref->mem.ref);
> +         fprintf (dump_file, " from loop %d\n", loop->num);
> +       }
> +      return;
> +    }

why do you need this?  I think you instead want to adjust 'can_sm_ref_p'
where you want to use sth like

  for_all_locs_in_loop (loop, ref, ...)

and ... being a lambda or function that checks loc->stmt and if at least one
reference is executed in a hot part of 'loop' then we should apply store-motion.
Do this last because it looks somewhat expensive.

>
>     if (dump_file && (dump_flags & TDF_DETAILS))
>       {
> @@ -2241,7 +2291,12 @@ execute_sm_exit (class loop *loop, edge ex, vec<seq_entry> &seq,
>         }
>         else
>         {
> -         sm_aux *aux = *aux_map.get (ref);
> +         sm_aux **paux = aux_map.get (ref);
> +         sm_aux *aux;
> +         if (paux)
> +           aux = *paux;
> +         else
> +           continue;
>           if (!aux->store_flag || kind == sm_ord)
>             {
>               gassign *store;
> diff --git a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
> index 638bf38db8c..641c91e719e 100644
> --- a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
> +++ b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
> @@ -23,4 +23,4 @@ float h ()
>         F[0] += E / d;
>   }
>
> -/* { dg-final { scan-tree-dump-times " / " 1 "recip" } } */
> +/* { dg-final { scan-tree-dump-times " / " 5 "recip" } } */
> diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-16.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-16.c
> new file mode 100644
> index 00000000000..2303f3d5d86
> --- /dev/null
> +++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-16.c
> @@ -0,0 +1,20 @@
> +/* { dg-do compile } */
> +/* { dg-options "-O2 -fdump-tree-lim2-details" } */
> +
> +volatile int x;
> +void
> +assert_fail (int, char *, char *);
> +void
> +foo (int *a, int n, int k)
> +{
> +  int i;
> +
> +  for (i = 0; i < n; i++)
> +    {
> +      if (__builtin_expect (x, 0))
> +       assert_fail (k / 5, "one", "two");

I don't think these are very good testcases since 'assert' is usually
noreturn which would place the whole block outside of the loop (it's
a loop exit then).  But naming the function 'foo' would make it less
obviously pointless.

> +      a[i] = k;
> +    }
> +}
> +
> +/* { dg-final { scan-tree-dump-not "out of loop 1" "lim2" } } */
> diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-17.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-17.c
> new file mode 100644
> index 00000000000..3b1c7c0cb3e
> --- /dev/null
> +++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-17.c
> @@ -0,0 +1,26 @@
> +/* { dg-do compile } */
> +/* { dg-options "-O2 -fdump-tree-lim2-details" } */
> +
> +volatile int x;
> +void
> +assert_fail (int, char *, char *);
> +void
> +foo (int *a, int n, int m, int k, int s)
> +{
> +  int i;
> +  int j;
> +
> +  for (i = 0; i < m; i++)
> +    {
> +      if (__builtin_expect (x, 0))
> +       for (j = 0; j < n; j++)
> +         {
> +           assert_fail (k / 5, "one", "two");
> +         a[s] = k;
> +       }
> +      a[s] = s;
> +    }
> +}
> +
> +/* { dg-final { scan-tree-dump-times "out of loop 2" 4 "lim2" } } */
> +/* { dg-final { scan-tree-dump-times "out of loop 1" 3 "lim2" } } */
> --
> 2.27.0.90.geebb51ba8c
>
>

Re: [RFC] Don't move cold code out of loop by checking bb count

[Xionghu Luo]

On 2021/9/22 17:14, Richard Biener wrote:
> On Thu, Sep 9, 2021 at 3:56 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>>
>>
>>
>> On 2021/8/26 19:33, Richard Biener wrote:
>>> On Tue, Aug 10, 2021 at 4:03 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> On 2021/8/6 20:15, Richard Biener wrote:
>>>>> On Mon, Aug 2, 2021 at 7:05 AM Xiong Hu Luo <luoxhu@linux.ibm.com> wrote:
>>>>>>
>>>>>> There was a patch trying to avoid move cold block out of loop:
>>>>>>
>>>>>> https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html
>>>>>>
>>>>>> Richard suggested to "never hoist anything from a bb with lower execution
>>>>>> frequency to a bb with higher one in LIM invariantness_dom_walker
>>>>>> before_dom_children".
>>>>>>
>>>>>> This patch does this profile count check in both gimple LIM
>>>>>> move_computations_worker and RTL loop-invariant.c find_invariants_bb,
>>>>>> if the loop bb is colder than loop preheader, don't hoist it out of
>>>>>> loop.
>>>>>>
>>>>>> Also, the profile count in loop split pass should be corrected to avoid
>>>>>> lim2 and lim4 mismatch behavior, currently, the new loop preheader generated
>>>>>> by loop_version is set to "[count: 0]:", then lim4 after lsplt pass will
>>>>>> move statement out of loop unexpectely when lim2 didn't move it.  This
>>>>>> change could fix regression on 544.nab_r from -1.55% to +0.46%.
>>>>>>
>>>>>> SPEC2017 performance evaluation shows 1% performance improvement for
>>>>>> intrate GEOMEAN and no obvious regression for others.  Especially,
>>>>>> 500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
>>>>>> largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
>>>>>> on P8LE.
>>>>>>
>>>>>> Regression and bootstrap tested pass on P8LE, any comments?  Thanks.
>>>>>
>>>>> While I'm not familiar with the RTL invariant motion pass the patch there
>>>>> looks reasonable.  Note that we should assess the profile quality
>>>>> somehow - I'm not sure how to do that, CCed Honza for that.
>>>>
>>>> Thanks.
>>>>
>>>>>
>>>>> For the GIMPLE part the patch looks quite complicated - but note it
>>>>> probably has to be since LIM performs kind of a "CSE" on loads
>>>>> (and stores for store-motion), so when there are multiple stmts
>>>>> affected by a hoisting decision the biggest block count has to be
>>>>> accounted.  Likewise when there are dependent stmts involved
>>>>> that might include conditional stmts (a "PHI"), but the overall
>>>>> cost should be looked at.
>>>>
>>>> Currently, The gimple code check two situations with the patch:
>>>> 1) The statement or PHI‘s BB is *colder* then preheader, don't move it out
>>>> of loop;
>>>> 2) The statement or PHI's BB is *hotter* then preheader, but any of it's rhs
>>>> couldn't be moved out of loop, also don't move it out of loop to avoid definition
>>>> not dominates use error.
>>>
>>> But part 2) is obviously already done.  What I tried to say is your heuristic
>>> doesn't integrate nicely with the pass but I admitted that it might be a bit
>>> difficult to find a place to add this heuristic.
>>>
>>> There is lim_data->cost which we could bias negatively but then this is
>>> a cost that is independent on the hoisting distance.  But doing this would
>>> work at least for the case where the immediately enclosing loop preheader
>>> is hotter than the stmt and with this it would be a patch that's similarly
>>> simple as the RTL one.
>>>
>>> Another possibility is to simply only adjust PHI processing in
>>> compute_invariantness, capping movement according to the hotness
>>> heuristic.  The same could be done for regular stmts there but I'm
>>> not sure that will do good in the end since this function is supposed
>>> to compute "correctness" (well, it also has the cost stuff), and it's
>>> not the place to do overall cost considerations.
>>
>> Thanks.  I found that adding a function find_coldest_out_loop and check it in
>> outermost_invariant_loop to find the coldest invariant loop between outermost
>> loop and itself could also reach the purpose.  Then the gimple code check is
>> redundant and could be removed.
>>
>>>
>>>> May be I could collect the number of instructions not hoisted with the patch
>>>> on regression tests and SPEC2017 to do a estimation for "multiple stmts affected"
>>>> and "overall cost" need to be considered?  But it seems move_computations_worker
>>>> couldn't rollback if we still want to hoist multiple stmts out during the iterations?
>>>>
>>>>>
>>>>> Now - GIMPLE LIM "costing" is somewhat backward right now
>>>>> and it isn't set up to consider those multiple involved stmts.  Plus
>>>>> the store-motion part does not have any cost part (but it depends
>>>>> on previously decided invariant motions).
>>>>>
>>>>> I think the way you implemented the check will cause no hoisting
>>>>> to be performed instead of, say, hoisting to a different loop level
>>>>> only.  Possibly shown when you consider a loop nest like
>>>>>
>>>>>      for (;;)
>>>>>        if (unlikely_cond)
>>>>>          for (;;)
>>>>>             invariant;
>>>>>
>>>>> we want to hoist 'invariant' but only from the inner loop even if it
>>>>> is invariant also in the outer loop.
>>>>
>>>>
>>>> For this case, theorotically I think the master GCC will optimize it to:
>>>>
>>>>     invariant;
>>>>     for (;;)
>>>>       if (unlikely_cond)
>>>>         for (;;)
>>>>            ;
>>>>
>>>> 'invariant' is moved out of outer loop, but with the patch, it will get:
>>>>
>>>>     for (;;)
>>>>       if (unlikely_cond)
>>>>         {
>>>>           invariant;
>>>>           for (;;)
>>>>              ;
>>>>         }
>>>>
>>>> 'invariant' is *cold* for outer loop, but it is still *hot* for inner loop,
>>>> so hoist it out of inner loop, this is exactly what we want, right?
>>>
>>> Yes.  I had doubts your patch would achieve that.
>>>
>>
>>
>> The below updated patch could achieve it:
>>
>>
>> There was a patch trying to avoid move cold block out of loop:
>>
>> https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html
>>
>> Richard suggested to "never hoist anything from a bb with lower execution
>> frequency to a bb with higher one in LIM invariantness_dom_walker
>> before_dom_children".
>>
>> In gimple LIM analysis,  add find_coldest_out_loop to move invariants to
>> expected target loop, then  if profile count of the loop bb is colder
>> than target loop preheader, it won't be hoisted out of loop.
>>
>> SPEC2017 performance evaluation shows 1% performance improvement for
>> intrate GEOMEAN and no obvious regression for others.  Especially,
>> 500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
>> largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
>> on P8LE.
>>
>> Regression and bootstrap tested pass on P8LE, any comments?  Thanks.
> 
> Can you split the RTL and GIMPLE changes and measure them separately
> please?

I did that before and got below data, it is slightly different due to
using ratio instead of seconds, 500.perlbench_r obviously benefits
from the RTL part change, while gimple part only improves exchange2
and blender, with a regression on nab which requires the fix of loop
split,

https://gcc.gnu.org/pipermail/gcc-patches/2021-August/576566.html

Reason is lim2 doesn't hoist code out of loop, but loop split generates
duplicated loop with incorrect profile count on preheader and header bb,
then later lim4 hoists code out of loop to unexpected place.  Same loop
shows mismatch behavior in lim2 and lim4.  With that patch, the regression
is gone.


               Gimple+RTL |  Gimple lim | RTL loop-invariant
500.perlbench_r     8.03%    0.67%    7.69%
502.gcc_r           0.56%    0.37%    0.19%
505.mcf_r           0.19%   -0.19%    0.39%
520.omnetpp_r       0.83%    0.83%    0.83%
523.xalancbmk_r    -0.78%    0.00%   -1.04%
525.x264_r          0.17%    0.00%    0.00%
531.deepsjeng_r     0.00%    0.31%    0.00%
541.leela_r         0.00%   -0.31%    0.31%
548.exchange2_r     2.08%    1.85%    0.23%
557.xz_r            0.97%    0.00%    0.65%
503.bwaves_r       -0.12%    0.00%   -0.23%
507.cactuBSSN_r     0.00%    0.14%    0.00%
508.namd_r          0.00%    0.00%    0.00%
510.parest_r       -0.16%   -0.65%    0.00%
511.povray_r        0.30%    0.91%    0.91%
519.lbm_r           0.15%    0.00%    0.00%
521.wrf_r           0.00%    0.00%   -0.80%
526.blender_r       1.84%    0.26%    0.52%
527.cam4_r          0.28%    0.00%    0.00%
538.imagick_r       0.20%    0.00%    0.00%
544.nab_r          -1.55%   -0.78%    0.00%
549.fotonik3d_r    -0.25%    0.00%    0.00%
554.roms_r         -0.84%    0.00%   -0.63%
INT GEAMEAN         1.16%    0.35%    0.90%
FLOAT GEOMEAN      -0.01%   -0.01%   -0.02%
GEOMEAN             0.50%    0.15%    0.38%


Will address other comments in later reply.  Thanks.


-- 
Thanks,
Xionghu

Re: [RFC] Don't move cold code out of loop by checking bb count

[Xionghu Luo]

On 2021/9/23 10:13, Xionghu Luo via Gcc-patches wrote:
> 
> 
> On 2021/9/22 17:14, Richard Biener wrote:
>> On Thu, Sep 9, 2021 at 3:56 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>>>
>>>
>>>
>>> On 2021/8/26 19:33, Richard Biener wrote:
>>>> On Tue, Aug 10, 2021 at 4:03 AM Xionghu Luo <luoxhu@linux.ibm.com> 
>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> On 2021/8/6 20:15, Richard Biener wrote:
>>>>>> On Mon, Aug 2, 2021 at 7:05 AM Xiong Hu Luo <luoxhu@linux.ibm.com> 
>>>>>> wrote:
>>>>>>>
>>>>>>> There was a patch trying to avoid move cold block out of loop:
>>>>>>>
>>>>>>> https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html
>>>>>>>
>>>>>>> Richard suggested to "never hoist anything from a bb with lower 
>>>>>>> execution
>>>>>>> frequency to a bb with higher one in LIM invariantness_dom_walker
>>>>>>> before_dom_children".
>>>>>>>
>>>>>>> This patch does this profile count check in both gimple LIM
>>>>>>> move_computations_worker and RTL loop-invariant.c 
>>>>>>> find_invariants_bb,
>>>>>>> if the loop bb is colder than loop preheader, don't hoist it out of
>>>>>>> loop.
>>>>>>>
>>>>>>> Also, the profile count in loop split pass should be corrected to 
>>>>>>> avoid
>>>>>>> lim2 and lim4 mismatch behavior, currently, the new loop 
>>>>>>> preheader generated
>>>>>>> by loop_version is set to "[count: 0]:", then lim4 after lsplt 
>>>>>>> pass will
>>>>>>> move statement out of loop unexpectely when lim2 didn't move it.  
>>>>>>> This
>>>>>>> change could fix regression on 544.nab_r from -1.55% to +0.46%.
>>>>>>>
>>>>>>> SPEC2017 performance evaluation shows 1% performance improvement for
>>>>>>> intrate GEOMEAN and no obvious regression for others.  Especially,
>>>>>>> 500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
>>>>>>> largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
>>>>>>> on P8LE.
>>>>>>>
>>>>>>> Regression and bootstrap tested pass on P8LE, any comments?  Thanks.
>>>>>>
>>>>>> While I'm not familiar with the RTL invariant motion pass the 
>>>>>> patch there
>>>>>> looks reasonable.  Note that we should assess the profile quality
>>>>>> somehow - I'm not sure how to do that, CCed Honza for that.
>>>>>
>>>>> Thanks.
>>>>>
>>>>>>
>>>>>> For the GIMPLE part the patch looks quite complicated - but note it
>>>>>> probably has to be since LIM performs kind of a "CSE" on loads
>>>>>> (and stores for store-motion), so when there are multiple stmts
>>>>>> affected by a hoisting decision the biggest block count has to be
>>>>>> accounted.  Likewise when there are dependent stmts involved
>>>>>> that might include conditional stmts (a "PHI"), but the overall
>>>>>> cost should be looked at.
>>>>>
>>>>> Currently, The gimple code check two situations with the patch:
>>>>> 1) The statement or PHI‘s BB is *colder* then preheader, don't move 
>>>>> it out
>>>>> of loop;
>>>>> 2) The statement or PHI's BB is *hotter* then preheader, but any of 
>>>>> it's rhs
>>>>> couldn't be moved out of loop, also don't move it out of loop to 
>>>>> avoid definition
>>>>> not dominates use error.
>>>>
>>>> But part 2) is obviously already done.  What I tried to say is your 
>>>> heuristic
>>>> doesn't integrate nicely with the pass but I admitted that it might 
>>>> be a bit
>>>> difficult to find a place to add this heuristic.
>>>>
>>>> There is lim_data->cost which we could bias negatively but then this is
>>>> a cost that is independent on the hoisting distance.  But doing this 
>>>> would
>>>> work at least for the case where the immediately enclosing loop 
>>>> preheader
>>>> is hotter than the stmt and with this it would be a patch that's 
>>>> similarly
>>>> simple as the RTL one.
>>>>
>>>> Another possibility is to simply only adjust PHI processing in
>>>> compute_invariantness, capping movement according to the hotness
>>>> heuristic.  The same could be done for regular stmts there but I'm
>>>> not sure that will do good in the end since this function is supposed
>>>> to compute "correctness" (well, it also has the cost stuff), and it's
>>>> not the place to do overall cost considerations.
>>>
>>> Thanks.  I found that adding a function find_coldest_out_loop and 
>>> check it in
>>> outermost_invariant_loop to find the coldest invariant loop between 
>>> outermost
>>> loop and itself could also reach the purpose.  Then the gimple code 
>>> check is
>>> redundant and could be removed.
>>>
>>>>
>>>>> May be I could collect the number of instructions not hoisted with 
>>>>> the patch
>>>>> on regression tests and SPEC2017 to do a estimation for "multiple 
>>>>> stmts affected"
>>>>> and "overall cost" need to be considered?  But it seems 
>>>>> move_computations_worker
>>>>> couldn't rollback if we still want to hoist multiple stmts out 
>>>>> during the iterations?
>>>>>
>>>>>>
>>>>>> Now - GIMPLE LIM "costing" is somewhat backward right now
>>>>>> and it isn't set up to consider those multiple involved stmts.  Plus
>>>>>> the store-motion part does not have any cost part (but it depends
>>>>>> on previously decided invariant motions).
>>>>>>
>>>>>> I think the way you implemented the check will cause no hoisting
>>>>>> to be performed instead of, say, hoisting to a different loop level
>>>>>> only.  Possibly shown when you consider a loop nest like
>>>>>>
>>>>>>      for (;;)
>>>>>>        if (unlikely_cond)
>>>>>>          for (;;)
>>>>>>             invariant;
>>>>>>
>>>>>> we want to hoist 'invariant' but only from the inner loop even if it
>>>>>> is invariant also in the outer loop.
>>>>>
>>>>>
>>>>> For this case, theorotically I think the master GCC will optimize 
>>>>> it to:
>>>>>
>>>>>     invariant;
>>>>>     for (;;)
>>>>>       if (unlikely_cond)
>>>>>         for (;;)
>>>>>            ;
>>>>>
>>>>> 'invariant' is moved out of outer loop, but with the patch, it will 
>>>>> get:
>>>>>
>>>>>     for (;;)
>>>>>       if (unlikely_cond)
>>>>>         {
>>>>>           invariant;
>>>>>           for (;;)
>>>>>              ;
>>>>>         }
>>>>>
>>>>> 'invariant' is *cold* for outer loop, but it is still *hot* for 
>>>>> inner loop,
>>>>> so hoist it out of inner loop, this is exactly what we want, right?
>>>>
>>>> Yes.  I had doubts your patch would achieve that.
>>>>
>>>
>>>
>>> The below updated patch could achieve it:
>>>
>>>
>>> There was a patch trying to avoid move cold block out of loop:
>>>
>>> https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html
>>>
>>> Richard suggested to "never hoist anything from a bb with lower 
>>> execution
>>> frequency to a bb with higher one in LIM invariantness_dom_walker
>>> before_dom_children".
>>>
>>> In gimple LIM analysis,  add find_coldest_out_loop to move invariants to
>>> expected target loop, then  if profile count of the loop bb is colder
>>> than target loop preheader, it won't be hoisted out of loop.
>>>
>>> SPEC2017 performance evaluation shows 1% performance improvement for
>>> intrate GEOMEAN and no obvious regression for others.  Especially,
>>> 500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
>>> largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
>>> on P8LE.
>>>
>>> Regression and bootstrap tested pass on P8LE, any comments?  Thanks.
>>
>> Can you split the RTL and GIMPLE changes and measure them separately
>> please?
> 
> I did that before and got below data, it is slightly different due to
> using ratio instead of seconds, 500.perlbench_r obviously benefits
> from the RTL part change, while gimple part only improves exchange2
> and blender, with a regression on nab which requires the fix of loop
> split,
> 
> https://gcc.gnu.org/pipermail/gcc-patches/2021-August/576566.html
> 
> Reason is lim2 doesn't hoist code out of loop, but loop split generates
> duplicated loop with incorrect profile count on preheader and header bb,
> then later lim4 hoists code out of loop to unexpected place.  Same loop
> shows mismatch behavior in lim2 and lim4.  With that patch, the regression
> is gone.
> 
> 
>                Gimple+RTL |  Gimple lim | RTL loop-invariant
> 500.perlbench_r     8.03%    0.67%    7.69%
> 502.gcc_r           0.56%    0.37%    0.19%
> 505.mcf_r           0.19%   -0.19%    0.39%
> 520.omnetpp_r       0.83%    0.83%    0.83%
> 523.xalancbmk_r    -0.78%    0.00%   -1.04%
> 525.x264_r          0.17%    0.00%    0.00%
> 531.deepsjeng_r     0.00%    0.31%    0.00%
> 541.leela_r         0.00%   -0.31%    0.31%
> 548.exchange2_r     2.08%    1.85%    0.23%
> 557.xz_r            0.97%    0.00%    0.65%
> 503.bwaves_r       -0.12%    0.00%   -0.23%
> 507.cactuBSSN_r     0.00%    0.14%    0.00%
> 508.namd_r          0.00%    0.00%    0.00%
> 510.parest_r       -0.16%   -0.65%    0.00%
> 511.povray_r        0.30%    0.91%    0.91%
> 519.lbm_r           0.15%    0.00%    0.00%
> 521.wrf_r           0.00%    0.00%   -0.80%
> 526.blender_r       1.84%    0.26%    0.52%
> 527.cam4_r          0.28%    0.00%    0.00%
> 538.imagick_r       0.20%    0.00%    0.00%
> 544.nab_r          -1.55%   -0.78%    0.00%
> 549.fotonik3d_r    -0.25%    0.00%    0.00%
> 554.roms_r         -0.84%    0.00%   -0.63%
> INT GEAMEAN         1.16%    0.35%    0.90%
> FLOAT GEOMEAN      -0.01%   -0.01%   -0.02%
> GEOMEAN             0.50%    0.15%    0.38%


BTW, feedback from other platform:


      I do see ~8% performance improvement for 500.perlbench on aarch64.


> 
> 
> Will address other comments in later reply.  Thanks.
> 
> 

-- 
Thanks,
Xionghu

Re: [RFC] Don't move cold code out of loop by checking bb count

[Xionghu Luo]

Update the patch to v3, not sure whether you prefer the paste style
and continue to link the previous thread as Segher dislikes this...


[PATCH v3] Don't move cold code out of loop by checking bb count


Changes:
1. Handle max_loop in determine_max_movement instead of
outermost_invariant_loop.
2. Remove unnecessary changes.
3. Add for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body) in can_sm_ref_p.
4. "gsi_next (&bsi);" in move_computations_worker is kept since it caused
infinite loop when implementing v1 and the iteration is missed to be
updated actually.

v1: https://gcc.gnu.org/pipermail/gcc-patches/2021-August/576488.html
v2: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/579086.html

There was a patch trying to avoid move cold block out of loop:

https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html

Richard suggested to "never hoist anything from a bb with lower execution
frequency to a bb with higher one in LIM invariantness_dom_walker
before_dom_children".

In gimple LIM analysis, add find_coldest_out_loop to move invariants to
expected target loop, if profile count of the loop bb is colder
than target loop preheader, it won't be hoisted out of loop.
Likely for store motion, if all locations of the REF in loop is cold,
don't do store motion of it.

SPEC2017 performance evaluation shows 1% performance improvement for
intrate GEOMEAN and no obvious regression for others.  Especially,
500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
on P8LE.

gcc/ChangeLog:

	* loop-invariant.c (find_invariants_bb): Check profile count
	before motion.
	(find_invariants_body): Add argument.
	* tree-ssa-loop-im.c (find_coldest_out_loop): New function.
	(determine_max_movement): Use find_coldest_out_loop.
	(move_computations_worker): Adjust and fix iteration udpate.
	(execute_sm_exit): Check pointer validness.
	(class ref_in_loop_hot_body): New functor.
	(ref_in_loop_hot_body::operator): New.
	(can_sm_ref_p): Use for_all_locs_in_loop.

gcc/testsuite/ChangeLog:

	* gcc.dg/tree-ssa/recip-3.c: Adjust.
	* gcc.dg/tree-ssa/ssa-lim-18.c: New test.
	* gcc.dg/tree-ssa/ssa-lim-19.c: New test.
	* gcc.dg/tree-ssa/ssa-lim-20.c: New test.
---
 gcc/loop-invariant.c                       | 10 ++--
 gcc/tree-ssa-loop-im.c                     | 61 ++++++++++++++++++++--
 gcc/testsuite/gcc.dg/tree-ssa/recip-3.c    |  2 +-
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c | 20 +++++++
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c | 27 ++++++++++
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c | 25 +++++++++
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c | 28 ++++++++++
 7 files changed, 165 insertions(+), 8 deletions(-)
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c

diff --git a/gcc/loop-invariant.c b/gcc/loop-invariant.c
index fca0c2b24be..5c3be7bf0eb 100644
--- a/gcc/loop-invariant.c
+++ b/gcc/loop-invariant.c
@@ -1183,9 +1183,14 @@ find_invariants_insn (rtx_insn *insn, bool always_reached, bool always_executed)
    call.  */
 
 static void
-find_invariants_bb (basic_block bb, bool always_reached, bool always_executed)
+find_invariants_bb (class loop *loop, basic_block bb, bool always_reached,
+		    bool always_executed)
 {
   rtx_insn *insn;
+  basic_block preheader = loop_preheader_edge (loop)->src;
+
+  if (preheader->count > bb->count)
+    return;
 
   FOR_BB_INSNS (bb, insn)
     {
@@ -1214,8 +1219,7 @@ find_invariants_body (class loop *loop, basic_block *body,
   unsigned i;
 
   for (i = 0; i < loop->num_nodes; i++)
-    find_invariants_bb (body[i],
-			bitmap_bit_p (always_reached, i),
+    find_invariants_bb (loop, body[i], bitmap_bit_p (always_reached, i),
 			bitmap_bit_p (always_executed, i));
 }
 
diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
index 4b187c2cdaf..655fab03442 100644
--- a/gcc/tree-ssa-loop-im.c
+++ b/gcc/tree-ssa-loop-im.c
@@ -417,6 +417,28 @@ movement_possibility (gimple *stmt)
   return ret;
 }
 
+/* Find coldest loop between outmost_loop and loop by comapring profile count.  */
+
+static class loop *
+find_coldest_out_loop (class loop *outmost_loop, class loop *loop,
+		       basic_block curr_bb)
+{
+  class loop *cold_loop, *min_loop;
+  cold_loop = min_loop = outmost_loop;
+  profile_count min_count = loop_preheader_edge (min_loop)->src->count;
+
+  if (curr_bb && curr_bb->count < loop_preheader_edge (loop)->src->count)
+    return NULL;
+
+  while (min_loop != loop)
+    {
+      min_loop = superloop_at_depth (loop, loop_depth (min_loop) + 1);
+      if (loop_preheader_edge (min_loop)->src->count < min_count)
+	cold_loop = min_loop;
+    }
+  return cold_loop;
+}
+
 /* Suppose that operand DEF is used inside the LOOP.  Returns the outermost
    loop to that we could move the expression using DEF if it did not have
    other operands, i.e. the outermost loop enclosing LOOP in that the value
@@ -685,7 +707,9 @@ determine_max_movement (gimple *stmt, bool must_preserve_exec)
     level = ALWAYS_EXECUTED_IN (bb);
   else
     level = superloop_at_depth (loop, 1);
-  lim_data->max_loop = level;
+  lim_data->max_loop = find_coldest_out_loop (level, loop, bb);
+  if (!lim_data->max_loop)
+    return false;
 
   if (gphi *phi = dyn_cast <gphi *> (stmt))
     {
@@ -1198,7 +1222,6 @@ move_computations_worker (basic_block bb)
   for (gimple_stmt_iterator bsi = gsi_start_bb (bb); !gsi_end_p (bsi); )
     {
       edge e;
-
       gimple *stmt = gsi_stmt (bsi);
 
       lim_data = get_lim_data (stmt);
@@ -1221,7 +1244,10 @@ move_computations_worker (basic_block bb)
       /* We do not really want to move conditionals out of the loop; we just
 	 placed it here to force its operands to be moved if necessary.  */
       if (gimple_code (stmt) == GIMPLE_COND)
-	continue;
+	{
+	  gsi_next (&bsi);
+	  continue;
+	}
 
       if (dump_file && (dump_flags & TDF_DETAILS))
 	{
@@ -2241,7 +2267,12 @@ execute_sm_exit (class loop *loop, edge ex, vec<seq_entry> &seq,
 	}
       else
 	{
-	  sm_aux *aux = *aux_map.get (ref);
+	  sm_aux **paux = aux_map.get (ref);
+	  sm_aux *aux;
+	  if (paux)
+	    aux = *paux;
+	  else
+	    continue;
 	  if (!aux->store_flag || kind == sm_ord)
 	    {
 	      gassign *store;
@@ -2887,6 +2918,25 @@ ref_indep_loop_p (class loop *loop, im_mem_ref *ref, dep_kind kind)
   return indep_p;
 }
 
+class ref_in_loop_hot_body
+{
+public:
+  ref_in_loop_hot_body (loop *loop_) : l (loop_) {}
+  bool operator () (mem_ref_loc *loc);
+  class loop *l;
+};
+
+bool
+ref_in_loop_hot_body::operator () (mem_ref_loc *loc)
+{
+  basic_block curr_bb = gimple_bb (loc->stmt);
+  edge e = loop_preheader_edge (l);
+  if (e->src->count > curr_bb->count)
+    return false;
+  else
+    return true;
+}
+
 
 /* Returns true if we can perform store motion of REF from LOOP.  */
 
@@ -2941,6 +2991,9 @@ can_sm_ref_p (class loop *loop, im_mem_ref *ref)
   if (!ref_indep_loop_p (loop, ref, sm_war))
     return false;
 
+  if (!for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body (loop)))
+    return false;
+
   return true;
 }
 
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
index 638bf38db8c..641c91e719e 100644
--- a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
+++ b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
@@ -23,4 +23,4 @@ float h ()
 	F[0] += E / d;
 }
 
-/* { dg-final { scan-tree-dump-times " / " 1 "recip" } } */
+/* { dg-final { scan-tree-dump-times " / " 5 "recip" } } */
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
new file mode 100644
index 00000000000..7326a230b3f
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
@@ -0,0 +1,20 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+volatile int x;
+void
+bar (int, char *, char *);
+void
+foo (int *a, int n, int k)
+{
+  int i;
+
+  for (i = 0; i < n; i++)
+    {
+      if (__builtin_expect (x, 0))
+	bar (k / 5, "one", "two");
+      a[i] = k;
+    }
+}
+
+/* { dg-final { scan-tree-dump-not "out of loop 1" "lim2" } } */
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
new file mode 100644
index 00000000000..f0a99fa42b4
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
@@ -0,0 +1,27 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+volatile int x;
+void
+bar (int, char *, char *);
+void
+foo (int *a, int n, int m, int k, int s)
+{
+  int i;
+  int j;
+
+  for (i = 0; i < m; i++)
+    {
+      if (__builtin_expect (x, 0))
+	for (j = 0; j < n; j++)
+	  {
+	    bar (k / 5, "one", "two");
+	  a[s] = k;
+	}
+      a[s] = s;
+    }
+}
+
+/* { dg-final { scan-tree-dump-times "out of loop 2" 4 "lim2" } } */
+/* { dg-final { scan-tree-dump-times "out of loop 1" 3 "lim2" } } */
+
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
new file mode 100644
index 00000000000..bc60a040a70
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
@@ -0,0 +1,25 @@
+/* { dg-do compile  } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+/* Test that `count' is not hoisted out of loop when bb is cold.  */
+
+int count;
+volatile int x;
+
+struct obj {
+  int data;
+  struct obj *next;
+
+} *q;
+
+void
+func (int m)
+{
+  struct obj *p;
+  for (int i = 0; i < m; i++)
+    if (__builtin_expect (x, 0))
+      count++;
+
+}
+
+/* { dg-final { scan-tree-dump-not "Executing store motion of" "lim2"  }  } */
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
new file mode 100644
index 00000000000..fedaa3b7119
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
@@ -0,0 +1,28 @@
+/* { dg-do compile  } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+/* Test that `count' is hoisted out of loop when one of it's used bb is hot. */
+
+int count;
+volatile int x;
+
+struct obj {
+  int data;
+  struct obj *next;
+
+} *q;
+
+void
+func (int m, int n)
+{
+  struct obj *p;
+  for (int i = 0; i < m; i++)
+  {
+    if (__builtin_expect (x, 0))
+      count++;
+    count += n;
+  }
+}
+
+/* { dg-final { scan-tree-dump-times "Executing store motion of" 1 "lim2"  }  } */
+
-- 
2.27.0.90.geebb51ba8c

Re: [RFC] Don't move cold code out of loop by checking bb count

[Richard Biener]

On Fri, Sep 24, 2021 at 8:29 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>
> Update the patch to v3, not sure whether you prefer the paste style
> and continue to link the previous thread as Segher dislikes this...
>
>
> [PATCH v3] Don't move cold code out of loop by checking bb count
>
>
> Changes:
> 1. Handle max_loop in determine_max_movement instead of
> outermost_invariant_loop.
> 2. Remove unnecessary changes.
> 3. Add for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body) in can_sm_ref_p.
> 4. "gsi_next (&bsi);" in move_computations_worker is kept since it caused
> infinite loop when implementing v1 and the iteration is missed to be
> updated actually.
>
> v1: https://gcc.gnu.org/pipermail/gcc-patches/2021-August/576488.html
> v2: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/579086.html
>
> There was a patch trying to avoid move cold block out of loop:
>
> https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html
>
> Richard suggested to "never hoist anything from a bb with lower execution
> frequency to a bb with higher one in LIM invariantness_dom_walker
> before_dom_children".
>
> In gimple LIM analysis, add find_coldest_out_loop to move invariants to
> expected target loop, if profile count of the loop bb is colder
> than target loop preheader, it won't be hoisted out of loop.
> Likely for store motion, if all locations of the REF in loop is cold,
> don't do store motion of it.
>
> SPEC2017 performance evaluation shows 1% performance improvement for
> intrate GEOMEAN and no obvious regression for others.  Especially,
> 500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
> largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
> on P8LE.
>
> gcc/ChangeLog:
>
>         * loop-invariant.c (find_invariants_bb): Check profile count
>         before motion.
>         (find_invariants_body): Add argument.
>         * tree-ssa-loop-im.c (find_coldest_out_loop): New function.
>         (determine_max_movement): Use find_coldest_out_loop.
>         (move_computations_worker): Adjust and fix iteration udpate.
>         (execute_sm_exit): Check pointer validness.
>         (class ref_in_loop_hot_body): New functor.
>         (ref_in_loop_hot_body::operator): New.
>         (can_sm_ref_p): Use for_all_locs_in_loop.
>
> gcc/testsuite/ChangeLog:
>
>         * gcc.dg/tree-ssa/recip-3.c: Adjust.
>         * gcc.dg/tree-ssa/ssa-lim-18.c: New test.
>         * gcc.dg/tree-ssa/ssa-lim-19.c: New test.
>         * gcc.dg/tree-ssa/ssa-lim-20.c: New test.
> ---
>  gcc/loop-invariant.c                       | 10 ++--
>  gcc/tree-ssa-loop-im.c                     | 61 ++++++++++++++++++++--
>  gcc/testsuite/gcc.dg/tree-ssa/recip-3.c    |  2 +-
>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c | 20 +++++++
>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c | 27 ++++++++++
>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c | 25 +++++++++
>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c | 28 ++++++++++
>  7 files changed, 165 insertions(+), 8 deletions(-)
>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
>
> diff --git a/gcc/loop-invariant.c b/gcc/loop-invariant.c
> index fca0c2b24be..5c3be7bf0eb 100644
> --- a/gcc/loop-invariant.c
> +++ b/gcc/loop-invariant.c
> @@ -1183,9 +1183,14 @@ find_invariants_insn (rtx_insn *insn, bool always_reached, bool always_executed)
>     call.  */
>
>  static void
> -find_invariants_bb (basic_block bb, bool always_reached, bool always_executed)
> +find_invariants_bb (class loop *loop, basic_block bb, bool always_reached,
> +                   bool always_executed)
>  {
>    rtx_insn *insn;
> +  basic_block preheader = loop_preheader_edge (loop)->src;
> +
> +  if (preheader->count > bb->count)
> +    return;
>
>    FOR_BB_INSNS (bb, insn)
>      {
> @@ -1214,8 +1219,7 @@ find_invariants_body (class loop *loop, basic_block *body,
>    unsigned i;
>
>    for (i = 0; i < loop->num_nodes; i++)
> -    find_invariants_bb (body[i],
> -                       bitmap_bit_p (always_reached, i),
> +    find_invariants_bb (loop, body[i], bitmap_bit_p (always_reached, i),
>                         bitmap_bit_p (always_executed, i));
>  }
>
> diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
> index 4b187c2cdaf..655fab03442 100644
> --- a/gcc/tree-ssa-loop-im.c
> +++ b/gcc/tree-ssa-loop-im.c
> @@ -417,6 +417,28 @@ movement_possibility (gimple *stmt)
>    return ret;
>  }
>
> +/* Find coldest loop between outmost_loop and loop by comapring profile count.  */
> +
> +static class loop *
> +find_coldest_out_loop (class loop *outmost_loop, class loop *loop,
> +                      basic_block curr_bb)
> +{
> +  class loop *cold_loop, *min_loop;
> +  cold_loop = min_loop = outmost_loop;
> +  profile_count min_count = loop_preheader_edge (min_loop)->src->count;
> +
> +  if (curr_bb && curr_bb->count < loop_preheader_edge (loop)->src->count)

Honza - can you comment on whether we should compare BB counts this way?

I would suspect that for, say,

  for (...)
     if (a)
       X;
     else
       Y;

that the counts for X and Y will be less than that of the preheader of the loop
only when the loop is estimated to run once.  That is, should we really compare
the to the preheader count or maybe better to the _header_ count which
would keep the number of iterations out of the equation?

If we look at maybe_hot_count_p that's a quite sophisticated thing to
compare a count to the "IPA hot", here we're comparing two counts
within a function where it actually matters whether we use a<b or
!(a>=b) since 'unordered' is mapped to false (but there's no ordered_p
function).

Xionghu, you error on the side of not hoisting for unordered counts here

> +    return NULL;
> +
> +  while (min_loop != loop)
> +    {
> +      min_loop = superloop_at_depth (loop, loop_depth (min_loop) + 1);
> +      if (loop_preheader_edge (min_loop)->src->count < min_count)

but in the other direction here and on the side of not hoisting
in ref_in_loop_hot_body.

The three-state relational operator overloads are probably not the
very best idea...
(see profile-count.h for them)

> +       cold_loop = min_loop;
> +    }
> +  return cold_loop;
> +}
> +
>  /* Suppose that operand DEF is used inside the LOOP.  Returns the outermost
>     loop to that we could move the expression using DEF if it did not have
>     other operands, i.e. the outermost loop enclosing LOOP in that the value
> @@ -685,7 +707,9 @@ determine_max_movement (gimple *stmt, bool must_preserve_exec)
>      level = ALWAYS_EXECUTED_IN (bb);
>    else
>      level = superloop_at_depth (loop, 1);
> -  lim_data->max_loop = level;
> +  lim_data->max_loop = find_coldest_out_loop (level, loop, bb);
> +  if (!lim_data->max_loop)
> +    return false;
>
>    if (gphi *phi = dyn_cast <gphi *> (stmt))
>      {
> @@ -1198,7 +1222,6 @@ move_computations_worker (basic_block bb)
>    for (gimple_stmt_iterator bsi = gsi_start_bb (bb); !gsi_end_p (bsi); )
>      {
>        edge e;
> -
>        gimple *stmt = gsi_stmt (bsi);
>
>        lim_data = get_lim_data (stmt);
> @@ -1221,7 +1244,10 @@ move_computations_worker (basic_block bb)
>        /* We do not really want to move conditionals out of the loop; we just
>          placed it here to force its operands to be moved if necessary.  */
>        if (gimple_code (stmt) == GIMPLE_COND)
> -       continue;
> +       {
> +         gsi_next (&bsi);
> +         continue;
> +       }
>
>        if (dump_file && (dump_flags & TDF_DETAILS))
>         {
> @@ -2241,7 +2267,12 @@ execute_sm_exit (class loop *loop, edge ex, vec<seq_entry> &seq,
>         }
>        else
>         {
> -         sm_aux *aux = *aux_map.get (ref);
> +         sm_aux **paux = aux_map.get (ref);
> +         sm_aux *aux;
> +         if (paux)
> +           aux = *paux;
> +         else
> +           continue;

do you really need this?  I doubt so.

>           if (!aux->store_flag || kind == sm_ord)
>             {
>               gassign *store;
> @@ -2887,6 +2918,25 @@ ref_indep_loop_p (class loop *loop, im_mem_ref *ref, dep_kind kind)
>    return indep_p;
>  }
>
> +class ref_in_loop_hot_body
> +{
> +public:
> +  ref_in_loop_hot_body (loop *loop_) : l (loop_) {}
> +  bool operator () (mem_ref_loc *loc);
> +  class loop *l;
> +};
> +
> +bool
> +ref_in_loop_hot_body::operator () (mem_ref_loc *loc)
> +{
> +  basic_block curr_bb = gimple_bb (loc->stmt);
> +  edge e = loop_preheader_edge (l);
> +  if (e->src->count > curr_bb->count)
> +    return false;
> +  else
> +    return true;
> +}
> +
>
>  /* Returns true if we can perform store motion of REF from LOOP.  */
>
> @@ -2941,6 +2991,9 @@ can_sm_ref_p (class loop *loop, im_mem_ref *ref)
>    if (!ref_indep_loop_p (loop, ref, sm_war))
>      return false;
>

Add a comment here what this is about.

Otherwise the GIMPLE invariant motion parts look sensible, but I'd
really like to have
the issue on the profile_count API sorted out.

Can you split out the RTL invariant motion part to a separate patch please?

Thanks,
Richard.

> +  if (!for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body (loop)))
> +    return false;
> +
>    return true;
>  }
>
> diff --git a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
> index 638bf38db8c..641c91e719e 100644
> --- a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
> +++ b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
> @@ -23,4 +23,4 @@ float h ()
>         F[0] += E / d;
>  }
>
> -/* { dg-final { scan-tree-dump-times " / " 1 "recip" } } */
> +/* { dg-final { scan-tree-dump-times " / " 5 "recip" } } */
> diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
> new file mode 100644
> index 00000000000..7326a230b3f
> --- /dev/null
> +++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
> @@ -0,0 +1,20 @@
> +/* { dg-do compile } */
> +/* { dg-options "-O2 -fdump-tree-lim2-details" } */
> +
> +volatile int x;
> +void
> +bar (int, char *, char *);
> +void
> +foo (int *a, int n, int k)
> +{
> +  int i;
> +
> +  for (i = 0; i < n; i++)
> +    {
> +      if (__builtin_expect (x, 0))
> +       bar (k / 5, "one", "two");
> +      a[i] = k;
> +    }
> +}
> +
> +/* { dg-final { scan-tree-dump-not "out of loop 1" "lim2" } } */
> diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
> new file mode 100644
> index 00000000000..f0a99fa42b4
> --- /dev/null
> +++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
> @@ -0,0 +1,27 @@
> +/* { dg-do compile } */
> +/* { dg-options "-O2 -fdump-tree-lim2-details" } */
> +
> +volatile int x;
> +void
> +bar (int, char *, char *);
> +void
> +foo (int *a, int n, int m, int k, int s)
> +{
> +  int i;
> +  int j;
> +
> +  for (i = 0; i < m; i++)
> +    {
> +      if (__builtin_expect (x, 0))
> +       for (j = 0; j < n; j++)
> +         {
> +           bar (k / 5, "one", "two");
> +         a[s] = k;
> +       }
> +      a[s] = s;
> +    }
> +}
> +
> +/* { dg-final { scan-tree-dump-times "out of loop 2" 4 "lim2" } } */
> +/* { dg-final { scan-tree-dump-times "out of loop 1" 3 "lim2" } } */
> +
> diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
> new file mode 100644
> index 00000000000..bc60a040a70
> --- /dev/null
> +++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
> @@ -0,0 +1,25 @@
> +/* { dg-do compile  } */
> +/* { dg-options "-O2 -fdump-tree-lim2-details" } */
> +
> +/* Test that `count' is not hoisted out of loop when bb is cold.  */
> +
> +int count;
> +volatile int x;
> +
> +struct obj {
> +  int data;
> +  struct obj *next;
> +
> +} *q;
> +
> +void
> +func (int m)
> +{
> +  struct obj *p;
> +  for (int i = 0; i < m; i++)
> +    if (__builtin_expect (x, 0))
> +      count++;
> +
> +}
> +
> +/* { dg-final { scan-tree-dump-not "Executing store motion of" "lim2"  }  } */
> diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
> new file mode 100644
> index 00000000000..fedaa3b7119
> --- /dev/null
> +++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
> @@ -0,0 +1,28 @@
> +/* { dg-do compile  } */
> +/* { dg-options "-O2 -fdump-tree-lim2-details" } */
> +
> +/* Test that `count' is hoisted out of loop when one of it's used bb is hot. */
> +
> +int count;
> +volatile int x;
> +
> +struct obj {
> +  int data;
> +  struct obj *next;
> +
> +} *q;
> +
> +void
> +func (int m, int n)
> +{
> +  struct obj *p;
> +  for (int i = 0; i < m; i++)
> +  {
> +    if (__builtin_expect (x, 0))
> +      count++;
> +    count += n;
> +  }
> +}
> +
> +/* { dg-final { scan-tree-dump-times "Executing store motion of" 1 "lim2"  }  } */
> +
> --
> 2.27.0.90.geebb51ba8c
>
>

Re: [RFC] Don't move cold code out of loop by checking bb count

[Xionghu Luo]

[-- Attachment #1: Type: text/plain, Size: 10321 bytes --]

Hi,

On 2021/9/28 20:09, Richard Biener wrote:
> On Fri, Sep 24, 2021 at 8:29 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>>
>> Update the patch to v3, not sure whether you prefer the paste style
>> and continue to link the previous thread as Segher dislikes this...
>>
>>
>> [PATCH v3] Don't move cold code out of loop by checking bb count
>>
>>
>> Changes:
>> 1. Handle max_loop in determine_max_movement instead of
>> outermost_invariant_loop.
>> 2. Remove unnecessary changes.
>> 3. Add for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body) in can_sm_ref_p.
>> 4. "gsi_next (&bsi);" in move_computations_worker is kept since it caused
>> infinite loop when implementing v1 and the iteration is missed to be
>> updated actually.
>>
>> v1: https://gcc.gnu.org/pipermail/gcc-patches/2021-August/576488.html
>> v2: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/579086.html
>>
>> There was a patch trying to avoid move cold block out of loop:
>>
>> https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html
>>
>> Richard suggested to "never hoist anything from a bb with lower execution
>> frequency to a bb with higher one in LIM invariantness_dom_walker
>> before_dom_children".
>>
>> In gimple LIM analysis, add find_coldest_out_loop to move invariants to
>> expected target loop, if profile count of the loop bb is colder
>> than target loop preheader, it won't be hoisted out of loop.
>> Likely for store motion, if all locations of the REF in loop is cold,
>> don't do store motion of it.
>>
>> SPEC2017 performance evaluation shows 1% performance improvement for
>> intrate GEOMEAN and no obvious regression for others.  Especially,
>> 500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
>> largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
>> on P8LE.
>>
>> gcc/ChangeLog:
>>
>>         * loop-invariant.c (find_invariants_bb): Check profile count
>>         before motion.
>>         (find_invariants_body): Add argument.
>>         * tree-ssa-loop-im.c (find_coldest_out_loop): New function.
>>         (determine_max_movement): Use find_coldest_out_loop.
>>         (move_computations_worker): Adjust and fix iteration udpate.
>>         (execute_sm_exit): Check pointer validness.
>>         (class ref_in_loop_hot_body): New functor.
>>         (ref_in_loop_hot_body::operator): New.
>>         (can_sm_ref_p): Use for_all_locs_in_loop.
>>
>> gcc/testsuite/ChangeLog:
>>
>>         * gcc.dg/tree-ssa/recip-3.c: Adjust.
>>         * gcc.dg/tree-ssa/ssa-lim-18.c: New test.
>>         * gcc.dg/tree-ssa/ssa-lim-19.c: New test.
>>         * gcc.dg/tree-ssa/ssa-lim-20.c: New test.
>> ---
>>  gcc/loop-invariant.c                       | 10 ++--
>>  gcc/tree-ssa-loop-im.c                     | 61 ++++++++++++++++++++--
>>  gcc/testsuite/gcc.dg/tree-ssa/recip-3.c    |  2 +-
>>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c | 20 +++++++
>>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c | 27 ++++++++++
>>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c | 25 +++++++++
>>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c | 28 ++++++++++
>>  7 files changed, 165 insertions(+), 8 deletions(-)
>>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
>>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
>>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
>>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
>>
>> diff --git a/gcc/loop-invariant.c b/gcc/loop-invariant.c
>> index fca0c2b24be..5c3be7bf0eb 100644
>> --- a/gcc/loop-invariant.c
>> +++ b/gcc/loop-invariant.c
>> @@ -1183,9 +1183,14 @@ find_invariants_insn (rtx_insn *insn, bool always_reached, bool always_executed)
>>     call.  */
>>
>>  static void
>> -find_invariants_bb (basic_block bb, bool always_reached, bool always_executed)
>> +find_invariants_bb (class loop *loop, basic_block bb, bool always_reached,
>> +                   bool always_executed)
>>  {
>>    rtx_insn *insn;
>> +  basic_block preheader = loop_preheader_edge (loop)->src;
>> +
>> +  if (preheader->count > bb->count)
>> +    return;
>>
>>    FOR_BB_INSNS (bb, insn)
>>      {
>> @@ -1214,8 +1219,7 @@ find_invariants_body (class loop *loop, basic_block *body,
>>    unsigned i;
>>
>>    for (i = 0; i < loop->num_nodes; i++)
>> -    find_invariants_bb (body[i],
>> -                       bitmap_bit_p (always_reached, i),
>> +    find_invariants_bb (loop, body[i], bitmap_bit_p (always_reached, i),
>>                         bitmap_bit_p (always_executed, i));
>>  }
>>
>> diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
>> index 4b187c2cdaf..655fab03442 100644
>> --- a/gcc/tree-ssa-loop-im.c
>> +++ b/gcc/tree-ssa-loop-im.c
>> @@ -417,6 +417,28 @@ movement_possibility (gimple *stmt)
>>    return ret;
>>  }
>>
>> +/* Find coldest loop between outmost_loop and loop by comapring profile count.  */
>> +
>> +static class loop *
>> +find_coldest_out_loop (class loop *outmost_loop, class loop *loop,
>> +                      basic_block curr_bb)
>> +{
>> +  class loop *cold_loop, *min_loop;
>> +  cold_loop = min_loop = outmost_loop;
>> +  profile_count min_count = loop_preheader_edge (min_loop)->src->count;
>> +
>> +  if (curr_bb && curr_bb->count < loop_preheader_edge (loop)->src->count)
> 
> Honza - can you comment on whether we should compare BB counts this way?
> 
> I would suspect that for, say,
> 
>   for (...)
>      if (a)
>        X;
>      else
>        Y;
> 
> that the counts for X and Y will be less than that of the preheader of the loop
> only when the loop is estimated to run once.  That is, should we really compare
> the to the preheader count or maybe better to the _header_ count which
> would keep the number of iterations out of the equation?

I quickly tried to replace all the loop_preheader_edge (loop)->src with
loop_preheader_edge (loop)->dest, it will cause many failures in
gcc.dg/tree-ssa/ssa-lim-*.c, I didn't go deep to investigate, but it seems
reasonable to compare the bb count with preheader count as both gimple lim
and RTL loop-invariant move instructions to *preheader* instead of *header*
after analysis?

> 
> If we look at maybe_hot_count_p that's a quite sophisticated thing to
> compare a count to the "IPA hot", here we're comparing two counts
> within a function where it actually matters whether we use a<b or
> !(a>=b) since 'unordered' is mapped to false (but there's no ordered_p
> function).
> 
> Xionghu, you error on the side of not hoisting for unordered counts here
> 
>> +    return NULL;
>> +
>> +  while (min_loop != loop)
>> +    {
>> +      min_loop = superloop_at_depth (loop, loop_depth (min_loop) + 1);
>> +      if (loop_preheader_edge (min_loop)->src->count < min_count)
> 
> but in the other direction here and on the side of not hoisting
> in ref_in_loop_hot_body.
> 
> The three-state relational operator overloads are probably not the
> very best idea...
> (see profile-count.h for them)
> 
Added new function bb_colder_than_loop_preheader to encapsulate the comparision,
if FALSE is returned due to three-state inequality,  find_coldest_out_loop
will return the original input to lim->max_loop, and ref_in_loop_hot_body::operator ()
will return true to continue perform store motion, both preserve the previous
behavior.


>> +       cold_loop = min_loop;
>> +    }
>> +  return cold_loop;
>> +}
>> +
>>  /* Suppose that operand DEF is used inside the LOOP.  Returns the outermost
>>     loop to that we could move the expression using DEF if it did not have
>>     other operands, i.e. the outermost loop enclosing LOOP in that the value
>> @@ -685,7 +707,9 @@ determine_max_movement (gimple *stmt, bool must_preserve_exec)
>>      level = ALWAYS_EXECUTED_IN (bb);
>>    else
>>      level = superloop_at_depth (loop, 1);
>> -  lim_data->max_loop = level;
>> +  lim_data->max_loop = find_coldest_out_loop (level, loop, bb);
>> +  if (!lim_data->max_loop)
>> +    return false;
>>
>>    if (gphi *phi = dyn_cast <gphi *> (stmt))
>>      {
>> @@ -1198,7 +1222,6 @@ move_computations_worker (basic_block bb)
>>    for (gimple_stmt_iterator bsi = gsi_start_bb (bb); !gsi_end_p (bsi); )
>>      {
>>        edge e;
>> -
>>        gimple *stmt = gsi_stmt (bsi);
>>
>>        lim_data = get_lim_data (stmt);
>> @@ -1221,7 +1244,10 @@ move_computations_worker (basic_block bb)
>>        /* We do not really want to move conditionals out of the loop; we just
>>          placed it here to force its operands to be moved if necessary.  */
>>        if (gimple_code (stmt) == GIMPLE_COND)
>> -       continue;
>> +       {
>> +         gsi_next (&bsi);
>> +         continue;
>> +       }
>>
>>        if (dump_file && (dump_flags & TDF_DETAILS))
>>         {
>> @@ -2241,7 +2267,12 @@ execute_sm_exit (class loop *loop, edge ex, vec<seq_entry> &seq,
>>         }
>>        else
>>         {
>> -         sm_aux *aux = *aux_map.get (ref);
>> +         sm_aux **paux = aux_map.get (ref);
>> +         sm_aux *aux;
>> +         if (paux)
>> +           aux = *paux;
>> +         else
>> +           continue;
> 
> do you really need this?  I doubt so.

Removed.

> 
>>           if (!aux->store_flag || kind == sm_ord)
>>             {
>>               gassign *store;
>> @@ -2887,6 +2918,25 @@ ref_indep_loop_p (class loop *loop, im_mem_ref *ref, dep_kind kind)
>>    return indep_p;
>>  }
>>
>> +class ref_in_loop_hot_body
>> +{
>> +public:
>> +  ref_in_loop_hot_body (loop *loop_) : l (loop_) {}
>> +  bool operator () (mem_ref_loc *loc);
>> +  class loop *l;
>> +};
>> +
>> +bool
>> +ref_in_loop_hot_body::operator () (mem_ref_loc *loc)
>> +{
>> +  basic_block curr_bb = gimple_bb (loc->stmt);
>> +  edge e = loop_preheader_edge (l);
>> +  if (e->src->count > curr_bb->count)
>> +    return false;
>> +  else
>> +    return true;
>> +}
>> +
>>
>>  /* Returns true if we can perform store motion of REF from LOOP.  */
>>
>> @@ -2941,6 +2991,9 @@ can_sm_ref_p (class loop *loop, im_mem_ref *ref)
>>    if (!ref_indep_loop_p (loop, ref, sm_war))
>>      return false;
>>
> 
> Add a comment here what this is about.

Done.

> 
> Otherwise the GIMPLE invariant motion parts look sensible, but I'd
> really like to have
> the issue on the profile_count API sorted out.
> 
> Can you split out the RTL invariant motion part to a separate patch please?

Done.  Attached the two patches, thanks.


BR,
Xionghu

[-- Attachment #2: v4-0001-Don-t-move-cold-code-out-of-loop-by-checking-bb-c.patch --]
[-- Type: text/plain, Size: 10945 bytes --]

From 092d6df49c0027001c3ed9343f0d1e8c02232d95 Mon Sep 17 00:00:00 2001
From: Xiong Hu Luo <luoxhu@linux.ibm.com>
Date: Mon, 5 Jul 2021 03:57:11 -0500
Subject: [PATCH v4 1/2] Don't move cold code out of loop by checking bb count

v4 changes:
1. Sort out profile_count comparision to function bb_cold_than_loop_preheader.
2. Update ref_in_loop_hot_body::operator () to find cold_loop before compare.
3. Split RTL invariant motion part out.
4. Remove aux changes.

v3 changes:
1. Handle max_loop in determine_max_movement instead of outermost_invariant_loop.
2. Remove unnecessary changes.
3. Add for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body) in can_sm_ref_p.
4. "gsi_next (&bsi);" in move_computations_worker is kept since it caused
infinite loop when implementing v1 and the iteration is missed to be
updated actually.

v1: https://gcc.gnu.org/pipermail/gcc-patches/2021-August/576488.html
v2: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/579086.html
v3: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/580211.html

There was a patch trying to avoid move cold block out of loop:

https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html

Richard suggested to "never hoist anything from a bb with lower execution
frequency to a bb with higher one in LIM invariantness_dom_walker
before_dom_children".

In gimple LIM analysis, add find_coldest_out_loop to move invariants to
expected target loop, if profile count of the loop bb is colder
than target loop preheader, it won't be hoisted out of loop.
Likely for store motion, if all locations of the REF in loop is cold,
don't do store motion of it.

SPEC2017 performance evaluation shows 1% performance improvement for
intrate GEOMEAN and no obvious regression for others.  Especially,
500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
on P8LE.

gcc/ChangeLog:

	(find_invariants_body): Add argument.
	* tree-ssa-loop-im.c (bb_colder_than_loop_preheader): New
	function.
	(find_coldest_out_loop): New function.
	(determine_max_movement): Use find_coldest_out_loop.
	(move_computations_worker): Adjust and fix iteration udpate.
	(execute_sm_exit): Check pointer validness.
	(class ref_in_loop_hot_body): New functor.
	(ref_in_loop_hot_body::operator): New.
	(can_sm_ref_p): Use for_all_locs_in_loop.

gcc/testsuite/ChangeLog:

	* gcc.dg/tree-ssa/recip-3.c: Adjust.
	* gcc.dg/tree-ssa/ssa-lim-18.c: New test.
	* gcc.dg/tree-ssa/ssa-lim-19.c: New test.
	* gcc.dg/tree-ssa/ssa-lim-20.c: New test.
	* gcc.dg/tree-ssa/ssa-lim-21.c: New test.
---
 gcc/tree-ssa-loop-im.c                     | 85 +++++++++++++++++++++-
 gcc/testsuite/gcc.dg/tree-ssa/recip-3.c    |  2 +-
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c | 20 +++++
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c | 27 +++++++
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c | 25 +++++++
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c | 35 +++++++++
 6 files changed, 191 insertions(+), 3 deletions(-)
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c

diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
index 4b187c2cdaf..870e0a00512 100644
--- a/gcc/tree-ssa-loop-im.c
+++ b/gcc/tree-ssa-loop-im.c
@@ -417,6 +417,46 @@ movement_possibility (gimple *stmt)
   return ret;
 }
 
+/* Compare the profile count inequality of COUNT1 and COUNT2, it is three-state
+   as stated in profile-count.h, FALSE is returned if inequality cannot be
+   decided.  */
+bool bb_colder_than_loop_preheader (profile_count count1, profile_count count2)
+{
+  if (count1 < count2)
+    return true;
+  else
+    return false;
+}
+
+/* Find coldest loop between OUTMOST_LOOP and LOOP by comapring profile count.
+ */
+
+static class loop *
+find_coldest_out_loop (class loop *outmost_loop, class loop *loop,
+		       basic_block curr_bb)
+{
+  class loop *cold_loop, *min_loop;
+  cold_loop = min_loop = outmost_loop;
+  profile_count min_count = loop_preheader_edge (min_loop)->src->count;
+
+  /* If bb_colder_than_loop_preheader returns false due to three-state
+    comparision, OUTMOST_LOOP is returned finally to preserve the behavior.
+    Otherwise, return the coldest loop between OUTMOST_LOOP and LOOP.  */
+  if (curr_bb
+      && bb_colder_than_loop_preheader (curr_bb->count,
+					loop_preheader_edge (loop)->src->count))
+    return NULL;
+
+  while (min_loop != loop)
+    {
+      min_loop = superloop_at_depth (loop, loop_depth (min_loop) + 1);
+      if (bb_colder_than_loop_preheader (
+	    loop_preheader_edge (min_loop)->src->count, min_count))
+	cold_loop = min_loop;
+    }
+  return cold_loop;
+}
+
 /* Suppose that operand DEF is used inside the LOOP.  Returns the outermost
    loop to that we could move the expression using DEF if it did not have
    other operands, i.e. the outermost loop enclosing LOOP in that the value
@@ -685,7 +725,9 @@ determine_max_movement (gimple *stmt, bool must_preserve_exec)
     level = ALWAYS_EXECUTED_IN (bb);
   else
     level = superloop_at_depth (loop, 1);
-  lim_data->max_loop = level;
+  lim_data->max_loop = find_coldest_out_loop (level, loop, bb);
+  if (!lim_data->max_loop)
+    return false;
 
   if (gphi *phi = dyn_cast <gphi *> (stmt))
     {
@@ -1221,7 +1263,10 @@ move_computations_worker (basic_block bb)
       /* We do not really want to move conditionals out of the loop; we just
 	 placed it here to force its operands to be moved if necessary.  */
       if (gimple_code (stmt) == GIMPLE_COND)
-	continue;
+	{
+	  gsi_next (&bsi);
+	  continue;
+	}
 
       if (dump_file && (dump_flags & TDF_DETAILS))
 	{
@@ -2887,6 +2932,35 @@ ref_indep_loop_p (class loop *loop, im_mem_ref *ref, dep_kind kind)
   return indep_p;
 }
 
+class ref_in_loop_hot_body
+{
+public:
+  ref_in_loop_hot_body (loop *loop_) : l (loop_) {}
+  bool operator () (mem_ref_loc *loc);
+  class loop *l;
+};
+
+/* Find out the coldest loop between loop L and innermost loop, compare the
+   hotness between current BB and coldest loop preheader by profile count.  */
+bool
+ref_in_loop_hot_body::operator () (mem_ref_loc *loc)
+{
+  basic_block curr_bb = gimple_bb (loc->stmt);
+  class loop *inner_loop = curr_bb->loop_father;
+  class loop *cold_loop = l;
+  if (l != inner_loop)
+    cold_loop = find_coldest_out_loop (l, inner_loop, curr_bb);
+  if (!cold_loop)
+    return false;
+  edge e = loop_preheader_edge (cold_loop);
+  /*  If bb_colder_than_loop_preheader is false due to three-state inequality
+     comparision, TRUE is returned to continue perform store motion.  */
+  if (bb_colder_than_loop_preheader (curr_bb->count, e->src->count))
+    return false;
+  else
+    return true;
+}
+
 
 /* Returns true if we can perform store motion of REF from LOOP.  */
 
@@ -2941,6 +3015,13 @@ can_sm_ref_p (class loop *loop, im_mem_ref *ref)
   if (!ref_indep_loop_p (loop, ref, sm_war))
     return false;
 
+  /* Verify whether the candidate is hot for LOOP.  Only do store motion if the
+    candidate's profile count is hot.  Statement in cold BB shouldn't be moved
+    out of it's loop_father, also it shouldn't be moved out of LOOP if it is
+    colder than LOOP's preheader.  See ssa-lim-21.c.  */
+  if (!for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body (loop)))
+    return false;
+
   return true;
 }
 
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
index 638bf38db8c..641c91e719e 100644
--- a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
+++ b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
@@ -23,4 +23,4 @@ float h ()
 	F[0] += E / d;
 }
 
-/* { dg-final { scan-tree-dump-times " / " 1 "recip" } } */
+/* { dg-final { scan-tree-dump-times " / " 5 "recip" } } */
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
new file mode 100644
index 00000000000..7326a230b3f
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
@@ -0,0 +1,20 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+volatile int x;
+void
+bar (int, char *, char *);
+void
+foo (int *a, int n, int k)
+{
+  int i;
+
+  for (i = 0; i < n; i++)
+    {
+      if (__builtin_expect (x, 0))
+	bar (k / 5, "one", "two");
+      a[i] = k;
+    }
+}
+
+/* { dg-final { scan-tree-dump-not "out of loop 1" "lim2" } } */
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
new file mode 100644
index 00000000000..f0a99fa42b4
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
@@ -0,0 +1,27 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+volatile int x;
+void
+bar (int, char *, char *);
+void
+foo (int *a, int n, int m, int k, int s)
+{
+  int i;
+  int j;
+
+  for (i = 0; i < m; i++)
+    {
+      if (__builtin_expect (x, 0))
+	for (j = 0; j < n; j++)
+	  {
+	    bar (k / 5, "one", "two");
+	  a[s] = k;
+	}
+      a[s] = s;
+    }
+}
+
+/* { dg-final { scan-tree-dump-times "out of loop 2" 4 "lim2" } } */
+/* { dg-final { scan-tree-dump-times "out of loop 1" 3 "lim2" } } */
+
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
new file mode 100644
index 00000000000..bc60a040a70
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
@@ -0,0 +1,25 @@
+/* { dg-do compile  } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+/* Test that `count' is not hoisted out of loop when bb is cold.  */
+
+int count;
+volatile int x;
+
+struct obj {
+  int data;
+  struct obj *next;
+
+} *q;
+
+void
+func (int m)
+{
+  struct obj *p;
+  for (int i = 0; i < m; i++)
+    if (__builtin_expect (x, 0))
+      count++;
+
+}
+
+/* { dg-final { scan-tree-dump-not "Executing store motion of" "lim2"  }  } */
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
new file mode 100644
index 00000000000..c38a858283f
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
@@ -0,0 +1,35 @@
+/* { dg-do compile  } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+/* Test that `count' is not hoisted out of inner loop and outer loop  when it is
+   in cold loop.  */
+
+int count;
+volatile int x;
+
+struct obj {
+  int data;
+  int data1;
+  struct obj *next;
+};
+
+void
+func (int m, int n, int k, struct obj *a)
+{
+  struct obj *q = a;
+  for (int j = 0; j < m; j++)
+    if (__builtin_expect (m, 0))
+      for (int i = 0; i < m; i++)
+	{
+	  if (__builtin_expect (x, 0))
+	    {
+	      count++;
+	      q->data += 3; /* Not hoisted out to inner loop. */
+	    }
+	  count += n;
+	  q->data1 += k; /* Not hoisted out to outer loop. */
+	}
+}
+
+/* { dg-final { scan-tree-dump-not "Executing store motion of" "lim2"  }  } */
+
-- 
2.27.0.90.geebb51ba8c


[-- Attachment #3: v4-0002-loop-invariant-Don-t-move-cold-bb-instructions-to.patch --]
[-- Type: text/plain, Size: 1463 bytes --]

From 468e0b252a6b4a8b648c4a49850ed337ab5e03e1 Mon Sep 17 00:00:00 2001
From: Xiong Hu Luo <luoxhu@linux.ibm.com>
Date: Fri, 8 Oct 2021 22:05:39 -0500
Subject: [PATCH v4 2/2] loop-invariant: Don't move cold bb instructions to
 preheader in RTL

gcc/ChangeLog:

	* loop-invariant.c (find_invariants_bb): Check profile count
	before motion.
	(find_invariants_body): Add argument.
---
 gcc/loop-invariant.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/gcc/loop-invariant.c b/gcc/loop-invariant.c
index fca0c2b24be..5c3be7bf0eb 100644
--- a/gcc/loop-invariant.c
+++ b/gcc/loop-invariant.c
@@ -1183,9 +1183,14 @@ find_invariants_insn (rtx_insn *insn, bool always_reached, bool always_executed)
    call.  */
 
 static void
-find_invariants_bb (basic_block bb, bool always_reached, bool always_executed)
+find_invariants_bb (class loop *loop, basic_block bb, bool always_reached,
+		    bool always_executed)
 {
   rtx_insn *insn;
+  basic_block preheader = loop_preheader_edge (loop)->src;
+
+  if (preheader->count > bb->count)
+    return;
 
   FOR_BB_INSNS (bb, insn)
     {
@@ -1214,8 +1219,7 @@ find_invariants_body (class loop *loop, basic_block *body,
   unsigned i;
 
   for (i = 0; i < loop->num_nodes; i++)
-    find_invariants_bb (body[i],
-			bitmap_bit_p (always_reached, i),
+    find_invariants_bb (loop, body[i], bitmap_bit_p (always_reached, i),
 			bitmap_bit_p (always_executed, i));
 }
 
-- 
2.27.0.90.geebb51ba8c

Re: [RFC] Don't move cold code out of loop by checking bb count

[Richard Biener]

On Sat, Oct 9, 2021 at 5:45 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>
> Hi,
>
> On 2021/9/28 20:09, Richard Biener wrote:
> > On Fri, Sep 24, 2021 at 8:29 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
> >>
> >> Update the patch to v3, not sure whether you prefer the paste style
> >> and continue to link the previous thread as Segher dislikes this...
> >>
> >>
> >> [PATCH v3] Don't move cold code out of loop by checking bb count
> >>
> >>
> >> Changes:
> >> 1. Handle max_loop in determine_max_movement instead of
> >> outermost_invariant_loop.
> >> 2. Remove unnecessary changes.
> >> 3. Add for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body) in can_sm_ref_p.
> >> 4. "gsi_next (&bsi);" in move_computations_worker is kept since it caused
> >> infinite loop when implementing v1 and the iteration is missed to be
> >> updated actually.
> >>
> >> v1: https://gcc.gnu.org/pipermail/gcc-patches/2021-August/576488.html
> >> v2: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/579086.html
> >>
> >> There was a patch trying to avoid move cold block out of loop:
> >>
> >> https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html
> >>
> >> Richard suggested to "never hoist anything from a bb with lower execution
> >> frequency to a bb with higher one in LIM invariantness_dom_walker
> >> before_dom_children".
> >>
> >> In gimple LIM analysis, add find_coldest_out_loop to move invariants to
> >> expected target loop, if profile count of the loop bb is colder
> >> than target loop preheader, it won't be hoisted out of loop.
> >> Likely for store motion, if all locations of the REF in loop is cold,
> >> don't do store motion of it.
> >>
> >> SPEC2017 performance evaluation shows 1% performance improvement for
> >> intrate GEOMEAN and no obvious regression for others.  Especially,
> >> 500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
> >> largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
> >> on P8LE.
> >>
> >> gcc/ChangeLog:
> >>
> >>         * loop-invariant.c (find_invariants_bb): Check profile count
> >>         before motion.
> >>         (find_invariants_body): Add argument.
> >>         * tree-ssa-loop-im.c (find_coldest_out_loop): New function.
> >>         (determine_max_movement): Use find_coldest_out_loop.
> >>         (move_computations_worker): Adjust and fix iteration udpate.
> >>         (execute_sm_exit): Check pointer validness.
> >>         (class ref_in_loop_hot_body): New functor.
> >>         (ref_in_loop_hot_body::operator): New.
> >>         (can_sm_ref_p): Use for_all_locs_in_loop.
> >>
> >> gcc/testsuite/ChangeLog:
> >>
> >>         * gcc.dg/tree-ssa/recip-3.c: Adjust.
> >>         * gcc.dg/tree-ssa/ssa-lim-18.c: New test.
> >>         * gcc.dg/tree-ssa/ssa-lim-19.c: New test.
> >>         * gcc.dg/tree-ssa/ssa-lim-20.c: New test.
> >> ---
> >>  gcc/loop-invariant.c                       | 10 ++--
> >>  gcc/tree-ssa-loop-im.c                     | 61 ++++++++++++++++++++--
> >>  gcc/testsuite/gcc.dg/tree-ssa/recip-3.c    |  2 +-
> >>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c | 20 +++++++
> >>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c | 27 ++++++++++
> >>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c | 25 +++++++++
> >>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c | 28 ++++++++++
> >>  7 files changed, 165 insertions(+), 8 deletions(-)
> >>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
> >>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
> >>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
> >>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
> >>
> >> diff --git a/gcc/loop-invariant.c b/gcc/loop-invariant.c
> >> index fca0c2b24be..5c3be7bf0eb 100644
> >> --- a/gcc/loop-invariant.c
> >> +++ b/gcc/loop-invariant.c
> >> @@ -1183,9 +1183,14 @@ find_invariants_insn (rtx_insn *insn, bool always_reached, bool always_executed)
> >>     call.  */
> >>
> >>  static void
> >> -find_invariants_bb (basic_block bb, bool always_reached, bool always_executed)
> >> +find_invariants_bb (class loop *loop, basic_block bb, bool always_reached,
> >> +                   bool always_executed)
> >>  {
> >>    rtx_insn *insn;
> >> +  basic_block preheader = loop_preheader_edge (loop)->src;
> >> +
> >> +  if (preheader->count > bb->count)
> >> +    return;
> >>
> >>    FOR_BB_INSNS (bb, insn)
> >>      {
> >> @@ -1214,8 +1219,7 @@ find_invariants_body (class loop *loop, basic_block *body,
> >>    unsigned i;
> >>
> >>    for (i = 0; i < loop->num_nodes; i++)
> >> -    find_invariants_bb (body[i],
> >> -                       bitmap_bit_p (always_reached, i),
> >> +    find_invariants_bb (loop, body[i], bitmap_bit_p (always_reached, i),
> >>                         bitmap_bit_p (always_executed, i));
> >>  }
> >>
> >> diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
> >> index 4b187c2cdaf..655fab03442 100644
> >> --- a/gcc/tree-ssa-loop-im.c
> >> +++ b/gcc/tree-ssa-loop-im.c
> >> @@ -417,6 +417,28 @@ movement_possibility (gimple *stmt)
> >>    return ret;
> >>  }
> >>
> >> +/* Find coldest loop between outmost_loop and loop by comapring profile count.  */
> >> +
> >> +static class loop *
> >> +find_coldest_out_loop (class loop *outmost_loop, class loop *loop,
> >> +                      basic_block curr_bb)
> >> +{
> >> +  class loop *cold_loop, *min_loop;
> >> +  cold_loop = min_loop = outmost_loop;
> >> +  profile_count min_count = loop_preheader_edge (min_loop)->src->count;
> >> +
> >> +  if (curr_bb && curr_bb->count < loop_preheader_edge (loop)->src->count)
> >
> > Honza - can you comment on whether we should compare BB counts this way?
> >
> > I would suspect that for, say,
> >
> >   for (...)
> >      if (a)
> >        X;
> >      else
> >        Y;
> >
> > that the counts for X and Y will be less than that of the preheader of the loop
> > only when the loop is estimated to run once.  That is, should we really compare
> > the to the preheader count or maybe better to the _header_ count which
> > would keep the number of iterations out of the equation?
>
> I quickly tried to replace all the loop_preheader_edge (loop)->src with
> loop_preheader_edge (loop)->dest, it will cause many failures in
> gcc.dg/tree-ssa/ssa-lim-*.c, I didn't go deep to investigate, but it seems
> reasonable to compare the bb count with preheader count as both gimple lim
> and RTL loop-invariant move instructions to *preheader* instead of *header*
> after analysis?

Hmm, yeah - guess I was confused here.

> >
> > If we look at maybe_hot_count_p that's a quite sophisticated thing to
> > compare a count to the "IPA hot", here we're comparing two counts
> > within a function where it actually matters whether we use a<b or
> > !(a>=b) since 'unordered' is mapped to false (but there's no ordered_p
> > function).
> >
> > Xionghu, you error on the side of not hoisting for unordered counts here
> >
> >> +    return NULL;
> >> +
> >> +  while (min_loop != loop)
> >> +    {
> >> +      min_loop = superloop_at_depth (loop, loop_depth (min_loop) + 1);
> >> +      if (loop_preheader_edge (min_loop)->src->count < min_count)
> >
> > but in the other direction here and on the side of not hoisting
> > in ref_in_loop_hot_body.
> >
> > The three-state relational operator overloads are probably not the
> > very best idea...
> > (see profile-count.h for them)
> >
> Added new function bb_colder_than_loop_preheader to encapsulate the comparision,
> if FALSE is returned due to three-state inequality,  find_coldest_out_loop
> will return the original input to lim->max_loop, and ref_in_loop_hot_body::operator ()
> will return true to continue perform store motion, both preserve the previous
> behavior.

Thanks.  But I don't think the abstraction as written is useful:

+/* Compare the profile count inequality of COUNT1 and COUNT2, it is three-state
+   as stated in profile-count.h, FALSE is returned if inequality cannot be
+   decided.  */
+bool bb_colder_than_loop_preheader (profile_count count1, profile_count count2)
+{
+  if (count1 < count2)
+    return true;
+  else
+    return false;
+}

given the following seems to pass the preheader count in place of the BB count.

+      if (bb_colder_than_loop_preheader (
+           loop_preheader_edge (min_loop)->src->count, min_count))
+       cold_loop = min_loop;

find_coldest_out_loop is also a bit weird, I think we want to find
the outermost loop between outmost_loop and loop that has a
lower count than the curr_bb count but

+  while (min_loop != loop)
+    {
+      min_loop = superloop_at_depth (loop, loop_depth (min_loop) + 1);
+      if (bb_colder_than_loop_preheader (
+           loop_preheader_edge (min_loop)->src->count, min_count))
+       cold_loop = min_loop;

compares the outermost loops count (min_count) against the preheader
count?  So we're searching for a cold loop with respect to its enclosing loop
here?

Why is this function not simply

+static class loop *
+find_coldest_out_loop (class loop *outmost_loop, class loop *loop,
+                      basic_block curr_bb)
+{
     while (bb_colder_than_loop_preheader (curr_bb->count,
               loop_preheader_edge (outermost_loop)->src->count))
        {
            if (outermost_loop == loop)
              return NULL;
            outermost_loop = superloop_at_depth (loop, loop_depth
(outermost_loop) + 1);
        }
     return outermost_loop;
}

?

Likewise I wonder why ref_in_loop_hot_body::operator () needs to call
find_coldest_out_loop and why it not simply does

+bool
+ref_in_loop_hot_body::operator () (mem_ref_loc *loc)
+{
+  basic_block curr_bb = gimple_bb (loc->stmt);
    if (bb_colder_than_loop_preheader (curr_bb->count,
loop_preheader_edge (l)->src->count))
      return false;
   return true;
   }

?
>
> >> +       cold_loop = min_loop;
> >> +    }
> >> +  return cold_loop;
> >> +}
> >> +
> >>  /* Suppose that operand DEF is used inside the LOOP.  Returns the outermost
> >>     loop to that we could move the expression using DEF if it did not have
> >>     other operands, i.e. the outermost loop enclosing LOOP in that the value
> >> @@ -685,7 +707,9 @@ determine_max_movement (gimple *stmt, bool must_preserve_exec)
> >>      level = ALWAYS_EXECUTED_IN (bb);
> >>    else
> >>      level = superloop_at_depth (loop, 1);
> >> -  lim_data->max_loop = level;
> >> +  lim_data->max_loop = find_coldest_out_loop (level, loop, bb);
> >> +  if (!lim_data->max_loop)
> >> +    return false;
> >>
> >>    if (gphi *phi = dyn_cast <gphi *> (stmt))
> >>      {
> >> @@ -1198,7 +1222,6 @@ move_computations_worker (basic_block bb)
> >>    for (gimple_stmt_iterator bsi = gsi_start_bb (bb); !gsi_end_p (bsi); )
> >>      {
> >>        edge e;
> >> -
> >>        gimple *stmt = gsi_stmt (bsi);
> >>
> >>        lim_data = get_lim_data (stmt);
> >> @@ -1221,7 +1244,10 @@ move_computations_worker (basic_block bb)
> >>        /* We do not really want to move conditionals out of the loop; we just
> >>          placed it here to force its operands to be moved if necessary.  */
> >>        if (gimple_code (stmt) == GIMPLE_COND)
> >> -       continue;
> >> +       {
> >> +         gsi_next (&bsi);
> >> +         continue;
> >> +       }
> >>
> >>        if (dump_file && (dump_flags & TDF_DETAILS))
> >>         {
> >> @@ -2241,7 +2267,12 @@ execute_sm_exit (class loop *loop, edge ex, vec<seq_entry> &seq,
> >>         }
> >>        else
> >>         {
> >> -         sm_aux *aux = *aux_map.get (ref);
> >> +         sm_aux **paux = aux_map.get (ref);
> >> +         sm_aux *aux;
> >> +         if (paux)
> >> +           aux = *paux;
> >> +         else
> >> +           continue;
> >
> > do you really need this?  I doubt so.
>
> Removed.
>
> >
> >>           if (!aux->store_flag || kind == sm_ord)
> >>             {
> >>               gassign *store;
> >> @@ -2887,6 +2918,25 @@ ref_indep_loop_p (class loop *loop, im_mem_ref *ref, dep_kind kind)
> >>    return indep_p;
> >>  }
> >>
> >> +class ref_in_loop_hot_body
> >> +{
> >> +public:
> >> +  ref_in_loop_hot_body (loop *loop_) : l (loop_) {}
> >> +  bool operator () (mem_ref_loc *loc);
> >> +  class loop *l;
> >> +};
> >> +
> >> +bool
> >> +ref_in_loop_hot_body::operator () (mem_ref_loc *loc)
> >> +{
> >> +  basic_block curr_bb = gimple_bb (loc->stmt);
> >> +  edge e = loop_preheader_edge (l);
> >> +  if (e->src->count > curr_bb->count)
> >> +    return false;
> >> +  else
> >> +    return true;
> >> +}
> >> +
> >>
> >>  /* Returns true if we can perform store motion of REF from LOOP.  */
> >>
> >> @@ -2941,6 +2991,9 @@ can_sm_ref_p (class loop *loop, im_mem_ref *ref)
> >>    if (!ref_indep_loop_p (loop, ref, sm_war))
> >>      return false;
> >>
> >
> > Add a comment here what this is about.
>
> Done.
>
> >
> > Otherwise the GIMPLE invariant motion parts look sensible, but I'd
> > really like to have
> > the issue on the profile_count API sorted out.
> >
> > Can you split out the RTL invariant motion part to a separate patch please?
>
> Done.  Attached the two patches, thanks.
>
>
> BR,
> Xionghu

Re: [RFC] Don't move cold code out of loop by checking bb count

[Xionghu Luo]

On 2021/10/15 16:11, Richard Biener wrote:
> On Sat, Oct 9, 2021 at 5:45 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>>
>> Hi,
>>
>> On 2021/9/28 20:09, Richard Biener wrote:
>>> On Fri, Sep 24, 2021 at 8:29 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>>>>
>>>> Update the patch to v3, not sure whether you prefer the paste style
>>>> and continue to link the previous thread as Segher dislikes this...
>>>>
>>>>
>>>> [PATCH v3] Don't move cold code out of loop by checking bb count
>>>>
>>>>
>>>> Changes:
>>>> 1. Handle max_loop in determine_max_movement instead of
>>>> outermost_invariant_loop.
>>>> 2. Remove unnecessary changes.
>>>> 3. Add for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body) in can_sm_ref_p.
>>>> 4. "gsi_next (&bsi);" in move_computations_worker is kept since it caused
>>>> infinite loop when implementing v1 and the iteration is missed to be
>>>> updated actually.
>>>>
>>>> v1: https://gcc.gnu.org/pipermail/gcc-patches/2021-August/576488.html
>>>> v2: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/579086.html
>>>>
>>>> There was a patch trying to avoid move cold block out of loop:
>>>>
>>>> https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html
>>>>
>>>> Richard suggested to "never hoist anything from a bb with lower execution
>>>> frequency to a bb with higher one in LIM invariantness_dom_walker
>>>> before_dom_children".
>>>>
>>>> In gimple LIM analysis, add find_coldest_out_loop to move invariants to
>>>> expected target loop, if profile count of the loop bb is colder
>>>> than target loop preheader, it won't be hoisted out of loop.
>>>> Likely for store motion, if all locations of the REF in loop is cold,
>>>> don't do store motion of it.
>>>>
>>>> SPEC2017 performance evaluation shows 1% performance improvement for
>>>> intrate GEOMEAN and no obvious regression for others.  Especially,
>>>> 500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
>>>> largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
>>>> on P8LE.
>>>>
>>>> gcc/ChangeLog:
>>>>
>>>>         * loop-invariant.c (find_invariants_bb): Check profile count
>>>>         before motion.
>>>>         (find_invariants_body): Add argument.
>>>>         * tree-ssa-loop-im.c (find_coldest_out_loop): New function.
>>>>         (determine_max_movement): Use find_coldest_out_loop.
>>>>         (move_computations_worker): Adjust and fix iteration udpate.
>>>>         (execute_sm_exit): Check pointer validness.
>>>>         (class ref_in_loop_hot_body): New functor.
>>>>         (ref_in_loop_hot_body::operator): New.
>>>>         (can_sm_ref_p): Use for_all_locs_in_loop.
>>>>
>>>> gcc/testsuite/ChangeLog:
>>>>
>>>>         * gcc.dg/tree-ssa/recip-3.c: Adjust.
>>>>         * gcc.dg/tree-ssa/ssa-lim-18.c: New test.
>>>>         * gcc.dg/tree-ssa/ssa-lim-19.c: New test.
>>>>         * gcc.dg/tree-ssa/ssa-lim-20.c: New test.
>>>> ---
>>>>  gcc/loop-invariant.c                       | 10 ++--
>>>>  gcc/tree-ssa-loop-im.c                     | 61 ++++++++++++++++++++--
>>>>  gcc/testsuite/gcc.dg/tree-ssa/recip-3.c    |  2 +-
>>>>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c | 20 +++++++
>>>>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c | 27 ++++++++++
>>>>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c | 25 +++++++++
>>>>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c | 28 ++++++++++
>>>>  7 files changed, 165 insertions(+), 8 deletions(-)
>>>>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
>>>>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
>>>>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
>>>>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
>>>>
>>>> diff --git a/gcc/loop-invariant.c b/gcc/loop-invariant.c
>>>> index fca0c2b24be..5c3be7bf0eb 100644
>>>> --- a/gcc/loop-invariant.c
>>>> +++ b/gcc/loop-invariant.c
>>>> @@ -1183,9 +1183,14 @@ find_invariants_insn (rtx_insn *insn, bool always_reached, bool always_executed)
>>>>     call.  */
>>>>
>>>>  static void
>>>> -find_invariants_bb (basic_block bb, bool always_reached, bool always_executed)
>>>> +find_invariants_bb (class loop *loop, basic_block bb, bool always_reached,
>>>> +                   bool always_executed)
>>>>  {
>>>>    rtx_insn *insn;
>>>> +  basic_block preheader = loop_preheader_edge (loop)->src;
>>>> +
>>>> +  if (preheader->count > bb->count)
>>>> +    return;
>>>>
>>>>    FOR_BB_INSNS (bb, insn)
>>>>      {
>>>> @@ -1214,8 +1219,7 @@ find_invariants_body (class loop *loop, basic_block *body,
>>>>    unsigned i;
>>>>
>>>>    for (i = 0; i < loop->num_nodes; i++)
>>>> -    find_invariants_bb (body[i],
>>>> -                       bitmap_bit_p (always_reached, i),
>>>> +    find_invariants_bb (loop, body[i], bitmap_bit_p (always_reached, i),
>>>>                         bitmap_bit_p (always_executed, i));
>>>>  }
>>>>
>>>> diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
>>>> index 4b187c2cdaf..655fab03442 100644
>>>> --- a/gcc/tree-ssa-loop-im.c
>>>> +++ b/gcc/tree-ssa-loop-im.c
>>>> @@ -417,6 +417,28 @@ movement_possibility (gimple *stmt)
>>>>    return ret;
>>>>  }
>>>>
>>>> +/* Find coldest loop between outmost_loop and loop by comapring profile count.  */
>>>> +
>>>> +static class loop *
>>>> +find_coldest_out_loop (class loop *outmost_loop, class loop *loop,
>>>> +                      basic_block curr_bb)
>>>> +{
>>>> +  class loop *cold_loop, *min_loop;
>>>> +  cold_loop = min_loop = outmost_loop;
>>>> +  profile_count min_count = loop_preheader_edge (min_loop)->src->count;
>>>> +
>>>> +  if (curr_bb && curr_bb->count < loop_preheader_edge (loop)->src->count)
>>>
>>> Honza - can you comment on whether we should compare BB counts this way?
>>>
>>> I would suspect that for, say,
>>>
>>>   for (...)
>>>      if (a)
>>>        X;
>>>      else
>>>        Y;
>>>
>>> that the counts for X and Y will be less than that of the preheader of the loop
>>> only when the loop is estimated to run once.  That is, should we really compare
>>> the to the preheader count or maybe better to the _header_ count which
>>> would keep the number of iterations out of the equation?
>>
>> I quickly tried to replace all the loop_preheader_edge (loop)->src with
>> loop_preheader_edge (loop)->dest, it will cause many failures in
>> gcc.dg/tree-ssa/ssa-lim-*.c, I didn't go deep to investigate, but it seems
>> reasonable to compare the bb count with preheader count as both gimple lim
>> and RTL loop-invariant move instructions to *preheader* instead of *header*
>> after analysis?
> 
> Hmm, yeah - guess I was confused here.
> 
>>>
>>> If we look at maybe_hot_count_p that's a quite sophisticated thing to
>>> compare a count to the "IPA hot", here we're comparing two counts
>>> within a function where it actually matters whether we use a<b or
>>> !(a>=b) since 'unordered' is mapped to false (but there's no ordered_p
>>> function).
>>>
>>> Xionghu, you error on the side of not hoisting for unordered counts here
>>>
>>>> +    return NULL;
>>>> +
>>>> +  while (min_loop != loop)
>>>> +    {
>>>> +      min_loop = superloop_at_depth (loop, loop_depth (min_loop) + 1);
>>>> +      if (loop_preheader_edge (min_loop)->src->count < min_count)
>>>
>>> but in the other direction here and on the side of not hoisting
>>> in ref_in_loop_hot_body.
>>>
>>> The three-state relational operator overloads are probably not the
>>> very best idea...
>>> (see profile-count.h for them)
>>>
>> Added new function bb_colder_than_loop_preheader to encapsulate the comparision,
>> if FALSE is returned due to three-state inequality,  find_coldest_out_loop
>> will return the original input to lim->max_loop, and ref_in_loop_hot_body::operator ()
>> will return true to continue perform store motion, both preserve the previous
>> behavior.
> 
> Thanks.  But I don't think the abstraction as written is useful:
> 
> +/* Compare the profile count inequality of COUNT1 and COUNT2, it is three-state
> +   as stated in profile-count.h, FALSE is returned if inequality cannot be
> +   decided.  */
> +bool bb_colder_than_loop_preheader (profile_count count1, profile_count count2)
> +{
> +  if (count1 < count2)
> +    return true;
> +  else
> +    return false;
> +}
> 
> given the following seems to pass the preheader count in place of the BB count.
> 
> +      if (bb_colder_than_loop_preheader (
> +           loop_preheader_edge (min_loop)->src->count, min_count))
> +       cold_loop = min_loop;
> 
> find_coldest_out_loop is also a bit weird, I think we want to find
> the outermost loop between outmost_loop and loop that has a
> lower count than the curr_bb count but
> 
> +  while (min_loop != loop)
> +    {
> +      min_loop = superloop_at_depth (loop, loop_depth (min_loop) + 1);
> +      if (bb_colder_than_loop_preheader (
> +           loop_preheader_edge (min_loop)->src->count, min_count))
> +       cold_loop = min_loop;
> 
> compares the outermost loops count (min_count) against the preheader
> count?  So we're searching for a cold loop with respect to its enclosing loop
> here?

Let me try to explain how it works :)

find_coldest_out_loop does two steps check:
1) Check whether curr_bb is cold in it's own loop_father, if it is cold,
just return NULL which means it should not be moved out at all;
2)  curr_bb is NOT cold, assuming the current loop L[m] is the coldest first, 
than try to find a cold loop to be hoisted to from {L[1], L[2], ... L[m]}, 
if L[i]->count < L[m]->count, set the cold_loop to L[i] until find the loop
that has smallest profile_count.


Take the updated ssa-lim-19.c as example, check whether curr_bb(bb 5) is cold in
loop 3, if it is cold, just return NULL, otherwise select the coldest loop in
{loop1, loop2, loop3} and find that loop2 is colder than loop3, return loop2 to
be the target hoist loop.  The first check could AVOID hoist if curr_bb is colder
than loop3, but it is still hot than loop1 and loop2.  Not sure whether it is possible
to construct such cases?


gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c

volatile int x;
void
bar (int, char *, char *);
void
foo (int *a, int n, int m, int s, int t)
{
  int i;
  int j;
  int k;

  for (i = 0; i < m; i++)  // loop 1
    {
      if (__builtin_expect (x, 0))
        for (j = 0; j < n; j++)   // loop 2
          for (k = 0; k < n; k++)   // loop 3
           {
             bar (s / 5, "one", "two");  // curr_bb 
             a[t] = s;
           }
      a[t] = t;  // curr_bb2
    }
}

The 4 invariant statements are moved to bb 11(loop2) instead of bb 10(loop1)
with this patch.
There are totally 6 combinations when curr_bb is hotter than loop 3.  We need
to compare the "Loop preheader hotness" instead of "every Loop[i] and curr_bb hotness",
returning the coldest loop for this function find_coldest_out_loop, otherwise
unexpected behavior happens.

L1 > L2 > L3   =>  return L3
L1 > L3 > L2   =>  return L2
L2 > L1 > L3   =>  return L3
L2 > L3 > L1   =>  return L1
L3 > L1 > L2   =>  return L2
L3 > L2 > L1   =>  return L1

So bb_colder_than_loop_preheader does two kind of checks, one is checking
L3 preheader count with curr_bb count, another is checking L3 preheader count
with L1 preheader count, L2 preheader count, etc...


ssa-lim-19.c.138t.lim2:
...
   <bb 10> [local count: 16057869]:  // L1 preheader
-  _4 = s_22(D) / 5;
-  _5 = (long unsigned int) t_24(D);
-  _6 = _5 * 4;
-  _7 = a_25(D) + _6;
   _8 = (long unsigned int) t_24(D);
   _9 = _8 * 4;
   _10 = a_25(D) + _9;

   <bb 3> [local count: 145980626]:
   # i_34 = PHI <i_29(12), 0(10)>
   x.0_1 ={v} x;
   if (x.0_1 != 0)
     goto <bb 4>; [10.00%]
   else
     goto <bb 8>; [90.00%]

   <bb 4> [local count: 14598063]:
   if (n_20(D) > 0)
     goto <bb 11>; [89.00%]
   else
     goto <bb 8>; [11.00%]

   <bb 11> [local count: 12992276]:  // L2 preheader
+  _4 = s_22(D) / 5;
+  _5 = (long unsigned int) t_24(D);
+  _6 = _5 * 4;
+  _7 = a_25(D) + _6;
   goto <bb 7>; [100.00%]

   <bb 14> [local count: 850510901]:

   <bb 5> [local count: 955630225]:  // curr_bb
   # k_36 = PHI <k_27(14), 0(7)>
   bar (_4, "one", "two");
   *_7 = s_22(D);
   k_27 = k_36 + 1;
   if (n_20(D) > k_27)
     goto <bb 14>; [89.00%]
   else
     goto <bb 6>; [11.00%]

   <bb 6> [local count: 118111600]:
   j_21 = j_35 + 1;
   if (n_20(D) > j_21)
     goto <bb 13>; [89.00%]
   else
     goto <bb 8>; [11.00%]

   <bb 13> [local count: 105119324]:

   <bb 7> [local count: 118111600]:   // L3 preheader
   # j_35 = PHI <j_21(13), 0(11)>
   goto <bb 5>; [100.00%]

   <bb 8> [local count: 145980626]:
   *_10 = t_24(D);
   i_29 = i_34 + 1;

Re-paste the bb_colder_than_loop_preheader and find_coldest_out_loop:

+/* Compare the profile count inequality of COUNT1 and COUNT2, it is three-state
+   as stated in profile-count.h, FALSE is returned if inequality cannot be
+   decided.  */
+bool bb_colder_than_loop_preheader (profile_count count1, profile_count count2)
+{
+  if (count1 < count2)
+    return true;
+  else
+    return false;
+}
+
+/* Find coldest loop between OUTMOST_LOOP and LOOP by comapring profile count.
+ */
+
+static class loop *
+find_coldest_out_loop (class loop *outmost_loop, class loop *loop,
+		       basic_block curr_bb)
+{
+  class loop *cold_loop, *min_loop;
+  cold_loop = min_loop = outmost_loop;
+  profile_count min_count = loop_preheader_edge (min_loop)->src->count;
+
+  /* If bb_colder_than_loop_preheader returns false due to three-state
+    comparision, OUTMOST_LOOP is returned finally to preserve the behavior.
+    Otherwise, return the coldest loop between OUTMOST_LOOP and LOOP.  */
+  if (curr_bb
+      && bb_colder_than_loop_preheader (curr_bb->count,
+					loop_preheader_edge (loop)->src->count))
+    return NULL;
+
+  while (min_loop != loop)
+    {
+      min_loop = superloop_at_depth (loop, loop_depth (min_loop) + 1);
+      if (bb_colder_than_loop_preheader (
+	    loop_preheader_edge (min_loop)->src->count, min_count))
+	cold_loop = min_loop;
+    }
+  return cold_loop;
+}
+


> 
> Why is this function not simply
> 
> +static class loop *
> +find_coldest_out_loop (class loop *outmost_loop, class loop *loop,
> +                      basic_block curr_bb)
> +{
>      while (bb_colder_than_loop_preheader (curr_bb->count,
>                loop_preheader_edge (outermost_loop)->src->count))
>         {
>             if (outermost_loop == loop)
>               return NULL;
>             outermost_loop = superloop_at_depth (loop, loop_depth
> (outermost_loop) + 1);
>         }
>      return outermost_loop;
> }

If change like this, when processing curr_bb(5), outermost_loop will
return loop 1 since curr_bb->count > Loop1_prehead->count, the while
loop stopped.  This doesn't meet what we want.  

> 
> ?
> 
> Likewise I wonder why ref_in_loop_hot_body::operator () needs to call
> find_coldest_out_loop and why it not simply does
> 
> +bool
> +ref_in_loop_hot_body::operator () (mem_ref_loc *loc)
> +{
> +  basic_block curr_bb = gimple_bb (loc->stmt);
>     if (bb_colder_than_loop_preheader (curr_bb->count,
> loop_preheader_edge (l)->src->count))
>       return false;
>    return true;
>    }

Likely for this part,

+/* Find out the coldest loop between loop L and innermost loop, compare the
+   hotness between current BB and coldest loop preheader by profile count.  */
+bool
+ref_in_loop_hot_body::operator () (mem_ref_loc *loc)
+{
+  basic_block curr_bb = gimple_bb (loc->stmt);
+  class loop *inner_loop = curr_bb->loop_father;
+  class loop *cold_loop = l;
+  if (l != inner_loop)
+    cold_loop = find_coldest_out_loop (l, inner_loop, curr_bb);
+  if (!cold_loop)
+    return false;
+  edge e = loop_preheader_edge (cold_loop);
+  /*  If bb_colder_than_loop_preheader is false due to three-state inequality
+     comparision, TRUE is returned to continue perform store motion.  */
+  if (bb_colder_than_loop_preheader (curr_bb->count, e->src->count))
+    return false;
+  else
+    return true;
+}

l is the input of ref_in_loop_hot_body, it is an out loop, we need to find a
cold_loop between l and inner_loop.  Reason is there may be cold loop between
l and inner_loop, which means we shouldn't do store-motion from curr_bb to l
directly.
After reconsideration, I think the bb_colder_than_loop_preheader could be
 removed since curr_bb is checked in find_coldest_out_loop already.  And remove
the "l != inner_loop" check:

+/* Find out the coldest loop between loop L and innermost loop.  */
+bool
+ref_in_loop_hot_body::operator () (mem_ref_loc *loc)
+{
+  basic_block curr_bb = gimple_bb (loc->stmt);
+  class loop *inner_loop = curr_bb->loop_father;
+  class loop *cold_loop = l;
+  cold_loop = find_coldest_out_loop (l, inner_loop, curr_bb);
+  if (!cold_loop)
+    return false;
+  return true;
+}


-- 
Thanks,
Xionghu

[PATCH v5 2/2] Don't move cold code out of loop by checking bb count

[Xionghu Luo]

On 2021/10/18 12:29, Xionghu Luo via Gcc-patches wrote:
> 
> 
> On 2021/10/15 16:11, Richard Biener wrote:
>> On Sat, Oct 9, 2021 at 5:45 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>>>
>>> Hi,
>>>
>>> On 2021/9/28 20:09, Richard Biener wrote:
>>>> On Fri, Sep 24, 2021 at 8:29 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>>>>>
>>>>> Update the patch to v3, not sure whether you prefer the paste style
>>>>> and continue to link the previous thread as Segher dislikes this...
>>>>>
>>>>>
>>>>> [PATCH v3] Don't move cold code out of loop by checking bb count
>>>>>
>>>>>
>>>>> Changes:
>>>>> 1. Handle max_loop in determine_max_movement instead of
>>>>> outermost_invariant_loop.
>>>>> 2. Remove unnecessary changes.
>>>>> 3. Add for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body) in can_sm_ref_p.
>>>>> 4. "gsi_next (&bsi);" in move_computations_worker is kept since it caused
>>>>> infinite loop when implementing v1 and the iteration is missed to be
>>>>> updated actually.
>>>>>
>>>>> v1: https://gcc.gnu.org/pipermail/gcc-patches/2021-August/576488.html
>>>>> v2: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/579086.html
>>>>>
>>>>> There was a patch trying to avoid move cold block out of loop:
>>>>>
>>>>> https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html
>>>>>
>>>>> Richard suggested to "never hoist anything from a bb with lower execution
>>>>> frequency to a bb with higher one in LIM invariantness_dom_walker
>>>>> before_dom_children".
>>>>>
>>>>> In gimple LIM analysis, add find_coldest_out_loop to move invariants to
>>>>> expected target loop, if profile count of the loop bb is colder
>>>>> than target loop preheader, it won't be hoisted out of loop.
>>>>> Likely for store motion, if all locations of the REF in loop is cold,
>>>>> don't do store motion of it.
>>>>>
>>>>> SPEC2017 performance evaluation shows 1% performance improvement for
>>>>> intrate GEOMEAN and no obvious regression for others.  Especially,
>>>>> 500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
>>>>> largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
>>>>> on P8LE.
>>>>>
>>>>> gcc/ChangeLog:
>>>>>
>>>>>         * loop-invariant.c (find_invariants_bb): Check profile count
>>>>>         before motion.
>>>>>         (find_invariants_body): Add argument.
>>>>>         * tree-ssa-loop-im.c (find_coldest_out_loop): New function.
>>>>>         (determine_max_movement): Use find_coldest_out_loop.
>>>>>         (move_computations_worker): Adjust and fix iteration udpate.
>>>>>         (execute_sm_exit): Check pointer validness.
>>>>>         (class ref_in_loop_hot_body): New functor.
>>>>>         (ref_in_loop_hot_body::operator): New.
>>>>>         (can_sm_ref_p): Use for_all_locs_in_loop.
>>>>>
>>>>> gcc/testsuite/ChangeLog:
>>>>>
>>>>>         * gcc.dg/tree-ssa/recip-3.c: Adjust.
>>>>>         * gcc.dg/tree-ssa/ssa-lim-18.c: New test.
>>>>>         * gcc.dg/tree-ssa/ssa-lim-19.c: New test.
>>>>>         * gcc.dg/tree-ssa/ssa-lim-20.c: New test.
>>>>> ---
>>>>>  gcc/loop-invariant.c                       | 10 ++--
>>>>>  gcc/tree-ssa-loop-im.c                     | 61 ++++++++++++++++++++--
>>>>>  gcc/testsuite/gcc.dg/tree-ssa/recip-3.c    |  2 +-
>>>>>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c | 20 +++++++
>>>>>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c | 27 ++++++++++
>>>>>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c | 25 +++++++++
>>>>>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c | 28 ++++++++++
>>>>>  7 files changed, 165 insertions(+), 8 deletions(-)
>>>>>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
>>>>>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
>>>>>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
>>>>>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
>>>>>
>>>>> diff --git a/gcc/loop-invariant.c b/gcc/loop-invariant.c
>>>>> index fca0c2b24be..5c3be7bf0eb 100644
>>>>> --- a/gcc/loop-invariant.c
>>>>> +++ b/gcc/loop-invariant.c
>>>>> @@ -1183,9 +1183,14 @@ find_invariants_insn (rtx_insn *insn, bool always_reached, bool always_executed)
>>>>>     call.  */
>>>>>
>>>>>  static void
>>>>> -find_invariants_bb (basic_block bb, bool always_reached, bool always_executed)
>>>>> +find_invariants_bb (class loop *loop, basic_block bb, bool always_reached,
>>>>> +                   bool always_executed)
>>>>>  {
>>>>>    rtx_insn *insn;
>>>>> +  basic_block preheader = loop_preheader_edge (loop)->src;
>>>>> +
>>>>> +  if (preheader->count > bb->count)
>>>>> +    return;
>>>>>
>>>>>    FOR_BB_INSNS (bb, insn)
>>>>>      {
>>>>> @@ -1214,8 +1219,7 @@ find_invariants_body (class loop *loop, basic_block *body,
>>>>>    unsigned i;
>>>>>
>>>>>    for (i = 0; i < loop->num_nodes; i++)
>>>>> -    find_invariants_bb (body[i],
>>>>> -                       bitmap_bit_p (always_reached, i),
>>>>> +    find_invariants_bb (loop, body[i], bitmap_bit_p (always_reached, i),
>>>>>                         bitmap_bit_p (always_executed, i));
>>>>>  }
>>>>>
>>>>> diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
>>>>> index 4b187c2cdaf..655fab03442 100644
>>>>> --- a/gcc/tree-ssa-loop-im.c
>>>>> +++ b/gcc/tree-ssa-loop-im.c
>>>>> @@ -417,6 +417,28 @@ movement_possibility (gimple *stmt)
>>>>>    return ret;
>>>>>  }
>>>>>
>>>>> +/* Find coldest loop between outmost_loop and loop by comapring profile count.  */
>>>>> +
>>>>> +static class loop *
>>>>> +find_coldest_out_loop (class loop *outmost_loop, class loop *loop,
>>>>> +                      basic_block curr_bb)
>>>>> +{
>>>>> +  class loop *cold_loop, *min_loop;
>>>>> +  cold_loop = min_loop = outmost_loop;
>>>>> +  profile_count min_count = loop_preheader_edge (min_loop)->src->count;
>>>>> +
>>>>> +  if (curr_bb && curr_bb->count < loop_preheader_edge (loop)->src->count)
>>>>
>>>> Honza - can you comment on whether we should compare BB counts this way?
>>>>
>>>> I would suspect that for, say,
>>>>
>>>>   for (...)
>>>>      if (a)
>>>>        X;
>>>>      else
>>>>        Y;
>>>>
>>>> that the counts for X and Y will be less than that of the preheader of the loop
>>>> only when the loop is estimated to run once.  That is, should we really compare
>>>> the to the preheader count or maybe better to the _header_ count which
>>>> would keep the number of iterations out of the equation?
>>>
>>> I quickly tried to replace all the loop_preheader_edge (loop)->src with
>>> loop_preheader_edge (loop)->dest, it will cause many failures in
>>> gcc.dg/tree-ssa/ssa-lim-*.c, I didn't go deep to investigate, but it seems
>>> reasonable to compare the bb count with preheader count as both gimple lim
>>> and RTL loop-invariant move instructions to *preheader* instead of *header*
>>> after analysis?
>>
>> Hmm, yeah - guess I was confused here.
>>
>>>>
>>>> If we look at maybe_hot_count_p that's a quite sophisticated thing to
>>>> compare a count to the "IPA hot", here we're comparing two counts
>>>> within a function where it actually matters whether we use a<b or
>>>> !(a>=b) since 'unordered' is mapped to false (but there's no ordered_p
>>>> function).
>>>>
>>>> Xionghu, you error on the side of not hoisting for unordered counts here
>>>>
>>>>> +    return NULL;
>>>>> +
>>>>> +  while (min_loop != loop)
>>>>> +    {
>>>>> +      min_loop = superloop_at_depth (loop, loop_depth (min_loop) + 1);
>>>>> +      if (loop_preheader_edge (min_loop)->src->count < min_count)
>>>>
>>>> but in the other direction here and on the side of not hoisting
>>>> in ref_in_loop_hot_body.
>>>>
>>>> The three-state relational operator overloads are probably not the
>>>> very best idea...
>>>> (see profile-count.h for them)
>>>>
>>> Added new function bb_colder_than_loop_preheader to encapsulate the comparision,
>>> if FALSE is returned due to three-state inequality,  find_coldest_out_loop
>>> will return the original input to lim->max_loop, and ref_in_loop_hot_body::operator ()
>>> will return true to continue perform store motion, both preserve the previous
>>> behavior.
>>
>> Thanks.  But I don't think the abstraction as written is useful:
>>
>> +/* Compare the profile count inequality of COUNT1 and COUNT2, it is three-state
>> +   as stated in profile-count.h, FALSE is returned if inequality cannot be
>> +   decided.  */
>> +bool bb_colder_than_loop_preheader (profile_count count1, profile_count count2)
>> +{
>> +  if (count1 < count2)
>> +    return true;
>> +  else
>> +    return false;
>> +}
>>
>> given the following seems to pass the preheader count in place of the BB count.
>>
>> +      if (bb_colder_than_loop_preheader (
>> +           loop_preheader_edge (min_loop)->src->count, min_count))
>> +       cold_loop = min_loop;
>>
>> find_coldest_out_loop is also a bit weird, I think we want to find
>> the outermost loop between outmost_loop and loop that has a
>> lower count than the curr_bb count but
>>
>> +  while (min_loop != loop)
>> +    {
>> +      min_loop = superloop_at_depth (loop, loop_depth (min_loop) + 1);
>> +      if (bb_colder_than_loop_preheader (
>> +           loop_preheader_edge (min_loop)->src->count, min_count))
>> +       cold_loop = min_loop;
>>
>> compares the outermost loops count (min_count) against the preheader
>> count?  So we're searching for a cold loop with respect to its enclosing loop
>> here?
> 
> Let me try to explain how it works :)
> 
> find_coldest_out_loop does two steps check:
> 1) Check whether curr_bb is cold in it's own loop_father, if it is cold,
> just return NULL which means it should not be moved out at all;
> 2)  curr_bb is NOT cold, assuming the current loop L[m] is the coldest first, 
> than try to find a cold loop to be hoisted to from {L[1], L[2], ... L[m]}, 
> if L[i]->count < L[m]->count, set the cold_loop to L[i] until find the loop
> that has smallest profile_count.
> 
> 
> Take the updated ssa-lim-19.c as example, check whether curr_bb(bb 5) is cold in
> loop 3, if it is cold, just return NULL, otherwise select the coldest loop in
> {loop1, loop2, loop3} and find that loop2 is colder than loop3, return loop2 to
> be the target hoist loop.  The first check could AVOID hoist if curr_bb is colder
> than loop3, but it is still hot than loop1 and loop2.  Not sure whether it is possible
> to construct such cases?
> 
> 
> gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
> 
> volatile int x;
> void
> bar (int, char *, char *);
> void
> foo (int *a, int n, int m, int s, int t)
> {
>   int i;
>   int j;
>   int k;
> 
>   for (i = 0; i < m; i++)  // loop 1
>     {
>       if (__builtin_expect (x, 0))
>         for (j = 0; j < n; j++)   // loop 2
>           for (k = 0; k < n; k++)   // loop 3
>            {
>              bar (s / 5, "one", "two");  // curr_bb 
>              a[t] = s;
>            }
>       a[t] = t;  // curr_bb2
>     }
> }
> 
> The 4 invariant statements are moved to bb 11(loop2) instead of bb 10(loop1)
> with this patch.
> There are totally 6 combinations when curr_bb is hotter than loop 3.  We need
> to compare the "Loop preheader hotness" instead of "every Loop[i] and curr_bb hotness",
> returning the coldest loop for this function find_coldest_out_loop, otherwise
> unexpected behavior happens.
> 
> L1 > L2 > L3   =>  return L3
> L1 > L3 > L2   =>  return L2
> L2 > L1 > L3   =>  return L3
> L2 > L3 > L1   =>  return L1
> L3 > L1 > L2   =>  return L2
> L3 > L2 > L1   =>  return L1
> 
> So bb_colder_than_loop_preheader does two kind of checks, one is checking
> L3 preheader count with curr_bb count, another is checking L3 preheader count
> with L1 preheader count, L2 preheader count, etc...
> 
> 
> ssa-lim-19.c.138t.lim2:
> ...
>    <bb 10> [local count: 16057869]:  // L1 preheader
> -  _4 = s_22(D) / 5;
> -  _5 = (long unsigned int) t_24(D);
> -  _6 = _5 * 4;
> -  _7 = a_25(D) + _6;
>    _8 = (long unsigned int) t_24(D);
>    _9 = _8 * 4;
>    _10 = a_25(D) + _9;
> 
>    <bb 3> [local count: 145980626]:
>    # i_34 = PHI <i_29(12), 0(10)>
>    x.0_1 ={v} x;
>    if (x.0_1 != 0)
>      goto <bb 4>; [10.00%]
>    else
>      goto <bb 8>; [90.00%]
> 
>    <bb 4> [local count: 14598063]:
>    if (n_20(D) > 0)
>      goto <bb 11>; [89.00%]
>    else
>      goto <bb 8>; [11.00%]
> 
>    <bb 11> [local count: 12992276]:  // L2 preheader
> +  _4 = s_22(D) / 5;
> +  _5 = (long unsigned int) t_24(D);
> +  _6 = _5 * 4;
> +  _7 = a_25(D) + _6;
>    goto <bb 7>; [100.00%]
> 
>    <bb 14> [local count: 850510901]:
> 
>    <bb 5> [local count: 955630225]:  // curr_bb
>    # k_36 = PHI <k_27(14), 0(7)>
>    bar (_4, "one", "two");
>    *_7 = s_22(D);
>    k_27 = k_36 + 1;
>    if (n_20(D) > k_27)
>      goto <bb 14>; [89.00%]
>    else
>      goto <bb 6>; [11.00%]
> 
>    <bb 6> [local count: 118111600]:
>    j_21 = j_35 + 1;
>    if (n_20(D) > j_21)
>      goto <bb 13>; [89.00%]
>    else
>      goto <bb 8>; [11.00%]
> 
>    <bb 13> [local count: 105119324]:
> 
>    <bb 7> [local count: 118111600]:   // L3 preheader
>    # j_35 = PHI <j_21(13), 0(11)>
>    goto <bb 5>; [100.00%]
> 
>    <bb 8> [local count: 145980626]:
>    *_10 = t_24(D);
>    i_29 = i_34 + 1;
> 
> Re-paste the bb_colder_than_loop_preheader and find_coldest_out_loop:
> 
> +/* Compare the profile count inequality of COUNT1 and COUNT2, it is three-state
> +   as stated in profile-count.h, FALSE is returned if inequality cannot be
> +   decided.  */
> +bool bb_colder_than_loop_preheader (profile_count count1, profile_count count2)
> +{
> +  if (count1 < count2)
> +    return true;
> +  else
> +    return false;
> +}
> +
> +/* Find coldest loop between OUTMOST_LOOP and LOOP by comapring profile count.
> + */
> +
> +static class loop *
> +find_coldest_out_loop (class loop *outmost_loop, class loop *loop,
> +		       basic_block curr_bb)
> +{
> +  class loop *cold_loop, *min_loop;
> +  cold_loop = min_loop = outmost_loop;
> +  profile_count min_count = loop_preheader_edge (min_loop)->src->count;
> +
> +  /* If bb_colder_than_loop_preheader returns false due to three-state
> +    comparision, OUTMOST_LOOP is returned finally to preserve the behavior.
> +    Otherwise, return the coldest loop between OUTMOST_LOOP and LOOP.  */
> +  if (curr_bb
> +      && bb_colder_than_loop_preheader (curr_bb->count,
> +					loop_preheader_edge (loop)->src->count))
> +    return NULL;
> +
> +  while (min_loop != loop)
> +    {
> +      min_loop = superloop_at_depth (loop, loop_depth (min_loop) + 1);
> +      if (bb_colder_than_loop_preheader (
> +	    loop_preheader_edge (min_loop)->src->count, min_count))
> +	cold_loop = min_loop;
> +    }
> +  return cold_loop;
> +}
> +
> 
> 
>>
>> Why is this function not simply
>>
>> +static class loop *
>> +find_coldest_out_loop (class loop *outmost_loop, class loop *loop,
>> +                      basic_block curr_bb)
>> +{
>>      while (bb_colder_than_loop_preheader (curr_bb->count,
>>                loop_preheader_edge (outermost_loop)->src->count))
>>         {
>>             if (outermost_loop == loop)
>>               return NULL;
>>             outermost_loop = superloop_at_depth (loop, loop_depth
>> (outermost_loop) + 1);
>>         }
>>      return outermost_loop;
>> }
> 
> If change like this, when processing curr_bb(5), outermost_loop will
> return loop 1 since curr_bb->count > Loop1_prehead->count, the while
> loop stopped.  This doesn't meet what we want.  
> 
>>
>> ?
>>
>> Likewise I wonder why ref_in_loop_hot_body::operator () needs to call
>> find_coldest_out_loop and why it not simply does
>>
>> +bool
>> +ref_in_loop_hot_body::operator () (mem_ref_loc *loc)
>> +{
>> +  basic_block curr_bb = gimple_bb (loc->stmt);
>>     if (bb_colder_than_loop_preheader (curr_bb->count,
>> loop_preheader_edge (l)->src->count))
>>       return false;
>>    return true;
>>    }
> 
> Likely for this part,
> 
> +/* Find out the coldest loop between loop L and innermost loop, compare the
> +   hotness between current BB and coldest loop preheader by profile count.  */
> +bool
> +ref_in_loop_hot_body::operator () (mem_ref_loc *loc)
> +{
> +  basic_block curr_bb = gimple_bb (loc->stmt);
> +  class loop *inner_loop = curr_bb->loop_father;
> +  class loop *cold_loop = l;
> +  if (l != inner_loop)
> +    cold_loop = find_coldest_out_loop (l, inner_loop, curr_bb);
> +  if (!cold_loop)
> +    return false;
> +  edge e = loop_preheader_edge (cold_loop);
> +  /*  If bb_colder_than_loop_preheader is false due to three-state inequality
> +     comparision, TRUE is returned to continue perform store motion.  */
> +  if (bb_colder_than_loop_preheader (curr_bb->count, e->src->count))
> +    return false;
> +  else
> +    return true;
> +}
> 
> l is the input of ref_in_loop_hot_body, it is an out loop, we need to find a
> cold_loop between l and inner_loop.  Reason is there may be cold loop between
> l and inner_loop, which means we shouldn't do store-motion from curr_bb to l
> directly.
> After reconsideration, I think the bb_colder_than_loop_preheader could be
>  removed since curr_bb is checked in find_coldest_out_loop already.  And remove
> the "l != inner_loop" check:
> 
> +/* Find out the coldest loop between loop L and innermost loop.  */
> +bool
> +ref_in_loop_hot_body::operator () (mem_ref_loc *loc)
> +{
> +  basic_block curr_bb = gimple_bb (loc->stmt);
> +  class loop *inner_loop = curr_bb->loop_father;
> +  class loop *cold_loop = l;
> +  cold_loop = find_coldest_out_loop (l, inner_loop, curr_bb);
> +  if (!cold_loop)
> +    return false;
> +  return true;
> +}
> 
> 

Refined the functions and comments for better understanding in v5:


[PATCH v5 2/2] Don't move cold code out of loop by checking bb count


v5 changes:
1. Refine comments for new functions.
2. Use basic_block instead of count in bb_colder_than_loop_preheader
to align with function name.
3. Refine with simpler implementation for find_coldest_out_loop and
ref_in_loop_hot_body::operator for better understanding.

v4 changes:
1. Sort out profile_count comparision to function bb_cold_than_loop_preheader.
2. Update ref_in_loop_hot_body::operator () to find cold_loop before compare.
3. Split RTL invariant motion part out.
4. Remove aux changes.

v3 changes:
1. Handle max_loop in determine_max_movement instead of outermost_invariant_loop.
2. Remove unnecessary changes.
3. Add for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body) in can_sm_ref_p.
4. "gsi_next (&bsi);" in move_computations_worker is kept since it caused
infinite loop when implementing v1 and the iteration is missed to be
updated actually.

v1: https://gcc.gnu.org/pipermail/gcc-patches/2021-August/576488.html
v2: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/579086.html
v3: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/580211.html

There was a patch trying to avoid move cold block out of loop:

https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html

Richard suggested to "never hoist anything from a bb with lower execution
frequency to a bb with higher one in LIM invariantness_dom_walker
before_dom_children".

In gimple LIM analysis, add find_coldest_out_loop to move invariants to
expected target loop, if profile count of the loop bb is colder
than target loop preheader, it won't be hoisted out of loop.
Likely for store motion, if all locations of the REF in loop is cold,
don't do store motion of it.

SPEC2017 performance evaluation shows 1% performance improvement for
intrate GEOMEAN and no obvious regression for others.  Especially,
500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
on P8LE.

gcc/ChangeLog:

	* tree-ssa-loop-im.c (bb_colder_than_loop_preheader): New
	function.
	(find_coldest_out_loop): New function.
	(determine_max_movement): Use find_coldest_out_loop.
	(move_computations_worker): Adjust and fix iteration udpate.
	(class ref_in_loop_hot_body): New functor.
	(ref_in_loop_hot_body::operator): New.
	(can_sm_ref_p): Use for_all_locs_in_loop.

gcc/testsuite/ChangeLog:

	* gcc.dg/tree-ssa/recip-3.c: Adjust.
	* gcc.dg/tree-ssa/ssa-lim-18.c: New test.
	* gcc.dg/tree-ssa/ssa-lim-19.c: New test.
	* gcc.dg/tree-ssa/ssa-lim-20.c: New test.
	* gcc.dg/tree-ssa/ssa-lim-21.c: New test.
---
 gcc/tree-ssa-loop-im.c                     | 78 +++++++++++++++++++++-
 gcc/testsuite/gcc.dg/tree-ssa/recip-3.c    |  2 +-
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c | 20 ++++++
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c | 29 ++++++++
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c | 25 +++++++
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c | 35 ++++++++++
 6 files changed, 186 insertions(+), 3 deletions(-)
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c

diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
index 4b187c2cdaf..6de9be7d4e5 100644
--- a/gcc/tree-ssa-loop-im.c
+++ b/gcc/tree-ssa-loop-im.c
@@ -417,6 +417,49 @@ movement_possibility (gimple *stmt)
   return ret;
 }
 
+/* Compare the profile count inequality of bb and preheader, it is three-state
+   as stated in profile-count.h, FALSE is returned if inequality cannot be
+   decided.  */
+bool bb_colder_than_loop_preheader (basic_block bb, basic_block preheader)
+{
+  gcc_assert (bb && preheader);
+  return bb->count < preheader->count;
+}
+
+/* Find coldest loop between OUTMOST_LOOP and LOOP by comparing profile count.
+   It does two steps check:
+   1) Check whether CURR_BB is cold in it's own loop_father, if it is cold, just
+   return NULL which means it should not be moved out at all;
+   2)  CURR_BB is NOT cold, set LOOP to cold_loop, then iteratively search loops
+   from {L[outmost_loop], L[outmost_loop+1], ... L[loop]}, if L[i] is colder
+   than L[cold_loop], reset cold_loop to L[i] until get the loop that has
+   smallest profile_count.  */
+
+static class loop *
+find_coldest_out_loop (class loop *outmost_loop, class loop *loop,
+		       basic_block curr_bb)
+{
+  class loop *cold_loop;
+
+  /* If bb_colder_than_loop_preheader returns false due to three-state
+    comparision, OUTMOST_LOOP is returned finally to preserve the behavior.
+    Otherwise, return the coldest loop between OUTMOST_LOOP and LOOP.  */
+  if (curr_bb
+      && bb_colder_than_loop_preheader (curr_bb,
+					loop_preheader_edge (loop)->src))
+    return NULL;
+
+  cold_loop = loop;
+  while (outmost_loop != loop)
+    {
+      if (bb_colder_than_loop_preheader (loop_preheader_edge (outmost_loop)->src,
+					 loop_preheader_edge (cold_loop)->src))
+	cold_loop = outmost_loop;
+      outmost_loop = superloop_at_depth (loop, loop_depth (outmost_loop) + 1);
+    }
+  return cold_loop;
+}
+
 /* Suppose that operand DEF is used inside the LOOP.  Returns the outermost
    loop to that we could move the expression using DEF if it did not have
    other operands, i.e. the outermost loop enclosing LOOP in that the value
@@ -685,7 +728,9 @@ determine_max_movement (gimple *stmt, bool must_preserve_exec)
     level = ALWAYS_EXECUTED_IN (bb);
   else
     level = superloop_at_depth (loop, 1);
-  lim_data->max_loop = level;
+  lim_data->max_loop = find_coldest_out_loop (level, loop, bb);
+  if (!lim_data->max_loop)
+    return false;
 
   if (gphi *phi = dyn_cast <gphi *> (stmt))
     {
@@ -1221,7 +1266,10 @@ move_computations_worker (basic_block bb)
       /* We do not really want to move conditionals out of the loop; we just
 	 placed it here to force its operands to be moved if necessary.  */
       if (gimple_code (stmt) == GIMPLE_COND)
-	continue;
+	{
+	  gsi_next (&bsi);
+	  continue;
+	}
 
       if (dump_file && (dump_flags & TDF_DETAILS))
 	{
@@ -2887,6 +2935,26 @@ ref_indep_loop_p (class loop *loop, im_mem_ref *ref, dep_kind kind)
   return indep_p;
 }
 
+class ref_in_loop_hot_body
+{
+public:
+  ref_in_loop_hot_body (loop *loop_) : l (loop_) {}
+  bool operator () (mem_ref_loc *loc);
+  class loop *l;
+};
+
+/* Find the coldest loop between loop L and innermost loop.  If there is one
+   cold loop between L and INNER_LOOP, store motion can be performed, otherwise
+   no cold loop means no store motion.  find_coldest_out_loop also handles cases
+   when l is inner_loop.  */
+bool
+ref_in_loop_hot_body::operator () (mem_ref_loc *loc)
+{
+  basic_block curr_bb = gimple_bb (loc->stmt);
+  class loop *inner_loop = curr_bb->loop_father;
+  return find_coldest_out_loop (l, inner_loop, curr_bb);
+}
+
 
 /* Returns true if we can perform store motion of REF from LOOP.  */
 
@@ -2941,6 +3009,12 @@ can_sm_ref_p (class loop *loop, im_mem_ref *ref)
   if (!ref_indep_loop_p (loop, ref, sm_war))
     return false;
 
+  /* Verify whether the candidate is hot for LOOP.  Only do store motion if the
+    candidate's profile count is hot.  Statement in cold BB shouldn't be moved
+    out of it's loop_father.  */
+  if (!for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body (loop)))
+    return false;
+
   return true;
 }
 
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
index 638bf38db8c..641c91e719e 100644
--- a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
+++ b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
@@ -23,4 +23,4 @@ float h ()
 	F[0] += E / d;
 }
 
-/* { dg-final { scan-tree-dump-times " / " 1 "recip" } } */
+/* { dg-final { scan-tree-dump-times " / " 5 "recip" } } */
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
new file mode 100644
index 00000000000..7326a230b3f
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
@@ -0,0 +1,20 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+volatile int x;
+void
+bar (int, char *, char *);
+void
+foo (int *a, int n, int k)
+{
+  int i;
+
+  for (i = 0; i < n; i++)
+    {
+      if (__builtin_expect (x, 0))
+	bar (k / 5, "one", "two");
+      a[i] = k;
+    }
+}
+
+/* { dg-final { scan-tree-dump-not "out of loop 1" "lim2" } } */
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
new file mode 100644
index 00000000000..51c1913d003
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
@@ -0,0 +1,29 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+volatile int x;
+void
+bar (int, char *, char *);
+void
+foo (int *a, int n, int m, int s, int t)
+{
+  int i;
+  int j;
+  int k;
+
+  for (i = 0; i < m; i++) // Loop 1
+    {
+      if (__builtin_expect (x, 0))
+	for (j = 0; j < n; j++) // Loop 2
+	  for (k = 0; k < n; k++) // Loop 3
+	    {
+	      bar (s / 5, "one", "two");
+	      a[t] = s;
+	    }
+      a[t] = t;
+    }
+}
+
+/* { dg-final { scan-tree-dump-times "out of loop 2" 4 "lim2" } } */
+/* { dg-final { scan-tree-dump-times "out of loop 1" 3 "lim2" } } */
+
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
new file mode 100644
index 00000000000..bc60a040a70
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
@@ -0,0 +1,25 @@
+/* { dg-do compile  } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+/* Test that `count' is not hoisted out of loop when bb is cold.  */
+
+int count;
+volatile int x;
+
+struct obj {
+  int data;
+  struct obj *next;
+
+} *q;
+
+void
+func (int m)
+{
+  struct obj *p;
+  for (int i = 0; i < m; i++)
+    if (__builtin_expect (x, 0))
+      count++;
+
+}
+
+/* { dg-final { scan-tree-dump-not "Executing store motion of" "lim2"  }  } */
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
new file mode 100644
index 00000000000..d843fc55368
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
@@ -0,0 +1,35 @@
+/* { dg-do compile  } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+/* Test that `data' and 'data1' is not hoisted out of inner loop and outer loop
+   when it is in cold loop.  */
+
+int count;
+volatile int x;
+
+struct obj {
+  int data;
+  int data1;
+  struct obj *next;
+};
+
+void
+func (int m, int n, int k, struct obj *a)
+{
+  struct obj *q = a;
+  for (int j = 0; j < m; j++)
+    if (__builtin_expect (m, 0))
+      for (int i = 0; i < m; i++)
+	{
+	  if (__builtin_expect (x, 0))
+	    {
+	      count++;
+	      q->data += 3; /* Not hoisted out to inner loop. */
+	    }
+	  count += n;
+	  q->data1 += k; /* Not hoisted out to outer loop. */
+	}
+}
+
+/* { dg-final { scan-tree-dump-not "Executing store motion of" "lim2"  }  } */
+
-- 
2.27.0.90.geebb51ba8c

Re: [RFC] Don't move cold code out of loop by checking bb count

[Richard Biener]

On Mon, Oct 18, 2021 at 6:29 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>
>
>
> On 2021/10/15 16:11, Richard Biener wrote:
> > On Sat, Oct 9, 2021 at 5:45 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
> >>
> >> Hi,
> >>
> >> On 2021/9/28 20:09, Richard Biener wrote:
> >>> On Fri, Sep 24, 2021 at 8:29 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
> >>>>
> >>>> Update the patch to v3, not sure whether you prefer the paste style
> >>>> and continue to link the previous thread as Segher dislikes this...
> >>>>
> >>>>
> >>>> [PATCH v3] Don't move cold code out of loop by checking bb count
> >>>>
> >>>>
> >>>> Changes:
> >>>> 1. Handle max_loop in determine_max_movement instead of
> >>>> outermost_invariant_loop.
> >>>> 2. Remove unnecessary changes.
> >>>> 3. Add for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body) in can_sm_ref_p.
> >>>> 4. "gsi_next (&bsi);" in move_computations_worker is kept since it caused
> >>>> infinite loop when implementing v1 and the iteration is missed to be
> >>>> updated actually.
> >>>>
> >>>> v1: https://gcc.gnu.org/pipermail/gcc-patches/2021-August/576488.html
> >>>> v2: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/579086.html
> >>>>
> >>>> There was a patch trying to avoid move cold block out of loop:
> >>>>
> >>>> https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html
> >>>>
> >>>> Richard suggested to "never hoist anything from a bb with lower execution
> >>>> frequency to a bb with higher one in LIM invariantness_dom_walker
> >>>> before_dom_children".
> >>>>
> >>>> In gimple LIM analysis, add find_coldest_out_loop to move invariants to
> >>>> expected target loop, if profile count of the loop bb is colder
> >>>> than target loop preheader, it won't be hoisted out of loop.
> >>>> Likely for store motion, if all locations of the REF in loop is cold,
> >>>> don't do store motion of it.
> >>>>
> >>>> SPEC2017 performance evaluation shows 1% performance improvement for
> >>>> intrate GEOMEAN and no obvious regression for others.  Especially,
> >>>> 500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
> >>>> largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
> >>>> on P8LE.
> >>>>
> >>>> gcc/ChangeLog:
> >>>>
> >>>>         * loop-invariant.c (find_invariants_bb): Check profile count
> >>>>         before motion.
> >>>>         (find_invariants_body): Add argument.
> >>>>         * tree-ssa-loop-im.c (find_coldest_out_loop): New function.
> >>>>         (determine_max_movement): Use find_coldest_out_loop.
> >>>>         (move_computations_worker): Adjust and fix iteration udpate.
> >>>>         (execute_sm_exit): Check pointer validness.
> >>>>         (class ref_in_loop_hot_body): New functor.
> >>>>         (ref_in_loop_hot_body::operator): New.
> >>>>         (can_sm_ref_p): Use for_all_locs_in_loop.
> >>>>
> >>>> gcc/testsuite/ChangeLog:
> >>>>
> >>>>         * gcc.dg/tree-ssa/recip-3.c: Adjust.
> >>>>         * gcc.dg/tree-ssa/ssa-lim-18.c: New test.
> >>>>         * gcc.dg/tree-ssa/ssa-lim-19.c: New test.
> >>>>         * gcc.dg/tree-ssa/ssa-lim-20.c: New test.
> >>>> ---
> >>>>  gcc/loop-invariant.c                       | 10 ++--
> >>>>  gcc/tree-ssa-loop-im.c                     | 61 ++++++++++++++++++++--
> >>>>  gcc/testsuite/gcc.dg/tree-ssa/recip-3.c    |  2 +-
> >>>>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c | 20 +++++++
> >>>>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c | 27 ++++++++++
> >>>>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c | 25 +++++++++
> >>>>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c | 28 ++++++++++
> >>>>  7 files changed, 165 insertions(+), 8 deletions(-)
> >>>>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
> >>>>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
> >>>>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
> >>>>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
> >>>>
> >>>> diff --git a/gcc/loop-invariant.c b/gcc/loop-invariant.c
> >>>> index fca0c2b24be..5c3be7bf0eb 100644
> >>>> --- a/gcc/loop-invariant.c
> >>>> +++ b/gcc/loop-invariant.c
> >>>> @@ -1183,9 +1183,14 @@ find_invariants_insn (rtx_insn *insn, bool always_reached, bool always_executed)
> >>>>     call.  */
> >>>>
> >>>>  static void
> >>>> -find_invariants_bb (basic_block bb, bool always_reached, bool always_executed)
> >>>> +find_invariants_bb (class loop *loop, basic_block bb, bool always_reached,
> >>>> +                   bool always_executed)
> >>>>  {
> >>>>    rtx_insn *insn;
> >>>> +  basic_block preheader = loop_preheader_edge (loop)->src;
> >>>> +
> >>>> +  if (preheader->count > bb->count)
> >>>> +    return;
> >>>>
> >>>>    FOR_BB_INSNS (bb, insn)
> >>>>      {
> >>>> @@ -1214,8 +1219,7 @@ find_invariants_body (class loop *loop, basic_block *body,
> >>>>    unsigned i;
> >>>>
> >>>>    for (i = 0; i < loop->num_nodes; i++)
> >>>> -    find_invariants_bb (body[i],
> >>>> -                       bitmap_bit_p (always_reached, i),
> >>>> +    find_invariants_bb (loop, body[i], bitmap_bit_p (always_reached, i),
> >>>>                         bitmap_bit_p (always_executed, i));
> >>>>  }
> >>>>
> >>>> diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
> >>>> index 4b187c2cdaf..655fab03442 100644
> >>>> --- a/gcc/tree-ssa-loop-im.c
> >>>> +++ b/gcc/tree-ssa-loop-im.c
> >>>> @@ -417,6 +417,28 @@ movement_possibility (gimple *stmt)
> >>>>    return ret;
> >>>>  }
> >>>>
> >>>> +/* Find coldest loop between outmost_loop and loop by comapring profile count.  */
> >>>> +
> >>>> +static class loop *
> >>>> +find_coldest_out_loop (class loop *outmost_loop, class loop *loop,
> >>>> +                      basic_block curr_bb)
> >>>> +{
> >>>> +  class loop *cold_loop, *min_loop;
> >>>> +  cold_loop = min_loop = outmost_loop;
> >>>> +  profile_count min_count = loop_preheader_edge (min_loop)->src->count;
> >>>> +
> >>>> +  if (curr_bb && curr_bb->count < loop_preheader_edge (loop)->src->count)
> >>>
> >>> Honza - can you comment on whether we should compare BB counts this way?
> >>>
> >>> I would suspect that for, say,
> >>>
> >>>   for (...)
> >>>      if (a)
> >>>        X;
> >>>      else
> >>>        Y;
> >>>
> >>> that the counts for X and Y will be less than that of the preheader of the loop
> >>> only when the loop is estimated to run once.  That is, should we really compare
> >>> the to the preheader count or maybe better to the _header_ count which
> >>> would keep the number of iterations out of the equation?
> >>
> >> I quickly tried to replace all the loop_preheader_edge (loop)->src with
> >> loop_preheader_edge (loop)->dest, it will cause many failures in
> >> gcc.dg/tree-ssa/ssa-lim-*.c, I didn't go deep to investigate, but it seems
> >> reasonable to compare the bb count with preheader count as both gimple lim
> >> and RTL loop-invariant move instructions to *preheader* instead of *header*
> >> after analysis?
> >
> > Hmm, yeah - guess I was confused here.
> >
> >>>
> >>> If we look at maybe_hot_count_p that's a quite sophisticated thing to
> >>> compare a count to the "IPA hot", here we're comparing two counts
> >>> within a function where it actually matters whether we use a<b or
> >>> !(a>=b) since 'unordered' is mapped to false (but there's no ordered_p
> >>> function).
> >>>
> >>> Xionghu, you error on the side of not hoisting for unordered counts here
> >>>
> >>>> +    return NULL;
> >>>> +
> >>>> +  while (min_loop != loop)
> >>>> +    {
> >>>> +      min_loop = superloop_at_depth (loop, loop_depth (min_loop) + 1);
> >>>> +      if (loop_preheader_edge (min_loop)->src->count < min_count)
> >>>
> >>> but in the other direction here and on the side of not hoisting
> >>> in ref_in_loop_hot_body.
> >>>
> >>> The three-state relational operator overloads are probably not the
> >>> very best idea...
> >>> (see profile-count.h for them)
> >>>
> >> Added new function bb_colder_than_loop_preheader to encapsulate the comparision,
> >> if FALSE is returned due to three-state inequality,  find_coldest_out_loop
> >> will return the original input to lim->max_loop, and ref_in_loop_hot_body::operator ()
> >> will return true to continue perform store motion, both preserve the previous
> >> behavior.
> >
> > Thanks.  But I don't think the abstraction as written is useful:
> >
> > +/* Compare the profile count inequality of COUNT1 and COUNT2, it is three-state
> > +   as stated in profile-count.h, FALSE is returned if inequality cannot be
> > +   decided.  */
> > +bool bb_colder_than_loop_preheader (profile_count count1, profile_count count2)
> > +{
> > +  if (count1 < count2)
> > +    return true;
> > +  else
> > +    return false;
> > +}
> >
> > given the following seems to pass the preheader count in place of the BB count.
> >
> > +      if (bb_colder_than_loop_preheader (
> > +           loop_preheader_edge (min_loop)->src->count, min_count))
> > +       cold_loop = min_loop;
> >
> > find_coldest_out_loop is also a bit weird, I think we want to find
> > the outermost loop between outmost_loop and loop that has a
> > lower count than the curr_bb count but
> >
> > +  while (min_loop != loop)
> > +    {
> > +      min_loop = superloop_at_depth (loop, loop_depth (min_loop) + 1);
> > +      if (bb_colder_than_loop_preheader (
> > +           loop_preheader_edge (min_loop)->src->count, min_count))
> > +       cold_loop = min_loop;
> >
> > compares the outermost loops count (min_count) against the preheader
> > count?  So we're searching for a cold loop with respect to its enclosing loop
> > here?
>
> Let me try to explain how it works :)
>
> find_coldest_out_loop does two steps check:
> 1) Check whether curr_bb is cold in it's own loop_father, if it is cold,
> just return NULL which means it should not be moved out at all;
> 2)  curr_bb is NOT cold, assuming the current loop L[m] is the coldest first,
> than try to find a cold loop to be hoisted to from {L[1], L[2], ... L[m]},
> if L[i]->count < L[m]->count, set the cold_loop to L[i] until find the loop
> that has smallest profile_count.
>
>
> Take the updated ssa-lim-19.c as example, check whether curr_bb(bb 5) is cold in
> loop 3, if it is cold, just return NULL, otherwise select the coldest loop in
> {loop1, loop2, loop3} and find that loop2 is colder than loop3, return loop2 to
> be the target hoist loop.  The first check could AVOID hoist if curr_bb is colder
> than loop3, but it is still hot than loop1 and loop2.  Not sure whether it is possible
> to construct such cases?
>
>
> gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
>
> volatile int x;
> void
> bar (int, char *, char *);
> void
> foo (int *a, int n, int m, int s, int t)
> {
>   int i;
>   int j;
>   int k;
>
>   for (i = 0; i < m; i++)  // loop 1
>     {
>       if (__builtin_expect (x, 0))
>         for (j = 0; j < n; j++)   // loop 2
>           for (k = 0; k < n; k++)   // loop 3
>            {
>              bar (s / 5, "one", "two");  // curr_bb
>              a[t] = s;
>            }
>       a[t] = t;  // curr_bb2
>     }
> }
>
> The 4 invariant statements are moved to bb 11(loop2) instead of bb 10(loop1)
> with this patch.
> There are totally 6 combinations when curr_bb is hotter than loop 3.  We need
> to compare the "Loop preheader hotness" instead of "every Loop[i] and curr_bb hotness",
> returning the coldest loop for this function find_coldest_out_loop, otherwise
> unexpected behavior happens.
>
> L1 > L2 > L3   =>  return L3
> L1 > L3 > L2   =>  return L2
> L2 > L1 > L3   =>  return L3
> L2 > L3 > L1   =>  return L1
> L3 > L1 > L2   =>  return L2
> L3 > L2 > L1   =>  return L1
>
> So bb_colder_than_loop_preheader does two kind of checks, one is checking
> L3 preheader count with curr_bb count, another is checking L3 preheader count
> with L1 preheader count, L2 preheader count, etc...
>
>
> ssa-lim-19.c.138t.lim2:
> ...
>    <bb 10> [local count: 16057869]:  // L1 preheader
> -  _4 = s_22(D) / 5;
> -  _5 = (long unsigned int) t_24(D);
> -  _6 = _5 * 4;
> -  _7 = a_25(D) + _6;
>    _8 = (long unsigned int) t_24(D);
>    _9 = _8 * 4;
>    _10 = a_25(D) + _9;
>
>    <bb 3> [local count: 145980626]:
>    # i_34 = PHI <i_29(12), 0(10)>
>    x.0_1 ={v} x;
>    if (x.0_1 != 0)
>      goto <bb 4>; [10.00%]
>    else
>      goto <bb 8>; [90.00%]
>
>    <bb 4> [local count: 14598063]:
>    if (n_20(D) > 0)
>      goto <bb 11>; [89.00%]
>    else
>      goto <bb 8>; [11.00%]
>
>    <bb 11> [local count: 12992276]:  // L2 preheader
> +  _4 = s_22(D) / 5;
> +  _5 = (long unsigned int) t_24(D);
> +  _6 = _5 * 4;
> +  _7 = a_25(D) + _6;
>    goto <bb 7>; [100.00%]
>
>    <bb 14> [local count: 850510901]:
>
>    <bb 5> [local count: 955630225]:  // curr_bb
>    # k_36 = PHI <k_27(14), 0(7)>
>    bar (_4, "one", "two");
>    *_7 = s_22(D);
>    k_27 = k_36 + 1;
>    if (n_20(D) > k_27)
>      goto <bb 14>; [89.00%]
>    else
>      goto <bb 6>; [11.00%]
>
>    <bb 6> [local count: 118111600]:
>    j_21 = j_35 + 1;
>    if (n_20(D) > j_21)
>      goto <bb 13>; [89.00%]
>    else
>      goto <bb 8>; [11.00%]
>
>    <bb 13> [local count: 105119324]:
>
>    <bb 7> [local count: 118111600]:   // L3 preheader
>    # j_35 = PHI <j_21(13), 0(11)>
>    goto <bb 5>; [100.00%]
>
>    <bb 8> [local count: 145980626]:
>    *_10 = t_24(D);
>    i_29 = i_34 + 1;
>
> Re-paste the bb_colder_than_loop_preheader and find_coldest_out_loop:
>
> +/* Compare the profile count inequality of COUNT1 and COUNT2, it is three-state
> +   as stated in profile-count.h, FALSE is returned if inequality cannot be
> +   decided.  */
> +bool bb_colder_than_loop_preheader (profile_count count1, profile_count count2)
> +{
> +  if (count1 < count2)
> +    return true;
> +  else
> +    return false;
> +}
> +
> +/* Find coldest loop between OUTMOST_LOOP and LOOP by comapring profile count.
> + */
> +
> +static class loop *
> +find_coldest_out_loop (class loop *outmost_loop, class loop *loop,
> +                      basic_block curr_bb)
> +{
> +  class loop *cold_loop, *min_loop;
> +  cold_loop = min_loop = outmost_loop;
> +  profile_count min_count = loop_preheader_edge (min_loop)->src->count;
> +
> +  /* If bb_colder_than_loop_preheader returns false due to three-state
> +    comparision, OUTMOST_LOOP is returned finally to preserve the behavior.
> +    Otherwise, return the coldest loop between OUTMOST_LOOP and LOOP.  */
> +  if (curr_bb
> +      && bb_colder_than_loop_preheader (curr_bb->count,
> +                                       loop_preheader_edge (loop)->src->count))
> +    return NULL;
> +
> +  while (min_loop != loop)
> +    {
> +      min_loop = superloop_at_depth (loop, loop_depth (min_loop) + 1);
> +      if (bb_colder_than_loop_preheader (
> +           loop_preheader_edge (min_loop)->src->count, min_count))
> +       cold_loop = min_loop;
> +    }
> +  return cold_loop;
> +}
> +
>
>
> >
> > Why is this function not simply
> >
> > +static class loop *
> > +find_coldest_out_loop (class loop *outmost_loop, class loop *loop,
> > +                      basic_block curr_bb)
> > +{
> >      while (bb_colder_than_loop_preheader (curr_bb->count,
> >                loop_preheader_edge (outermost_loop)->src->count))
> >         {
> >             if (outermost_loop == loop)
> >               return NULL;
> >             outermost_loop = superloop_at_depth (loop, loop_depth
> > (outermost_loop) + 1);
> >         }
> >      return outermost_loop;
> > }
>
> If change like this, when processing curr_bb(5), outermost_loop will
> return loop 1 since curr_bb->count > Loop1_prehead->count, the while
> loop stopped.  This doesn't meet what we want.

Why?  curr_bb is executed at least as often as loop1 preheader if
we look at the counts?  So either the counts do not really tell us
anything of help or I am missing something.  Are you merely
looking for a block with a lower count on the path from the outermost
loop entry to the block in question and deciding you do not want to
hoist further than that?  So it's not about not hoisting to a hot place
but instead hoist to the coldest place within a loop nest?

So we have

  for (i = 0; i < m; i++)  // loop 1
    {
      if (__builtin_expect (x, 0))
        for (j = 0; j < n; j++)   // loop 2


   <bb 10> [local count: 16057869]:  // L1 preheader
       ...
 <bb 3> [local count: 145980626]:
   # i_34 = PHI <i_29(12), 0(10)>
 ...
   <bb 11> [local count: 12992276]:  // L2 preheader
   ...
    <bb 7> [local count: 118111600]:   // L3 preheader
   # j_35 = PHI <j_21(13), 0(11)>
   goto <bb 5>; [100.00%]

and we want to hoist to the L2 preheader because that's less frequently
executed than the L1 preheader (which is less frequently executed
than the L3 preheader or the block we are hoisting from).

I'm concerned with compile-time complexity re-evaluating counts on the
loop nest many times.  So it looks to me that we can pre-compute
this lowest-preheader-count loop for a loop nest at least for the
store-motion case where we know the outermost loop?


> >
> > ?
> >
> > Likewise I wonder why ref_in_loop_hot_body::operator () needs to call
> > find_coldest_out_loop and why it not simply does
> >
> > +bool
> > +ref_in_loop_hot_body::operator () (mem_ref_loc *loc)
> > +{
> > +  basic_block curr_bb = gimple_bb (loc->stmt);
> >     if (bb_colder_than_loop_preheader (curr_bb->count,
> > loop_preheader_edge (l)->src->count))
> >       return false;
> >    return true;
> >    }
>
> Likely for this part,
>
> +/* Find out the coldest loop between loop L and innermost loop, compare the
> +   hotness between current BB and coldest loop preheader by profile count.  */
> +bool
> +ref_in_loop_hot_body::operator () (mem_ref_loc *loc)
> +{
> +  basic_block curr_bb = gimple_bb (loc->stmt);
> +  class loop *inner_loop = curr_bb->loop_father;
> +  class loop *cold_loop = l;
> +  if (l != inner_loop)
> +    cold_loop = find_coldest_out_loop (l, inner_loop, curr_bb);
> +  if (!cold_loop)
> +    return false;
> +  edge e = loop_preheader_edge (cold_loop);
> +  /*  If bb_colder_than_loop_preheader is false due to three-state inequality
> +     comparision, TRUE is returned to continue perform store motion.  */
> +  if (bb_colder_than_loop_preheader (curr_bb->count, e->src->count))
> +    return false;
> +  else
> +    return true;
> +}
>
> l is the input of ref_in_loop_hot_body, it is an out loop, we need to find a
> cold_loop between l and inner_loop.  Reason is there may be cold loop between
> l and inner_loop, which means we shouldn't do store-motion from curr_bb to l
> directly.
> After reconsideration, I think the bb_colder_than_loop_preheader could be
>  removed since curr_bb is checked in find_coldest_out_loop already.  And remove
> the "l != inner_loop" check:
>
> +/* Find out the coldest loop between loop L and innermost loop.  */
> +bool
> +ref_in_loop_hot_body::operator () (mem_ref_loc *loc)
> +{
> +  basic_block curr_bb = gimple_bb (loc->stmt);
> +  class loop *inner_loop = curr_bb->loop_father;
> +  class loop *cold_loop = l;
> +  cold_loop = find_coldest_out_loop (l, inner_loop, curr_bb);
> +  if (!cold_loop)
> +    return false;
> +  return true;
> +}
>
>
> --
> Thanks,
> Xionghu

Re: [RFC] Don't move cold code out of loop by checking bb count

[Xionghu Luo]

On 2021/10/26 21:20, Richard Biener wrote:
> On Mon, Oct 18, 2021 at 6:29 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>>
>>
>>
>> On 2021/10/15 16:11, Richard Biener wrote:
>>> On Sat, Oct 9, 2021 at 5:45 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> On 2021/9/28 20:09, Richard Biener wrote:
>>>>> On Fri, Sep 24, 2021 at 8:29 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>>>>>>
>>>>>> Update the patch to v3, not sure whether you prefer the paste style
>>>>>> and continue to link the previous thread as Segher dislikes this...
>>>>>>
>>>>>>
>>>>>> [PATCH v3] Don't move cold code out of loop by checking bb count
>>>>>>
>>>>>>
>>>>>> Changes:
>>>>>> 1. Handle max_loop in determine_max_movement instead of
>>>>>> outermost_invariant_loop.
>>>>>> 2. Remove unnecessary changes.
>>>>>> 3. Add for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body) in can_sm_ref_p.
>>>>>> 4. "gsi_next (&bsi);" in move_computations_worker is kept since it caused
>>>>>> infinite loop when implementing v1 and the iteration is missed to be
>>>>>> updated actually.
>>>>>>
>>>>>> v1: https://gcc.gnu.org/pipermail/gcc-patches/2021-August/576488.html
>>>>>> v2: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/579086.html
>>>>>>
>>>>>> There was a patch trying to avoid move cold block out of loop:
>>>>>>
>>>>>> https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html
>>>>>>
>>>>>> Richard suggested to "never hoist anything from a bb with lower execution
>>>>>> frequency to a bb with higher one in LIM invariantness_dom_walker
>>>>>> before_dom_children".
>>>>>>
>>>>>> In gimple LIM analysis, add find_coldest_out_loop to move invariants to
>>>>>> expected target loop, if profile count of the loop bb is colder
>>>>>> than target loop preheader, it won't be hoisted out of loop.
>>>>>> Likely for store motion, if all locations of the REF in loop is cold,
>>>>>> don't do store motion of it.
>>>>>>
>>>>>> SPEC2017 performance evaluation shows 1% performance improvement for
>>>>>> intrate GEOMEAN and no obvious regression for others.  Especially,
>>>>>> 500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
>>>>>> largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
>>>>>> on P8LE.
>>>>>>
>>>>>> gcc/ChangeLog:
>>>>>>
>>>>>>         * loop-invariant.c (find_invariants_bb): Check profile count
>>>>>>         before motion.
>>>>>>         (find_invariants_body): Add argument.
>>>>>>         * tree-ssa-loop-im.c (find_coldest_out_loop): New function.
>>>>>>         (determine_max_movement): Use find_coldest_out_loop.
>>>>>>         (move_computations_worker): Adjust and fix iteration udpate.
>>>>>>         (execute_sm_exit): Check pointer validness.
>>>>>>         (class ref_in_loop_hot_body): New functor.
>>>>>>         (ref_in_loop_hot_body::operator): New.
>>>>>>         (can_sm_ref_p): Use for_all_locs_in_loop.
>>>>>>
>>>>>> gcc/testsuite/ChangeLog:
>>>>>>
>>>>>>         * gcc.dg/tree-ssa/recip-3.c: Adjust.
>>>>>>         * gcc.dg/tree-ssa/ssa-lim-18.c: New test.
>>>>>>         * gcc.dg/tree-ssa/ssa-lim-19.c: New test.
>>>>>>         * gcc.dg/tree-ssa/ssa-lim-20.c: New test.
>>>>>> ---
>>>>>>  gcc/loop-invariant.c                       | 10 ++--
>>>>>>  gcc/tree-ssa-loop-im.c                     | 61 ++++++++++++++++++++--
>>>>>>  gcc/testsuite/gcc.dg/tree-ssa/recip-3.c    |  2 +-
>>>>>>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c | 20 +++++++
>>>>>>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c | 27 ++++++++++
>>>>>>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c | 25 +++++++++
>>>>>>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c | 28 ++++++++++
>>>>>>  7 files changed, 165 insertions(+), 8 deletions(-)
>>>>>>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
>>>>>>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
>>>>>>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
>>>>>>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
>>>>>>
>>>>>> diff --git a/gcc/loop-invariant.c b/gcc/loop-invariant.c
>>>>>> index fca0c2b24be..5c3be7bf0eb 100644
>>>>>> --- a/gcc/loop-invariant.c
>>>>>> +++ b/gcc/loop-invariant.c
>>>>>> @@ -1183,9 +1183,14 @@ find_invariants_insn (rtx_insn *insn, bool always_reached, bool always_executed)
>>>>>>     call.  */
>>>>>>
>>>>>>  static void
>>>>>> -find_invariants_bb (basic_block bb, bool always_reached, bool always_executed)
>>>>>> +find_invariants_bb (class loop *loop, basic_block bb, bool always_reached,
>>>>>> +                   bool always_executed)
>>>>>>  {
>>>>>>    rtx_insn *insn;
>>>>>> +  basic_block preheader = loop_preheader_edge (loop)->src;
>>>>>> +
>>>>>> +  if (preheader->count > bb->count)
>>>>>> +    return;
>>>>>>
>>>>>>    FOR_BB_INSNS (bb, insn)
>>>>>>      {
>>>>>> @@ -1214,8 +1219,7 @@ find_invariants_body (class loop *loop, basic_block *body,
>>>>>>    unsigned i;
>>>>>>
>>>>>>    for (i = 0; i < loop->num_nodes; i++)
>>>>>> -    find_invariants_bb (body[i],
>>>>>> -                       bitmap_bit_p (always_reached, i),
>>>>>> +    find_invariants_bb (loop, body[i], bitmap_bit_p (always_reached, i),
>>>>>>                         bitmap_bit_p (always_executed, i));
>>>>>>  }
>>>>>>
>>>>>> diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
>>>>>> index 4b187c2cdaf..655fab03442 100644
>>>>>> --- a/gcc/tree-ssa-loop-im.c
>>>>>> +++ b/gcc/tree-ssa-loop-im.c
>>>>>> @@ -417,6 +417,28 @@ movement_possibility (gimple *stmt)
>>>>>>    return ret;
>>>>>>  }
>>>>>>
>>>>>> +/* Find coldest loop between outmost_loop and loop by comapring profile count.  */
>>>>>> +
>>>>>> +static class loop *
>>>>>> +find_coldest_out_loop (class loop *outmost_loop, class loop *loop,
>>>>>> +                      basic_block curr_bb)
>>>>>> +{
>>>>>> +  class loop *cold_loop, *min_loop;
>>>>>> +  cold_loop = min_loop = outmost_loop;
>>>>>> +  profile_count min_count = loop_preheader_edge (min_loop)->src->count;
>>>>>> +
>>>>>> +  if (curr_bb && curr_bb->count < loop_preheader_edge (loop)->src->count)
>>>>>
>>>>> Honza - can you comment on whether we should compare BB counts this way?
>>>>>
>>>>> I would suspect that for, say,
>>>>>
>>>>>   for (...)
>>>>>      if (a)
>>>>>        X;
>>>>>      else
>>>>>        Y;
>>>>>
>>>>> that the counts for X and Y will be less than that of the preheader of the loop
>>>>> only when the loop is estimated to run once.  That is, should we really compare
>>>>> the to the preheader count or maybe better to the _header_ count which
>>>>> would keep the number of iterations out of the equation?
>>>>
>>>> I quickly tried to replace all the loop_preheader_edge (loop)->src with
>>>> loop_preheader_edge (loop)->dest, it will cause many failures in
>>>> gcc.dg/tree-ssa/ssa-lim-*.c, I didn't go deep to investigate, but it seems
>>>> reasonable to compare the bb count with preheader count as both gimple lim
>>>> and RTL loop-invariant move instructions to *preheader* instead of *header*
>>>> after analysis?
>>>
>>> Hmm, yeah - guess I was confused here.
>>>
>>>>>
>>>>> If we look at maybe_hot_count_p that's a quite sophisticated thing to
>>>>> compare a count to the "IPA hot", here we're comparing two counts
>>>>> within a function where it actually matters whether we use a<b or
>>>>> !(a>=b) since 'unordered' is mapped to false (but there's no ordered_p
>>>>> function).
>>>>>
>>>>> Xionghu, you error on the side of not hoisting for unordered counts here
>>>>>
>>>>>> +    return NULL;
>>>>>> +
>>>>>> +  while (min_loop != loop)
>>>>>> +    {
>>>>>> +      min_loop = superloop_at_depth (loop, loop_depth (min_loop) + 1);
>>>>>> +      if (loop_preheader_edge (min_loop)->src->count < min_count)
>>>>>
>>>>> but in the other direction here and on the side of not hoisting
>>>>> in ref_in_loop_hot_body.
>>>>>
>>>>> The three-state relational operator overloads are probably not the
>>>>> very best idea...
>>>>> (see profile-count.h for them)
>>>>>
>>>> Added new function bb_colder_than_loop_preheader to encapsulate the comparision,
>>>> if FALSE is returned due to three-state inequality,  find_coldest_out_loop
>>>> will return the original input to lim->max_loop, and ref_in_loop_hot_body::operator ()
>>>> will return true to continue perform store motion, both preserve the previous
>>>> behavior.
>>>
>>> Thanks.  But I don't think the abstraction as written is useful:
>>>
>>> +/* Compare the profile count inequality of COUNT1 and COUNT2, it is three-state
>>> +   as stated in profile-count.h, FALSE is returned if inequality cannot be
>>> +   decided.  */
>>> +bool bb_colder_than_loop_preheader (profile_count count1, profile_count count2)
>>> +{
>>> +  if (count1 < count2)
>>> +    return true;
>>> +  else
>>> +    return false;
>>> +}
>>>
>>> given the following seems to pass the preheader count in place of the BB count.
>>>
>>> +      if (bb_colder_than_loop_preheader (
>>> +           loop_preheader_edge (min_loop)->src->count, min_count))
>>> +       cold_loop = min_loop;
>>>
>>> find_coldest_out_loop is also a bit weird, I think we want to find
>>> the outermost loop between outmost_loop and loop that has a
>>> lower count than the curr_bb count but
>>>
>>> +  while (min_loop != loop)
>>> +    {
>>> +      min_loop = superloop_at_depth (loop, loop_depth (min_loop) + 1);
>>> +      if (bb_colder_than_loop_preheader (
>>> +           loop_preheader_edge (min_loop)->src->count, min_count))
>>> +       cold_loop = min_loop;
>>>
>>> compares the outermost loops count (min_count) against the preheader
>>> count?  So we're searching for a cold loop with respect to its enclosing loop
>>> here?
>>
>> Let me try to explain how it works :)
>>
>> find_coldest_out_loop does two steps check:
>> 1) Check whether curr_bb is cold in it's own loop_father, if it is cold,
>> just return NULL which means it should not be moved out at all;
>> 2)  curr_bb is NOT cold, assuming the current loop L[m] is the coldest first,
>> than try to find a cold loop to be hoisted to from {L[1], L[2], ... L[m]},
>> if L[i]->count < L[m]->count, set the cold_loop to L[i] until find the loop
>> that has smallest profile_count.
>>
>>
>> Take the updated ssa-lim-19.c as example, check whether curr_bb(bb 5) is cold in
>> loop 3, if it is cold, just return NULL, otherwise select the coldest loop in
>> {loop1, loop2, loop3} and find that loop2 is colder than loop3, return loop2 to
>> be the target hoist loop.  The first check could AVOID hoist if curr_bb is colder
>> than loop3, but it is still hot than loop1 and loop2.  Not sure whether it is possible
>> to construct such cases?
>>
>>
>> gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
>>
>> volatile int x;
>> void
>> bar (int, char *, char *);
>> void
>> foo (int *a, int n, int m, int s, int t)
>> {
>>   int i;
>>   int j;
>>   int k;
>>
>>   for (i = 0; i < m; i++)  // loop 1
>>     {
>>       if (__builtin_expect (x, 0))
>>         for (j = 0; j < n; j++)   // loop 2
>>           for (k = 0; k < n; k++)   // loop 3
>>            {
>>              bar (s / 5, "one", "two");  // curr_bb
>>              a[t] = s;
>>            }
>>       a[t] = t;  // curr_bb2
>>     }
>> }
>>
>> The 4 invariant statements are moved to bb 11(loop2) instead of bb 10(loop1)
>> with this patch.
>> There are totally 6 combinations when curr_bb is hotter than loop 3.  We need
>> to compare the "Loop preheader hotness" instead of "every Loop[i] and curr_bb hotness",
>> returning the coldest loop for this function find_coldest_out_loop, otherwise
>> unexpected behavior happens.
>>
>> L1 > L2 > L3   =>  return L3
>> L1 > L3 > L2   =>  return L2
>> L2 > L1 > L3   =>  return L3
>> L2 > L3 > L1   =>  return L1
>> L3 > L1 > L2   =>  return L2
>> L3 > L2 > L1   =>  return L1
>>
>> So bb_colder_than_loop_preheader does two kind of checks, one is checking
>> L3 preheader count with curr_bb count, another is checking L3 preheader count
>> with L1 preheader count, L2 preheader count, etc...
>>
>>
>> ssa-lim-19.c.138t.lim2:
>> ...
>>    <bb 10> [local count: 16057869]:  // L1 preheader
>> -  _4 = s_22(D) / 5;
>> -  _5 = (long unsigned int) t_24(D);
>> -  _6 = _5 * 4;
>> -  _7 = a_25(D) + _6;
>>    _8 = (long unsigned int) t_24(D);
>>    _9 = _8 * 4;
>>    _10 = a_25(D) + _9;
>>
>>    <bb 3> [local count: 145980626]:
>>    # i_34 = PHI <i_29(12), 0(10)>
>>    x.0_1 ={v} x;
>>    if (x.0_1 != 0)
>>      goto <bb 4>; [10.00%]
>>    else
>>      goto <bb 8>; [90.00%]
>>
>>    <bb 4> [local count: 14598063]:
>>    if (n_20(D) > 0)
>>      goto <bb 11>; [89.00%]
>>    else
>>      goto <bb 8>; [11.00%]
>>
>>    <bb 11> [local count: 12992276]:  // L2 preheader
>> +  _4 = s_22(D) / 5;
>> +  _5 = (long unsigned int) t_24(D);
>> +  _6 = _5 * 4;
>> +  _7 = a_25(D) + _6;
>>    goto <bb 7>; [100.00%]
>>
>>    <bb 14> [local count: 850510901]:
>>
>>    <bb 5> [local count: 955630225]:  // curr_bb
>>    # k_36 = PHI <k_27(14), 0(7)>
>>    bar (_4, "one", "two");
>>    *_7 = s_22(D);
>>    k_27 = k_36 + 1;
>>    if (n_20(D) > k_27)
>>      goto <bb 14>; [89.00%]
>>    else
>>      goto <bb 6>; [11.00%]
>>
>>    <bb 6> [local count: 118111600]:
>>    j_21 = j_35 + 1;
>>    if (n_20(D) > j_21)
>>      goto <bb 13>; [89.00%]
>>    else
>>      goto <bb 8>; [11.00%]
>>
>>    <bb 13> [local count: 105119324]:
>>
>>    <bb 7> [local count: 118111600]:   // L3 preheader
>>    # j_35 = PHI <j_21(13), 0(11)>
>>    goto <bb 5>; [100.00%]
>>
>>    <bb 8> [local count: 145980626]:
>>    *_10 = t_24(D);
>>    i_29 = i_34 + 1;
>>
>> Re-paste the bb_colder_than_loop_preheader and find_coldest_out_loop:
>>
>> +/* Compare the profile count inequality of COUNT1 and COUNT2, it is three-state
>> +   as stated in profile-count.h, FALSE is returned if inequality cannot be
>> +   decided.  */
>> +bool bb_colder_than_loop_preheader (profile_count count1, profile_count count2)
>> +{
>> +  if (count1 < count2)
>> +    return true;
>> +  else
>> +    return false;
>> +}
>> +
>> +/* Find coldest loop between OUTMOST_LOOP and LOOP by comapring profile count.
>> + */
>> +
>> +static class loop *
>> +find_coldest_out_loop (class loop *outmost_loop, class loop *loop,
>> +                      basic_block curr_bb)
>> +{
>> +  class loop *cold_loop, *min_loop;
>> +  cold_loop = min_loop = outmost_loop;
>> +  profile_count min_count = loop_preheader_edge (min_loop)->src->count;
>> +
>> +  /* If bb_colder_than_loop_preheader returns false due to three-state
>> +    comparision, OUTMOST_LOOP is returned finally to preserve the behavior.
>> +    Otherwise, return the coldest loop between OUTMOST_LOOP and LOOP.  */
>> +  if (curr_bb
>> +      && bb_colder_than_loop_preheader (curr_bb->count,
>> +                                       loop_preheader_edge (loop)->src->count))
>> +    return NULL;
>> +
>> +  while (min_loop != loop)
>> +    {
>> +      min_loop = superloop_at_depth (loop, loop_depth (min_loop) + 1);
>> +      if (bb_colder_than_loop_preheader (
>> +           loop_preheader_edge (min_loop)->src->count, min_count))
>> +       cold_loop = min_loop;
>> +    }
>> +  return cold_loop;
>> +}
>> +
>>
>>
>>>
>>> Why is this function not simply
>>>
>>> +static class loop *
>>> +find_coldest_out_loop (class loop *outmost_loop, class loop *loop,
>>> +                      basic_block curr_bb)
>>> +{
>>>      while (bb_colder_than_loop_preheader (curr_bb->count,
>>>                loop_preheader_edge (outermost_loop)->src->count))
>>>         {
>>>             if (outermost_loop == loop)
>>>               return NULL;
>>>             outermost_loop = superloop_at_depth (loop, loop_depth
>>> (outermost_loop) + 1);
>>>         }
>>>      return outermost_loop;
>>> }
>>
>> If change like this, when processing curr_bb(5), outermost_loop will
>> return loop 1 since curr_bb->count > Loop1_prehead->count, the while
>> loop stopped.  This doesn't meet what we want.
> 
> Why?  curr_bb is executed at least as often as loop1 preheader if
> we look at the counts?  So either the counts do not really tell us
> anything of help or I am missing something.  Are you merely
> looking for a block with a lower count on the path from the outermost
> loop entry to the block in question and deciding you do not want to
> hoist further than that?  So it's not about not hoisting to a hot place
> but instead hoist to the coldest place within a loop nest?
> 
> So we have
> 
>   for (i = 0; i < m; i++)  // loop 1
>     {
>       if (__builtin_expect (x, 0))
>         for (j = 0; j < n; j++)   // loop 2
> 
> 
>    <bb 10> [local count: 16057869]:  // L1 preheader
>        ...
>  <bb 3> [local count: 145980626]:
>    # i_34 = PHI <i_29(12), 0(10)>
>  ...
>    <bb 11> [local count: 12992276]:  // L2 preheader
>    ...
>     <bb 7> [local count: 118111600]:   // L3 preheader
>    # j_35 = PHI <j_21(13), 0(11)>
>    goto <bb 5>; [100.00%]
> 
> and we want to hoist to the L2 preheader because that's less frequently
> executed than the L1 preheader (which is less frequently executed
> than the L3 preheader or the block we are hoisting from).

Yes, this is exactly what I want, sorry for not describe it clear before ;(

The updated patch[1] may reflect find_coldest_out_loop better:
It first check whether curr_bb is hotter than it's preheader, if false, return NULL
which means no need hoist at all; Then find a *coldest* preheader to hoist
within a loop nest from outmost_loop.


[1] https://gcc.gnu.org/pipermail/gcc-patches/2021-October/581961.html


+/* Find coldest loop between OUTMOST_LOOP and LOOP by comparing profile count.
+   It does two steps check:
+   1) Check whether CURR_BB is cold in it's own loop_father, if it is cold, just
+   return NULL which means it should not be moved out at all;
+   2)  CURR_BB is NOT cold, set LOOP to cold_loop, then iteratively search loops
+   from {L[outmost_loop], L[outmost_loop+1], ... L[loop]}, if L[i] is colder
+   than L[cold_loop], reset cold_loop to L[i] until get the loop that has
+   smallest profile_count.  */
+
+static class loop *
+find_coldest_out_loop (class loop *outmost_loop, class loop *loop,
+		       basic_block curr_bb)
+{
+  class loop *cold_loop;
+
+  /* If bb_colder_than_loop_preheader returns false due to three-state
+    comparision, OUTMOST_LOOP is returned finally to preserve the behavior.
+    Otherwise, return the coldest loop between OUTMOST_LOOP and LOOP.  */
+  if (curr_bb
+      && bb_colder_than_loop_preheader (curr_bb,
+					loop_preheader_edge (loop)->src))
+    return NULL;
+
+  cold_loop = loop;
+  while (outmost_loop != loop)
+    {
+      if (bb_colder_than_loop_preheader (loop_preheader_edge (outmost_loop)->src,
+					 loop_preheader_edge (cold_loop)->src))
+	cold_loop = outmost_loop;
+      outmost_loop = superloop_at_depth (loop, loop_depth (outmost_loop) + 1);
+    }
+  return cold_loop;
+}


> 
> I'm concerned with compile-time complexity re-evaluating counts on the
> loop nest many times.  So it looks to me that we can pre-compute
> this lowest-preheader-count loop for a loop nest at least for the
> store-motion case where we know the outermost loop?
> 
> 

But the lowest-preheader-count loop may change for a loop/bb with different
outermost loop.  For example if,

L1_preheader_count < L2_preheader_count < L3_preheader_count < L4_preheader_count < curr_bb_count

then,

find_coldest_out_loop (L1, loop, curr_bb)  => coldest preheader loop is L1
find_coldest_out_loop (L2, loop, curr_bb)  => coldest preheader loop is L2

So it will be a 1:N map?  Pre-compute it in find_coldest_out_loop
and save it also in lim_data with a new variable
coldest_preheader_loop[outmost_loop][coldest_preheader_loop]?
each call of find_coldest_out_loop will check whether that variable is set
already, only continue the search if
coldest_preheader_loop[outmost_loop][coldest_preheader_loop] is not set?
Seems a bit complicated and not sure whether it helps to reduce
compile-time complexity or I am misunderstanding...


-- 
Thanks,
Xionghu

Re: [RFC] Don't move cold code out of loop by checking bb count

[Richard Biener]

On Wed, Oct 27, 2021 at 4:40 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>
>
>
> On 2021/10/26 21:20, Richard Biener wrote:
> > On Mon, Oct 18, 2021 at 6:29 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
> >>
> >>
> >>
> >> On 2021/10/15 16:11, Richard Biener wrote:
> >>> On Sat, Oct 9, 2021 at 5:45 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
> >>>>
> >>>> Hi,
> >>>>
> >>>> On 2021/9/28 20:09, Richard Biener wrote:
> >>>>> On Fri, Sep 24, 2021 at 8:29 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
> >>>>>>
> >>>>>> Update the patch to v3, not sure whether you prefer the paste style
> >>>>>> and continue to link the previous thread as Segher dislikes this...
> >>>>>>
> >>>>>>
> >>>>>> [PATCH v3] Don't move cold code out of loop by checking bb count
> >>>>>>
> >>>>>>
> >>>>>> Changes:
> >>>>>> 1. Handle max_loop in determine_max_movement instead of
> >>>>>> outermost_invariant_loop.
> >>>>>> 2. Remove unnecessary changes.
> >>>>>> 3. Add for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body) in can_sm_ref_p.
> >>>>>> 4. "gsi_next (&bsi);" in move_computations_worker is kept since it caused
> >>>>>> infinite loop when implementing v1 and the iteration is missed to be
> >>>>>> updated actually.
> >>>>>>
> >>>>>> v1: https://gcc.gnu.org/pipermail/gcc-patches/2021-August/576488.html
> >>>>>> v2: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/579086.html
> >>>>>>
> >>>>>> There was a patch trying to avoid move cold block out of loop:
> >>>>>>
> >>>>>> https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html
> >>>>>>
> >>>>>> Richard suggested to "never hoist anything from a bb with lower execution
> >>>>>> frequency to a bb with higher one in LIM invariantness_dom_walker
> >>>>>> before_dom_children".
> >>>>>>
> >>>>>> In gimple LIM analysis, add find_coldest_out_loop to move invariants to
> >>>>>> expected target loop, if profile count of the loop bb is colder
> >>>>>> than target loop preheader, it won't be hoisted out of loop.
> >>>>>> Likely for store motion, if all locations of the REF in loop is cold,
> >>>>>> don't do store motion of it.
> >>>>>>
> >>>>>> SPEC2017 performance evaluation shows 1% performance improvement for
> >>>>>> intrate GEOMEAN and no obvious regression for others.  Especially,
> >>>>>> 500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
> >>>>>> largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
> >>>>>> on P8LE.
> >>>>>>
> >>>>>> gcc/ChangeLog:
> >>>>>>
> >>>>>>         * loop-invariant.c (find_invariants_bb): Check profile count
> >>>>>>         before motion.
> >>>>>>         (find_invariants_body): Add argument.
> >>>>>>         * tree-ssa-loop-im.c (find_coldest_out_loop): New function.
> >>>>>>         (determine_max_movement): Use find_coldest_out_loop.
> >>>>>>         (move_computations_worker): Adjust and fix iteration udpate.
> >>>>>>         (execute_sm_exit): Check pointer validness.
> >>>>>>         (class ref_in_loop_hot_body): New functor.
> >>>>>>         (ref_in_loop_hot_body::operator): New.
> >>>>>>         (can_sm_ref_p): Use for_all_locs_in_loop.
> >>>>>>
> >>>>>> gcc/testsuite/ChangeLog:
> >>>>>>
> >>>>>>         * gcc.dg/tree-ssa/recip-3.c: Adjust.
> >>>>>>         * gcc.dg/tree-ssa/ssa-lim-18.c: New test.
> >>>>>>         * gcc.dg/tree-ssa/ssa-lim-19.c: New test.
> >>>>>>         * gcc.dg/tree-ssa/ssa-lim-20.c: New test.
> >>>>>> ---
> >>>>>>  gcc/loop-invariant.c                       | 10 ++--
> >>>>>>  gcc/tree-ssa-loop-im.c                     | 61 ++++++++++++++++++++--
> >>>>>>  gcc/testsuite/gcc.dg/tree-ssa/recip-3.c    |  2 +-
> >>>>>>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c | 20 +++++++
> >>>>>>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c | 27 ++++++++++
> >>>>>>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c | 25 +++++++++
> >>>>>>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c | 28 ++++++++++
> >>>>>>  7 files changed, 165 insertions(+), 8 deletions(-)
> >>>>>>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
> >>>>>>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
> >>>>>>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
> >>>>>>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
> >>>>>>
> >>>>>> diff --git a/gcc/loop-invariant.c b/gcc/loop-invariant.c
> >>>>>> index fca0c2b24be..5c3be7bf0eb 100644
> >>>>>> --- a/gcc/loop-invariant.c
> >>>>>> +++ b/gcc/loop-invariant.c
> >>>>>> @@ -1183,9 +1183,14 @@ find_invariants_insn (rtx_insn *insn, bool always_reached, bool always_executed)
> >>>>>>     call.  */
> >>>>>>
> >>>>>>  static void
> >>>>>> -find_invariants_bb (basic_block bb, bool always_reached, bool always_executed)
> >>>>>> +find_invariants_bb (class loop *loop, basic_block bb, bool always_reached,
> >>>>>> +                   bool always_executed)
> >>>>>>  {
> >>>>>>    rtx_insn *insn;
> >>>>>> +  basic_block preheader = loop_preheader_edge (loop)->src;
> >>>>>> +
> >>>>>> +  if (preheader->count > bb->count)
> >>>>>> +    return;
> >>>>>>
> >>>>>>    FOR_BB_INSNS (bb, insn)
> >>>>>>      {
> >>>>>> @@ -1214,8 +1219,7 @@ find_invariants_body (class loop *loop, basic_block *body,
> >>>>>>    unsigned i;
> >>>>>>
> >>>>>>    for (i = 0; i < loop->num_nodes; i++)
> >>>>>> -    find_invariants_bb (body[i],
> >>>>>> -                       bitmap_bit_p (always_reached, i),
> >>>>>> +    find_invariants_bb (loop, body[i], bitmap_bit_p (always_reached, i),
> >>>>>>                         bitmap_bit_p (always_executed, i));
> >>>>>>  }
> >>>>>>
> >>>>>> diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
> >>>>>> index 4b187c2cdaf..655fab03442 100644
> >>>>>> --- a/gcc/tree-ssa-loop-im.c
> >>>>>> +++ b/gcc/tree-ssa-loop-im.c
> >>>>>> @@ -417,6 +417,28 @@ movement_possibility (gimple *stmt)
> >>>>>>    return ret;
> >>>>>>  }
> >>>>>>
> >>>>>> +/* Find coldest loop between outmost_loop and loop by comapring profile count.  */
> >>>>>> +
> >>>>>> +static class loop *
> >>>>>> +find_coldest_out_loop (class loop *outmost_loop, class loop *loop,
> >>>>>> +                      basic_block curr_bb)
> >>>>>> +{
> >>>>>> +  class loop *cold_loop, *min_loop;
> >>>>>> +  cold_loop = min_loop = outmost_loop;
> >>>>>> +  profile_count min_count = loop_preheader_edge (min_loop)->src->count;
> >>>>>> +
> >>>>>> +  if (curr_bb && curr_bb->count < loop_preheader_edge (loop)->src->count)
> >>>>>
> >>>>> Honza - can you comment on whether we should compare BB counts this way?
> >>>>>
> >>>>> I would suspect that for, say,
> >>>>>
> >>>>>   for (...)
> >>>>>      if (a)
> >>>>>        X;
> >>>>>      else
> >>>>>        Y;
> >>>>>
> >>>>> that the counts for X and Y will be less than that of the preheader of the loop
> >>>>> only when the loop is estimated to run once.  That is, should we really compare
> >>>>> the to the preheader count or maybe better to the _header_ count which
> >>>>> would keep the number of iterations out of the equation?
> >>>>
> >>>> I quickly tried to replace all the loop_preheader_edge (loop)->src with
> >>>> loop_preheader_edge (loop)->dest, it will cause many failures in
> >>>> gcc.dg/tree-ssa/ssa-lim-*.c, I didn't go deep to investigate, but it seems
> >>>> reasonable to compare the bb count with preheader count as both gimple lim
> >>>> and RTL loop-invariant move instructions to *preheader* instead of *header*
> >>>> after analysis?
> >>>
> >>> Hmm, yeah - guess I was confused here.
> >>>
> >>>>>
> >>>>> If we look at maybe_hot_count_p that's a quite sophisticated thing to
> >>>>> compare a count to the "IPA hot", here we're comparing two counts
> >>>>> within a function where it actually matters whether we use a<b or
> >>>>> !(a>=b) since 'unordered' is mapped to false (but there's no ordered_p
> >>>>> function).
> >>>>>
> >>>>> Xionghu, you error on the side of not hoisting for unordered counts here
> >>>>>
> >>>>>> +    return NULL;
> >>>>>> +
> >>>>>> +  while (min_loop != loop)
> >>>>>> +    {
> >>>>>> +      min_loop = superloop_at_depth (loop, loop_depth (min_loop) + 1);
> >>>>>> +      if (loop_preheader_edge (min_loop)->src->count < min_count)
> >>>>>
> >>>>> but in the other direction here and on the side of not hoisting
> >>>>> in ref_in_loop_hot_body.
> >>>>>
> >>>>> The three-state relational operator overloads are probably not the
> >>>>> very best idea...
> >>>>> (see profile-count.h for them)
> >>>>>
> >>>> Added new function bb_colder_than_loop_preheader to encapsulate the comparision,
> >>>> if FALSE is returned due to three-state inequality,  find_coldest_out_loop
> >>>> will return the original input to lim->max_loop, and ref_in_loop_hot_body::operator ()
> >>>> will return true to continue perform store motion, both preserve the previous
> >>>> behavior.
> >>>
> >>> Thanks.  But I don't think the abstraction as written is useful:
> >>>
> >>> +/* Compare the profile count inequality of COUNT1 and COUNT2, it is three-state
> >>> +   as stated in profile-count.h, FALSE is returned if inequality cannot be
> >>> +   decided.  */
> >>> +bool bb_colder_than_loop_preheader (profile_count count1, profile_count count2)
> >>> +{
> >>> +  if (count1 < count2)
> >>> +    return true;
> >>> +  else
> >>> +    return false;
> >>> +}
> >>>
> >>> given the following seems to pass the preheader count in place of the BB count.
> >>>
> >>> +      if (bb_colder_than_loop_preheader (
> >>> +           loop_preheader_edge (min_loop)->src->count, min_count))
> >>> +       cold_loop = min_loop;
> >>>
> >>> find_coldest_out_loop is also a bit weird, I think we want to find
> >>> the outermost loop between outmost_loop and loop that has a
> >>> lower count than the curr_bb count but
> >>>
> >>> +  while (min_loop != loop)
> >>> +    {
> >>> +      min_loop = superloop_at_depth (loop, loop_depth (min_loop) + 1);
> >>> +      if (bb_colder_than_loop_preheader (
> >>> +           loop_preheader_edge (min_loop)->src->count, min_count))
> >>> +       cold_loop = min_loop;
> >>>
> >>> compares the outermost loops count (min_count) against the preheader
> >>> count?  So we're searching for a cold loop with respect to its enclosing loop
> >>> here?
> >>
> >> Let me try to explain how it works :)
> >>
> >> find_coldest_out_loop does two steps check:
> >> 1) Check whether curr_bb is cold in it's own loop_father, if it is cold,
> >> just return NULL which means it should not be moved out at all;
> >> 2)  curr_bb is NOT cold, assuming the current loop L[m] is the coldest first,
> >> than try to find a cold loop to be hoisted to from {L[1], L[2], ... L[m]},
> >> if L[i]->count < L[m]->count, set the cold_loop to L[i] until find the loop
> >> that has smallest profile_count.
> >>
> >>
> >> Take the updated ssa-lim-19.c as example, check whether curr_bb(bb 5) is cold in
> >> loop 3, if it is cold, just return NULL, otherwise select the coldest loop in
> >> {loop1, loop2, loop3} and find that loop2 is colder than loop3, return loop2 to
> >> be the target hoist loop.  The first check could AVOID hoist if curr_bb is colder
> >> than loop3, but it is still hot than loop1 and loop2.  Not sure whether it is possible
> >> to construct such cases?
> >>
> >>
> >> gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
> >>
> >> volatile int x;
> >> void
> >> bar (int, char *, char *);
> >> void
> >> foo (int *a, int n, int m, int s, int t)
> >> {
> >>   int i;
> >>   int j;
> >>   int k;
> >>
> >>   for (i = 0; i < m; i++)  // loop 1
> >>     {
> >>       if (__builtin_expect (x, 0))
> >>         for (j = 0; j < n; j++)   // loop 2
> >>           for (k = 0; k < n; k++)   // loop 3
> >>            {
> >>              bar (s / 5, "one", "two");  // curr_bb
> >>              a[t] = s;
> >>            }
> >>       a[t] = t;  // curr_bb2
> >>     }
> >> }
> >>
> >> The 4 invariant statements are moved to bb 11(loop2) instead of bb 10(loop1)
> >> with this patch.
> >> There are totally 6 combinations when curr_bb is hotter than loop 3.  We need
> >> to compare the "Loop preheader hotness" instead of "every Loop[i] and curr_bb hotness",
> >> returning the coldest loop for this function find_coldest_out_loop, otherwise
> >> unexpected behavior happens.
> >>
> >> L1 > L2 > L3   =>  return L3
> >> L1 > L3 > L2   =>  return L2
> >> L2 > L1 > L3   =>  return L3
> >> L2 > L3 > L1   =>  return L1
> >> L3 > L1 > L2   =>  return L2
> >> L3 > L2 > L1   =>  return L1
> >>
> >> So bb_colder_than_loop_preheader does two kind of checks, one is checking
> >> L3 preheader count with curr_bb count, another is checking L3 preheader count
> >> with L1 preheader count, L2 preheader count, etc...
> >>
> >>
> >> ssa-lim-19.c.138t.lim2:
> >> ...
> >>    <bb 10> [local count: 16057869]:  // L1 preheader
> >> -  _4 = s_22(D) / 5;
> >> -  _5 = (long unsigned int) t_24(D);
> >> -  _6 = _5 * 4;
> >> -  _7 = a_25(D) + _6;
> >>    _8 = (long unsigned int) t_24(D);
> >>    _9 = _8 * 4;
> >>    _10 = a_25(D) + _9;
> >>
> >>    <bb 3> [local count: 145980626]:
> >>    # i_34 = PHI <i_29(12), 0(10)>
> >>    x.0_1 ={v} x;
> >>    if (x.0_1 != 0)
> >>      goto <bb 4>; [10.00%]
> >>    else
> >>      goto <bb 8>; [90.00%]
> >>
> >>    <bb 4> [local count: 14598063]:
> >>    if (n_20(D) > 0)
> >>      goto <bb 11>; [89.00%]
> >>    else
> >>      goto <bb 8>; [11.00%]
> >>
> >>    <bb 11> [local count: 12992276]:  // L2 preheader
> >> +  _4 = s_22(D) / 5;
> >> +  _5 = (long unsigned int) t_24(D);
> >> +  _6 = _5 * 4;
> >> +  _7 = a_25(D) + _6;
> >>    goto <bb 7>; [100.00%]
> >>
> >>    <bb 14> [local count: 850510901]:
> >>
> >>    <bb 5> [local count: 955630225]:  // curr_bb
> >>    # k_36 = PHI <k_27(14), 0(7)>
> >>    bar (_4, "one", "two");
> >>    *_7 = s_22(D);
> >>    k_27 = k_36 + 1;
> >>    if (n_20(D) > k_27)
> >>      goto <bb 14>; [89.00%]
> >>    else
> >>      goto <bb 6>; [11.00%]
> >>
> >>    <bb 6> [local count: 118111600]:
> >>    j_21 = j_35 + 1;
> >>    if (n_20(D) > j_21)
> >>      goto <bb 13>; [89.00%]
> >>    else
> >>      goto <bb 8>; [11.00%]
> >>
> >>    <bb 13> [local count: 105119324]:
> >>
> >>    <bb 7> [local count: 118111600]:   // L3 preheader
> >>    # j_35 = PHI <j_21(13), 0(11)>
> >>    goto <bb 5>; [100.00%]
> >>
> >>    <bb 8> [local count: 145980626]:
> >>    *_10 = t_24(D);
> >>    i_29 = i_34 + 1;
> >>
> >> Re-paste the bb_colder_than_loop_preheader and find_coldest_out_loop:
> >>
> >> +/* Compare the profile count inequality of COUNT1 and COUNT2, it is three-state
> >> +   as stated in profile-count.h, FALSE is returned if inequality cannot be
> >> +   decided.  */
> >> +bool bb_colder_than_loop_preheader (profile_count count1, profile_count count2)
> >> +{
> >> +  if (count1 < count2)
> >> +    return true;
> >> +  else
> >> +    return false;
> >> +}
> >> +
> >> +/* Find coldest loop between OUTMOST_LOOP and LOOP by comapring profile count.
> >> + */
> >> +
> >> +static class loop *
> >> +find_coldest_out_loop (class loop *outmost_loop, class loop *loop,
> >> +                      basic_block curr_bb)
> >> +{
> >> +  class loop *cold_loop, *min_loop;
> >> +  cold_loop = min_loop = outmost_loop;
> >> +  profile_count min_count = loop_preheader_edge (min_loop)->src->count;
> >> +
> >> +  /* If bb_colder_than_loop_preheader returns false due to three-state
> >> +    comparision, OUTMOST_LOOP is returned finally to preserve the behavior.
> >> +    Otherwise, return the coldest loop between OUTMOST_LOOP and LOOP.  */
> >> +  if (curr_bb
> >> +      && bb_colder_than_loop_preheader (curr_bb->count,
> >> +                                       loop_preheader_edge (loop)->src->count))
> >> +    return NULL;
> >> +
> >> +  while (min_loop != loop)
> >> +    {
> >> +      min_loop = superloop_at_depth (loop, loop_depth (min_loop) + 1);
> >> +      if (bb_colder_than_loop_preheader (
> >> +           loop_preheader_edge (min_loop)->src->count, min_count))
> >> +       cold_loop = min_loop;
> >> +    }
> >> +  return cold_loop;
> >> +}
> >> +
> >>
> >>
> >>>
> >>> Why is this function not simply
> >>>
> >>> +static class loop *
> >>> +find_coldest_out_loop (class loop *outmost_loop, class loop *loop,
> >>> +                      basic_block curr_bb)
> >>> +{
> >>>      while (bb_colder_than_loop_preheader (curr_bb->count,
> >>>                loop_preheader_edge (outermost_loop)->src->count))
> >>>         {
> >>>             if (outermost_loop == loop)
> >>>               return NULL;
> >>>             outermost_loop = superloop_at_depth (loop, loop_depth
> >>> (outermost_loop) + 1);
> >>>         }
> >>>      return outermost_loop;
> >>> }
> >>
> >> If change like this, when processing curr_bb(5), outermost_loop will
> >> return loop 1 since curr_bb->count > Loop1_prehead->count, the while
> >> loop stopped.  This doesn't meet what we want.
> >
> > Why?  curr_bb is executed at least as often as loop1 preheader if
> > we look at the counts?  So either the counts do not really tell us
> > anything of help or I am missing something.  Are you merely
> > looking for a block with a lower count on the path from the outermost
> > loop entry to the block in question and deciding you do not want to
> > hoist further than that?  So it's not about not hoisting to a hot place
> > but instead hoist to the coldest place within a loop nest?
> >
> > So we have
> >
> >   for (i = 0; i < m; i++)  // loop 1
> >     {
> >       if (__builtin_expect (x, 0))
> >         for (j = 0; j < n; j++)   // loop 2
> >
> >
> >    <bb 10> [local count: 16057869]:  // L1 preheader
> >        ...
> >  <bb 3> [local count: 145980626]:
> >    # i_34 = PHI <i_29(12), 0(10)>
> >  ...
> >    <bb 11> [local count: 12992276]:  // L2 preheader
> >    ...
> >     <bb 7> [local count: 118111600]:   // L3 preheader
> >    # j_35 = PHI <j_21(13), 0(11)>
> >    goto <bb 5>; [100.00%]
> >
> > and we want to hoist to the L2 preheader because that's less frequently
> > executed than the L1 preheader (which is less frequently executed
> > than the L3 preheader or the block we are hoisting from).
>
> Yes, this is exactly what I want, sorry for not describe it clear before ;(

OK, thanks for confirming ;)

> The updated patch[1] may reflect find_coldest_out_loop better:
> It first check whether curr_bb is hotter than it's preheader, if false, return NULL
> which means no need hoist at all; Then find a *coldest* preheader to hoist
> within a loop nest from outmost_loop.
>
>
> [1] https://gcc.gnu.org/pipermail/gcc-patches/2021-October/581961.html
>
>
> +/* Find coldest loop between OUTMOST_LOOP and LOOP by comparing profile count.
> +   It does two steps check:
> +   1) Check whether CURR_BB is cold in it's own loop_father, if it is cold, just
> +   return NULL which means it should not be moved out at all;
> +   2)  CURR_BB is NOT cold, set LOOP to cold_loop, then iteratively search loops
> +   from {L[outmost_loop], L[outmost_loop+1], ... L[loop]}, if L[i] is colder
> +   than L[cold_loop], reset cold_loop to L[i] until get the loop that has
> +   smallest profile_count.  */
> +
> +static class loop *
> +find_coldest_out_loop (class loop *outmost_loop, class loop *loop,
> +                      basic_block curr_bb)
> +{
> +  class loop *cold_loop;
> +
> +  /* If bb_colder_than_loop_preheader returns false due to three-state
> +    comparision, OUTMOST_LOOP is returned finally to preserve the behavior.
> +    Otherwise, return the coldest loop between OUTMOST_LOOP and LOOP.  */
> +  if (curr_bb
> +      && bb_colder_than_loop_preheader (curr_bb,
> +                                       loop_preheader_edge (loop)->src))
> +    return NULL;
> +
> +  cold_loop = loop;
> +  while (outmost_loop != loop)
> +    {
> +      if (bb_colder_than_loop_preheader (loop_preheader_edge (outmost_loop)->src,
> +                                        loop_preheader_edge (cold_loop)->src))
> +       cold_loop = outmost_loop;
> +      outmost_loop = superloop_at_depth (loop, loop_depth (outmost_loop) + 1);
> +    }
> +  return cold_loop;
> +}
>
>
> >
> > I'm concerned with compile-time complexity re-evaluating counts on the
> > loop nest many times.  So it looks to me that we can pre-compute
> > this lowest-preheader-count loop for a loop nest at least for the
> > store-motion case where we know the outermost loop?
> >
> >
>
> But the lowest-preheader-count loop may change for a loop/bb with different
> outermost loop.  For example if,
>
> L1_preheader_count < L2_preheader_count < L3_preheader_count < L4_preheader_count < curr_bb_count
>
> then,
>
> find_coldest_out_loop (L1, loop, curr_bb)  => coldest preheader loop is L1
> find_coldest_out_loop (L2, loop, curr_bb)  => coldest preheader loop is L2
>
> So it will be a 1:N map?

I'm talking about the can_sm_ref_p call, in that context 'loop' will
be the outermost loop of
interest, and we are calling this for all stores in a loop.  We're doing

+bool
+ref_in_loop_hot_body::operator () (mem_ref_loc *loc)
+{
+  basic_block curr_bb = gimple_bb (loc->stmt);
+  class loop *inner_loop = curr_bb->loop_father;
+  return find_coldest_out_loop (l, inner_loop, curr_bb);

for each location the ref is accessed and the intent was to see
whether there's at least one
that we would like to move to 'loop'.  Indeed since we only know the
common outer loop
but not the inner we are hosting from there's not a single "coldest"
loop to cache and so
any caching we might want to perform could be applied to the other case as well.

I suppose the most natural thing to cache is for each loop the outer loop where
its outer loop preheader would be hotter than the outer loops preheader so that

+  while (outmost_loop != loop)
+    {
+      if (bb_colder_than_loop_preheader (loop_preheader_edge
(outmost_loop)->src,
+                                        loop_preheader_edge (cold_loop)->src))
+       cold_loop = outmost_loop;
+      outmost_loop = superloop_at_depth (loop, loop_depth (outmost_loop) + 1);
+    }

could be instead written as

  coldest_loop = coldest_outermost_loop[loop->num];
  if (loop_depth (coldest_loop) < loop_depth (outermost_loop))
    return outermost_loop;
  return coldest_loop;

?  And in the usual case coldest_outermost_loop[L] would be the loop tree root.
It should be possible to compute such cache in a DFS walk of the loop tree
(the loop iterator by default visits in such order).

>  Pre-compute it in find_coldest_out_loop
> and save it also in lim_data with a new variable
> coldest_preheader_loop[outmost_loop][coldest_preheader_loop]?
> each call of find_coldest_out_loop will check whether that variable is set
> already, only continue the search if
> coldest_preheader_loop[outmost_loop][coldest_preheader_loop] is not set?
> Seems a bit complicated and not sure whether it helps to reduce
> compile-time complexity or I am misunderstanding...
>
>
> --
> Thanks,
> Xionghu

Re: [RFC] Don't move cold code out of loop by checking bb count

[Xionghu Luo]

On 2021/10/29 19:48, Richard Biener wrote:
> I'm talking about the can_sm_ref_p call, in that context 'loop' will
> be the outermost loop of
> interest, and we are calling this for all stores in a loop.  We're doing
> 
> +bool
> +ref_in_loop_hot_body::operator () (mem_ref_loc *loc)
> +{
> +  basic_block curr_bb = gimple_bb (loc->stmt);
> +  class loop *inner_loop = curr_bb->loop_father;
> +  return find_coldest_out_loop (l, inner_loop, curr_bb);
> 
> for each location the ref is accessed and the intent was to see
> whether there's at least one
> that we would like to move to 'loop'.  Indeed since we only know the
> common outer loop
> but not the inner we are hosting from there's not a single "coldest"
> loop to cache and so
> any caching we might want to perform could be applied to the other case as well.
> 
> I suppose the most natural thing to cache is for each loop the outer loop where
> its outer loop preheader would be hotter than the outer loops preheader so that
> 
> +  while (outmost_loop != loop)
> +    {
> +      if (bb_colder_than_loop_preheader (loop_preheader_edge
> (outmost_loop)->src,
> +                                        loop_preheader_edge (cold_loop)->src))
> +       cold_loop = outmost_loop;
> +      outmost_loop = superloop_at_depth (loop, loop_depth (outmost_loop) + 1);
> +    }
> 
> could be instead written as
> 
>   coldest_loop = coldest_outermost_loop[loop->num];
>   if (loop_depth (coldest_loop) < loop_depth (outermost_loop))
>     return outermost_loop;
>   return coldest_loop;
> 
> ?  And in the usual case coldest_outermost_loop[L] would be the loop tree root.
> It should be possible to compute such cache in a DFS walk of the loop tree
> (the loop iterator by default visits in such order).


Thanks.  Updated the patch with your suggestion.  Not sure whether it strictly
conforms to your comments.  Though the patch passed all my added tests(coverage not enough),
I am still a bit worried if pre-computed coldest_loop is outside of outermost_loop, but
outermost_loop is not the COLDEST LOOP, i.e. (outer->inner)

 [loop tree root, coldest_loop, outermost_loop,..., second_coldest_loop, ..., loop],

then function find_coldest_out_loop will return a loop NOT accord with our
expectation, that should return second_coldest_loop instead of outermost_loop?


Changes:
1. Add function fill_coldest_out_loop to pre compute the coldest
outermost loop for each loop.
2. Rename find_coldest_out_loop to get_coldest_out_loop.
3. Add testcase ssa-lim-22.c to differentiate with ssa-lim-19.c.

v5 changes:
1. Refine comments for new functions.
2. Use basic_block instead of count in bb_colder_than_loop_preheader
to align with function name.
3. Refine with simpler implementation for get_coldest_out_loop and
ref_in_loop_hot_body::operator for better understanding.

v4 changes:
1. Sort out profile_count comparision to function bb_cold_than_loop_preheader.
2. Update ref_in_loop_hot_body::operator () to find cold_loop before compare.
3. Split RTL invariant motion part out.
4. Remove aux changes.

v3 changes:
1. Handle max_loop in determine_max_movement instead of outermost_invariant_loop.
2. Remove unnecessary changes.
3. Add for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body) in can_sm_ref_p.
4. "gsi_next (&bsi);" in move_computations_worker is kept since it caused
infinite loop when implementing v1 and the iteration is missed to be
updated actually.

v1: https://gcc.gnu.org/pipermail/gcc-patches/2021-August/576488.html
v2: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/579086.html
v3: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/580211.html
v4: https://gcc.gnu.org/pipermail/gcc-patches/2021-October/581231.html
v5: https://gcc.gnu.org/pipermail/gcc-patches/2021-October/581961.html

There was a patch trying to avoid move cold block out of loop:

https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html

Richard suggested to "never hoist anything from a bb with lower execution
frequency to a bb with higher one in LIM invariantness_dom_walker
before_dom_children".

In gimple LIM analysis, add get_coldest_out_loop to move invariants to
expected target loop, if profile count of the loop bb is colder
than target loop preheader, it won't be hoisted out of loop.
Likely for store motion, if all locations of the REF in loop is cold,
don't do store motion of it.

SPEC2017 performance evaluation shows 1% performance improvement for
intrate GEOMEAN and no obvious regression for others.  Especially,
500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
on P8LE.

gcc/ChangeLog:

	* tree-ssa-loop-im.c (bb_colder_than_loop_preheader): New
	function.
	(get_coldest_out_loop): New function.
	(determine_max_movement): Use get_coldest_out_loop.
	(move_computations_worker): Adjust and fix iteration udpate.
	(class ref_in_loop_hot_body): New functor.
	(ref_in_loop_hot_body::operator): New.
	(can_sm_ref_p): Use for_all_locs_in_loop.
	(fill_coldest_out_loop): New.
	(loop_invariant_motion_in_fun): Call fill_coldest_out_loop.

gcc/testsuite/ChangeLog:

	* gcc.dg/tree-ssa/recip-3.c: Adjust.
	* gcc.dg/tree-ssa/ssa-lim-18.c: New test.
	* gcc.dg/tree-ssa/ssa-lim-19.c: New test.
	* gcc.dg/tree-ssa/ssa-lim-20.c: New test.
	* gcc.dg/tree-ssa/ssa-lim-21.c: New test.
	* gcc.dg/tree-ssa/ssa-lim-22.c: New test.
---
 gcc/tree-ssa-loop-im.c                     | 111 ++++++++++++++++++++-
 gcc/testsuite/gcc.dg/tree-ssa/recip-3.c    |   2 +-
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c |  20 ++++
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c |  29 ++++++
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c |  25 +++++
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c |  35 +++++++
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c |  32 ++++++
 7 files changed, 251 insertions(+), 3 deletions(-)
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c

diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
index 4b187c2cdaf..d3390385fd9 100644
--- a/gcc/tree-ssa-loop-im.c
+++ b/gcc/tree-ssa-loop-im.c
@@ -146,6 +146,9 @@ public:
 enum dep_kind { lim_raw, sm_war, sm_waw };
 enum dep_state { dep_unknown, dep_independent, dep_dependent };
 
+/* coldest outermost loop for given loop.  */
+class loop **coldest_outermost_loop;
+
 /* Populate the loop dependence cache of REF for LOOP, KIND with STATE.  */
 
 static void
@@ -417,6 +420,43 @@ movement_possibility (gimple *stmt)
   return ret;
 }
 
+/* Compare the profile count inequality of bb and preheader, it is three-state
+   as stated in profile-count.h, FALSE is returned if inequality cannot be
+   decided.  */
+bool bb_colder_than_loop_preheader (basic_block bb, basic_block preheader)
+{
+  gcc_assert (bb && preheader);
+  return bb->count < preheader->count;
+}
+
+/* Check coldest loop between OUTMOST_LOOP and LOOP by comparing profile count.
+   It does two steps check:
+   1) Check whether CURR_BB is cold in it's own loop_father, if it is cold, just
+   return NULL which means it should not be moved out at all;
+   2)  CURR_BB is NOT cold, check if pre-computed COLDEST_LOOP is outside of
+   OUTMOST_LOOP.  */
+
+static class loop *
+get_coldest_out_loop (class loop *outmost_loop, class loop *loop,
+		      basic_block curr_bb)
+{
+  gcc_assert (outmost_loop == loop || flow_loop_nested_p (outmost_loop, loop));
+  class loop *cold_loop;
+
+  /* If bb_colder_than_loop_preheader returns false due to three-state
+    comparision, OUTMOST_LOOP is returned finally to preserve the behavior.
+    Otherwise, return the coldest loop between OUTMOST_LOOP and LOOP.  */
+  if (curr_bb
+      && bb_colder_than_loop_preheader (curr_bb,
+					loop_preheader_edge (loop)->src))
+    return NULL;
+
+  class loop *coldest_loop = coldest_outermost_loop[loop->num];
+  if (loop_depth (coldest_loop) < loop_depth (outmost_loop))
+    return outmost_loop;
+  return coldest_loop;
+}
+
 /* Suppose that operand DEF is used inside the LOOP.  Returns the outermost
    loop to that we could move the expression using DEF if it did not have
    other operands, i.e. the outermost loop enclosing LOOP in that the value
@@ -685,7 +725,9 @@ determine_max_movement (gimple *stmt, bool must_preserve_exec)
     level = ALWAYS_EXECUTED_IN (bb);
   else
     level = superloop_at_depth (loop, 1);
-  lim_data->max_loop = level;
+  lim_data->max_loop = get_coldest_out_loop (level, loop, bb);
+  if (!lim_data->max_loop)
+    return false;
 
   if (gphi *phi = dyn_cast <gphi *> (stmt))
     {
@@ -1221,7 +1263,10 @@ move_computations_worker (basic_block bb)
       /* We do not really want to move conditionals out of the loop; we just
 	 placed it here to force its operands to be moved if necessary.  */
       if (gimple_code (stmt) == GIMPLE_COND)
-	continue;
+	{
+	  gsi_next (&bsi);
+	  continue;
+	}
 
       if (dump_file && (dump_flags & TDF_DETAILS))
 	{
@@ -2887,6 +2932,26 @@ ref_indep_loop_p (class loop *loop, im_mem_ref *ref, dep_kind kind)
   return indep_p;
 }
 
+class ref_in_loop_hot_body
+{
+public:
+  ref_in_loop_hot_body (loop *loop_) : l (loop_) {}
+  bool operator () (mem_ref_loc *loc);
+  class loop *l;
+};
+
+/* Check the coldest loop between loop L and innermost loop.  If there is one
+   cold loop between L and INNER_LOOP, store motion can be performed, otherwise
+   no cold loop means no store motion.  get_coldest_out_loop also handles cases
+   when l is inner_loop.  */
+bool
+ref_in_loop_hot_body::operator () (mem_ref_loc *loc)
+{
+  basic_block curr_bb = gimple_bb (loc->stmt);
+  class loop *inner_loop = curr_bb->loop_father;
+  return get_coldest_out_loop (l, inner_loop, curr_bb);
+}
+
 
 /* Returns true if we can perform store motion of REF from LOOP.  */
 
@@ -2941,6 +3006,12 @@ can_sm_ref_p (class loop *loop, im_mem_ref *ref)
   if (!ref_indep_loop_p (loop, ref, sm_war))
     return false;
 
+  /* Verify whether the candidate is hot for LOOP.  Only do store motion if the
+    candidate's profile count is hot.  Statement in cold BB shouldn't be moved
+    out of it's loop_father.  */
+  if (!for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body (loop)))
+    return false;
+
   return true;
 }
 
@@ -3153,6 +3224,34 @@ fill_always_executed_in (void)
     fill_always_executed_in_1 (loop, contains_call);
 }
 
+/* Find the coldest loop preheader from loop tree root to LOOP.  Set LOOP to
+   cold_loop, then iteratively search loops from {L[outmost_loop],
+   L[outmost_loop+1], ... L[loop]}, if L[i] is colder than L[cold_loop], reset
+   cold_loop to L[i] until get the loop that has smallest profile_count.
+   Then recursively set each inner loop.  */
+
+void
+fill_coldest_out_loop (class loop *loop)
+{
+  class loop *outmost_loop = current_loops->tree_root->inner;
+  class loop *cold_loop = loop;
+  while (outmost_loop != loop)
+    {
+      if (bb_colder_than_loop_preheader (
+	    loop_preheader_edge (outmost_loop)->src,
+	    loop_preheader_edge (cold_loop)->src))
+	cold_loop = outmost_loop;
+      outmost_loop = superloop_at_depth (loop, loop_depth (outmost_loop) + 1);
+    }
+
+  coldest_outermost_loop[loop->num] = cold_loop;
+  if (dump_enabled_p ())
+    dump_printf (MSG_NOTE, "loop %d's coldest outermost loop is %d\n",
+		 loop->num, cold_loop->num);
+
+  for (loop = loop->inner; loop; loop = loop->next)
+    fill_coldest_out_loop (loop);
+}
 
 /* Compute the global information needed by the loop invariant motion pass.  */
 
@@ -3237,6 +3336,8 @@ tree_ssa_lim_finalize (void)
     free_affine_expand_cache (&memory_accesses.ttae_cache);
 
   free (bb_loop_postorder);
+
+  free (coldest_outermost_loop);
 }
 
 /* Moves invariants from loops.  Only "expensive" invariants are moved out --
@@ -3256,6 +3357,12 @@ loop_invariant_motion_in_fun (function *fun, bool store_motion)
   /* Fills ALWAYS_EXECUTED_IN information for basic blocks.  */
   fill_always_executed_in ();
 
+  /* Pre-compute coldest outermost loop of each loop.  */
+  class loop *loop;
+  coldest_outermost_loop = XNEWVEC (class loop *, number_of_loops (cfun));
+  for (loop = current_loops->tree_root->inner; loop != NULL; loop = loop->next)
+    fill_coldest_out_loop (loop);
+
   int *rpo = XNEWVEC (int, last_basic_block_for_fn (fun));
   int n = pre_and_rev_post_order_compute_fn (fun, NULL, rpo, false);
 
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
index 638bf38db8c..641c91e719e 100644
--- a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
+++ b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
@@ -23,4 +23,4 @@ float h ()
 	F[0] += E / d;
 }
 
-/* { dg-final { scan-tree-dump-times " / " 1 "recip" } } */
+/* { dg-final { scan-tree-dump-times " / " 5 "recip" } } */
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
new file mode 100644
index 00000000000..7326a230b3f
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
@@ -0,0 +1,20 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+volatile int x;
+void
+bar (int, char *, char *);
+void
+foo (int *a, int n, int k)
+{
+  int i;
+
+  for (i = 0; i < n; i++)
+    {
+      if (__builtin_expect (x, 0))
+	bar (k / 5, "one", "two");
+      a[i] = k;
+    }
+}
+
+/* { dg-final { scan-tree-dump-not "out of loop 1" "lim2" } } */
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
new file mode 100644
index 00000000000..51c1913d003
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
@@ -0,0 +1,29 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+volatile int x;
+void
+bar (int, char *, char *);
+void
+foo (int *a, int n, int m, int s, int t)
+{
+  int i;
+  int j;
+  int k;
+
+  for (i = 0; i < m; i++) // Loop 1
+    {
+      if (__builtin_expect (x, 0))
+	for (j = 0; j < n; j++) // Loop 2
+	  for (k = 0; k < n; k++) // Loop 3
+	    {
+	      bar (s / 5, "one", "two");
+	      a[t] = s;
+	    }
+      a[t] = t;
+    }
+}
+
+/* { dg-final { scan-tree-dump-times "out of loop 2" 4 "lim2" } } */
+/* { dg-final { scan-tree-dump-times "out of loop 1" 3 "lim2" } } */
+
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
new file mode 100644
index 00000000000..bc60a040a70
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
@@ -0,0 +1,25 @@
+/* { dg-do compile  } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+/* Test that `count' is not hoisted out of loop when bb is cold.  */
+
+int count;
+volatile int x;
+
+struct obj {
+  int data;
+  struct obj *next;
+
+} *q;
+
+void
+func (int m)
+{
+  struct obj *p;
+  for (int i = 0; i < m; i++)
+    if (__builtin_expect (x, 0))
+      count++;
+
+}
+
+/* { dg-final { scan-tree-dump-not "Executing store motion of" "lim2"  }  } */
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
new file mode 100644
index 00000000000..ffe6f8f699d
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
@@ -0,0 +1,35 @@
+/* { dg-do compile  } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+/* Test that `data' and 'data1' is not hoisted out of inner loop and outer loop
+   when it is in cold loop.  */
+
+int count;
+volatile int x;
+
+struct obj {
+  int data;
+  int data1;
+  struct obj *next;
+};
+
+void
+func (int m, int n, int k, struct obj *a)
+{
+  struct obj *q = a;
+  for (int j = 0; j < m; j++)
+    if (__builtin_expect (m, 0))
+      for (int i = 0; i < m; i++)
+	{
+	  if (__builtin_expect (x, 0))
+	    {
+	      count++;
+	      q->data += 3; /* Not hoisted out to inner loop. */
+	    }
+	  count += n;
+	  q->data1 += k; /* Not hoisted out to outer loop. */
+	}
+}
+
+/* { dg-final { scan-tree-dump-not "Executing store motion of" "lim2"  }  } */
+
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c
new file mode 100644
index 00000000000..16ba4ceb8ab
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c
@@ -0,0 +1,32 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+volatile int x;
+volatile int y;
+void
+bar (int, char *, char *);
+void
+foo (int *a, int n, int m, int s, int t)
+{
+  int i;
+  int j;
+  int k;
+
+  for (i = 0; i < m; i++) // Loop 1
+    {
+      if (__builtin_expect (x, 0))
+	for (j = 0; j < n; j++) // Loop 2
+	  if (__builtin_expect (y, 0))
+	    for (k = 0; k < n; k++) // Loop 3
+	      {
+		bar (s / 5, "one", "two");
+		a[t] = s;
+	      }
+      a[t] = t;
+    }
+}
+
+/* { dg-final { scan-tree-dump-times "out of loop 3" 4 "lim2" } } */
+/* { dg-final { scan-tree-dump-times "out of loop 1" 3 "lim2" } } */
+
+
-- 
2.27.0.90.geebb51ba8c

Re: [RFC] Don't move cold code out of loop by checking bb count

[Xionghu Luo]

On 2021/10/29 19:48, Richard Biener wrote:
> I'm talking about the can_sm_ref_p call, in that context 'loop' will
> be the outermost loop of
> interest, and we are calling this for all stores in a loop.  We're doing
> 
> +bool
> +ref_in_loop_hot_body::operator () (mem_ref_loc *loc)
> +{
> +  basic_block curr_bb = gimple_bb (loc->stmt);
> +  class loop *inner_loop = curr_bb->loop_father;
> +  return find_coldest_out_loop (l, inner_loop, curr_bb);
> 
> for each location the ref is accessed and the intent was to see
> whether there's at least one
> that we would like to move to 'loop'.  Indeed since we only know the
> common outer loop
> but not the inner we are hosting from there's not a single "coldest"
> loop to cache and so
> any caching we might want to perform could be applied to the other case as well.
> 
> I suppose the most natural thing to cache is for each loop the outer loop where
> its outer loop preheader would be hotter than the outer loops preheader so that
> 
> +  while (outmost_loop != loop)
> +    {
> +      if (bb_colder_than_loop_preheader (loop_preheader_edge
> (outmost_loop)->src,
> +                                        loop_preheader_edge (cold_loop)->src))
> +       cold_loop = outmost_loop;
> +      outmost_loop = superloop_at_depth (loop, loop_depth (outmost_loop) + 1);
> +    }
> 
> could be instead written as
> 
>   coldest_loop = coldest_outermost_loop[loop->num];
>   if (loop_depth (coldest_loop) < loop_depth (outermost_loop))
>     return outermost_loop;
>   return coldest_loop;
> 
> ?  And in the usual case coldest_outermost_loop[L] would be the loop tree root.
> It should be possible to compute such cache in a DFS walk of the loop tree
> (the loop iterator by default visits in such order).



Thanks.  Updated the patch with your suggestion.  Not sure whether it strictly
conforms to your comments.  Though the patch passed all my added tests(coverage not enough),
I am still a bit worried if pre-computed coldest_loop is outside of outermost_loop, but
outermost_loop is not the COLDEST LOOP, i.e. (outer->inner)

 [loop tree root, coldest_loop, outermost_loop,..., second_coldest_loop, ..., loop],

then function find_coldest_out_loop will return a loop NOT accord with our
expectation, that should return second_coldest_loop instead of outermost_loop?


Changes:
1. Add function fill_coldest_out_loop to pre compute the coldest
outermost loop for each loop.
2. Rename find_coldest_out_loop to get_coldest_out_loop.
3. Add testcase ssa-lim-22.c to differentiate with ssa-lim-19.c.

v5 changes:
1. Refine comments for new functions.
2. Use basic_block instead of count in bb_colder_than_loop_preheader
to align with function name.
3. Refine with simpler implementation for get_coldest_out_loop and
ref_in_loop_hot_body::operator for better understanding.

v4 changes:
1. Sort out profile_count comparision to function bb_cold_than_loop_preheader.
2. Update ref_in_loop_hot_body::operator () to find cold_loop before compare.
3. Split RTL invariant motion part out.
4. Remove aux changes.

v3 changes:
1. Handle max_loop in determine_max_movement instead of outermost_invariant_loop.
2. Remove unnecessary changes.
3. Add for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body) in can_sm_ref_p.
4. "gsi_next (&bsi);" in move_computations_worker is kept since it caused
infinite loop when implementing v1 and the iteration is missed to be
updated actually.

v1: https://gcc.gnu.org/pipermail/gcc-patches/2021-August/576488.html
v2: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/579086.html
v3: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/580211.html
v4: https://gcc.gnu.org/pipermail/gcc-patches/2021-October/581231.html
v5: https://gcc.gnu.org/pipermail/gcc-patches/2021-October/581961.html

There was a patch trying to avoid move cold block out of loop:

https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html

Richard suggested to "never hoist anything from a bb with lower execution
frequency to a bb with higher one in LIM invariantness_dom_walker
before_dom_children".

In gimple LIM analysis, add get_coldest_out_loop to move invariants to
expected target loop, if profile count of the loop bb is colder
than target loop preheader, it won't be hoisted out of loop.
Likely for store motion, if all locations of the REF in loop is cold,
don't do store motion of it.

SPEC2017 performance evaluation shows 1% performance improvement for
intrate GEOMEAN and no obvious regression for others.  Especially,
500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
on P8LE.

gcc/ChangeLog:

	* tree-ssa-loop-im.c (bb_colder_than_loop_preheader): New
	function.
	(get_coldest_out_loop): New function.
	(determine_max_movement): Use get_coldest_out_loop.
	(move_computations_worker): Adjust and fix iteration udpate.
	(class ref_in_loop_hot_body): New functor.
	(ref_in_loop_hot_body::operator): New.
	(can_sm_ref_p): Use for_all_locs_in_loop.
	(fill_coldest_out_loop): New.
	(loop_invariant_motion_in_fun): Call fill_coldest_out_loop.

gcc/testsuite/ChangeLog:

	* gcc.dg/tree-ssa/recip-3.c: Adjust.
	* gcc.dg/tree-ssa/ssa-lim-18.c: New test.
	* gcc.dg/tree-ssa/ssa-lim-19.c: New test.
	* gcc.dg/tree-ssa/ssa-lim-20.c: New test.
	* gcc.dg/tree-ssa/ssa-lim-21.c: New test.
	* gcc.dg/tree-ssa/ssa-lim-22.c: New test.
---
 gcc/tree-ssa-loop-im.c                     | 111 ++++++++++++++++++++-
 gcc/testsuite/gcc.dg/tree-ssa/recip-3.c    |   2 +-
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c |  20 ++++
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c |  29 ++++++
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c |  25 +++++
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c |  35 +++++++
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c |  32 ++++++
 7 files changed, 251 insertions(+), 3 deletions(-)
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c

diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
index 4b187c2cdaf..d3390385fd9 100644
--- a/gcc/tree-ssa-loop-im.c
+++ b/gcc/tree-ssa-loop-im.c
@@ -146,6 +146,9 @@ public:
 enum dep_kind { lim_raw, sm_war, sm_waw };
 enum dep_state { dep_unknown, dep_independent, dep_dependent };
 
+/* coldest outermost loop for given loop.  */
+class loop **coldest_outermost_loop;
+
 /* Populate the loop dependence cache of REF for LOOP, KIND with STATE.  */
 
 static void
@@ -417,6 +420,43 @@ movement_possibility (gimple *stmt)
   return ret;
 }
 
+/* Compare the profile count inequality of bb and preheader, it is three-state
+   as stated in profile-count.h, FALSE is returned if inequality cannot be
+   decided.  */
+bool bb_colder_than_loop_preheader (basic_block bb, basic_block preheader)
+{
+  gcc_assert (bb && preheader);
+  return bb->count < preheader->count;
+}
+
+/* Check coldest loop between OUTMOST_LOOP and LOOP by comparing profile count.
+   It does two steps check:
+   1) Check whether CURR_BB is cold in it's own loop_father, if it is cold, just
+   return NULL which means it should not be moved out at all;
+   2)  CURR_BB is NOT cold, check if pre-computed COLDEST_LOOP is outside of
+   OUTMOST_LOOP.  */
+
+static class loop *
+get_coldest_out_loop (class loop *outmost_loop, class loop *loop,
+		      basic_block curr_bb)
+{
+  gcc_assert (outmost_loop == loop || flow_loop_nested_p (outmost_loop, loop));
+  class loop *cold_loop;
+
+  /* If bb_colder_than_loop_preheader returns false due to three-state
+    comparision, OUTMOST_LOOP is returned finally to preserve the behavior.
+    Otherwise, return the coldest loop between OUTMOST_LOOP and LOOP.  */
+  if (curr_bb
+      && bb_colder_than_loop_preheader (curr_bb,
+					loop_preheader_edge (loop)->src))
+    return NULL;
+
+  class loop *coldest_loop = coldest_outermost_loop[loop->num];
+  if (loop_depth (coldest_loop) < loop_depth (outmost_loop))
+    return outmost_loop;
+  return coldest_loop;
+}
+
 /* Suppose that operand DEF is used inside the LOOP.  Returns the outermost
    loop to that we could move the expression using DEF if it did not have
    other operands, i.e. the outermost loop enclosing LOOP in that the value
@@ -685,7 +725,9 @@ determine_max_movement (gimple *stmt, bool must_preserve_exec)
     level = ALWAYS_EXECUTED_IN (bb);
   else
     level = superloop_at_depth (loop, 1);
-  lim_data->max_loop = level;
+  lim_data->max_loop = get_coldest_out_loop (level, loop, bb);
+  if (!lim_data->max_loop)
+    return false;
 
   if (gphi *phi = dyn_cast <gphi *> (stmt))
     {
@@ -1221,7 +1263,10 @@ move_computations_worker (basic_block bb)
       /* We do not really want to move conditionals out of the loop; we just
 	 placed it here to force its operands to be moved if necessary.  */
       if (gimple_code (stmt) == GIMPLE_COND)
-	continue;
+	{
+	  gsi_next (&bsi);
+	  continue;
+	}
 
       if (dump_file && (dump_flags & TDF_DETAILS))
 	{
@@ -2887,6 +2932,26 @@ ref_indep_loop_p (class loop *loop, im_mem_ref *ref, dep_kind kind)
   return indep_p;
 }
 
+class ref_in_loop_hot_body
+{
+public:
+  ref_in_loop_hot_body (loop *loop_) : l (loop_) {}
+  bool operator () (mem_ref_loc *loc);
+  class loop *l;
+};
+
+/* Check the coldest loop between loop L and innermost loop.  If there is one
+   cold loop between L and INNER_LOOP, store motion can be performed, otherwise
+   no cold loop means no store motion.  get_coldest_out_loop also handles cases
+   when l is inner_loop.  */
+bool
+ref_in_loop_hot_body::operator () (mem_ref_loc *loc)
+{
+  basic_block curr_bb = gimple_bb (loc->stmt);
+  class loop *inner_loop = curr_bb->loop_father;
+  return get_coldest_out_loop (l, inner_loop, curr_bb);
+}
+
 
 /* Returns true if we can perform store motion of REF from LOOP.  */
 
@@ -2941,6 +3006,12 @@ can_sm_ref_p (class loop *loop, im_mem_ref *ref)
   if (!ref_indep_loop_p (loop, ref, sm_war))
     return false;
 
+  /* Verify whether the candidate is hot for LOOP.  Only do store motion if the
+    candidate's profile count is hot.  Statement in cold BB shouldn't be moved
+    out of it's loop_father.  */
+  if (!for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body (loop)))
+    return false;
+
   return true;
 }
 
@@ -3153,6 +3224,34 @@ fill_always_executed_in (void)
     fill_always_executed_in_1 (loop, contains_call);
 }
 
+/* Find the coldest loop preheader from loop tree root to LOOP.  Set LOOP to
+   cold_loop, then iteratively search loops from {L[outmost_loop],
+   L[outmost_loop+1], ... L[loop]}, if L[i] is colder than L[cold_loop], reset
+   cold_loop to L[i] until get the loop that has smallest profile_count.
+   Then recursively set each inner loop.  */
+
+void
+fill_coldest_out_loop (class loop *loop)
+{
+  class loop *outmost_loop = current_loops->tree_root->inner;
+  class loop *cold_loop = loop;
+  while (outmost_loop != loop)
+    {
+      if (bb_colder_than_loop_preheader (
+	    loop_preheader_edge (outmost_loop)->src,
+	    loop_preheader_edge (cold_loop)->src))
+	cold_loop = outmost_loop;
+      outmost_loop = superloop_at_depth (loop, loop_depth (outmost_loop) + 1);
+    }
+
+  coldest_outermost_loop[loop->num] = cold_loop;
+  if (dump_enabled_p ())
+    dump_printf (MSG_NOTE, "loop %d's coldest outermost loop is %d\n",
+		 loop->num, cold_loop->num);
+
+  for (loop = loop->inner; loop; loop = loop->next)
+    fill_coldest_out_loop (loop);
+}
 
 /* Compute the global information needed by the loop invariant motion pass.  */
 
@@ -3237,6 +3336,8 @@ tree_ssa_lim_finalize (void)
     free_affine_expand_cache (&memory_accesses.ttae_cache);
 
   free (bb_loop_postorder);
+
+  free (coldest_outermost_loop);
 }
 
 /* Moves invariants from loops.  Only "expensive" invariants are moved out --
@@ -3256,6 +3357,12 @@ loop_invariant_motion_in_fun (function *fun, bool store_motion)
   /* Fills ALWAYS_EXECUTED_IN information for basic blocks.  */
   fill_always_executed_in ();
 
+  /* Pre-compute coldest outermost loop of each loop.  */
+  class loop *loop;
+  coldest_outermost_loop = XNEWVEC (class loop *, number_of_loops (cfun));
+  for (loop = current_loops->tree_root->inner; loop != NULL; loop = loop->next)
+    fill_coldest_out_loop (loop);
+
   int *rpo = XNEWVEC (int, last_basic_block_for_fn (fun));
   int n = pre_and_rev_post_order_compute_fn (fun, NULL, rpo, false);
 
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
index 638bf38db8c..641c91e719e 100644
--- a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
+++ b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
@@ -23,4 +23,4 @@ float h ()
 	F[0] += E / d;
 }
 
-/* { dg-final { scan-tree-dump-times " / " 1 "recip" } } */
+/* { dg-final { scan-tree-dump-times " / " 5 "recip" } } */
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
new file mode 100644
index 00000000000..7326a230b3f
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
@@ -0,0 +1,20 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+volatile int x;
+void
+bar (int, char *, char *);
+void
+foo (int *a, int n, int k)
+{
+  int i;
+
+  for (i = 0; i < n; i++)
+    {
+      if (__builtin_expect (x, 0))
+	bar (k / 5, "one", "two");
+      a[i] = k;
+    }
+}
+
+/* { dg-final { scan-tree-dump-not "out of loop 1" "lim2" } } */
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
new file mode 100644
index 00000000000..51c1913d003
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
@@ -0,0 +1,29 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+volatile int x;
+void
+bar (int, char *, char *);
+void
+foo (int *a, int n, int m, int s, int t)
+{
+  int i;
+  int j;
+  int k;
+
+  for (i = 0; i < m; i++) // Loop 1
+    {
+      if (__builtin_expect (x, 0))
+	for (j = 0; j < n; j++) // Loop 2
+	  for (k = 0; k < n; k++) // Loop 3
+	    {
+	      bar (s / 5, "one", "two");
+	      a[t] = s;
+	    }
+      a[t] = t;
+    }
+}
+
+/* { dg-final { scan-tree-dump-times "out of loop 2" 4 "lim2" } } */
+/* { dg-final { scan-tree-dump-times "out of loop 1" 3 "lim2" } } */
+
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
new file mode 100644
index 00000000000..bc60a040a70
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
@@ -0,0 +1,25 @@
+/* { dg-do compile  } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+/* Test that `count' is not hoisted out of loop when bb is cold.  */
+
+int count;
+volatile int x;
+
+struct obj {
+  int data;
+  struct obj *next;
+
+} *q;
+
+void
+func (int m)
+{
+  struct obj *p;
+  for (int i = 0; i < m; i++)
+    if (__builtin_expect (x, 0))
+      count++;
+
+}
+
+/* { dg-final { scan-tree-dump-not "Executing store motion of" "lim2"  }  } */
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
new file mode 100644
index 00000000000..ffe6f8f699d
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
@@ -0,0 +1,35 @@
+/* { dg-do compile  } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+/* Test that `data' and 'data1' is not hoisted out of inner loop and outer loop
+   when it is in cold loop.  */
+
+int count;
+volatile int x;
+
+struct obj {
+  int data;
+  int data1;
+  struct obj *next;
+};
+
+void
+func (int m, int n, int k, struct obj *a)
+{
+  struct obj *q = a;
+  for (int j = 0; j < m; j++)
+    if (__builtin_expect (m, 0))
+      for (int i = 0; i < m; i++)
+	{
+	  if (__builtin_expect (x, 0))
+	    {
+	      count++;
+	      q->data += 3; /* Not hoisted out to inner loop. */
+	    }
+	  count += n;
+	  q->data1 += k; /* Not hoisted out to outer loop. */
+	}
+}
+
+/* { dg-final { scan-tree-dump-not "Executing store motion of" "lim2"  }  } */
+
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c
new file mode 100644
index 00000000000..16ba4ceb8ab
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c
@@ -0,0 +1,32 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+volatile int x;
+volatile int y;
+void
+bar (int, char *, char *);
+void
+foo (int *a, int n, int m, int s, int t)
+{
+  int i;
+  int j;
+  int k;
+
+  for (i = 0; i < m; i++) // Loop 1
+    {
+      if (__builtin_expect (x, 0))
+	for (j = 0; j < n; j++) // Loop 2
+	  if (__builtin_expect (y, 0))
+	    for (k = 0; k < n; k++) // Loop 3
+	      {
+		bar (s / 5, "one", "two");
+		a[t] = s;
+	      }
+      a[t] = t;
+    }
+}
+
+/* { dg-final { scan-tree-dump-times "out of loop 3" 4 "lim2" } } */
+/* { dg-final { scan-tree-dump-times "out of loop 1" 3 "lim2" } } */
+
+
-- 
2.27.0.90.geebb51ba8c

Re: [RFC] Don't move cold code out of loop by checking bb count

[Richard Biener]

On Wed, Nov 3, 2021 at 2:29 PM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>
>
>
> On 2021/10/29 19:48, Richard Biener wrote:
> > I'm talking about the can_sm_ref_p call, in that context 'loop' will
> > be the outermost loop of
> > interest, and we are calling this for all stores in a loop.  We're doing
> >
> > +bool
> > +ref_in_loop_hot_body::operator () (mem_ref_loc *loc)
> > +{
> > +  basic_block curr_bb = gimple_bb (loc->stmt);
> > +  class loop *inner_loop = curr_bb->loop_father;
> > +  return find_coldest_out_loop (l, inner_loop, curr_bb);
> >
> > for each location the ref is accessed and the intent was to see
> > whether there's at least one
> > that we would like to move to 'loop'.  Indeed since we only know the
> > common outer loop
> > but not the inner we are hosting from there's not a single "coldest"
> > loop to cache and so
> > any caching we might want to perform could be applied to the other case as well.
> >
> > I suppose the most natural thing to cache is for each loop the outer loop where
> > its outer loop preheader would be hotter than the outer loops preheader so that
> >
> > +  while (outmost_loop != loop)
> > +    {
> > +      if (bb_colder_than_loop_preheader (loop_preheader_edge
> > (outmost_loop)->src,
> > +                                        loop_preheader_edge (cold_loop)->src))
> > +       cold_loop = outmost_loop;
> > +      outmost_loop = superloop_at_depth (loop, loop_depth (outmost_loop) + 1);
> > +    }
> >
> > could be instead written as
> >
> >   coldest_loop = coldest_outermost_loop[loop->num];
> >   if (loop_depth (coldest_loop) < loop_depth (outermost_loop))
> >     return outermost_loop;
> >   return coldest_loop;
> >
> > ?  And in the usual case coldest_outermost_loop[L] would be the loop tree root.
> > It should be possible to compute such cache in a DFS walk of the loop tree
> > (the loop iterator by default visits in such order).
>
>
>
> Thanks.  Updated the patch with your suggestion.  Not sure whether it strictly
> conforms to your comments.  Though the patch passed all my added tests(coverage not enough),
> I am still a bit worried if pre-computed coldest_loop is outside of outermost_loop, but
> outermost_loop is not the COLDEST LOOP, i.e. (outer->inner)
>
>  [loop tree root, coldest_loop, outermost_loop,..., second_coldest_loop, ..., loop],
>
> then function find_coldest_out_loop will return a loop NOT accord with our
> expectation, that should return second_coldest_loop instead of outermost_loop?

Hmm, interesting - yes.  I guess the common case will be that the pre-computed
outermost loop will be the loop at depth 1 since outer loops tend to
be colder than
inner loops?  That would then defeat the whole exercise.

To optimize the common case but not avoiding iteration in the cases we care
about we could instead cache the next outermost loop that is _not_ colder
than loop.  So for your [ ... ] example above we'd have
hotter_than_inner_loop[loop] == outer (second_coldest_loop), where the
candidate would then be 'second_coldest_loop' and we'd then iterate
to hotter_than_inner_loop[hotter_than_inner_loop[loop]] to find the next
cold candidate we can compare against?  For the common case we'd
have hotter_than_inner_loop[looo] == NULL (no such loop) and we then
simply pick 'outermost_loop'.

One comment on the patch itself below.

>
>
> Changes:
> 1. Add function fill_coldest_out_loop to pre compute the coldest
> outermost loop for each loop.
> 2. Rename find_coldest_out_loop to get_coldest_out_loop.
> 3. Add testcase ssa-lim-22.c to differentiate with ssa-lim-19.c.
>
> v5 changes:
> 1. Refine comments for new functions.
> 2. Use basic_block instead of count in bb_colder_than_loop_preheader
> to align with function name.
> 3. Refine with simpler implementation for get_coldest_out_loop and
> ref_in_loop_hot_body::operator for better understanding.
>
> v4 changes:
> 1. Sort out profile_count comparision to function bb_cold_than_loop_preheader.
> 2. Update ref_in_loop_hot_body::operator () to find cold_loop before compare.
> 3. Split RTL invariant motion part out.
> 4. Remove aux changes.
>
> v3 changes:
> 1. Handle max_loop in determine_max_movement instead of outermost_invariant_loop.
> 2. Remove unnecessary changes.
> 3. Add for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body) in can_sm_ref_p.
> 4. "gsi_next (&bsi);" in move_computations_worker is kept since it caused
> infinite loop when implementing v1 and the iteration is missed to be
> updated actually.
>
> v1: https://gcc.gnu.org/pipermail/gcc-patches/2021-August/576488.html
> v2: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/579086.html
> v3: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/580211.html
> v4: https://gcc.gnu.org/pipermail/gcc-patches/2021-October/581231.html
> v5: https://gcc.gnu.org/pipermail/gcc-patches/2021-October/581961.html
>
> There was a patch trying to avoid move cold block out of loop:
>
> https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html
>
> Richard suggested to "never hoist anything from a bb with lower execution
> frequency to a bb with higher one in LIM invariantness_dom_walker
> before_dom_children".
>
> In gimple LIM analysis, add get_coldest_out_loop to move invariants to
> expected target loop, if profile count of the loop bb is colder
> than target loop preheader, it won't be hoisted out of loop.
> Likely for store motion, if all locations of the REF in loop is cold,
> don't do store motion of it.
>
> SPEC2017 performance evaluation shows 1% performance improvement for
> intrate GEOMEAN and no obvious regression for others.  Especially,
> 500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
> largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
> on P8LE.
>
> gcc/ChangeLog:
>
>         * tree-ssa-loop-im.c (bb_colder_than_loop_preheader): New
>         function.
>         (get_coldest_out_loop): New function.
>         (determine_max_movement): Use get_coldest_out_loop.
>         (move_computations_worker): Adjust and fix iteration udpate.
>         (class ref_in_loop_hot_body): New functor.
>         (ref_in_loop_hot_body::operator): New.
>         (can_sm_ref_p): Use for_all_locs_in_loop.
>         (fill_coldest_out_loop): New.
>         (loop_invariant_motion_in_fun): Call fill_coldest_out_loop.
>
> gcc/testsuite/ChangeLog:
>
>         * gcc.dg/tree-ssa/recip-3.c: Adjust.
>         * gcc.dg/tree-ssa/ssa-lim-18.c: New test.
>         * gcc.dg/tree-ssa/ssa-lim-19.c: New test.
>         * gcc.dg/tree-ssa/ssa-lim-20.c: New test.
>         * gcc.dg/tree-ssa/ssa-lim-21.c: New test.
>         * gcc.dg/tree-ssa/ssa-lim-22.c: New test.
> ---
>  gcc/tree-ssa-loop-im.c                     | 111 ++++++++++++++++++++-
>  gcc/testsuite/gcc.dg/tree-ssa/recip-3.c    |   2 +-
>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c |  20 ++++
>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c |  29 ++++++
>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c |  25 +++++
>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c |  35 +++++++
>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c |  32 ++++++
>  7 files changed, 251 insertions(+), 3 deletions(-)
>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c
>
> diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
> index 4b187c2cdaf..d3390385fd9 100644
> --- a/gcc/tree-ssa-loop-im.c
> +++ b/gcc/tree-ssa-loop-im.c
> @@ -146,6 +146,9 @@ public:
>  enum dep_kind { lim_raw, sm_war, sm_waw };
>  enum dep_state { dep_unknown, dep_independent, dep_dependent };
>
> +/* coldest outermost loop for given loop.  */
> +class loop **coldest_outermost_loop;
> +
>  /* Populate the loop dependence cache of REF for LOOP, KIND with STATE.  */
>
>  static void
> @@ -417,6 +420,43 @@ movement_possibility (gimple *stmt)
>    return ret;
>  }
>
> +/* Compare the profile count inequality of bb and preheader, it is three-state
> +   as stated in profile-count.h, FALSE is returned if inequality cannot be
> +   decided.  */
> +bool bb_colder_than_loop_preheader (basic_block bb, basic_block preheader)
> +{
> +  gcc_assert (bb && preheader);
> +  return bb->count < preheader->count;
> +}
> +
> +/* Check coldest loop between OUTMOST_LOOP and LOOP by comparing profile count.
> +   It does two steps check:
> +   1) Check whether CURR_BB is cold in it's own loop_father, if it is cold, just
> +   return NULL which means it should not be moved out at all;
> +   2)  CURR_BB is NOT cold, check if pre-computed COLDEST_LOOP is outside of
> +   OUTMOST_LOOP.  */
> +
> +static class loop *
> +get_coldest_out_loop (class loop *outmost_loop, class loop *loop,
> +                     basic_block curr_bb)
> +{
> +  gcc_assert (outmost_loop == loop || flow_loop_nested_p (outmost_loop, loop));
> +  class loop *cold_loop;
> +
> +  /* If bb_colder_than_loop_preheader returns false due to three-state
> +    comparision, OUTMOST_LOOP is returned finally to preserve the behavior.
> +    Otherwise, return the coldest loop between OUTMOST_LOOP and LOOP.  */
> +  if (curr_bb
> +      && bb_colder_than_loop_preheader (curr_bb,
> +                                       loop_preheader_edge (loop)->src))
> +    return NULL;
> +
> +  class loop *coldest_loop = coldest_outermost_loop[loop->num];
> +  if (loop_depth (coldest_loop) < loop_depth (outmost_loop))
> +    return outmost_loop;
> +  return coldest_loop;
> +}
> +
>  /* Suppose that operand DEF is used inside the LOOP.  Returns the outermost
>     loop to that we could move the expression using DEF if it did not have
>     other operands, i.e. the outermost loop enclosing LOOP in that the value
> @@ -685,7 +725,9 @@ determine_max_movement (gimple *stmt, bool must_preserve_exec)
>      level = ALWAYS_EXECUTED_IN (bb);
>    else
>      level = superloop_at_depth (loop, 1);
> -  lim_data->max_loop = level;
> +  lim_data->max_loop = get_coldest_out_loop (level, loop, bb);
> +  if (!lim_data->max_loop)
> +    return false;
>
>    if (gphi *phi = dyn_cast <gphi *> (stmt))
>      {
> @@ -1221,7 +1263,10 @@ move_computations_worker (basic_block bb)
>        /* We do not really want to move conditionals out of the loop; we just
>          placed it here to force its operands to be moved if necessary.  */
>        if (gimple_code (stmt) == GIMPLE_COND)
> -       continue;
> +       {
> +         gsi_next (&bsi);
> +         continue;
> +       }
>
>        if (dump_file && (dump_flags & TDF_DETAILS))
>         {
> @@ -2887,6 +2932,26 @@ ref_indep_loop_p (class loop *loop, im_mem_ref *ref, dep_kind kind)
>    return indep_p;
>  }
>
> +class ref_in_loop_hot_body
> +{
> +public:
> +  ref_in_loop_hot_body (loop *loop_) : l (loop_) {}
> +  bool operator () (mem_ref_loc *loc);
> +  class loop *l;
> +};
> +
> +/* Check the coldest loop between loop L and innermost loop.  If there is one
> +   cold loop between L and INNER_LOOP, store motion can be performed, otherwise
> +   no cold loop means no store motion.  get_coldest_out_loop also handles cases
> +   when l is inner_loop.  */
> +bool
> +ref_in_loop_hot_body::operator () (mem_ref_loc *loc)
> +{
> +  basic_block curr_bb = gimple_bb (loc->stmt);
> +  class loop *inner_loop = curr_bb->loop_father;
> +  return get_coldest_out_loop (l, inner_loop, curr_bb);
> +}
> +
>
>  /* Returns true if we can perform store motion of REF from LOOP.  */
>
> @@ -2941,6 +3006,12 @@ can_sm_ref_p (class loop *loop, im_mem_ref *ref)
>    if (!ref_indep_loop_p (loop, ref, sm_war))
>      return false;
>
> +  /* Verify whether the candidate is hot for LOOP.  Only do store motion if the
> +    candidate's profile count is hot.  Statement in cold BB shouldn't be moved
> +    out of it's loop_father.  */
> +  if (!for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body (loop)))
> +    return false;
> +
>    return true;
>  }
>
> @@ -3153,6 +3224,34 @@ fill_always_executed_in (void)
>      fill_always_executed_in_1 (loop, contains_call);
>  }
>
> +/* Find the coldest loop preheader from loop tree root to LOOP.  Set LOOP to
> +   cold_loop, then iteratively search loops from {L[outmost_loop],
> +   L[outmost_loop+1], ... L[loop]}, if L[i] is colder than L[cold_loop], reset
> +   cold_loop to L[i] until get the loop that has smallest profile_count.
> +   Then recursively set each inner loop.  */
> +
> +void
> +fill_coldest_out_loop (class loop *loop)
> +{
> +  class loop *outmost_loop = current_loops->tree_root->inner;

that should be superloop_at_depth (loop, 1), otherwise it's wrong
when the function has more than one loop at the outermost level.
I was also hoping to avoid this loop by passing down the current
coldest loop as the single one to compare against.

But with the above discussion things will look different anyway I guess.

> +  class loop *cold_loop = loop;
> +  while (outmost_loop != loop)
> +    {
> +      if (bb_colder_than_loop_preheader (
> +           loop_preheader_edge (outmost_loop)->src,
> +           loop_preheader_edge (cold_loop)->src))
> +       cold_loop = outmost_loop;
> +      outmost_loop = superloop_at_depth (loop, loop_depth (outmost_loop) + 1);
> +    }
> +
> +  coldest_outermost_loop[loop->num] = cold_loop;
> +  if (dump_enabled_p ())
> +    dump_printf (MSG_NOTE, "loop %d's coldest outermost loop is %d\n",
> +                loop->num, cold_loop->num);
> +
> +  for (loop = loop->inner; loop; loop = loop->next)
> +    fill_coldest_out_loop (loop);
> +}
>
>  /* Compute the global information needed by the loop invariant motion pass.  */
>
> @@ -3237,6 +3336,8 @@ tree_ssa_lim_finalize (void)
>      free_affine_expand_cache (&memory_accesses.ttae_cache);
>
>    free (bb_loop_postorder);
> +
> +  free (coldest_outermost_loop);
>  }
>
>  /* Moves invariants from loops.  Only "expensive" invariants are moved out --
> @@ -3256,6 +3357,12 @@ loop_invariant_motion_in_fun (function *fun, bool store_motion)
>    /* Fills ALWAYS_EXECUTED_IN information for basic blocks.  */
>    fill_always_executed_in ();
>
> +  /* Pre-compute coldest outermost loop of each loop.  */
> +  class loop *loop;
> +  coldest_outermost_loop = XNEWVEC (class loop *, number_of_loops (cfun));
> +  for (loop = current_loops->tree_root->inner; loop != NULL; loop = loop->next)
> +    fill_coldest_out_loop (loop);
> +
>    int *rpo = XNEWVEC (int, last_basic_block_for_fn (fun));
>    int n = pre_and_rev_post_order_compute_fn (fun, NULL, rpo, false);
>
> diff --git a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
> index 638bf38db8c..641c91e719e 100644
> --- a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
> +++ b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
> @@ -23,4 +23,4 @@ float h ()
>         F[0] += E / d;
>  }
>
> -/* { dg-final { scan-tree-dump-times " / " 1 "recip" } } */
> +/* { dg-final { scan-tree-dump-times " / " 5 "recip" } } */
> diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
> new file mode 100644
> index 00000000000..7326a230b3f
> --- /dev/null
> +++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
> @@ -0,0 +1,20 @@
> +/* { dg-do compile } */
> +/* { dg-options "-O2 -fdump-tree-lim2-details" } */
> +
> +volatile int x;
> +void
> +bar (int, char *, char *);
> +void
> +foo (int *a, int n, int k)
> +{
> +  int i;
> +
> +  for (i = 0; i < n; i++)
> +    {
> +      if (__builtin_expect (x, 0))
> +       bar (k / 5, "one", "two");
> +      a[i] = k;
> +    }
> +}
> +
> +/* { dg-final { scan-tree-dump-not "out of loop 1" "lim2" } } */
> diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
> new file mode 100644
> index 00000000000..51c1913d003
> --- /dev/null
> +++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
> @@ -0,0 +1,29 @@
> +/* { dg-do compile } */
> +/* { dg-options "-O2 -fdump-tree-lim2-details" } */
> +
> +volatile int x;
> +void
> +bar (int, char *, char *);
> +void
> +foo (int *a, int n, int m, int s, int t)
> +{
> +  int i;
> +  int j;
> +  int k;
> +
> +  for (i = 0; i < m; i++) // Loop 1
> +    {
> +      if (__builtin_expect (x, 0))
> +       for (j = 0; j < n; j++) // Loop 2
> +         for (k = 0; k < n; k++) // Loop 3
> +           {
> +             bar (s / 5, "one", "two");
> +             a[t] = s;
> +           }
> +      a[t] = t;
> +    }
> +}
> +
> +/* { dg-final { scan-tree-dump-times "out of loop 2" 4 "lim2" } } */
> +/* { dg-final { scan-tree-dump-times "out of loop 1" 3 "lim2" } } */
> +
> diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
> new file mode 100644
> index 00000000000..bc60a040a70
> --- /dev/null
> +++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
> @@ -0,0 +1,25 @@
> +/* { dg-do compile  } */
> +/* { dg-options "-O2 -fdump-tree-lim2-details" } */
> +
> +/* Test that `count' is not hoisted out of loop when bb is cold.  */
> +
> +int count;
> +volatile int x;
> +
> +struct obj {
> +  int data;
> +  struct obj *next;
> +
> +} *q;
> +
> +void
> +func (int m)
> +{
> +  struct obj *p;
> +  for (int i = 0; i < m; i++)
> +    if (__builtin_expect (x, 0))
> +      count++;
> +
> +}
> +
> +/* { dg-final { scan-tree-dump-not "Executing store motion of" "lim2"  }  } */
> diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
> new file mode 100644
> index 00000000000..ffe6f8f699d
> --- /dev/null
> +++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
> @@ -0,0 +1,35 @@
> +/* { dg-do compile  } */
> +/* { dg-options "-O2 -fdump-tree-lim2-details" } */
> +
> +/* Test that `data' and 'data1' is not hoisted out of inner loop and outer loop
> +   when it is in cold loop.  */
> +
> +int count;
> +volatile int x;
> +
> +struct obj {
> +  int data;
> +  int data1;
> +  struct obj *next;
> +};
> +
> +void
> +func (int m, int n, int k, struct obj *a)
> +{
> +  struct obj *q = a;
> +  for (int j = 0; j < m; j++)
> +    if (__builtin_expect (m, 0))
> +      for (int i = 0; i < m; i++)
> +       {
> +         if (__builtin_expect (x, 0))
> +           {
> +             count++;
> +             q->data += 3; /* Not hoisted out to inner loop. */
> +           }
> +         count += n;
> +         q->data1 += k; /* Not hoisted out to outer loop. */
> +       }
> +}
> +
> +/* { dg-final { scan-tree-dump-not "Executing store motion of" "lim2"  }  } */
> +
> diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c
> new file mode 100644
> index 00000000000..16ba4ceb8ab
> --- /dev/null
> +++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c
> @@ -0,0 +1,32 @@
> +/* { dg-do compile } */
> +/* { dg-options "-O2 -fdump-tree-lim2-details" } */
> +
> +volatile int x;
> +volatile int y;
> +void
> +bar (int, char *, char *);
> +void
> +foo (int *a, int n, int m, int s, int t)
> +{
> +  int i;
> +  int j;
> +  int k;
> +
> +  for (i = 0; i < m; i++) // Loop 1
> +    {
> +      if (__builtin_expect (x, 0))
> +       for (j = 0; j < n; j++) // Loop 2
> +         if (__builtin_expect (y, 0))
> +           for (k = 0; k < n; k++) // Loop 3
> +             {
> +               bar (s / 5, "one", "two");
> +               a[t] = s;
> +             }
> +      a[t] = t;
> +    }
> +}
> +
> +/* { dg-final { scan-tree-dump-times "out of loop 3" 4 "lim2" } } */
> +/* { dg-final { scan-tree-dump-times "out of loop 1" 3 "lim2" } } */
> +
> +
> --
> 2.27.0.90.geebb51ba8c
>
>
>

[PATCH v7 2/2] Don't move cold code out of loop by checking bb count

[Xionghu Luo]

On 2021/11/4 21:00, Richard Biener wrote:
> On Wed, Nov 3, 2021 at 2:29 PM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>>
>>
>>> +  while (outmost_loop != loop)
>>> +    {
>>> +      if (bb_colder_than_loop_preheader (loop_preheader_edge
>>> (outmost_loop)->src,
>>> +                                        loop_preheader_edge (cold_loop)->src))
>>> +       cold_loop = outmost_loop;
>>> +      outmost_loop = superloop_at_depth (loop, loop_depth (outmost_loop) + 1);
>>> +    }
>>>
>>> could be instead written as
>>>
>>>   coldest_loop = coldest_outermost_loop[loop->num];
>>>   if (loop_depth (coldest_loop) < loop_depth (outermost_loop))
>>>     return outermost_loop;
>>>   return coldest_loop;
>>>
>>> ?  And in the usual case coldest_outermost_loop[L] would be the loop tree root.
>>> It should be possible to compute such cache in a DFS walk of the loop tree
>>> (the loop iterator by default visits in such order).
>>
>>
>> Thanks.  Updated the patch with your suggestion.  Not sure whether it strictly
>> conforms to your comments.  Though the patch passed all my added tests(coverage not enough),
>> I am still a bit worried if pre-computed coldest_loop is outside of outermost_loop, but
>> outermost_loop is not the COLDEST LOOP, i.e. (outer->inner)
>>
>>  [loop tree root, coldest_loop, outermost_loop,..., second_coldest_loop, ..., loop],
>>
>> then function find_coldest_out_loop will return a loop NOT accord with our
>> expectation, that should return second_coldest_loop instead of outermost_loop?
> Hmm, interesting - yes.  I guess the common case will be that the pre-computed
> outermost loop will be the loop at depth 1 since outer loops tend to
> be colder than
> inner loops?  That would then defeat the whole exercise.

It is not easy to construct such cases, But finally I got below results,

1) many cases inner loop is hotter than outer loop, for example:

loop 1's coldest_outermost_loop is 1, colder_than_inner_loop is NULL
loop 2's coldest_outermost_loop is 1, colder_than_inner_loop is 1
loop 3's coldest_outermost_loop is 1, colder_than_inner_loop is 2
loop 4's coldest_outermost_loop is 1, colder_than_inner_loop is 2


2) But there are also cases inner loop is colder than outer loop, like:

loop 1's coldest outermost loop is 1, colder_than_inner_loop is NULL
loop 2's coldest outermost loop is 2, colder_than_inner_loop is NULL
loop 3's coldest outermost loop is 3, colder_than_inner_loop is NULL


> 
> To optimize the common case but not avoiding iteration in the cases we care
> about we could instead cache the next outermost loop that is _not_ colder
> than loop.  So for your [ ... ] example above we'd have> hotter_than_inner_loop[loop] == outer (second_coldest_loop), where the
> candidate would then be 'second_coldest_loop' and we'd then iterate
> to hotter_than_inner_loop[hotter_than_inner_loop[loop]] to find the next
> cold candidate we can compare against?  For the common case we'd
> have hotter_than_inner_loop[looo] == NULL (no such loop) and we then
> simply pick 'outermost_loop'.

Thanks.  It was difficult to understand, but finally I got to know what you
want to express :)

We should cache the next loop that is *colder* than loop instead of '_not_ colder
than loop', and 'hotter_than_inner_loop' should be 'colder_than_inner_loop',
then it makes sense if the coldest loop is outside of outermost loop, continue to
find a colder loop between outermost loop and current loop in
colder_than_inner_loop[loop->num]?  Hope I understood you correctly...

> 
> One comment on the patch itself below.
> 

The loop in fill_cold_out_loop is also removed in the updated v7 patch.



[PATCH v7 2/2] Don't move cold code out of loop by checking bb count

From: Xiong Hu Luo <luoxhu@linux.ibm.com>

v7 changes:
1. Refine get_coldest_out_loop to replace loop with checking
pre-computed coldest_outermost_loop and colder_than_inner_loop.
2. Add function fill_cold_out_loop, compute coldest_outermost_loop and
colder_than_inner_loop recursively without loop.

v6 changes:
1. Add function fill_coldest_out_loop to pre compute the coldest
outermost loop for each loop.
2. Rename find_coldest_out_loop to get_coldest_out_loop.
3. Add testcase ssa-lim-22.c to differentiate with ssa-lim-19.c.

v5 changes:
1. Refine comments for new functions.
2. Use basic_block instead of count in bb_colder_than_loop_preheader
to align with function name.
3. Refine with simpler implementation for get_coldest_out_loop and
ref_in_loop_hot_body::operator for better understanding.

v4 changes:
1. Sort out profile_count comparision to function bb_cold_than_loop_preheader.
2. Update ref_in_loop_hot_body::operator () to find cold_loop before compare.
3. Split RTL invariant motion part out.
4. Remove aux changes.

v3 changes:
1. Handle max_loop in determine_max_movement instead of outermost_invariant_loop.
2. Remove unnecessary changes.
3. Add for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body) in can_sm_ref_p.
4. "gsi_next (&bsi);" in move_computations_worker is kept since it caused
infinite loop when implementing v1 and the iteration is missed to be
updated actually.

v1: https://gcc.gnu.org/pipermail/gcc-patches/2021-August/576488.html
v2: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/579086.html
v3: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/580211.html
v4: https://gcc.gnu.org/pipermail/gcc-patches/2021-October/581231.html
v5: https://gcc.gnu.org/pipermail/gcc-patches/2021-October/581961.html

There was a patch trying to avoid move cold block out of loop:

https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html

Richard suggested to "never hoist anything from a bb with lower execution
frequency to a bb with higher one in LIM invariantness_dom_walker
before_dom_children".

In gimple LIM analysis, add get_coldest_out_loop to move invariants to
expected target loop, if profile count of the loop bb is colder
than target loop preheader, it won't be hoisted out of loop.
Likely for store motion, if all locations of the REF in loop is cold,
don't do store motion of it.

SPEC2017 performance evaluation shows 1% performance improvement for
intrate GEOMEAN and no obvious regression for others.  Especially,
500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
on P8LE.

gcc/ChangeLog:

	* tree-ssa-loop-im.c (bb_colder_than_loop_preheader): New
	function.
	(get_coldest_out_loop): New function.
	(determine_max_movement): Use get_coldest_out_loop.
	(move_computations_worker): Adjust and fix iteration udpate.
	(class ref_in_loop_hot_body): New functor.
	(ref_in_loop_hot_body::operator): New.
	(can_sm_ref_p): Use for_all_locs_in_loop.
	(fill_cold_out_loop): New.
	(tree_ssa_lim_finalize): Free coldest_outermost_loop and
	colder_than_inner_loop.
	(loop_invariant_motion_in_fun): Call fill_cold_out_loop.

gcc/testsuite/ChangeLog:

	* gcc.dg/tree-ssa/recip-3.c: Adjust.
	* gcc.dg/tree-ssa/ssa-lim-18.c: New test.
	* gcc.dg/tree-ssa/ssa-lim-19.c: New test.
	* gcc.dg/tree-ssa/ssa-lim-20.c: New test.
	* gcc.dg/tree-ssa/ssa-lim-21.c: New test.
	* gcc.dg/tree-ssa/ssa-lim-22.c: New test.
---
 gcc/tree-ssa-loop-im.c                     | 140 ++++++++++++++++++++-
 gcc/testsuite/gcc.dg/tree-ssa/recip-3.c    |   2 +-
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c |  20 +++
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c |  29 +++++
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c |  25 ++++
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c |  35 ++++++
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c |  32 +++++
 7 files changed, 280 insertions(+), 3 deletions(-)
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c

diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
index 4b187c2cdaf..e9b3d0cba93 100644
--- a/gcc/tree-ssa-loop-im.c
+++ b/gcc/tree-ssa-loop-im.c
@@ -146,6 +146,11 @@ public:
 enum dep_kind { lim_raw, sm_war, sm_waw };
 enum dep_state { dep_unknown, dep_independent, dep_dependent };
 
+/* coldest outermost loop for given loop.  */
+class loop **coldest_outermost_loop;
+/* colder outer loop nerest to given loop.  */
+class loop **colder_than_inner_loop;
+
 /* Populate the loop dependence cache of REF for LOOP, KIND with STATE.  */
 
 static void
@@ -417,6 +422,53 @@ movement_possibility (gimple *stmt)
   return ret;
 }
 
+/* Compare the profile count inequality of bb and preheader, it is three-state
+   as stated in profile-count.h, FALSE is returned if inequality cannot be
+   decided.  */
+bool bb_colder_than_loop_preheader (basic_block bb, basic_block preheader)
+{
+  gcc_assert (bb && preheader);
+  return bb->count < preheader->count;
+}
+
+/* Check coldest loop between OUTERMOST_LOOP and LOOP by comparing profile
+   count.
+  It does three steps check:
+  1) Check whether CURR_BB is cold in it's own loop_father, if it is cold, just
+  return NULL which means it should not be moved out at all;
+  2) CURR_BB is NOT cold, check if pre-computed COLDEST_LOOP is outside of
+  OUTERMOST_LOOP, if it is inside of OUTERMOST_LOOP, return the COLDEST_LOOP;
+  3) If COLDEST_LOOP is outside of OUTERMOST_LOOP, check whether there is a
+  colder loop between OUTERMOST_LOOP and loop in pre-computed
+  COLDER_THAN_INNER_LOOP, return it, otherwise return OUTERMOST_LOOP.  */
+
+static class loop *
+get_coldest_out_loop (class loop *outermost_loop, class loop *loop,
+		      basic_block curr_bb)
+{
+  gcc_assert (outermost_loop == loop
+	      || flow_loop_nested_p (outermost_loop, loop));
+
+  /* If bb_colder_than_loop_preheader returns false due to three-state
+    comparision, OUTERMOST_LOOP is returned finally to preserve the behavior.
+    Otherwise, return the coldest loop between OUTERMOST_LOOP and LOOP.  */
+  if (curr_bb
+      && bb_colder_than_loop_preheader (curr_bb,
+					loop_preheader_edge (loop)->src))
+    return NULL;
+
+  class loop *coldest_loop = coldest_outermost_loop[loop->num];
+  if (loop_depth (coldest_loop) < loop_depth (outermost_loop))
+    {
+      if (colder_than_inner_loop[loop->num] != NULL
+	  && loop_depth (outermost_loop)
+	       < loop_depth (colder_than_inner_loop[loop->num]))
+	return colder_than_inner_loop[loop->num];
+      return outermost_loop;
+    }
+  return coldest_loop;
+}
+
 /* Suppose that operand DEF is used inside the LOOP.  Returns the outermost
    loop to that we could move the expression using DEF if it did not have
    other operands, i.e. the outermost loop enclosing LOOP in that the value
@@ -685,7 +737,9 @@ determine_max_movement (gimple *stmt, bool must_preserve_exec)
     level = ALWAYS_EXECUTED_IN (bb);
   else
     level = superloop_at_depth (loop, 1);
-  lim_data->max_loop = level;
+  lim_data->max_loop = get_coldest_out_loop (level, loop, bb);
+  if (!lim_data->max_loop)
+    return false;
 
   if (gphi *phi = dyn_cast <gphi *> (stmt))
     {
@@ -1221,7 +1275,10 @@ move_computations_worker (basic_block bb)
       /* We do not really want to move conditionals out of the loop; we just
 	 placed it here to force its operands to be moved if necessary.  */
       if (gimple_code (stmt) == GIMPLE_COND)
-	continue;
+	{
+	  gsi_next (&bsi);
+	  continue;
+	}
 
       if (dump_file && (dump_flags & TDF_DETAILS))
 	{
@@ -2887,6 +2944,26 @@ ref_indep_loop_p (class loop *loop, im_mem_ref *ref, dep_kind kind)
   return indep_p;
 }
 
+class ref_in_loop_hot_body
+{
+public:
+  ref_in_loop_hot_body (loop *loop_) : l (loop_) {}
+  bool operator () (mem_ref_loc *loc);
+  class loop *l;
+};
+
+/* Check the coldest loop between loop L and innermost loop.  If there is one
+   cold loop between L and INNER_LOOP, store motion can be performed, otherwise
+   no cold loop means no store motion.  get_coldest_out_loop also handles cases
+   when l is inner_loop.  */
+bool
+ref_in_loop_hot_body::operator () (mem_ref_loc *loc)
+{
+  basic_block curr_bb = gimple_bb (loc->stmt);
+  class loop *inner_loop = curr_bb->loop_father;
+  return get_coldest_out_loop (l, inner_loop, curr_bb);
+}
+
 
 /* Returns true if we can perform store motion of REF from LOOP.  */
 
@@ -2941,6 +3018,12 @@ can_sm_ref_p (class loop *loop, im_mem_ref *ref)
   if (!ref_indep_loop_p (loop, ref, sm_war))
     return false;
 
+  /* Verify whether the candidate is hot for LOOP.  Only do store motion if the
+    candidate's profile count is hot.  Statement in cold BB shouldn't be moved
+    out of it's loop_father.  */
+  if (!for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body (loop)))
+    return false;
+
   return true;
 }
 
@@ -3153,6 +3236,48 @@ fill_always_executed_in (void)
     fill_always_executed_in_1 (loop, contains_call);
 }
 
+/* Find the coldest loop preheader for LOOP, also find the nearest colder loop
+   to LOOP.  Then recursively iterate each inner loop.  */
+
+void
+fill_cold_out_loop (class loop *coldest_loop, class loop *outer_loop,
+		    class loop *loop)
+{
+  class loop *colder_loop = NULL;
+  if (outer_loop)
+    {
+      if (bb_colder_than_loop_preheader (loop_preheader_edge (outer_loop)->src,
+					 loop_preheader_edge (loop)->src))
+	colder_loop = outer_loop;
+      else if (colder_than_inner_loop[outer_loop->num]
+	       && bb_colder_than_loop_preheader (
+		 loop_preheader_edge (colder_than_inner_loop[outer_loop->num])
+		   ->src,
+		 loop_preheader_edge (loop)->src))
+	colder_loop = colder_than_inner_loop[outer_loop->num];
+    }
+
+  if (bb_colder_than_loop_preheader (loop_preheader_edge (loop)->src,
+				     loop_preheader_edge (coldest_loop)->src))
+    coldest_loop = loop;
+
+  coldest_outermost_loop[loop->num] = coldest_loop;
+  colder_than_inner_loop[loop->num] = colder_loop;
+  if (dump_enabled_p ())
+    {
+      dump_printf (MSG_NOTE, "loop %d's coldest_outermost_loop is %d, ",
+		   loop->num, coldest_loop->num);
+      if (colder_loop)
+	dump_printf (MSG_NOTE, "colder_than_inner_loop is %d\n",
+		     colder_loop->num);
+      else
+	dump_printf (MSG_NOTE, "colder_than_inner_loop is NULL\n");
+    }
+
+  class loop *inner_loop;
+  for (inner_loop = loop->inner; inner_loop; inner_loop = inner_loop->next)
+    fill_cold_out_loop (coldest_loop, loop, inner_loop);
+}
 
 /* Compute the global information needed by the loop invariant motion pass.  */
 
@@ -3237,6 +3362,9 @@ tree_ssa_lim_finalize (void)
     free_affine_expand_cache (&memory_accesses.ttae_cache);
 
   free (bb_loop_postorder);
+
+  free (coldest_outermost_loop);
+  free (colder_than_inner_loop);
 }
 
 /* Moves invariants from loops.  Only "expensive" invariants are moved out --
@@ -3256,6 +3384,14 @@ loop_invariant_motion_in_fun (function *fun, bool store_motion)
   /* Fills ALWAYS_EXECUTED_IN information for basic blocks.  */
   fill_always_executed_in ();
 
+  /* Pre-compute coldest outermost loop and nearest colder loop  of each loop.
+   */
+  class loop *loop;
+  coldest_outermost_loop = XNEWVEC (class loop *, number_of_loops (cfun));
+  colder_than_inner_loop = XNEWVEC (class loop *, number_of_loops (cfun));
+  for (loop = current_loops->tree_root->inner; loop != NULL; loop = loop->next)
+    fill_cold_out_loop (loop, NULL, loop);
+
   int *rpo = XNEWVEC (int, last_basic_block_for_fn (fun));
   int n = pre_and_rev_post_order_compute_fn (fun, NULL, rpo, false);
 
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
index 638bf38db8c..641c91e719e 100644
--- a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
+++ b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
@@ -23,4 +23,4 @@ float h ()
 	F[0] += E / d;
 }
 
-/* { dg-final { scan-tree-dump-times " / " 1 "recip" } } */
+/* { dg-final { scan-tree-dump-times " / " 5 "recip" } } */
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
new file mode 100644
index 00000000000..7326a230b3f
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
@@ -0,0 +1,20 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+volatile int x;
+void
+bar (int, char *, char *);
+void
+foo (int *a, int n, int k)
+{
+  int i;
+
+  for (i = 0; i < n; i++)
+    {
+      if (__builtin_expect (x, 0))
+	bar (k / 5, "one", "two");
+      a[i] = k;
+    }
+}
+
+/* { dg-final { scan-tree-dump-not "out of loop 1" "lim2" } } */
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
new file mode 100644
index 00000000000..51c1913d003
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
@@ -0,0 +1,29 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+volatile int x;
+void
+bar (int, char *, char *);
+void
+foo (int *a, int n, int m, int s, int t)
+{
+  int i;
+  int j;
+  int k;
+
+  for (i = 0; i < m; i++) // Loop 1
+    {
+      if (__builtin_expect (x, 0))
+	for (j = 0; j < n; j++) // Loop 2
+	  for (k = 0; k < n; k++) // Loop 3
+	    {
+	      bar (s / 5, "one", "two");
+	      a[t] = s;
+	    }
+      a[t] = t;
+    }
+}
+
+/* { dg-final { scan-tree-dump-times "out of loop 2" 4 "lim2" } } */
+/* { dg-final { scan-tree-dump-times "out of loop 1" 3 "lim2" } } */
+
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
new file mode 100644
index 00000000000..bc60a040a70
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
@@ -0,0 +1,25 @@
+/* { dg-do compile  } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+/* Test that `count' is not hoisted out of loop when bb is cold.  */
+
+int count;
+volatile int x;
+
+struct obj {
+  int data;
+  struct obj *next;
+
+} *q;
+
+void
+func (int m)
+{
+  struct obj *p;
+  for (int i = 0; i < m; i++)
+    if (__builtin_expect (x, 0))
+      count++;
+
+}
+
+/* { dg-final { scan-tree-dump-not "Executing store motion of" "lim2"  }  } */
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
new file mode 100644
index 00000000000..ffe6f8f699d
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
@@ -0,0 +1,35 @@
+/* { dg-do compile  } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+/* Test that `data' and 'data1' is not hoisted out of inner loop and outer loop
+   when it is in cold loop.  */
+
+int count;
+volatile int x;
+
+struct obj {
+  int data;
+  int data1;
+  struct obj *next;
+};
+
+void
+func (int m, int n, int k, struct obj *a)
+{
+  struct obj *q = a;
+  for (int j = 0; j < m; j++)
+    if (__builtin_expect (m, 0))
+      for (int i = 0; i < m; i++)
+	{
+	  if (__builtin_expect (x, 0))
+	    {
+	      count++;
+	      q->data += 3; /* Not hoisted out to inner loop. */
+	    }
+	  count += n;
+	  q->data1 += k; /* Not hoisted out to outer loop. */
+	}
+}
+
+/* { dg-final { scan-tree-dump-not "Executing store motion of" "lim2"  }  } */
+
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c
new file mode 100644
index 00000000000..16ba4ceb8ab
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c
@@ -0,0 +1,32 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+volatile int x;
+volatile int y;
+void
+bar (int, char *, char *);
+void
+foo (int *a, int n, int m, int s, int t)
+{
+  int i;
+  int j;
+  int k;
+
+  for (i = 0; i < m; i++) // Loop 1
+    {
+      if (__builtin_expect (x, 0))
+	for (j = 0; j < n; j++) // Loop 2
+	  if (__builtin_expect (y, 0))
+	    for (k = 0; k < n; k++) // Loop 3
+	      {
+		bar (s / 5, "one", "two");
+		a[t] = s;
+	      }
+      a[t] = t;
+    }
+}
+
+/* { dg-final { scan-tree-dump-times "out of loop 3" 4 "lim2" } } */
+/* { dg-final { scan-tree-dump-times "out of loop 1" 3 "lim2" } } */
+
+
-- 
2.27.0.90.geebb51ba8c

Ping: [PATCH v7 2/2] Don't move cold code out of loop by checking bb count

[Xionghu Luo]

Gentle ping and is this patch still suitable for stage 3?  Thanks.


[PATCH v7 2/2] Don't move cold code out of loop by checking bb count

https://gcc.gnu.org/pipermail/gcc-patches/2021-November/583911.html



On 2021/11/10 11:08, Xionghu Luo via Gcc-patches wrote:
> 
> 
> On 2021/11/4 21:00, Richard Biener wrote:
>> On Wed, Nov 3, 2021 at 2:29 PM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>>>
>>>
>>>> +  while (outmost_loop != loop)
>>>> +    {
>>>> +      if (bb_colder_than_loop_preheader (loop_preheader_edge
>>>> (outmost_loop)->src,
>>>> +                                        loop_preheader_edge (cold_loop)->src))
>>>> +       cold_loop = outmost_loop;
>>>> +      outmost_loop = superloop_at_depth (loop, loop_depth (outmost_loop) + 1);
>>>> +    }
>>>>
>>>> could be instead written as
>>>>
>>>>   coldest_loop = coldest_outermost_loop[loop->num];
>>>>   if (loop_depth (coldest_loop) < loop_depth (outermost_loop))
>>>>     return outermost_loop;
>>>>   return coldest_loop;
>>>>
>>>> ?  And in the usual case coldest_outermost_loop[L] would be the loop tree root.
>>>> It should be possible to compute such cache in a DFS walk of the loop tree
>>>> (the loop iterator by default visits in such order).
>>>
>>>
>>> Thanks.  Updated the patch with your suggestion.  Not sure whether it strictly
>>> conforms to your comments.  Though the patch passed all my added tests(coverage not enough),
>>> I am still a bit worried if pre-computed coldest_loop is outside of outermost_loop, but
>>> outermost_loop is not the COLDEST LOOP, i.e. (outer->inner)
>>>
>>>  [loop tree root, coldest_loop, outermost_loop,..., second_coldest_loop, ..., loop],
>>>
>>> then function find_coldest_out_loop will return a loop NOT accord with our
>>> expectation, that should return second_coldest_loop instead of outermost_loop?
>> Hmm, interesting - yes.  I guess the common case will be that the pre-computed
>> outermost loop will be the loop at depth 1 since outer loops tend to
>> be colder than
>> inner loops?  That would then defeat the whole exercise.
> 
> It is not easy to construct such cases, But finally I got below results,
> 
> 1) many cases inner loop is hotter than outer loop, for example:
> 
> loop 1's coldest_outermost_loop is 1, colder_than_inner_loop is NULL
> loop 2's coldest_outermost_loop is 1, colder_than_inner_loop is 1
> loop 3's coldest_outermost_loop is 1, colder_than_inner_loop is 2
> loop 4's coldest_outermost_loop is 1, colder_than_inner_loop is 2
> 
> 
> 2) But there are also cases inner loop is colder than outer loop, like:
> 
> loop 1's coldest outermost loop is 1, colder_than_inner_loop is NULL
> loop 2's coldest outermost loop is 2, colder_than_inner_loop is NULL
> loop 3's coldest outermost loop is 3, colder_than_inner_loop is NULL
> 
> 
>>
>> To optimize the common case but not avoiding iteration in the cases we care
>> about we could instead cache the next outermost loop that is _not_ colder
>> than loop.  So for your [ ... ] example above we'd have> hotter_than_inner_loop[loop] == outer (second_coldest_loop), where the
>> candidate would then be 'second_coldest_loop' and we'd then iterate
>> to hotter_than_inner_loop[hotter_than_inner_loop[loop]] to find the next
>> cold candidate we can compare against?  For the common case we'd
>> have hotter_than_inner_loop[looo] == NULL (no such loop) and we then
>> simply pick 'outermost_loop'.
> 
> Thanks.  It was difficult to understand, but finally I got to know what you
> want to express :)
> 
> We should cache the next loop that is *colder* than loop instead of '_not_ colder
> than loop', and 'hotter_than_inner_loop' should be 'colder_than_inner_loop',
> then it makes sense if the coldest loop is outside of outermost loop, continue to
> find a colder loop between outermost loop and current loop in
> colder_than_inner_loop[loop->num]?  Hope I understood you correctly...
> 
>>
>> One comment on the patch itself below.
>>
> 
> The loop in fill_cold_out_loop is also removed in the updated v7 patch.
> 
> 
> 
> [PATCH v7 2/2] Don't move cold code out of loop by checking bb count
> 
> From: Xiong Hu Luo <luoxhu@linux.ibm.com>
> 
> v7 changes:
> 1. Refine get_coldest_out_loop to replace loop with checking
> pre-computed coldest_outermost_loop and colder_than_inner_loop.
> 2. Add function fill_cold_out_loop, compute coldest_outermost_loop and
> colder_than_inner_loop recursively without loop.
> 
> v6 changes:
> 1. Add function fill_coldest_out_loop to pre compute the coldest
> outermost loop for each loop.
> 2. Rename find_coldest_out_loop to get_coldest_out_loop.
> 3. Add testcase ssa-lim-22.c to differentiate with ssa-lim-19.c.
> 
> v5 changes:
> 1. Refine comments for new functions.
> 2. Use basic_block instead of count in bb_colder_than_loop_preheader
> to align with function name.
> 3. Refine with simpler implementation for get_coldest_out_loop and
> ref_in_loop_hot_body::operator for better understanding.
> 
> v4 changes:
> 1. Sort out profile_count comparision to function bb_cold_than_loop_preheader.
> 2. Update ref_in_loop_hot_body::operator () to find cold_loop before compare.
> 3. Split RTL invariant motion part out.
> 4. Remove aux changes.
> 
> v3 changes:
> 1. Handle max_loop in determine_max_movement instead of outermost_invariant_loop.
> 2. Remove unnecessary changes.
> 3. Add for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body) in can_sm_ref_p.
> 4. "gsi_next (&bsi);" in move_computations_worker is kept since it caused
> infinite loop when implementing v1 and the iteration is missed to be
> updated actually.
> 
> v1: https://gcc.gnu.org/pipermail/gcc-patches/2021-August/576488.html
> v2: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/579086.html
> v3: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/580211.html
> v4: https://gcc.gnu.org/pipermail/gcc-patches/2021-October/581231.html
> v5: https://gcc.gnu.org/pipermail/gcc-patches/2021-October/581961.html
> 
> There was a patch trying to avoid move cold block out of loop:
> 
> https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html
> 
> Richard suggested to "never hoist anything from a bb with lower execution
> frequency to a bb with higher one in LIM invariantness_dom_walker
> before_dom_children".
> 
> In gimple LIM analysis, add get_coldest_out_loop to move invariants to
> expected target loop, if profile count of the loop bb is colder
> than target loop preheader, it won't be hoisted out of loop.
> Likely for store motion, if all locations of the REF in loop is cold,
> don't do store motion of it.
> 
> SPEC2017 performance evaluation shows 1% performance improvement for
> intrate GEOMEAN and no obvious regression for others.  Especially,
> 500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
> largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
> on P8LE.
> 
> gcc/ChangeLog:
> 
> 	* tree-ssa-loop-im.c (bb_colder_than_loop_preheader): New
> 	function.
> 	(get_coldest_out_loop): New function.
> 	(determine_max_movement): Use get_coldest_out_loop.
> 	(move_computations_worker): Adjust and fix iteration udpate.
> 	(class ref_in_loop_hot_body): New functor.
> 	(ref_in_loop_hot_body::operator): New.
> 	(can_sm_ref_p): Use for_all_locs_in_loop.
> 	(fill_cold_out_loop): New.
> 	(tree_ssa_lim_finalize): Free coldest_outermost_loop and
> 	colder_than_inner_loop.
> 	(loop_invariant_motion_in_fun): Call fill_cold_out_loop.
> 
> gcc/testsuite/ChangeLog:
> 
> 	* gcc.dg/tree-ssa/recip-3.c: Adjust.
> 	* gcc.dg/tree-ssa/ssa-lim-18.c: New test.
> 	* gcc.dg/tree-ssa/ssa-lim-19.c: New test.
> 	* gcc.dg/tree-ssa/ssa-lim-20.c: New test.
> 	* gcc.dg/tree-ssa/ssa-lim-21.c: New test.
> 	* gcc.dg/tree-ssa/ssa-lim-22.c: New test.
> ---
>  gcc/tree-ssa-loop-im.c                     | 140 ++++++++++++++++++++-
>  gcc/testsuite/gcc.dg/tree-ssa/recip-3.c    |   2 +-
>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c |  20 +++
>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c |  29 +++++
>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c |  25 ++++
>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c |  35 ++++++
>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c |  32 +++++
>  7 files changed, 280 insertions(+), 3 deletions(-)
>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c
> 
> diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
> index 4b187c2cdaf..e9b3d0cba93 100644
> --- a/gcc/tree-ssa-loop-im.c
> +++ b/gcc/tree-ssa-loop-im.c
> @@ -146,6 +146,11 @@ public:
>  enum dep_kind { lim_raw, sm_war, sm_waw };
>  enum dep_state { dep_unknown, dep_independent, dep_dependent };
> 
> +/* coldest outermost loop for given loop.  */
> +class loop **coldest_outermost_loop;
> +/* colder outer loop nerest to given loop.  */
> +class loop **colder_than_inner_loop;
> +
>  /* Populate the loop dependence cache of REF for LOOP, KIND with STATE.  */
> 
>  static void
> @@ -417,6 +422,53 @@ movement_possibility (gimple *stmt)
>    return ret;
>  }
> 
> +/* Compare the profile count inequality of bb and preheader, it is three-state
> +   as stated in profile-count.h, FALSE is returned if inequality cannot be
> +   decided.  */
> +bool bb_colder_than_loop_preheader (basic_block bb, basic_block preheader)
> +{
> +  gcc_assert (bb && preheader);
> +  return bb->count < preheader->count;
> +}
> +
> +/* Check coldest loop between OUTERMOST_LOOP and LOOP by comparing profile
> +   count.
> +  It does three steps check:
> +  1) Check whether CURR_BB is cold in it's own loop_father, if it is cold, just
> +  return NULL which means it should not be moved out at all;
> +  2) CURR_BB is NOT cold, check if pre-computed COLDEST_LOOP is outside of
> +  OUTERMOST_LOOP, if it is inside of OUTERMOST_LOOP, return the COLDEST_LOOP;
> +  3) If COLDEST_LOOP is outside of OUTERMOST_LOOP, check whether there is a
> +  colder loop between OUTERMOST_LOOP and loop in pre-computed
> +  COLDER_THAN_INNER_LOOP, return it, otherwise return OUTERMOST_LOOP.  */
> +
> +static class loop *
> +get_coldest_out_loop (class loop *outermost_loop, class loop *loop,
> +		      basic_block curr_bb)
> +{
> +  gcc_assert (outermost_loop == loop
> +	      || flow_loop_nested_p (outermost_loop, loop));
> +
> +  /* If bb_colder_than_loop_preheader returns false due to three-state
> +    comparision, OUTERMOST_LOOP is returned finally to preserve the behavior.
> +    Otherwise, return the coldest loop between OUTERMOST_LOOP and LOOP.  */
> +  if (curr_bb
> +      && bb_colder_than_loop_preheader (curr_bb,
> +					loop_preheader_edge (loop)->src))
> +    return NULL;
> +
> +  class loop *coldest_loop = coldest_outermost_loop[loop->num];
> +  if (loop_depth (coldest_loop) < loop_depth (outermost_loop))
> +    {
> +      if (colder_than_inner_loop[loop->num] != NULL
> +	  && loop_depth (outermost_loop)
> +	       < loop_depth (colder_than_inner_loop[loop->num]))
> +	return colder_than_inner_loop[loop->num];
> +      return outermost_loop;
> +    }
> +  return coldest_loop;
> +}
> +
>  /* Suppose that operand DEF is used inside the LOOP.  Returns the outermost
>     loop to that we could move the expression using DEF if it did not have
>     other operands, i.e. the outermost loop enclosing LOOP in that the value
> @@ -685,7 +737,9 @@ determine_max_movement (gimple *stmt, bool must_preserve_exec)
>      level = ALWAYS_EXECUTED_IN (bb);
>    else
>      level = superloop_at_depth (loop, 1);
> -  lim_data->max_loop = level;
> +  lim_data->max_loop = get_coldest_out_loop (level, loop, bb);
> +  if (!lim_data->max_loop)
> +    return false;
> 
>    if (gphi *phi = dyn_cast <gphi *> (stmt))
>      {
> @@ -1221,7 +1275,10 @@ move_computations_worker (basic_block bb)
>        /* We do not really want to move conditionals out of the loop; we just
>  	 placed it here to force its operands to be moved if necessary.  */
>        if (gimple_code (stmt) == GIMPLE_COND)
> -	continue;
> +	{
> +	  gsi_next (&bsi);
> +	  continue;
> +	}
> 
>        if (dump_file && (dump_flags & TDF_DETAILS))
>  	{
> @@ -2887,6 +2944,26 @@ ref_indep_loop_p (class loop *loop, im_mem_ref *ref, dep_kind kind)
>    return indep_p;
>  }
> 
> +class ref_in_loop_hot_body
> +{
> +public:
> +  ref_in_loop_hot_body (loop *loop_) : l (loop_) {}
> +  bool operator () (mem_ref_loc *loc);
> +  class loop *l;
> +};
> +
> +/* Check the coldest loop between loop L and innermost loop.  If there is one
> +   cold loop between L and INNER_LOOP, store motion can be performed, otherwise
> +   no cold loop means no store motion.  get_coldest_out_loop also handles cases
> +   when l is inner_loop.  */
> +bool
> +ref_in_loop_hot_body::operator () (mem_ref_loc *loc)
> +{
> +  basic_block curr_bb = gimple_bb (loc->stmt);
> +  class loop *inner_loop = curr_bb->loop_father;
> +  return get_coldest_out_loop (l, inner_loop, curr_bb);
> +}
> +
> 
>  /* Returns true if we can perform store motion of REF from LOOP.  */
> 
> @@ -2941,6 +3018,12 @@ can_sm_ref_p (class loop *loop, im_mem_ref *ref)
>    if (!ref_indep_loop_p (loop, ref, sm_war))
>      return false;
> 
> +  /* Verify whether the candidate is hot for LOOP.  Only do store motion if the
> +    candidate's profile count is hot.  Statement in cold BB shouldn't be moved
> +    out of it's loop_father.  */
> +  if (!for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body (loop)))
> +    return false;
> +
>    return true;
>  }
> 
> @@ -3153,6 +3236,48 @@ fill_always_executed_in (void)
>      fill_always_executed_in_1 (loop, contains_call);
>  }
> 
> +/* Find the coldest loop preheader for LOOP, also find the nearest colder loop
> +   to LOOP.  Then recursively iterate each inner loop.  */
> +
> +void
> +fill_cold_out_loop (class loop *coldest_loop, class loop *outer_loop,
> +		    class loop *loop)
> +{
> +  class loop *colder_loop = NULL;
> +  if (outer_loop)
> +    {
> +      if (bb_colder_than_loop_preheader (loop_preheader_edge (outer_loop)->src,
> +					 loop_preheader_edge (loop)->src))
> +	colder_loop = outer_loop;
> +      else if (colder_than_inner_loop[outer_loop->num]
> +	       && bb_colder_than_loop_preheader (
> +		 loop_preheader_edge (colder_than_inner_loop[outer_loop->num])
> +		   ->src,
> +		 loop_preheader_edge (loop)->src))
> +	colder_loop = colder_than_inner_loop[outer_loop->num];
> +    }
> +
> +  if (bb_colder_than_loop_preheader (loop_preheader_edge (loop)->src,
> +				     loop_preheader_edge (coldest_loop)->src))
> +    coldest_loop = loop;
> +
> +  coldest_outermost_loop[loop->num] = coldest_loop;
> +  colder_than_inner_loop[loop->num] = colder_loop;
> +  if (dump_enabled_p ())
> +    {
> +      dump_printf (MSG_NOTE, "loop %d's coldest_outermost_loop is %d, ",
> +		   loop->num, coldest_loop->num);
> +      if (colder_loop)
> +	dump_printf (MSG_NOTE, "colder_than_inner_loop is %d\n",
> +		     colder_loop->num);
> +      else
> +	dump_printf (MSG_NOTE, "colder_than_inner_loop is NULL\n");
> +    }
> +
> +  class loop *inner_loop;
> +  for (inner_loop = loop->inner; inner_loop; inner_loop = inner_loop->next)
> +    fill_cold_out_loop (coldest_loop, loop, inner_loop);
> +}
> 
>  /* Compute the global information needed by the loop invariant motion pass.  */
> 
> @@ -3237,6 +3362,9 @@ tree_ssa_lim_finalize (void)
>      free_affine_expand_cache (&memory_accesses.ttae_cache);
> 
>    free (bb_loop_postorder);
> +
> +  free (coldest_outermost_loop);
> +  free (colder_than_inner_loop);
>  }
> 
>  /* Moves invariants from loops.  Only "expensive" invariants are moved out --
> @@ -3256,6 +3384,14 @@ loop_invariant_motion_in_fun (function *fun, bool store_motion)
>    /* Fills ALWAYS_EXECUTED_IN information for basic blocks.  */
>    fill_always_executed_in ();
> 
> +  /* Pre-compute coldest outermost loop and nearest colder loop  of each loop.
> +   */
> +  class loop *loop;
> +  coldest_outermost_loop = XNEWVEC (class loop *, number_of_loops (cfun));
> +  colder_than_inner_loop = XNEWVEC (class loop *, number_of_loops (cfun));
> +  for (loop = current_loops->tree_root->inner; loop != NULL; loop = loop->next)
> +    fill_cold_out_loop (loop, NULL, loop);
> +
>    int *rpo = XNEWVEC (int, last_basic_block_for_fn (fun));
>    int n = pre_and_rev_post_order_compute_fn (fun, NULL, rpo, false);
> 
> diff --git a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
> index 638bf38db8c..641c91e719e 100644
> --- a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
> +++ b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
> @@ -23,4 +23,4 @@ float h ()
>  	F[0] += E / d;
>  }
> 
> -/* { dg-final { scan-tree-dump-times " / " 1 "recip" } } */
> +/* { dg-final { scan-tree-dump-times " / " 5 "recip" } } */
> diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
> new file mode 100644
> index 00000000000..7326a230b3f
> --- /dev/null
> +++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
> @@ -0,0 +1,20 @@
> +/* { dg-do compile } */
> +/* { dg-options "-O2 -fdump-tree-lim2-details" } */
> +
> +volatile int x;
> +void
> +bar (int, char *, char *);
> +void
> +foo (int *a, int n, int k)
> +{
> +  int i;
> +
> +  for (i = 0; i < n; i++)
> +    {
> +      if (__builtin_expect (x, 0))
> +	bar (k / 5, "one", "two");
> +      a[i] = k;
> +    }
> +}
> +
> +/* { dg-final { scan-tree-dump-not "out of loop 1" "lim2" } } */
> diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
> new file mode 100644
> index 00000000000..51c1913d003
> --- /dev/null
> +++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
> @@ -0,0 +1,29 @@
> +/* { dg-do compile } */
> +/* { dg-options "-O2 -fdump-tree-lim2-details" } */
> +
> +volatile int x;
> +void
> +bar (int, char *, char *);
> +void
> +foo (int *a, int n, int m, int s, int t)
> +{
> +  int i;
> +  int j;
> +  int k;
> +
> +  for (i = 0; i < m; i++) // Loop 1
> +    {
> +      if (__builtin_expect (x, 0))
> +	for (j = 0; j < n; j++) // Loop 2
> +	  for (k = 0; k < n; k++) // Loop 3
> +	    {
> +	      bar (s / 5, "one", "two");
> +	      a[t] = s;
> +	    }
> +      a[t] = t;
> +    }
> +}
> +
> +/* { dg-final { scan-tree-dump-times "out of loop 2" 4 "lim2" } } */
> +/* { dg-final { scan-tree-dump-times "out of loop 1" 3 "lim2" } } */
> +
> diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
> new file mode 100644
> index 00000000000..bc60a040a70
> --- /dev/null
> +++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
> @@ -0,0 +1,25 @@
> +/* { dg-do compile  } */
> +/* { dg-options "-O2 -fdump-tree-lim2-details" } */
> +
> +/* Test that `count' is not hoisted out of loop when bb is cold.  */
> +
> +int count;
> +volatile int x;
> +
> +struct obj {
> +  int data;
> +  struct obj *next;
> +
> +} *q;
> +
> +void
> +func (int m)
> +{
> +  struct obj *p;
> +  for (int i = 0; i < m; i++)
> +    if (__builtin_expect (x, 0))
> +      count++;
> +
> +}
> +
> +/* { dg-final { scan-tree-dump-not "Executing store motion of" "lim2"  }  } */
> diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
> new file mode 100644
> index 00000000000..ffe6f8f699d
> --- /dev/null
> +++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
> @@ -0,0 +1,35 @@
> +/* { dg-do compile  } */
> +/* { dg-options "-O2 -fdump-tree-lim2-details" } */
> +
> +/* Test that `data' and 'data1' is not hoisted out of inner loop and outer loop
> +   when it is in cold loop.  */
> +
> +int count;
> +volatile int x;
> +
> +struct obj {
> +  int data;
> +  int data1;
> +  struct obj *next;
> +};
> +
> +void
> +func (int m, int n, int k, struct obj *a)
> +{
> +  struct obj *q = a;
> +  for (int j = 0; j < m; j++)
> +    if (__builtin_expect (m, 0))
> +      for (int i = 0; i < m; i++)
> +	{
> +	  if (__builtin_expect (x, 0))
> +	    {
> +	      count++;
> +	      q->data += 3; /* Not hoisted out to inner loop. */
> +	    }
> +	  count += n;
> +	  q->data1 += k; /* Not hoisted out to outer loop. */
> +	}
> +}
> +
> +/* { dg-final { scan-tree-dump-not "Executing store motion of" "lim2"  }  } */
> +
> diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c
> new file mode 100644
> index 00000000000..16ba4ceb8ab
> --- /dev/null
> +++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c
> @@ -0,0 +1,32 @@
> +/* { dg-do compile } */
> +/* { dg-options "-O2 -fdump-tree-lim2-details" } */
> +
> +volatile int x;
> +volatile int y;
> +void
> +bar (int, char *, char *);
> +void
> +foo (int *a, int n, int m, int s, int t)
> +{
> +  int i;
> +  int j;
> +  int k;
> +
> +  for (i = 0; i < m; i++) // Loop 1
> +    {
> +      if (__builtin_expect (x, 0))
> +	for (j = 0; j < n; j++) // Loop 2
> +	  if (__builtin_expect (y, 0))
> +	    for (k = 0; k < n; k++) // Loop 3
> +	      {
> +		bar (s / 5, "one", "two");
> +		a[t] = s;
> +	      }
> +      a[t] = t;
> +    }
> +}
> +
> +/* { dg-final { scan-tree-dump-times "out of loop 3" 4 "lim2" } } */
> +/* { dg-final { scan-tree-dump-times "out of loop 1" 3 "lim2" } } */
> +
> +
> 

-- 
Thanks,
Xionghu

Re: Ping: [PATCH v7 2/2] Don't move cold code out of loop by checking bb count

[Richard Biener]

On Wed, Nov 24, 2021 at 6:15 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>
> Gentle ping and is this patch still suitable for stage 3?  Thanks.

It's on my list to look at, yes.  I'm worried about accuracy of profile counts
though which is why I keep pushing it back - I see the various profile count
fixes are still pending.

I'm thinking of requiring reliable_p () on the counts (but that will restrict
this to FDO).

Anyway, still on my list to look at - sorry for the delays.

Richard.

>
> [PATCH v7 2/2] Don't move cold code out of loop by checking bb count
>
> https://gcc.gnu.org/pipermail/gcc-patches/2021-November/583911.html
>
>
>
> On 2021/11/10 11:08, Xionghu Luo via Gcc-patches wrote:
> >
> >
> > On 2021/11/4 21:00, Richard Biener wrote:
> >> On Wed, Nov 3, 2021 at 2:29 PM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
> >>>
> >>>
> >>>> +  while (outmost_loop != loop)
> >>>> +    {
> >>>> +      if (bb_colder_than_loop_preheader (loop_preheader_edge
> >>>> (outmost_loop)->src,
> >>>> +                                        loop_preheader_edge (cold_loop)->src))
> >>>> +       cold_loop = outmost_loop;
> >>>> +      outmost_loop = superloop_at_depth (loop, loop_depth (outmost_loop) + 1);
> >>>> +    }
> >>>>
> >>>> could be instead written as
> >>>>
> >>>>   coldest_loop = coldest_outermost_loop[loop->num];
> >>>>   if (loop_depth (coldest_loop) < loop_depth (outermost_loop))
> >>>>     return outermost_loop;
> >>>>   return coldest_loop;
> >>>>
> >>>> ?  And in the usual case coldest_outermost_loop[L] would be the loop tree root.
> >>>> It should be possible to compute such cache in a DFS walk of the loop tree
> >>>> (the loop iterator by default visits in such order).
> >>>
> >>>
> >>> Thanks.  Updated the patch with your suggestion.  Not sure whether it strictly
> >>> conforms to your comments.  Though the patch passed all my added tests(coverage not enough),
> >>> I am still a bit worried if pre-computed coldest_loop is outside of outermost_loop, but
> >>> outermost_loop is not the COLDEST LOOP, i.e. (outer->inner)
> >>>
> >>>  [loop tree root, coldest_loop, outermost_loop,..., second_coldest_loop, ..., loop],
> >>>
> >>> then function find_coldest_out_loop will return a loop NOT accord with our
> >>> expectation, that should return second_coldest_loop instead of outermost_loop?
> >> Hmm, interesting - yes.  I guess the common case will be that the pre-computed
> >> outermost loop will be the loop at depth 1 since outer loops tend to
> >> be colder than
> >> inner loops?  That would then defeat the whole exercise.
> >
> > It is not easy to construct such cases, But finally I got below results,
> >
> > 1) many cases inner loop is hotter than outer loop, for example:
> >
> > loop 1's coldest_outermost_loop is 1, colder_than_inner_loop is NULL
> > loop 2's coldest_outermost_loop is 1, colder_than_inner_loop is 1
> > loop 3's coldest_outermost_loop is 1, colder_than_inner_loop is 2
> > loop 4's coldest_outermost_loop is 1, colder_than_inner_loop is 2
> >
> >
> > 2) But there are also cases inner loop is colder than outer loop, like:
> >
> > loop 1's coldest outermost loop is 1, colder_than_inner_loop is NULL
> > loop 2's coldest outermost loop is 2, colder_than_inner_loop is NULL
> > loop 3's coldest outermost loop is 3, colder_than_inner_loop is NULL
> >
> >
> >>
> >> To optimize the common case but not avoiding iteration in the cases we care
> >> about we could instead cache the next outermost loop that is _not_ colder
> >> than loop.  So for your [ ... ] example above we'd have> hotter_than_inner_loop[loop] == outer (second_coldest_loop), where the
> >> candidate would then be 'second_coldest_loop' and we'd then iterate
> >> to hotter_than_inner_loop[hotter_than_inner_loop[loop]] to find the next
> >> cold candidate we can compare against?  For the common case we'd
> >> have hotter_than_inner_loop[looo] == NULL (no such loop) and we then
> >> simply pick 'outermost_loop'.
> >
> > Thanks.  It was difficult to understand, but finally I got to know what you
> > want to express :)
> >
> > We should cache the next loop that is *colder* than loop instead of '_not_ colder
> > than loop', and 'hotter_than_inner_loop' should be 'colder_than_inner_loop',
> > then it makes sense if the coldest loop is outside of outermost loop, continue to
> > find a colder loop between outermost loop and current loop in
> > colder_than_inner_loop[loop->num]?  Hope I understood you correctly...
> >
> >>
> >> One comment on the patch itself below.
> >>
> >
> > The loop in fill_cold_out_loop is also removed in the updated v7 patch.
> >
> >
> >
> > [PATCH v7 2/2] Don't move cold code out of loop by checking bb count
> >
> > From: Xiong Hu Luo <luoxhu@linux.ibm.com>
> >
> > v7 changes:
> > 1. Refine get_coldest_out_loop to replace loop with checking
> > pre-computed coldest_outermost_loop and colder_than_inner_loop.
> > 2. Add function fill_cold_out_loop, compute coldest_outermost_loop and
> > colder_than_inner_loop recursively without loop.
> >
> > v6 changes:
> > 1. Add function fill_coldest_out_loop to pre compute the coldest
> > outermost loop for each loop.
> > 2. Rename find_coldest_out_loop to get_coldest_out_loop.
> > 3. Add testcase ssa-lim-22.c to differentiate with ssa-lim-19.c.
> >
> > v5 changes:
> > 1. Refine comments for new functions.
> > 2. Use basic_block instead of count in bb_colder_than_loop_preheader
> > to align with function name.
> > 3. Refine with simpler implementation for get_coldest_out_loop and
> > ref_in_loop_hot_body::operator for better understanding.
> >
> > v4 changes:
> > 1. Sort out profile_count comparision to function bb_cold_than_loop_preheader.
> > 2. Update ref_in_loop_hot_body::operator () to find cold_loop before compare.
> > 3. Split RTL invariant motion part out.
> > 4. Remove aux changes.
> >
> > v3 changes:
> > 1. Handle max_loop in determine_max_movement instead of outermost_invariant_loop.
> > 2. Remove unnecessary changes.
> > 3. Add for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body) in can_sm_ref_p.
> > 4. "gsi_next (&bsi);" in move_computations_worker is kept since it caused
> > infinite loop when implementing v1 and the iteration is missed to be
> > updated actually.
> >
> > v1: https://gcc.gnu.org/pipermail/gcc-patches/2021-August/576488.html
> > v2: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/579086.html
> > v3: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/580211.html
> > v4: https://gcc.gnu.org/pipermail/gcc-patches/2021-October/581231.html
> > v5: https://gcc.gnu.org/pipermail/gcc-patches/2021-October/581961.html
> >
> > There was a patch trying to avoid move cold block out of loop:
> >
> > https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html
> >
> > Richard suggested to "never hoist anything from a bb with lower execution
> > frequency to a bb with higher one in LIM invariantness_dom_walker
> > before_dom_children".
> >
> > In gimple LIM analysis, add get_coldest_out_loop to move invariants to
> > expected target loop, if profile count of the loop bb is colder
> > than target loop preheader, it won't be hoisted out of loop.
> > Likely for store motion, if all locations of the REF in loop is cold,
> > don't do store motion of it.
> >
> > SPEC2017 performance evaluation shows 1% performance improvement for
> > intrate GEOMEAN and no obvious regression for others.  Especially,
> > 500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
> > largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
> > on P8LE.
> >
> > gcc/ChangeLog:
> >
> >       * tree-ssa-loop-im.c (bb_colder_than_loop_preheader): New
> >       function.
> >       (get_coldest_out_loop): New function.
> >       (determine_max_movement): Use get_coldest_out_loop.
> >       (move_computations_worker): Adjust and fix iteration udpate.
> >       (class ref_in_loop_hot_body): New functor.
> >       (ref_in_loop_hot_body::operator): New.
> >       (can_sm_ref_p): Use for_all_locs_in_loop.
> >       (fill_cold_out_loop): New.
> >       (tree_ssa_lim_finalize): Free coldest_outermost_loop and
> >       colder_than_inner_loop.
> >       (loop_invariant_motion_in_fun): Call fill_cold_out_loop.
> >
> > gcc/testsuite/ChangeLog:
> >
> >       * gcc.dg/tree-ssa/recip-3.c: Adjust.
> >       * gcc.dg/tree-ssa/ssa-lim-18.c: New test.
> >       * gcc.dg/tree-ssa/ssa-lim-19.c: New test.
> >       * gcc.dg/tree-ssa/ssa-lim-20.c: New test.
> >       * gcc.dg/tree-ssa/ssa-lim-21.c: New test.
> >       * gcc.dg/tree-ssa/ssa-lim-22.c: New test.
> > ---
> >  gcc/tree-ssa-loop-im.c                     | 140 ++++++++++++++++++++-
> >  gcc/testsuite/gcc.dg/tree-ssa/recip-3.c    |   2 +-
> >  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c |  20 +++
> >  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c |  29 +++++
> >  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c |  25 ++++
> >  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c |  35 ++++++
> >  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c |  32 +++++
> >  7 files changed, 280 insertions(+), 3 deletions(-)
> >  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
> >  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
> >  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
> >  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
> >  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c
> >
> > diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
> > index 4b187c2cdaf..e9b3d0cba93 100644
> > --- a/gcc/tree-ssa-loop-im.c
> > +++ b/gcc/tree-ssa-loop-im.c
> > @@ -146,6 +146,11 @@ public:
> >  enum dep_kind { lim_raw, sm_war, sm_waw };
> >  enum dep_state { dep_unknown, dep_independent, dep_dependent };
> >
> > +/* coldest outermost loop for given loop.  */
> > +class loop **coldest_outermost_loop;
> > +/* colder outer loop nerest to given loop.  */
> > +class loop **colder_than_inner_loop;
> > +
> >  /* Populate the loop dependence cache of REF for LOOP, KIND with STATE.  */
> >
> >  static void
> > @@ -417,6 +422,53 @@ movement_possibility (gimple *stmt)
> >    return ret;
> >  }
> >
> > +/* Compare the profile count inequality of bb and preheader, it is three-state
> > +   as stated in profile-count.h, FALSE is returned if inequality cannot be
> > +   decided.  */
> > +bool bb_colder_than_loop_preheader (basic_block bb, basic_block preheader)
> > +{
> > +  gcc_assert (bb && preheader);
> > +  return bb->count < preheader->count;
> > +}
> > +
> > +/* Check coldest loop between OUTERMOST_LOOP and LOOP by comparing profile
> > +   count.
> > +  It does three steps check:
> > +  1) Check whether CURR_BB is cold in it's own loop_father, if it is cold, just
> > +  return NULL which means it should not be moved out at all;
> > +  2) CURR_BB is NOT cold, check if pre-computed COLDEST_LOOP is outside of
> > +  OUTERMOST_LOOP, if it is inside of OUTERMOST_LOOP, return the COLDEST_LOOP;
> > +  3) If COLDEST_LOOP is outside of OUTERMOST_LOOP, check whether there is a
> > +  colder loop between OUTERMOST_LOOP and loop in pre-computed
> > +  COLDER_THAN_INNER_LOOP, return it, otherwise return OUTERMOST_LOOP.  */
> > +
> > +static class loop *
> > +get_coldest_out_loop (class loop *outermost_loop, class loop *loop,
> > +                   basic_block curr_bb)
> > +{
> > +  gcc_assert (outermost_loop == loop
> > +           || flow_loop_nested_p (outermost_loop, loop));
> > +
> > +  /* If bb_colder_than_loop_preheader returns false due to three-state
> > +    comparision, OUTERMOST_LOOP is returned finally to preserve the behavior.
> > +    Otherwise, return the coldest loop between OUTERMOST_LOOP and LOOP.  */
> > +  if (curr_bb
> > +      && bb_colder_than_loop_preheader (curr_bb,
> > +                                     loop_preheader_edge (loop)->src))
> > +    return NULL;
> > +
> > +  class loop *coldest_loop = coldest_outermost_loop[loop->num];
> > +  if (loop_depth (coldest_loop) < loop_depth (outermost_loop))
> > +    {
> > +      if (colder_than_inner_loop[loop->num] != NULL
> > +       && loop_depth (outermost_loop)
> > +            < loop_depth (colder_than_inner_loop[loop->num]))
> > +     return colder_than_inner_loop[loop->num];
> > +      return outermost_loop;
> > +    }
> > +  return coldest_loop;
> > +}
> > +
> >  /* Suppose that operand DEF is used inside the LOOP.  Returns the outermost
> >     loop to that we could move the expression using DEF if it did not have
> >     other operands, i.e. the outermost loop enclosing LOOP in that the value
> > @@ -685,7 +737,9 @@ determine_max_movement (gimple *stmt, bool must_preserve_exec)
> >      level = ALWAYS_EXECUTED_IN (bb);
> >    else
> >      level = superloop_at_depth (loop, 1);
> > -  lim_data->max_loop = level;
> > +  lim_data->max_loop = get_coldest_out_loop (level, loop, bb);
> > +  if (!lim_data->max_loop)
> > +    return false;
> >
> >    if (gphi *phi = dyn_cast <gphi *> (stmt))
> >      {
> > @@ -1221,7 +1275,10 @@ move_computations_worker (basic_block bb)
> >        /* We do not really want to move conditionals out of the loop; we just
> >        placed it here to force its operands to be moved if necessary.  */
> >        if (gimple_code (stmt) == GIMPLE_COND)
> > -     continue;
> > +     {
> > +       gsi_next (&bsi);
> > +       continue;
> > +     }
> >
> >        if (dump_file && (dump_flags & TDF_DETAILS))
> >       {
> > @@ -2887,6 +2944,26 @@ ref_indep_loop_p (class loop *loop, im_mem_ref *ref, dep_kind kind)
> >    return indep_p;
> >  }
> >
> > +class ref_in_loop_hot_body
> > +{
> > +public:
> > +  ref_in_loop_hot_body (loop *loop_) : l (loop_) {}
> > +  bool operator () (mem_ref_loc *loc);
> > +  class loop *l;
> > +};
> > +
> > +/* Check the coldest loop between loop L and innermost loop.  If there is one
> > +   cold loop between L and INNER_LOOP, store motion can be performed, otherwise
> > +   no cold loop means no store motion.  get_coldest_out_loop also handles cases
> > +   when l is inner_loop.  */
> > +bool
> > +ref_in_loop_hot_body::operator () (mem_ref_loc *loc)
> > +{
> > +  basic_block curr_bb = gimple_bb (loc->stmt);
> > +  class loop *inner_loop = curr_bb->loop_father;
> > +  return get_coldest_out_loop (l, inner_loop, curr_bb);
> > +}
> > +
> >
> >  /* Returns true if we can perform store motion of REF from LOOP.  */
> >
> > @@ -2941,6 +3018,12 @@ can_sm_ref_p (class loop *loop, im_mem_ref *ref)
> >    if (!ref_indep_loop_p (loop, ref, sm_war))
> >      return false;
> >
> > +  /* Verify whether the candidate is hot for LOOP.  Only do store motion if the
> > +    candidate's profile count is hot.  Statement in cold BB shouldn't be moved
> > +    out of it's loop_father.  */
> > +  if (!for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body (loop)))
> > +    return false;
> > +
> >    return true;
> >  }
> >
> > @@ -3153,6 +3236,48 @@ fill_always_executed_in (void)
> >      fill_always_executed_in_1 (loop, contains_call);
> >  }
> >
> > +/* Find the coldest loop preheader for LOOP, also find the nearest colder loop
> > +   to LOOP.  Then recursively iterate each inner loop.  */
> > +
> > +void
> > +fill_cold_out_loop (class loop *coldest_loop, class loop *outer_loop,
> > +                 class loop *loop)
> > +{
> > +  class loop *colder_loop = NULL;
> > +  if (outer_loop)
> > +    {
> > +      if (bb_colder_than_loop_preheader (loop_preheader_edge (outer_loop)->src,
> > +                                      loop_preheader_edge (loop)->src))
> > +     colder_loop = outer_loop;
> > +      else if (colder_than_inner_loop[outer_loop->num]
> > +            && bb_colder_than_loop_preheader (
> > +              loop_preheader_edge (colder_than_inner_loop[outer_loop->num])
> > +                ->src,
> > +              loop_preheader_edge (loop)->src))
> > +     colder_loop = colder_than_inner_loop[outer_loop->num];
> > +    }
> > +
> > +  if (bb_colder_than_loop_preheader (loop_preheader_edge (loop)->src,
> > +                                  loop_preheader_edge (coldest_loop)->src))
> > +    coldest_loop = loop;
> > +
> > +  coldest_outermost_loop[loop->num] = coldest_loop;
> > +  colder_than_inner_loop[loop->num] = colder_loop;
> > +  if (dump_enabled_p ())
> > +    {
> > +      dump_printf (MSG_NOTE, "loop %d's coldest_outermost_loop is %d, ",
> > +                loop->num, coldest_loop->num);
> > +      if (colder_loop)
> > +     dump_printf (MSG_NOTE, "colder_than_inner_loop is %d\n",
> > +                  colder_loop->num);
> > +      else
> > +     dump_printf (MSG_NOTE, "colder_than_inner_loop is NULL\n");
> > +    }
> > +
> > +  class loop *inner_loop;
> > +  for (inner_loop = loop->inner; inner_loop; inner_loop = inner_loop->next)
> > +    fill_cold_out_loop (coldest_loop, loop, inner_loop);
> > +}
> >
> >  /* Compute the global information needed by the loop invariant motion pass.  */
> >
> > @@ -3237,6 +3362,9 @@ tree_ssa_lim_finalize (void)
> >      free_affine_expand_cache (&memory_accesses.ttae_cache);
> >
> >    free (bb_loop_postorder);
> > +
> > +  free (coldest_outermost_loop);
> > +  free (colder_than_inner_loop);
> >  }
> >
> >  /* Moves invariants from loops.  Only "expensive" invariants are moved out --
> > @@ -3256,6 +3384,14 @@ loop_invariant_motion_in_fun (function *fun, bool store_motion)
> >    /* Fills ALWAYS_EXECUTED_IN information for basic blocks.  */
> >    fill_always_executed_in ();
> >
> > +  /* Pre-compute coldest outermost loop and nearest colder loop  of each loop.
> > +   */
> > +  class loop *loop;
> > +  coldest_outermost_loop = XNEWVEC (class loop *, number_of_loops (cfun));
> > +  colder_than_inner_loop = XNEWVEC (class loop *, number_of_loops (cfun));
> > +  for (loop = current_loops->tree_root->inner; loop != NULL; loop = loop->next)
> > +    fill_cold_out_loop (loop, NULL, loop);
> > +
> >    int *rpo = XNEWVEC (int, last_basic_block_for_fn (fun));
> >    int n = pre_and_rev_post_order_compute_fn (fun, NULL, rpo, false);
> >
> > diff --git a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
> > index 638bf38db8c..641c91e719e 100644
> > --- a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
> > +++ b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
> > @@ -23,4 +23,4 @@ float h ()
> >       F[0] += E / d;
> >  }
> >
> > -/* { dg-final { scan-tree-dump-times " / " 1 "recip" } } */
> > +/* { dg-final { scan-tree-dump-times " / " 5 "recip" } } */
> > diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
> > new file mode 100644
> > index 00000000000..7326a230b3f
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
> > @@ -0,0 +1,20 @@
> > +/* { dg-do compile } */
> > +/* { dg-options "-O2 -fdump-tree-lim2-details" } */
> > +
> > +volatile int x;
> > +void
> > +bar (int, char *, char *);
> > +void
> > +foo (int *a, int n, int k)
> > +{
> > +  int i;
> > +
> > +  for (i = 0; i < n; i++)
> > +    {
> > +      if (__builtin_expect (x, 0))
> > +     bar (k / 5, "one", "two");
> > +      a[i] = k;
> > +    }
> > +}
> > +
> > +/* { dg-final { scan-tree-dump-not "out of loop 1" "lim2" } } */
> > diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
> > new file mode 100644
> > index 00000000000..51c1913d003
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
> > @@ -0,0 +1,29 @@
> > +/* { dg-do compile } */
> > +/* { dg-options "-O2 -fdump-tree-lim2-details" } */
> > +
> > +volatile int x;
> > +void
> > +bar (int, char *, char *);
> > +void
> > +foo (int *a, int n, int m, int s, int t)
> > +{
> > +  int i;
> > +  int j;
> > +  int k;
> > +
> > +  for (i = 0; i < m; i++) // Loop 1
> > +    {
> > +      if (__builtin_expect (x, 0))
> > +     for (j = 0; j < n; j++) // Loop 2
> > +       for (k = 0; k < n; k++) // Loop 3
> > +         {
> > +           bar (s / 5, "one", "two");
> > +           a[t] = s;
> > +         }
> > +      a[t] = t;
> > +    }
> > +}
> > +
> > +/* { dg-final { scan-tree-dump-times "out of loop 2" 4 "lim2" } } */
> > +/* { dg-final { scan-tree-dump-times "out of loop 1" 3 "lim2" } } */
> > +
> > diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
> > new file mode 100644
> > index 00000000000..bc60a040a70
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
> > @@ -0,0 +1,25 @@
> > +/* { dg-do compile  } */
> > +/* { dg-options "-O2 -fdump-tree-lim2-details" } */
> > +
> > +/* Test that `count' is not hoisted out of loop when bb is cold.  */
> > +
> > +int count;
> > +volatile int x;
> > +
> > +struct obj {
> > +  int data;
> > +  struct obj *next;
> > +
> > +} *q;
> > +
> > +void
> > +func (int m)
> > +{
> > +  struct obj *p;
> > +  for (int i = 0; i < m; i++)
> > +    if (__builtin_expect (x, 0))
> > +      count++;
> > +
> > +}
> > +
> > +/* { dg-final { scan-tree-dump-not "Executing store motion of" "lim2"  }  } */
> > diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
> > new file mode 100644
> > index 00000000000..ffe6f8f699d
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
> > @@ -0,0 +1,35 @@
> > +/* { dg-do compile  } */
> > +/* { dg-options "-O2 -fdump-tree-lim2-details" } */
> > +
> > +/* Test that `data' and 'data1' is not hoisted out of inner loop and outer loop
> > +   when it is in cold loop.  */
> > +
> > +int count;
> > +volatile int x;
> > +
> > +struct obj {
> > +  int data;
> > +  int data1;
> > +  struct obj *next;
> > +};
> > +
> > +void
> > +func (int m, int n, int k, struct obj *a)
> > +{
> > +  struct obj *q = a;
> > +  for (int j = 0; j < m; j++)
> > +    if (__builtin_expect (m, 0))
> > +      for (int i = 0; i < m; i++)
> > +     {
> > +       if (__builtin_expect (x, 0))
> > +         {
> > +           count++;
> > +           q->data += 3; /* Not hoisted out to inner loop. */
> > +         }
> > +       count += n;
> > +       q->data1 += k; /* Not hoisted out to outer loop. */
> > +     }
> > +}
> > +
> > +/* { dg-final { scan-tree-dump-not "Executing store motion of" "lim2"  }  } */
> > +
> > diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c
> > new file mode 100644
> > index 00000000000..16ba4ceb8ab
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c
> > @@ -0,0 +1,32 @@
> > +/* { dg-do compile } */
> > +/* { dg-options "-O2 -fdump-tree-lim2-details" } */
> > +
> > +volatile int x;
> > +volatile int y;
> > +void
> > +bar (int, char *, char *);
> > +void
> > +foo (int *a, int n, int m, int s, int t)
> > +{
> > +  int i;
> > +  int j;
> > +  int k;
> > +
> > +  for (i = 0; i < m; i++) // Loop 1
> > +    {
> > +      if (__builtin_expect (x, 0))
> > +     for (j = 0; j < n; j++) // Loop 2
> > +       if (__builtin_expect (y, 0))
> > +         for (k = 0; k < n; k++) // Loop 3
> > +           {
> > +             bar (s / 5, "one", "two");
> > +             a[t] = s;
> > +           }
> > +      a[t] = t;
> > +    }
> > +}
> > +
> > +/* { dg-final { scan-tree-dump-times "out of loop 3" 4 "lim2" } } */
> > +/* { dg-final { scan-tree-dump-times "out of loop 1" 3 "lim2" } } */
> > +
> > +
> >
>
> --
> Thanks,
> Xionghu

Re: [PATCH v7 2/2] Don't move cold code out of loop by checking bb count

[Richard Biener]

On Wed, Nov 10, 2021 at 4:08 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>
>
>
> On 2021/11/4 21:00, Richard Biener wrote:
> > On Wed, Nov 3, 2021 at 2:29 PM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
> >>
> >>
> >>> +  while (outmost_loop != loop)
> >>> +    {
> >>> +      if (bb_colder_than_loop_preheader (loop_preheader_edge
> >>> (outmost_loop)->src,
> >>> +                                        loop_preheader_edge (cold_loop)->src))
> >>> +       cold_loop = outmost_loop;
> >>> +      outmost_loop = superloop_at_depth (loop, loop_depth (outmost_loop) + 1);
> >>> +    }
> >>>
> >>> could be instead written as
> >>>
> >>>   coldest_loop = coldest_outermost_loop[loop->num];
> >>>   if (loop_depth (coldest_loop) < loop_depth (outermost_loop))
> >>>     return outermost_loop;
> >>>   return coldest_loop;
> >>>
> >>> ?  And in the usual case coldest_outermost_loop[L] would be the loop tree root.
> >>> It should be possible to compute such cache in a DFS walk of the loop tree
> >>> (the loop iterator by default visits in such order).
> >>
> >>
> >> Thanks.  Updated the patch with your suggestion.  Not sure whether it strictly
> >> conforms to your comments.  Though the patch passed all my added tests(coverage not enough),
> >> I am still a bit worried if pre-computed coldest_loop is outside of outermost_loop, but
> >> outermost_loop is not the COLDEST LOOP, i.e. (outer->inner)
> >>
> >>  [loop tree root, coldest_loop, outermost_loop,..., second_coldest_loop, ..., loop],
> >>
> >> then function find_coldest_out_loop will return a loop NOT accord with our
> >> expectation, that should return second_coldest_loop instead of outermost_loop?
> > Hmm, interesting - yes.  I guess the common case will be that the pre-computed
> > outermost loop will be the loop at depth 1 since outer loops tend to
> > be colder than
> > inner loops?  That would then defeat the whole exercise.
>
> It is not easy to construct such cases, But finally I got below results,
>
> 1) many cases inner loop is hotter than outer loop, for example:
>
> loop 1's coldest_outermost_loop is 1, colder_than_inner_loop is NULL
> loop 2's coldest_outermost_loop is 1, colder_than_inner_loop is 1
> loop 3's coldest_outermost_loop is 1, colder_than_inner_loop is 2
> loop 4's coldest_outermost_loop is 1, colder_than_inner_loop is 2
>
>
> 2) But there are also cases inner loop is colder than outer loop, like:
>
> loop 1's coldest outermost loop is 1, colder_than_inner_loop is NULL
> loop 2's coldest outermost loop is 2, colder_than_inner_loop is NULL
> loop 3's coldest outermost loop is 3, colder_than_inner_loop is NULL
>
>
> >
> > To optimize the common case but not avoiding iteration in the cases we care
> > about we could instead cache the next outermost loop that is _not_ colder
> > than loop.  So for your [ ... ] example above we'd have> hotter_than_inner_loop[loop] == outer (second_coldest_loop), where the
> > candidate would then be 'second_coldest_loop' and we'd then iterate
> > to hotter_than_inner_loop[hotter_than_inner_loop[loop]] to find the next
> > cold candidate we can compare against?  For the common case we'd
> > have hotter_than_inner_loop[looo] == NULL (no such loop) and we then
> > simply pick 'outermost_loop'.
>
> Thanks.  It was difficult to understand, but finally I got to know what you
> want to express :)
>
> We should cache the next loop that is *colder* than loop instead of '_not_ colder
> than loop', and 'hotter_than_inner_loop' should be 'colder_than_inner_loop',
> then it makes sense if the coldest loop is outside of outermost loop, continue to
> find a colder loop between outermost loop and current loop in
> colder_than_inner_loop[loop->num]?  Hope I understood you correctly...

Heh, looking at the patch - I don't know.

To make the calls to bb_colder_than_loop_preheader more obvious can you
change that function to

+bool bb_colder_than_loop_preheader (basic_block bb, loop *loop)
+{
+  return bb->count < loop_preheader_edge (loop)->src->count;

?

+      if (colder_than_inner_loop[loop->num] != NULL
+         && loop_depth (outermost_loop)
+              < loop_depth (colder_than_inner_loop[loop->num]))
+       return colder_than_inner_loop[loop->num];

that is

+  3) If COLDEST_LOOP is outside of OUTERMOST_LOOP, check whether there is a
+  colder loop between OUTERMOST_LOOP and loop in pre-computed
+  COLDER_THAN_INNER_LOOP, return it, otherwise return OUTERMOST_LOOP.  */

since we index colder_than_inner_loop by loop->num as well this doesn't find the
coldest outer loop of 'loop' within the nest up to outermost_loop,
correct?  For the
"common" case of each successive outer loop being colder than the immediate
inner loop doesn't that result in only hoisting one level in case the
coldest outer loop
is outside of the outermost loop we can legally hoist to?

My suggestion to cache the next outer "hotter" loop was to make the search for
the optimal hoisting position O(1) for the common case (there is no
hotter outer loop),
allowing to do

   hot_loop = next_hot_loop[loop->num];
   if (!hot_loop || loop_depth (hot_loop) < loop_depth (outermost_loop))
     return outermost_loop;

and do the linear search in the other uncommon case (eventually the cache
can be exploited to optimize that search as well).

The cache should guarantee that for loops between next_hot_loop[loop->num]
and loop each loop is colder than its inner loop, so the immediate inner loop
of 'hot_loop' is the coldest place in the sub-nest.

Basically the cache should make the common case O(1) (well, I _do_ hope
that is the common case ;))

Btw, can you make the cache array a vec<class loop> * please so we get
index checking?

> >
> > One comment on the patch itself below.
> >
>
> The loop in fill_cold_out_loop is also removed in the updated v7 patch.
>
>
>
> [PATCH v7 2/2] Don't move cold code out of loop by checking bb count
>
> From: Xiong Hu Luo <luoxhu@linux.ibm.com>
>
> v7 changes:
> 1. Refine get_coldest_out_loop to replace loop with checking
> pre-computed coldest_outermost_loop and colder_than_inner_loop.
> 2. Add function fill_cold_out_loop, compute coldest_outermost_loop and
> colder_than_inner_loop recursively without loop.
>
> v6 changes:
> 1. Add function fill_coldest_out_loop to pre compute the coldest
> outermost loop for each loop.
> 2. Rename find_coldest_out_loop to get_coldest_out_loop.
> 3. Add testcase ssa-lim-22.c to differentiate with ssa-lim-19.c.
>
> v5 changes:
> 1. Refine comments for new functions.
> 2. Use basic_block instead of count in bb_colder_than_loop_preheader
> to align with function name.
> 3. Refine with simpler implementation for get_coldest_out_loop and
> ref_in_loop_hot_body::operator for better understanding.
>
> v4 changes:
> 1. Sort out profile_count comparision to function bb_cold_than_loop_preheader.
> 2. Update ref_in_loop_hot_body::operator () to find cold_loop before compare.
> 3. Split RTL invariant motion part out.
> 4. Remove aux changes.
>
> v3 changes:
> 1. Handle max_loop in determine_max_movement instead of outermost_invariant_loop.
> 2. Remove unnecessary changes.
> 3. Add for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body) in can_sm_ref_p.
> 4. "gsi_next (&bsi);" in move_computations_worker is kept since it caused
> infinite loop when implementing v1 and the iteration is missed to be
> updated actually.
>
> v1: https://gcc.gnu.org/pipermail/gcc-patches/2021-August/576488.html
> v2: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/579086.html
> v3: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/580211.html
> v4: https://gcc.gnu.org/pipermail/gcc-patches/2021-October/581231.html
> v5: https://gcc.gnu.org/pipermail/gcc-patches/2021-October/581961.html
>
> There was a patch trying to avoid move cold block out of loop:
>
> https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html
>
> Richard suggested to "never hoist anything from a bb with lower execution
> frequency to a bb with higher one in LIM invariantness_dom_walker
> before_dom_children".
>
> In gimple LIM analysis, add get_coldest_out_loop to move invariants to
> expected target loop, if profile count of the loop bb is colder
> than target loop preheader, it won't be hoisted out of loop.
> Likely for store motion, if all locations of the REF in loop is cold,
> don't do store motion of it.
>
> SPEC2017 performance evaluation shows 1% performance improvement for
> intrate GEOMEAN and no obvious regression for others.  Especially,
> 500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
> largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
> on P8LE.
>
> gcc/ChangeLog:
>
>         * tree-ssa-loop-im.c (bb_colder_than_loop_preheader): New
>         function.
>         (get_coldest_out_loop): New function.
>         (determine_max_movement): Use get_coldest_out_loop.
>         (move_computations_worker): Adjust and fix iteration udpate.
>         (class ref_in_loop_hot_body): New functor.
>         (ref_in_loop_hot_body::operator): New.
>         (can_sm_ref_p): Use for_all_locs_in_loop.
>         (fill_cold_out_loop): New.
>         (tree_ssa_lim_finalize): Free coldest_outermost_loop and
>         colder_than_inner_loop.
>         (loop_invariant_motion_in_fun): Call fill_cold_out_loop.
>
> gcc/testsuite/ChangeLog:
>
>         * gcc.dg/tree-ssa/recip-3.c: Adjust.
>         * gcc.dg/tree-ssa/ssa-lim-18.c: New test.
>         * gcc.dg/tree-ssa/ssa-lim-19.c: New test.
>         * gcc.dg/tree-ssa/ssa-lim-20.c: New test.
>         * gcc.dg/tree-ssa/ssa-lim-21.c: New test.
>         * gcc.dg/tree-ssa/ssa-lim-22.c: New test.
> ---
>  gcc/tree-ssa-loop-im.c                     | 140 ++++++++++++++++++++-
>  gcc/testsuite/gcc.dg/tree-ssa/recip-3.c    |   2 +-
>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c |  20 +++
>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c |  29 +++++
>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c |  25 ++++
>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c |  35 ++++++
>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c |  32 +++++
>  7 files changed, 280 insertions(+), 3 deletions(-)
>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c
>
> diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
> index 4b187c2cdaf..e9b3d0cba93 100644
> --- a/gcc/tree-ssa-loop-im.c
> +++ b/gcc/tree-ssa-loop-im.c
> @@ -146,6 +146,11 @@ public:
>  enum dep_kind { lim_raw, sm_war, sm_waw };
>  enum dep_state { dep_unknown, dep_independent, dep_dependent };
>
> +/* coldest outermost loop for given loop.  */
> +class loop **coldest_outermost_loop;
> +/* colder outer loop nerest to given loop.  */
> +class loop **colder_than_inner_loop;
> +
>  /* Populate the loop dependence cache of REF for LOOP, KIND with STATE.  */
>
>  static void
> @@ -417,6 +422,53 @@ movement_possibility (gimple *stmt)
>    return ret;
>  }
>
> +/* Compare the profile count inequality of bb and preheader, it is three-state
> +   as stated in profile-count.h, FALSE is returned if inequality cannot be
> +   decided.  */
> +bool bb_colder_than_loop_preheader (basic_block bb, basic_block preheader)
> +{
> +  gcc_assert (bb && preheader);
> +  return bb->count < preheader->count;
> +}
> +
> +/* Check coldest loop between OUTERMOST_LOOP and LOOP by comparing profile
> +   count.
> +  It does three steps check:
> +  1) Check whether CURR_BB is cold in it's own loop_father, if it is cold, just
> +  return NULL which means it should not be moved out at all;
> +  2) CURR_BB is NOT cold, check if pre-computed COLDEST_LOOP is outside of
> +  OUTERMOST_LOOP, if it is inside of OUTERMOST_LOOP, return the COLDEST_LOOP;
> +  3) If COLDEST_LOOP is outside of OUTERMOST_LOOP, check whether there is a
> +  colder loop between OUTERMOST_LOOP and loop in pre-computed
> +  COLDER_THAN_INNER_LOOP, return it, otherwise return OUTERMOST_LOOP.  */
> +
> +static class loop *
> +get_coldest_out_loop (class loop *outermost_loop, class loop *loop,
> +                     basic_block curr_bb)
> +{
> +  gcc_assert (outermost_loop == loop
> +             || flow_loop_nested_p (outermost_loop, loop));
> +
> +  /* If bb_colder_than_loop_preheader returns false due to three-state
> +    comparision, OUTERMOST_LOOP is returned finally to preserve the behavior.
> +    Otherwise, return the coldest loop between OUTERMOST_LOOP and LOOP.  */
> +  if (curr_bb
> +      && bb_colder_than_loop_preheader (curr_bb,
> +                                       loop_preheader_edge (loop)->src))
> +    return NULL;
> +
> +  class loop *coldest_loop = coldest_outermost_loop[loop->num];
> +  if (loop_depth (coldest_loop) < loop_depth (outermost_loop))
> +    {
> +      if (colder_than_inner_loop[loop->num] != NULL
> +         && loop_depth (outermost_loop)
> +              < loop_depth (colder_than_inner_loop[loop->num]))
> +       return colder_than_inner_loop[loop->num];
> +      return outermost_loop;
> +    }
> +  return coldest_loop;
> +}
> +
>  /* Suppose that operand DEF is used inside the LOOP.  Returns the outermost
>     loop to that we could move the expression using DEF if it did not have
>     other operands, i.e. the outermost loop enclosing LOOP in that the value
> @@ -685,7 +737,9 @@ determine_max_movement (gimple *stmt, bool must_preserve_exec)
>      level = ALWAYS_EXECUTED_IN (bb);
>    else
>      level = superloop_at_depth (loop, 1);
> -  lim_data->max_loop = level;
> +  lim_data->max_loop = get_coldest_out_loop (level, loop, bb);
> +  if (!lim_data->max_loop)
> +    return false;
>
>    if (gphi *phi = dyn_cast <gphi *> (stmt))
>      {
> @@ -1221,7 +1275,10 @@ move_computations_worker (basic_block bb)
>        /* We do not really want to move conditionals out of the loop; we just
>          placed it here to force its operands to be moved if necessary.  */
>        if (gimple_code (stmt) == GIMPLE_COND)
> -       continue;
> +       {
> +         gsi_next (&bsi);
> +         continue;
> +       }
>
>        if (dump_file && (dump_flags & TDF_DETAILS))
>         {
> @@ -2887,6 +2944,26 @@ ref_indep_loop_p (class loop *loop, im_mem_ref *ref, dep_kind kind)
>    return indep_p;
>  }
>
> +class ref_in_loop_hot_body
> +{
> +public:
> +  ref_in_loop_hot_body (loop *loop_) : l (loop_) {}
> +  bool operator () (mem_ref_loc *loc);
> +  class loop *l;
> +};
> +
> +/* Check the coldest loop between loop L and innermost loop.  If there is one
> +   cold loop between L and INNER_LOOP, store motion can be performed, otherwise
> +   no cold loop means no store motion.  get_coldest_out_loop also handles cases
> +   when l is inner_loop.  */
> +bool
> +ref_in_loop_hot_body::operator () (mem_ref_loc *loc)
> +{
> +  basic_block curr_bb = gimple_bb (loc->stmt);
> +  class loop *inner_loop = curr_bb->loop_father;
> +  return get_coldest_out_loop (l, inner_loop, curr_bb);
> +}
> +
>
>  /* Returns true if we can perform store motion of REF from LOOP.  */
>
> @@ -2941,6 +3018,12 @@ can_sm_ref_p (class loop *loop, im_mem_ref *ref)
>    if (!ref_indep_loop_p (loop, ref, sm_war))
>      return false;
>
> +  /* Verify whether the candidate is hot for LOOP.  Only do store motion if the
> +    candidate's profile count is hot.  Statement in cold BB shouldn't be moved
> +    out of it's loop_father.  */
> +  if (!for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body (loop)))
> +    return false;
> +
>    return true;
>  }
>
> @@ -3153,6 +3236,48 @@ fill_always_executed_in (void)
>      fill_always_executed_in_1 (loop, contains_call);
>  }
>
> +/* Find the coldest loop preheader for LOOP, also find the nearest colder loop
> +   to LOOP.  Then recursively iterate each inner loop.  */
> +
> +void
> +fill_cold_out_loop (class loop *coldest_loop, class loop *outer_loop,
> +                   class loop *loop)
> +{
> +  class loop *colder_loop = NULL;
> +  if (outer_loop)
> +    {
> +      if (bb_colder_than_loop_preheader (loop_preheader_edge (outer_loop)->src,
> +                                        loop_preheader_edge (loop)->src))
> +       colder_loop = outer_loop;
> +      else if (colder_than_inner_loop[outer_loop->num]
> +              && bb_colder_than_loop_preheader (
> +                loop_preheader_edge (colder_than_inner_loop[outer_loop->num])
> +                  ->src,
> +                loop_preheader_edge (loop)->src))
> +       colder_loop = colder_than_inner_loop[outer_loop->num];
> +    }
> +
> +  if (bb_colder_than_loop_preheader (loop_preheader_edge (loop)->src,
> +                                    loop_preheader_edge (coldest_loop)->src))
> +    coldest_loop = loop;
> +
> +  coldest_outermost_loop[loop->num] = coldest_loop;
> +  colder_than_inner_loop[loop->num] = colder_loop;
> +  if (dump_enabled_p ())
> +    {
> +      dump_printf (MSG_NOTE, "loop %d's coldest_outermost_loop is %d, ",
> +                  loop->num, coldest_loop->num);
> +      if (colder_loop)
> +       dump_printf (MSG_NOTE, "colder_than_inner_loop is %d\n",
> +                    colder_loop->num);
> +      else
> +       dump_printf (MSG_NOTE, "colder_than_inner_loop is NULL\n");
> +    }
> +
> +  class loop *inner_loop;
> +  for (inner_loop = loop->inner; inner_loop; inner_loop = inner_loop->next)
> +    fill_cold_out_loop (coldest_loop, loop, inner_loop);
> +}
>
>  /* Compute the global information needed by the loop invariant motion pass.  */
>
> @@ -3237,6 +3362,9 @@ tree_ssa_lim_finalize (void)
>      free_affine_expand_cache (&memory_accesses.ttae_cache);
>
>    free (bb_loop_postorder);
> +
> +  free (coldest_outermost_loop);
> +  free (colder_than_inner_loop);
>  }
>
>  /* Moves invariants from loops.  Only "expensive" invariants are moved out --
> @@ -3256,6 +3384,14 @@ loop_invariant_motion_in_fun (function *fun, bool store_motion)
>    /* Fills ALWAYS_EXECUTED_IN information for basic blocks.  */
>    fill_always_executed_in ();
>
> +  /* Pre-compute coldest outermost loop and nearest colder loop  of each loop.
> +   */
> +  class loop *loop;
> +  coldest_outermost_loop = XNEWVEC (class loop *, number_of_loops (cfun));
> +  colder_than_inner_loop = XNEWVEC (class loop *, number_of_loops (cfun));
> +  for (loop = current_loops->tree_root->inner; loop != NULL; loop = loop->next)
> +    fill_cold_out_loop (loop, NULL, loop);
> +
>    int *rpo = XNEWVEC (int, last_basic_block_for_fn (fun));
>    int n = pre_and_rev_post_order_compute_fn (fun, NULL, rpo, false);
>
> diff --git a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
> index 638bf38db8c..641c91e719e 100644
> --- a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
> +++ b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
> @@ -23,4 +23,4 @@ float h ()
>         F[0] += E / d;
>  }
>
> -/* { dg-final { scan-tree-dump-times " / " 1 "recip" } } */
> +/* { dg-final { scan-tree-dump-times " / " 5 "recip" } } */
> diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
> new file mode 100644
> index 00000000000..7326a230b3f
> --- /dev/null
> +++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
> @@ -0,0 +1,20 @@
> +/* { dg-do compile } */
> +/* { dg-options "-O2 -fdump-tree-lim2-details" } */
> +
> +volatile int x;
> +void
> +bar (int, char *, char *);
> +void
> +foo (int *a, int n, int k)
> +{
> +  int i;
> +
> +  for (i = 0; i < n; i++)
> +    {
> +      if (__builtin_expect (x, 0))
> +       bar (k / 5, "one", "two");
> +      a[i] = k;
> +    }
> +}
> +
> +/* { dg-final { scan-tree-dump-not "out of loop 1" "lim2" } } */
> diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
> new file mode 100644
> index 00000000000..51c1913d003
> --- /dev/null
> +++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
> @@ -0,0 +1,29 @@
> +/* { dg-do compile } */
> +/* { dg-options "-O2 -fdump-tree-lim2-details" } */
> +
> +volatile int x;
> +void
> +bar (int, char *, char *);
> +void
> +foo (int *a, int n, int m, int s, int t)
> +{
> +  int i;
> +  int j;
> +  int k;
> +
> +  for (i = 0; i < m; i++) // Loop 1
> +    {
> +      if (__builtin_expect (x, 0))
> +       for (j = 0; j < n; j++) // Loop 2
> +         for (k = 0; k < n; k++) // Loop 3
> +           {
> +             bar (s / 5, "one", "two");
> +             a[t] = s;
> +           }
> +      a[t] = t;
> +    }
> +}
> +
> +/* { dg-final { scan-tree-dump-times "out of loop 2" 4 "lim2" } } */
> +/* { dg-final { scan-tree-dump-times "out of loop 1" 3 "lim2" } } */
> +
> diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
> new file mode 100644
> index 00000000000..bc60a040a70
> --- /dev/null
> +++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
> @@ -0,0 +1,25 @@
> +/* { dg-do compile  } */
> +/* { dg-options "-O2 -fdump-tree-lim2-details" } */
> +
> +/* Test that `count' is not hoisted out of loop when bb is cold.  */
> +
> +int count;
> +volatile int x;
> +
> +struct obj {
> +  int data;
> +  struct obj *next;
> +
> +} *q;
> +
> +void
> +func (int m)
> +{
> +  struct obj *p;
> +  for (int i = 0; i < m; i++)
> +    if (__builtin_expect (x, 0))
> +      count++;
> +
> +}
> +
> +/* { dg-final { scan-tree-dump-not "Executing store motion of" "lim2"  }  } */
> diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
> new file mode 100644
> index 00000000000..ffe6f8f699d
> --- /dev/null
> +++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
> @@ -0,0 +1,35 @@
> +/* { dg-do compile  } */
> +/* { dg-options "-O2 -fdump-tree-lim2-details" } */
> +
> +/* Test that `data' and 'data1' is not hoisted out of inner loop and outer loop
> +   when it is in cold loop.  */
> +
> +int count;
> +volatile int x;
> +
> +struct obj {
> +  int data;
> +  int data1;
> +  struct obj *next;
> +};
> +
> +void
> +func (int m, int n, int k, struct obj *a)
> +{
> +  struct obj *q = a;
> +  for (int j = 0; j < m; j++)
> +    if (__builtin_expect (m, 0))
> +      for (int i = 0; i < m; i++)
> +       {
> +         if (__builtin_expect (x, 0))
> +           {
> +             count++;
> +             q->data += 3; /* Not hoisted out to inner loop. */
> +           }
> +         count += n;
> +         q->data1 += k; /* Not hoisted out to outer loop. */
> +       }
> +}
> +
> +/* { dg-final { scan-tree-dump-not "Executing store motion of" "lim2"  }  } */
> +
> diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c
> new file mode 100644
> index 00000000000..16ba4ceb8ab
> --- /dev/null
> +++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c
> @@ -0,0 +1,32 @@
> +/* { dg-do compile } */
> +/* { dg-options "-O2 -fdump-tree-lim2-details" } */
> +
> +volatile int x;
> +volatile int y;
> +void
> +bar (int, char *, char *);
> +void
> +foo (int *a, int n, int m, int s, int t)
> +{
> +  int i;
> +  int j;
> +  int k;
> +
> +  for (i = 0; i < m; i++) // Loop 1
> +    {
> +      if (__builtin_expect (x, 0))
> +       for (j = 0; j < n; j++) // Loop 2
> +         if (__builtin_expect (y, 0))
> +           for (k = 0; k < n; k++) // Loop 3
> +             {
> +               bar (s / 5, "one", "two");
> +               a[t] = s;
> +             }
> +      a[t] = t;
> +    }
> +}
> +
> +/* { dg-final { scan-tree-dump-times "out of loop 3" 4 "lim2" } } */
> +/* { dg-final { scan-tree-dump-times "out of loop 1" 3 "lim2" } } */
> +
> +
> --
> 2.27.0.90.geebb51ba8c
>

Re: [PATCH v8 2/2] Don't move cold code out of loop by checking bb count

[Xionghu Luo]

On 2021/12/1 18:09, Richard Biener wrote:
> On Wed, Nov 10, 2021 at 4:08 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>>
>>
>>
>> On 2021/11/4 21:00, Richard Biener wrote:
>>> On Wed, Nov 3, 2021 at 2:29 PM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>>>>
>>>>
>>>>> +  while (outmost_loop != loop)
>>>>> +    {
>>>>> +      if (bb_colder_than_loop_preheader (loop_preheader_edge
>>>>> (outmost_loop)->src,
>>>>> +                                        loop_preheader_edge (cold_loop)->src))
>>>>> +       cold_loop = outmost_loop;
>>>>> +      outmost_loop = superloop_at_depth (loop, loop_depth (outmost_loop) + 1);
>>>>> +    }
>>>>>
>>>>> could be instead written as
>>>>>
>>>>>   coldest_loop = coldest_outermost_loop[loop->num];
>>>>>   if (loop_depth (coldest_loop) < loop_depth (outermost_loop))
>>>>>     return outermost_loop;
>>>>>   return coldest_loop;
>>>>>
>>>>> ?  And in the usual case coldest_outermost_loop[L] would be the loop tree root.
>>>>> It should be possible to compute such cache in a DFS walk of the loop tree
>>>>> (the loop iterator by default visits in such order).
>>>>
>>>>
>>>> Thanks.  Updated the patch with your suggestion.  Not sure whether it strictly
>>>> conforms to your comments.  Though the patch passed all my added tests(coverage not enough),
>>>> I am still a bit worried if pre-computed coldest_loop is outside of outermost_loop, but
>>>> outermost_loop is not the COLDEST LOOP, i.e. (outer->inner)
>>>>
>>>>  [loop tree root, coldest_loop, outermost_loop,..., second_coldest_loop, ..., loop],
>>>>
>>>> then function find_coldest_out_loop will return a loop NOT accord with our
>>>> expectation, that should return second_coldest_loop instead of outermost_loop?
>>> Hmm, interesting - yes.  I guess the common case will be that the pre-computed
>>> outermost loop will be the loop at depth 1 since outer loops tend to
>>> be colder than
>>> inner loops?  That would then defeat the whole exercise.
>>
>> It is not easy to construct such cases, But finally I got below results,
>>
>> 1) many cases inner loop is hotter than outer loop, for example:
>>
>> loop 1's coldest_outermost_loop is 1, colder_than_inner_loop is NULL
>> loop 2's coldest_outermost_loop is 1, colder_than_inner_loop is 1
>> loop 3's coldest_outermost_loop is 1, colder_than_inner_loop is 2
>> loop 4's coldest_outermost_loop is 1, colder_than_inner_loop is 2
>>
>>
>> 2) But there are also cases inner loop is colder than outer loop, like:
>>
>> loop 1's coldest outermost loop is 1, colder_than_inner_loop is NULL
>> loop 2's coldest outermost loop is 2, colder_than_inner_loop is NULL
>> loop 3's coldest outermost loop is 3, colder_than_inner_loop is NULL
>>
>>
>>>
>>> To optimize the common case but not avoiding iteration in the cases we care
>>> about we could instead cache the next outermost loop that is _not_ colder
>>> than loop.  So for your [ ... ] example above we'd have> hotter_than_inner_loop[loop] == outer (second_coldest_loop), where the
>>> candidate would then be 'second_coldest_loop' and we'd then iterate
>>> to hotter_than_inner_loop[hotter_than_inner_loop[loop]] to find the next
>>> cold candidate we can compare against?  For the common case we'd
>>> have hotter_than_inner_loop[looo] == NULL (no such loop) and we then
>>> simply pick 'outermost_loop'.
>>
>> Thanks.  It was difficult to understand, but finally I got to know what you
>> want to express :)
>>
>> We should cache the next loop that is *colder* than loop instead of '_not_ colder
>> than loop', and 'hotter_than_inner_loop' should be 'colder_than_inner_loop',
>> then it makes sense if the coldest loop is outside of outermost loop, continue to
>> find a colder loop between outermost loop and current loop in
>> colder_than_inner_loop[loop->num]?  Hope I understood you correctly...
> 
> Heh, looking at the patch - I don't know.
> 
> To make the calls to bb_colder_than_loop_preheader more obvious can you
> change that function to
> 
> +bool bb_colder_than_loop_preheader (basic_block bb, loop *loop)
> +{
> +  return bb->count < loop_preheader_edge (loop)->src->count;
> 
> ?
> 
> +      if (colder_than_inner_loop[loop->num] != NULL
> +         && loop_depth (outermost_loop)
> +              < loop_depth (colder_than_inner_loop[loop->num]))
> +       return colder_than_inner_loop[loop->num];
> 
> that is
> 
> +  3) If COLDEST_LOOP is outside of OUTERMOST_LOOP, check whether there is a
> +  colder loop between OUTERMOST_LOOP and loop in pre-computed
> +  COLDER_THAN_INNER_LOOP, return it, otherwise return OUTERMOST_LOOP.  */
> 
> since we index colder_than_inner_loop by loop->num as well this doesn't find the
> coldest outer loop of 'loop' within the nest up to outermost_loop,
> correct?  For the
> "common" case of each successive outer loop being colder than the immediate
> inner loop doesn't that result in only hoisting one level in case the
> coldest outer loop
> is outside of the outermost loop we can legally hoist to?
> 
> My suggestion to cache the next outer "hotter" loop was to make the search for
> the optimal hoisting position O(1) for the common case (there is no
> hotter outer loop),
> allowing to do
> 
>    hot_loop = next_hot_loop[loop->num];
>    if (!hot_loop || loop_depth (hot_loop) < loop_depth (outermost_loop))
>      return outermost_loop;
> 
> and do the linear search in the other uncommon case (eventually the cache
> can be exploited to optimize that search as well).
> 
> The cache should guarantee that for loops between next_hot_loop[loop->num]
> and loop each loop is colder than its inner loop, so the immediate inner loop
> of 'hot_loop' is the coldest place in the sub-nest.
> 
> Basically the cache should make the common case O(1) (well, I _do_ hope
> that is the common case ;))
> 
> Btw, can you make the cache array a vec<class loop> * please so we get
> index checking?
> 

Thanks, updated with your comments to use hotter_than_inner_loop instead of
colder_than_inner_loop, it works.

For typical case like below:

for (int j = 0; j < m; j++) // Loop 1
  if (__builtin_expect (y, 0))
   for (int j2 = 0; j2 < 256; j2++) // Loop 2
      for (int i = 0; i < m; i++) // Loop 3
        for (int s = 0; s < m; s++) // Loop 4
          for (int s1 = 0; s1 < m; s1++) // Loop 5
            if (__builtin_expect (y, 0))
              for (int s2 = 0; s2 < m; s2++) // Loop 6


The output is:

loop 1's coldest_outermost_loop is 1, hotter_than_inner_loop is NULL
loop 2's coldest_outermost_loop is 2, hotter_than_inner_loop is 1
loop 3's coldest_outermost_loop is 2, hotter_than_inner_loop is NULL
loop 4's coldest_outermost_loop is 2, hotter_than_inner_loop is NULL
loop 5's coldest_outermost_loop is 2, hotter_than_inner_loop is NULL
loop 6's coldest_outermost_loop is 2, hotter_than_inner_loop is 5



v8-0002-Don-t-move-cold-code-out-of-loop-by-checking-bb-c.patch


From: Xiong Hu Luo <luoxhu@linux.ibm.com>

v8 changes:
1. Use hotter_than_inner_loop instead of colder to store a hotter loop
nearest to loop.
2. Update the logic in fill_coldest_and_hotter_out_loop and
get_coldest_out_loop to make common case O(1).
3. Update function argument bb_colder_than_loop_preheader.
4. Make cached array to vec<class *loop> for index checking.

v7 changes:
1. Refine get_coldest_out_loop to replace loop with checking
pre-computed coldest_outermost_loop and colder_than_inner_loop.
2. Add function fill_cold_out_loop, compute coldest_outermost_loop and
colder_than_inner_loop recursively without loop.

v6 changes:
1. Add function fill_coldest_out_loop to pre compute the coldest
outermost loop for each loop.
2. Rename find_coldest_out_loop to get_coldest_out_loop.
3. Add testcase ssa-lim-22.c to differentiate with ssa-lim-19.c.

v5 changes:
1. Refine comments for new functions.
2. Use basic_block instead of count in bb_colder_than_loop_preheader
to align with function name.
3. Refine with simpler implementation for get_coldest_out_loop and
ref_in_loop_hot_body::operator for better understanding.

v4 changes:
1. Sort out profile_count comparision to function bb_cold_than_loop_preheader.
2. Update ref_in_loop_hot_body::operator () to find cold_loop before compare.
3. Split RTL invariant motion part out.
4. Remove aux changes.

v3 changes:
1. Handle max_loop in determine_max_movement instead of outermost_invariant_loop.
2. Remove unnecessary changes.
3. Add for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body) in can_sm_ref_p.
4. "gsi_next (&bsi);" in move_computations_worker is kept since it caused
infinite loop when implementing v1 and the iteration is missed to be
updated actually.

v1: https://gcc.gnu.org/pipermail/gcc-patches/2021-August/576488.html
v2: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/579086.html
v3: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/580211.html
v4: https://gcc.gnu.org/pipermail/gcc-patches/2021-October/581231.html
v5: https://gcc.gnu.org/pipermail/gcc-patches/2021-October/581961.html

There was a patch trying to avoid move cold block out of loop:

https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html

Richard suggested to "never hoist anything from a bb with lower execution
frequency to a bb with higher one in LIM invariantness_dom_walker
before_dom_children".

In gimple LIM analysis, add get_coldest_out_loop to move invariants to
expected target loop, if profile count of the loop bb is colder
than target loop preheader, it won't be hoisted out of loop.
Likely for store motion, if all locations of the REF in loop is cold,
don't do store motion of it.

SPEC2017 performance evaluation shows 1% performance improvement for
intrate GEOMEAN and no obvious regression for others.  Especially,
500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
on P8LE.

gcc/ChangeLog:

	* tree-ssa-loop-im.c (bb_colder_than_loop_preheader): New
	function.
	(get_coldest_out_loop): New function.
	(determine_max_movement): Use get_coldest_out_loop.
	(move_computations_worker): Adjust and fix iteration udpate.
	(class ref_in_loop_hot_body): New functor.
	(ref_in_loop_hot_body::operator): New.
	(can_sm_ref_p): Use for_all_locs_in_loop.
	(fill_coldest_and_hotter_out_loop): New.
	(tree_ssa_lim_finalize): Free coldest_outermost_loop and
	hotter_than_inner_loop.
	(loop_invariant_motion_in_fun): Call fill_coldet_and_hotter_out_loop.

gcc/testsuite/ChangeLog:

	* gcc.dg/tree-ssa/recip-3.c: Adjust.
	* gcc.dg/tree-ssa/ssa-lim-19.c: New test.
	* gcc.dg/tree-ssa/ssa-lim-20.c: New test.
	* gcc.dg/tree-ssa/ssa-lim-21.c: New test.
	* gcc.dg/tree-ssa/ssa-lim-22.c: New test.
	* gcc.dg/tree-ssa/ssa-lim-23.c: New test.
---
 gcc/tree-ssa-loop-im.c                     | 150 ++++++++++++++++++++-
 gcc/testsuite/gcc.dg/tree-ssa/recip-3.c    |   2 +-
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c |  29 ++++
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c |  25 ++++
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c |  35 +++++
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c |  32 +++++
 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-23.c |  21 +++
 7 files changed, 291 insertions(+), 3 deletions(-)
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c
 create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-23.c

diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
index 4b187c2cdaf..0bcc3bca91b 100644
--- a/gcc/tree-ssa-loop-im.c
+++ b/gcc/tree-ssa-loop-im.c
@@ -146,6 +146,11 @@ public:
 enum dep_kind { lim_raw, sm_war, sm_waw };
 enum dep_state { dep_unknown, dep_independent, dep_dependent };
 
+/* coldest outermost loop for given loop.  */
+vec<class loop *> coldest_outermost_loop;
+/* hotter outer loop nearest to given loop.  */
+vec<class loop *> hotter_than_inner_loop;
+
 /* Populate the loop dependence cache of REF for LOOP, KIND with STATE.  */
 
 static void
@@ -417,6 +422,61 @@ movement_possibility (gimple *stmt)
   return ret;
 }
 
+/* Compare the profile count inequality of bb and loop's preheader, it is
+   three-state as stated in profile-count.h, FALSE is returned if inequality
+   cannot be decided.  */
+bool
+bb_colder_than_loop_preheader (basic_block bb, class loop *loop)
+{
+  gcc_assert (bb && loop);
+  return bb->count < loop_preheader_edge (loop)->src->count;
+}
+
+/* Check coldest loop between OUTERMOST_LOOP and LOOP by comparing profile
+   count.
+  It does three steps check:
+  1) Check whether CURR_BB is cold in it's own loop_father, if it is cold, just
+  return NULL which means it should not be moved out at all;
+  2) CURR_BB is NOT cold, check if pre-computed COLDEST_LOOP is outside of
+  OUTERMOST_LOOP, if it is inside of OUTERMOST_LOOP, return the COLDEST_LOOP;
+  3) If COLDEST_LOOP is outside of OUTERMOST_LOOP, check whether there is a
+  hotter loop between OUTERMOST_LOOP and loop in pre-computed
+  HOTTER_THAN_INNER_LOOP, return it's nested inner loop, otherwise return
+  OUTERMOST_LOOP.  */
+
+static class loop *
+get_coldest_out_loop (class loop *outermost_loop, class loop *loop,
+		      basic_block curr_bb)
+{
+  gcc_assert (outermost_loop == loop
+	      || flow_loop_nested_p (outermost_loop, loop));
+
+  /* If bb_colder_than_loop_preheader returns false due to three-state
+    comparision, OUTERMOST_LOOP is returned finally to preserve the behavior.
+    Otherwise, return the coldest loop between OUTERMOST_LOOP and LOOP.  */
+  if (curr_bb && bb_colder_than_loop_preheader (curr_bb, loop))
+    return NULL;
+
+  class loop *coldest_loop = coldest_outermost_loop[loop->num];
+  if (loop_depth (coldest_loop) < loop_depth (outermost_loop))
+    {
+      class loop *hotter_loop = hotter_than_inner_loop[loop->num];
+      if (!hotter_loop
+	  || loop_depth (hotter_loop) < loop_depth (outermost_loop))
+	return outermost_loop;
+
+      /*  hotter_loop is between OUTERMOST_LOOP and LOOP like:
+	[loop tree root, ..., coldest_loop, ..., outermost_loop, ...,
+	hotter_loop, second_coldest_loop, ..., loop]
+	return second_coldest_loop to be the hoist target.  */
+      class loop *aloop;
+      for (aloop = hotter_loop->inner; aloop; aloop = aloop->next)
+	if (flow_loop_nested_p (aloop, loop))
+	  return aloop;
+    }
+  return coldest_loop;
+}
+
 /* Suppose that operand DEF is used inside the LOOP.  Returns the outermost
    loop to that we could move the expression using DEF if it did not have
    other operands, i.e. the outermost loop enclosing LOOP in that the value
@@ -685,7 +745,9 @@ determine_max_movement (gimple *stmt, bool must_preserve_exec)
     level = ALWAYS_EXECUTED_IN (bb);
   else
     level = superloop_at_depth (loop, 1);
-  lim_data->max_loop = level;
+  lim_data->max_loop = get_coldest_out_loop (level, loop, bb);
+  if (!lim_data->max_loop)
+    return false;
 
   if (gphi *phi = dyn_cast <gphi *> (stmt))
     {
@@ -1221,7 +1283,10 @@ move_computations_worker (basic_block bb)
       /* We do not really want to move conditionals out of the loop; we just
 	 placed it here to force its operands to be moved if necessary.  */
       if (gimple_code (stmt) == GIMPLE_COND)
-	continue;
+	{
+	  gsi_next (&bsi);
+	  continue;
+	}
 
       if (dump_file && (dump_flags & TDF_DETAILS))
 	{
@@ -2887,6 +2952,26 @@ ref_indep_loop_p (class loop *loop, im_mem_ref *ref, dep_kind kind)
   return indep_p;
 }
 
+class ref_in_loop_hot_body
+{
+public:
+  ref_in_loop_hot_body (class loop *loop_) : l (loop_) {}
+  bool operator () (mem_ref_loc *loc);
+  class loop *l;
+};
+
+/* Check the coldest loop between loop L and innermost loop.  If there is one
+   cold loop between L and INNER_LOOP, store motion can be performed, otherwise
+   no cold loop means no store motion.  get_coldest_out_loop also handles cases
+   when l is inner_loop.  */
+bool
+ref_in_loop_hot_body::operator () (mem_ref_loc *loc)
+{
+  basic_block curr_bb = gimple_bb (loc->stmt);
+  class loop *inner_loop = curr_bb->loop_father;
+  return get_coldest_out_loop (l, inner_loop, curr_bb);
+}
+
 
 /* Returns true if we can perform store motion of REF from LOOP.  */
 
@@ -2941,6 +3026,12 @@ can_sm_ref_p (class loop *loop, im_mem_ref *ref)
   if (!ref_indep_loop_p (loop, ref, sm_war))
     return false;
 
+  /* Verify whether the candidate is hot for LOOP.  Only do store motion if the
+    candidate's profile count is hot.  Statement in cold BB shouldn't be moved
+    out of it's loop_father.  */
+  if (!for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body (loop)))
+    return false;
+
   return true;
 }
 
@@ -3153,6 +3244,48 @@ fill_always_executed_in (void)
     fill_always_executed_in_1 (loop, contains_call);
 }
 
+/* Find the coldest loop preheader for LOOP, also find the nearest hotter loop
+   to LOOP.  Then recursively iterate each inner loop.  */
+
+void
+fill_coldest_and_hotter_out_loop (class loop *coldest_loop,
+				  class loop *hotter_loop, class loop *loop)
+{
+  if (bb_colder_than_loop_preheader (loop_preheader_edge (loop)->src,
+				     coldest_loop))
+    coldest_loop = loop;
+
+  coldest_outermost_loop[loop->num] = coldest_loop;
+
+  hotter_than_inner_loop[loop->num] = NULL;
+  class loop *outer_loop = loop_outer (loop);
+  if (hotter_loop
+      && bb_colder_than_loop_preheader (loop_preheader_edge (loop)->src,
+					hotter_loop))
+    hotter_than_inner_loop[loop->num] = hotter_loop;
+
+  if (outer_loop && outer_loop != current_loops->tree_root
+      && bb_colder_than_loop_preheader (loop_preheader_edge (loop)->src,
+					outer_loop))
+    hotter_than_inner_loop[loop->num] = outer_loop;
+
+  if (dump_enabled_p ())
+    {
+      dump_printf (MSG_NOTE, "loop %d's coldest_outermost_loop is %d, ",
+		   loop->num, coldest_loop->num);
+      if (hotter_than_inner_loop[loop->num])
+	dump_printf (MSG_NOTE, "hotter_than_inner_loop is %d\n",
+		     hotter_than_inner_loop[loop->num]->num);
+      else
+	dump_printf (MSG_NOTE, "hotter_than_inner_loop is NULL\n");
+    }
+
+  class loop *inner_loop;
+  for (inner_loop = loop->inner; inner_loop; inner_loop = inner_loop->next)
+    fill_coldest_and_hotter_out_loop (coldest_loop,
+				      hotter_than_inner_loop[loop->num],
+				      inner_loop);
+}
 
 /* Compute the global information needed by the loop invariant motion pass.  */
 
@@ -3237,6 +3370,9 @@ tree_ssa_lim_finalize (void)
     free_affine_expand_cache (&memory_accesses.ttae_cache);
 
   free (bb_loop_postorder);
+
+  coldest_outermost_loop.release ();
+  hotter_than_inner_loop.release ();
 }
 
 /* Moves invariants from loops.  Only "expensive" invariants are moved out --
@@ -3256,6 +3392,16 @@ loop_invariant_motion_in_fun (function *fun, bool store_motion)
   /* Fills ALWAYS_EXECUTED_IN information for basic blocks.  */
   fill_always_executed_in ();
 
+  /* Pre-compute coldest outermost loop and nearest hotter loop of each loop.
+   */
+  class loop *loop;
+  coldest_outermost_loop.create (number_of_loops (cfun));
+  coldest_outermost_loop.safe_grow_cleared (number_of_loops (cfun));
+  hotter_than_inner_loop.create (number_of_loops (cfun));
+  hotter_than_inner_loop.safe_grow_cleared (number_of_loops (cfun));
+  for (loop = current_loops->tree_root->inner; loop != NULL; loop = loop->next)
+    fill_coldest_and_hotter_out_loop (loop, NULL, loop);
+
   int *rpo = XNEWVEC (int, last_basic_block_for_fn (fun));
   int n = pre_and_rev_post_order_compute_fn (fun, NULL, rpo, false);
 
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
index 638bf38db8c..641c91e719e 100644
--- a/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
+++ b/gcc/testsuite/gcc.dg/tree-ssa/recip-3.c
@@ -23,4 +23,4 @@ float h ()
 	F[0] += E / d;
 }
 
-/* { dg-final { scan-tree-dump-times " / " 1 "recip" } } */
+/* { dg-final { scan-tree-dump-times " / " 5 "recip" } } */
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
new file mode 100644
index 00000000000..51c1913d003
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
@@ -0,0 +1,29 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+volatile int x;
+void
+bar (int, char *, char *);
+void
+foo (int *a, int n, int m, int s, int t)
+{
+  int i;
+  int j;
+  int k;
+
+  for (i = 0; i < m; i++) // Loop 1
+    {
+      if (__builtin_expect (x, 0))
+	for (j = 0; j < n; j++) // Loop 2
+	  for (k = 0; k < n; k++) // Loop 3
+	    {
+	      bar (s / 5, "one", "two");
+	      a[t] = s;
+	    }
+      a[t] = t;
+    }
+}
+
+/* { dg-final { scan-tree-dump-times "out of loop 2" 4 "lim2" } } */
+/* { dg-final { scan-tree-dump-times "out of loop 1" 3 "lim2" } } */
+
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
new file mode 100644
index 00000000000..bc60a040a70
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
@@ -0,0 +1,25 @@
+/* { dg-do compile  } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+/* Test that `count' is not hoisted out of loop when bb is cold.  */
+
+int count;
+volatile int x;
+
+struct obj {
+  int data;
+  struct obj *next;
+
+} *q;
+
+void
+func (int m)
+{
+  struct obj *p;
+  for (int i = 0; i < m; i++)
+    if (__builtin_expect (x, 0))
+      count++;
+
+}
+
+/* { dg-final { scan-tree-dump-not "Executing store motion of" "lim2"  }  } */
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
new file mode 100644
index 00000000000..ffe6f8f699d
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
@@ -0,0 +1,35 @@
+/* { dg-do compile  } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+/* Test that `data' and 'data1' is not hoisted out of inner loop and outer loop
+   when it is in cold loop.  */
+
+int count;
+volatile int x;
+
+struct obj {
+  int data;
+  int data1;
+  struct obj *next;
+};
+
+void
+func (int m, int n, int k, struct obj *a)
+{
+  struct obj *q = a;
+  for (int j = 0; j < m; j++)
+    if (__builtin_expect (m, 0))
+      for (int i = 0; i < m; i++)
+	{
+	  if (__builtin_expect (x, 0))
+	    {
+	      count++;
+	      q->data += 3; /* Not hoisted out to inner loop. */
+	    }
+	  count += n;
+	  q->data1 += k; /* Not hoisted out to outer loop. */
+	}
+}
+
+/* { dg-final { scan-tree-dump-not "Executing store motion of" "lim2"  }  } */
+
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c
new file mode 100644
index 00000000000..16ba4ceb8ab
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c
@@ -0,0 +1,32 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+volatile int x;
+volatile int y;
+void
+bar (int, char *, char *);
+void
+foo (int *a, int n, int m, int s, int t)
+{
+  int i;
+  int j;
+  int k;
+
+  for (i = 0; i < m; i++) // Loop 1
+    {
+      if (__builtin_expect (x, 0))
+	for (j = 0; j < n; j++) // Loop 2
+	  if (__builtin_expect (y, 0))
+	    for (k = 0; k < n; k++) // Loop 3
+	      {
+		bar (s / 5, "one", "two");
+		a[t] = s;
+	      }
+      a[t] = t;
+    }
+}
+
+/* { dg-final { scan-tree-dump-times "out of loop 3" 4 "lim2" } } */
+/* { dg-final { scan-tree-dump-times "out of loop 1" 3 "lim2" } } */
+
+
diff --git a/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-23.c b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-23.c
new file mode 100644
index 00000000000..e7880746422
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-23.c
@@ -0,0 +1,21 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -fdump-tree-lim2-details" } */
+
+volatile int x;
+void
+bar (int, char *, char *);
+void
+foo (int *a, int n, int k)
+{
+  int i;
+
+  for (i = 0; i < n; i++)
+    {
+      if (__builtin_expect (x, 0))
+	bar (k / 5, "one", "two");
+      a[i] = k;
+    }
+}
+
+/* { dg-final { scan-tree-dump-not "out of loop 1" "lim2" } } */
+
-- 
2.27.0.90.geebb51ba8c

Re: [PATCH v8 2/2] Don't move cold code out of loop by checking bb count

[Xionghu Luo]

On 2021/12/6 13:09, Xionghu Luo via Gcc-patches wrote:
> 
> 
> On 2021/12/1 18:09, Richard Biener wrote:
>> On Wed, Nov 10, 2021 at 4:08 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>>>
>>>
>>>
>>> On 2021/11/4 21:00, Richard Biener wrote:
>>>> On Wed, Nov 3, 2021 at 2:29 PM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>>>>>
>>>>>
>>>>>> +  while (outmost_loop != loop)
>>>>>> +    {
>>>>>> +      if (bb_colder_than_loop_preheader (loop_preheader_edge
>>>>>> (outmost_loop)->src,
>>>>>> +                                        loop_preheader_edge (cold_loop)->src))
>>>>>> +       cold_loop = outmost_loop;
>>>>>> +      outmost_loop = superloop_at_depth (loop, loop_depth (outmost_loop) + 1);
>>>>>> +    }
>>>>>>
>>>>>> could be instead written as
>>>>>>
>>>>>>   coldest_loop = coldest_outermost_loop[loop->num];
>>>>>>   if (loop_depth (coldest_loop) < loop_depth (outermost_loop))
>>>>>>     return outermost_loop;
>>>>>>   return coldest_loop;
>>>>>>
>>>>>> ?  And in the usual case coldest_outermost_loop[L] would be the loop tree root.
>>>>>> It should be possible to compute such cache in a DFS walk of the loop tree
>>>>>> (the loop iterator by default visits in such order).
>>>>>
>>>>>
>>>>> Thanks.  Updated the patch with your suggestion.  Not sure whether it strictly
>>>>> conforms to your comments.  Though the patch passed all my added tests(coverage not enough),
>>>>> I am still a bit worried if pre-computed coldest_loop is outside of outermost_loop, but
>>>>> outermost_loop is not the COLDEST LOOP, i.e. (outer->inner)
>>>>>
>>>>>  [loop tree root, coldest_loop, outermost_loop,..., second_coldest_loop, ..., loop],
>>>>>
>>>>> then function find_coldest_out_loop will return a loop NOT accord with our
>>>>> expectation, that should return second_coldest_loop instead of outermost_loop?
>>>> Hmm, interesting - yes.  I guess the common case will be that the pre-computed
>>>> outermost loop will be the loop at depth 1 since outer loops tend to
>>>> be colder than
>>>> inner loops?  That would then defeat the whole exercise.
>>>
>>> It is not easy to construct such cases, But finally I got below results,
>>>
>>> 1) many cases inner loop is hotter than outer loop, for example:
>>>
>>> loop 1's coldest_outermost_loop is 1, colder_than_inner_loop is NULL
>>> loop 2's coldest_outermost_loop is 1, colder_than_inner_loop is 1
>>> loop 3's coldest_outermost_loop is 1, colder_than_inner_loop is 2
>>> loop 4's coldest_outermost_loop is 1, colder_than_inner_loop is 2
>>>
>>>
>>> 2) But there are also cases inner loop is colder than outer loop, like:
>>>
>>> loop 1's coldest outermost loop is 1, colder_than_inner_loop is NULL
>>> loop 2's coldest outermost loop is 2, colder_than_inner_loop is NULL
>>> loop 3's coldest outermost loop is 3, colder_than_inner_loop is NULL
>>>
>>>
>>>>
>>>> To optimize the common case but not avoiding iteration in the cases we care
>>>> about we could instead cache the next outermost loop that is _not_ colder
>>>> than loop.  So for your [ ... ] example above we'd have> hotter_than_inner_loop[loop] == outer (second_coldest_loop), where the
>>>> candidate would then be 'second_coldest_loop' and we'd then iterate
>>>> to hotter_than_inner_loop[hotter_than_inner_loop[loop]] to find the next
>>>> cold candidate we can compare against?  For the common case we'd
>>>> have hotter_than_inner_loop[looo] == NULL (no such loop) and we then
>>>> simply pick 'outermost_loop'.
>>>
>>> Thanks.  It was difficult to understand, but finally I got to know what you
>>> want to express :)
>>>
>>> We should cache the next loop that is *colder* than loop instead of '_not_ colder
>>> than loop', and 'hotter_than_inner_loop' should be 'colder_than_inner_loop',
>>> then it makes sense if the coldest loop is outside of outermost loop, continue to
>>> find a colder loop between outermost loop and current loop in
>>> colder_than_inner_loop[loop->num]?  Hope I understood you correctly...
>>
>> Heh, looking at the patch - I don't know.
>>
>> To make the calls to bb_colder_than_loop_preheader more obvious can you
>> change that function to
>>
>> +bool bb_colder_than_loop_preheader (basic_block bb, loop *loop)
>> +{
>> +  return bb->count < loop_preheader_edge (loop)->src->count;
>>
>> ?
>>
>> +      if (colder_than_inner_loop[loop->num] != NULL
>> +         && loop_depth (outermost_loop)
>> +              < loop_depth (colder_than_inner_loop[loop->num]))
>> +       return colder_than_inner_loop[loop->num];
>>
>> that is
>>
>> +  3) If COLDEST_LOOP is outside of OUTERMOST_LOOP, check whether there is a
>> +  colder loop between OUTERMOST_LOOP and loop in pre-computed
>> +  COLDER_THAN_INNER_LOOP, return it, otherwise return OUTERMOST_LOOP.  */
>>
>> since we index colder_than_inner_loop by loop->num as well this doesn't find the
>> coldest outer loop of 'loop' within the nest up to outermost_loop,
>> correct?  For the
>> "common" case of each successive outer loop being colder than the immediate
>> inner loop doesn't that result in only hoisting one level in case the
>> coldest outer loop
>> is outside of the outermost loop we can legally hoist to?
>>
>> My suggestion to cache the next outer "hotter" loop was to make the search for
>> the optimal hoisting position O(1) for the common case (there is no
>> hotter outer loop),
>> allowing to do
>>
>>    hot_loop = next_hot_loop[loop->num];
>>    if (!hot_loop || loop_depth (hot_loop) < loop_depth (outermost_loop))
>>      return outermost_loop;
>>
>> and do the linear search in the other uncommon case (eventually the cache
>> can be exploited to optimize that search as well).
>>
>> The cache should guarantee that for loops between next_hot_loop[loop->num]
>> and loop each loop is colder than its inner loop, so the immediate inner loop
>> of 'hot_loop' is the coldest place in the sub-nest.
>>
>> Basically the cache should make the common case O(1) (well, I _do_ hope
>> that is the common case ;))
>>
>> Btw, can you make the cache array a vec<class loop> * please so we get
>> index checking?
>>
> 
> Thanks, updated with your comments to use hotter_than_inner_loop instead of
> colder_than_inner_loop, it works.
> 
> For typical case like below:
> 
> for (int j = 0; j < m; j++) // Loop 1
>   if (__builtin_expect (y, 0))
>    for (int j2 = 0; j2 < 256; j2++) // Loop 2
>       for (int i = 0; i < m; i++) // Loop 3
>         for (int s = 0; s < m; s++) // Loop 4
>           for (int s1 = 0; s1 < m; s1++) // Loop 5
>             if (__builtin_expect (y, 0))
>               for (int s2 = 0; s2 < m; s2++) // Loop 6
> 
> 
> The output is:
> 
> loop 1's coldest_outermost_loop is 1, hotter_than_inner_loop is NULL
> loop 2's coldest_outermost_loop is 2, hotter_than_inner_loop is 1
> loop 3's coldest_outermost_loop is 2, hotter_than_inner_loop is NULL
> loop 4's coldest_outermost_loop is 2, hotter_than_inner_loop is NULL
> loop 5's coldest_outermost_loop is 2, hotter_than_inner_loop is NULL
> loop 6's coldest_outermost_loop is 2, hotter_than_inner_loop is 5
> 
> 
> 
> v8-0002-Don-t-move-cold-code-out-of-loop-by-checking-bb-c.patch
> 
> 
> From: Xiong Hu Luo <luoxhu@linux.ibm.com>
> 
> v8 changes:
> 1. Use hotter_than_inner_loop instead of colder to store a hotter loop
> nearest to loop.
> 2. Update the logic in fill_coldest_and_hotter_out_loop and
> get_coldest_out_loop to make common case O(1).
> 3. Update function argument bb_colder_than_loop_preheader.
> 4. Make cached array to vec<class *loop> for index checking.
> 
> v7 changes:
> 1. Refine get_coldest_out_loop to replace loop with checking
> pre-computed coldest_outermost_loop and colder_than_inner_loop.
> 2. Add function fill_cold_out_loop, compute coldest_outermost_loop and
> colder_than_inner_loop recursively without loop.
> 
> v6 changes:
> 1. Add function fill_coldest_out_loop to pre compute the coldest
> outermost loop for each loop.
> 2. Rename find_coldest_out_loop to get_coldest_out_loop.
> 3. Add testcase ssa-lim-22.c to differentiate with ssa-lim-19.c.
> 
> v5 changes:
> 1. Refine comments for new functions.
> 2. Use basic_block instead of count in bb_colder_than_loop_preheader
> to align with function name.
> 3. Refine with simpler implementation for get_coldest_out_loop and
> ref_in_loop_hot_body::operator for better understanding.
> 
> v4 changes:
> 1. Sort out profile_count comparision to function bb_cold_than_loop_preheader.
> 2. Update ref_in_loop_hot_body::operator () to find cold_loop before compare.
> 3. Split RTL invariant motion part out.
> 4. Remove aux changes.
> 
> v3 changes:
> 1. Handle max_loop in determine_max_movement instead of outermost_invariant_loop.
> 2. Remove unnecessary changes.
> 3. Add for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body) in can_sm_ref_p.
> 4. "gsi_next (&bsi);" in move_computations_worker is kept since it caused
> infinite loop when implementing v1 and the iteration is missed to be
> updated actually.
> 
> v1: https://gcc.gnu.org/pipermail/gcc-patches/2021-August/576488.html
> v2: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/579086.html
> v3: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/580211.html
> v4: https://gcc.gnu.org/pipermail/gcc-patches/2021-October/581231.html
> v5: https://gcc.gnu.org/pipermail/gcc-patches/2021-October/581961.html
> 
> There was a patch trying to avoid move cold block out of loop:
> 
> https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html
> 
> Richard suggested to "never hoist anything from a bb with lower execution
> frequency to a bb with higher one in LIM invariantness_dom_walker
> before_dom_children".
> 
> In gimple LIM analysis, add get_coldest_out_loop to move invariants to
> expected target loop, if profile count of the loop bb is colder
> than target loop preheader, it won't be hoisted out of loop.
> Likely for store motion, if all locations of the REF in loop is cold,
> don't do store motion of it.
> 
> SPEC2017 performance evaluation shows 1% performance improvement for
> intrate GEOMEAN and no obvious regression for others.  Especially,
> 500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
> largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
> on P8LE.
> 
> gcc/ChangeLog:
> 
> 	* tree-ssa-loop-im.c (bb_colder_than_loop_preheader): New
> 	function.
> 	(get_coldest_out_loop): New function.
> 	(determine_max_movement): Use get_coldest_out_loop.
> 	(move_computations_worker): Adjust and fix iteration udpate.
> 	(class ref_in_loop_hot_body): New functor.
> 	(ref_in_loop_hot_body::operator): New.
> 	(can_sm_ref_p): Use for_all_locs_in_loop.
> 	(fill_coldest_and_hotter_out_loop): New.
> 	(tree_ssa_lim_finalize): Free coldest_outermost_loop and
> 	hotter_than_inner_loop.
> 	(loop_invariant_motion_in_fun): Call fill_coldet_and_hotter_out_loop.
> 
> gcc/testsuite/ChangeLog:
> 
> 	* gcc.dg/tree-ssa/recip-3.c: Adjust.
> 	* gcc.dg/tree-ssa/ssa-lim-19.c: New test.
> 	* gcc.dg/tree-ssa/ssa-lim-20.c: New test.
> 	* gcc.dg/tree-ssa/ssa-lim-21.c: New test.
> 	* gcc.dg/tree-ssa/ssa-lim-22.c: New test.
> 	* gcc.dg/tree-ssa/ssa-lim-23.c: New test.
> ---
>  gcc/tree-ssa-loop-im.c                     | 150 ++++++++++++++++++++-
>  gcc/testsuite/gcc.dg/tree-ssa/recip-3.c    |   2 +-
>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c |  29 ++++
>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c |  25 ++++
>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c |  35 +++++
>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c |  32 +++++
>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-23.c |  21 +++
>  7 files changed, 291 insertions(+), 3 deletions(-)
>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c
>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-23.c
> 
> diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
> index 4b187c2cdaf..0bcc3bca91b 100644
> --- a/gcc/tree-ssa-loop-im.c
> +++ b/gcc/tree-ssa-loop-im.c
> @@ -146,6 +146,11 @@ public:
>  enum dep_kind { lim_raw, sm_war, sm_waw };
>  enum dep_state { dep_unknown, dep_independent, dep_dependent };
> 
> +/* coldest outermost loop for given loop.  */
> +vec<class loop *> coldest_outermost_loop;
> +/* hotter outer loop nearest to given loop.  */
> +vec<class loop *> hotter_than_inner_loop;
> +
>  /* Populate the loop dependence cache of REF for LOOP, KIND with STATE.  */
> 
>  static void
> @@ -417,6 +422,61 @@ movement_possibility (gimple *stmt)
>    return ret;
>  }
> 
> +/* Compare the profile count inequality of bb and loop's preheader, it is
> +   three-state as stated in profile-count.h, FALSE is returned if inequality
> +   cannot be decided.  */
> +bool
> +bb_colder_than_loop_preheader (basic_block bb, class loop *loop)
> +{
> +  gcc_assert (bb && loop);
> +  return bb->count < loop_preheader_edge (loop)->src->count;
> +}
> +
> +/* Check coldest loop between OUTERMOST_LOOP and LOOP by comparing profile
> +   count.
> +  It does three steps check:
> +  1) Check whether CURR_BB is cold in it's own loop_father, if it is cold, just
> +  return NULL which means it should not be moved out at all;
> +  2) CURR_BB is NOT cold, check if pre-computed COLDEST_LOOP is outside of
> +  OUTERMOST_LOOP, if it is inside of OUTERMOST_LOOP, return the COLDEST_LOOP;
> +  3) If COLDEST_LOOP is outside of OUTERMOST_LOOP, check whether there is a
> +  hotter loop between OUTERMOST_LOOP and loop in pre-computed
> +  HOTTER_THAN_INNER_LOOP, return it's nested inner loop, otherwise return
> +  OUTERMOST_LOOP.  */
> +
> +static class loop *
> +get_coldest_out_loop (class loop *outermost_loop, class loop *loop,
> +		      basic_block curr_bb)
> +{
> +  gcc_assert (outermost_loop == loop
> +	      || flow_loop_nested_p (outermost_loop, loop));
> +
> +  /* If bb_colder_than_loop_preheader returns false due to three-state
> +    comparision, OUTERMOST_LOOP is returned finally to preserve the behavior.
> +    Otherwise, return the coldest loop between OUTERMOST_LOOP and LOOP.  */
> +  if (curr_bb && bb_colder_than_loop_preheader (curr_bb, loop))
> +    return NULL;
> +
> +  class loop *coldest_loop = coldest_outermost_loop[loop->num];
> +  if (loop_depth (coldest_loop) < loop_depth (outermost_loop))
> +    {
> +      class loop *hotter_loop = hotter_than_inner_loop[loop->num];
> +      if (!hotter_loop
> +	  || loop_depth (hotter_loop) < loop_depth (outermost_loop))
> +	return outermost_loop;
> +
> +      /*  hotter_loop is between OUTERMOST_LOOP and LOOP like:
> +	[loop tree root, ..., coldest_loop, ..., outermost_loop, ...,
> +	hotter_loop, second_coldest_loop, ..., loop]
> +	return second_coldest_loop to be the hoist target.  */
> +      class loop *aloop;
> +      for (aloop = hotter_loop->inner; aloop; aloop = aloop->next)
> +	if (flow_loop_nested_p (aloop, loop))

should be:

        if (aloop == loop || flow_loop_nested_p (aloop, loop))


> +	  return aloop;
> +    }
> +  return coldest_loop;
> +}
> +


-- 
Thanks,
Xionghu

Re: [PATCH v8 2/2] Don't move cold code out of loop by checking bb count

[Richard Biener]

On Mon, Dec 6, 2021 at 6:26 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>
>
>
> On 2021/12/6 13:09, Xionghu Luo via Gcc-patches wrote:
> >
> >
> > On 2021/12/1 18:09, Richard Biener wrote:
> >> On Wed, Nov 10, 2021 at 4:08 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
> >>>
> >>>
> >>>
> >>> On 2021/11/4 21:00, Richard Biener wrote:
> >>>> On Wed, Nov 3, 2021 at 2:29 PM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
> >>>>>
> >>>>>
> >>>>>> +  while (outmost_loop != loop)
> >>>>>> +    {
> >>>>>> +      if (bb_colder_than_loop_preheader (loop_preheader_edge
> >>>>>> (outmost_loop)->src,
> >>>>>> +                                        loop_preheader_edge (cold_loop)->src))
> >>>>>> +       cold_loop = outmost_loop;
> >>>>>> +      outmost_loop = superloop_at_depth (loop, loop_depth (outmost_loop) + 1);
> >>>>>> +    }
> >>>>>>
> >>>>>> could be instead written as
> >>>>>>
> >>>>>>   coldest_loop = coldest_outermost_loop[loop->num];
> >>>>>>   if (loop_depth (coldest_loop) < loop_depth (outermost_loop))
> >>>>>>     return outermost_loop;
> >>>>>>   return coldest_loop;
> >>>>>>
> >>>>>> ?  And in the usual case coldest_outermost_loop[L] would be the loop tree root.
> >>>>>> It should be possible to compute such cache in a DFS walk of the loop tree
> >>>>>> (the loop iterator by default visits in such order).
> >>>>>
> >>>>>
> >>>>> Thanks.  Updated the patch with your suggestion.  Not sure whether it strictly
> >>>>> conforms to your comments.  Though the patch passed all my added tests(coverage not enough),
> >>>>> I am still a bit worried if pre-computed coldest_loop is outside of outermost_loop, but
> >>>>> outermost_loop is not the COLDEST LOOP, i.e. (outer->inner)
> >>>>>
> >>>>>  [loop tree root, coldest_loop, outermost_loop,..., second_coldest_loop, ..., loop],
> >>>>>
> >>>>> then function find_coldest_out_loop will return a loop NOT accord with our
> >>>>> expectation, that should return second_coldest_loop instead of outermost_loop?
> >>>> Hmm, interesting - yes.  I guess the common case will be that the pre-computed
> >>>> outermost loop will be the loop at depth 1 since outer loops tend to
> >>>> be colder than
> >>>> inner loops?  That would then defeat the whole exercise.
> >>>
> >>> It is not easy to construct such cases, But finally I got below results,
> >>>
> >>> 1) many cases inner loop is hotter than outer loop, for example:
> >>>
> >>> loop 1's coldest_outermost_loop is 1, colder_than_inner_loop is NULL
> >>> loop 2's coldest_outermost_loop is 1, colder_than_inner_loop is 1
> >>> loop 3's coldest_outermost_loop is 1, colder_than_inner_loop is 2
> >>> loop 4's coldest_outermost_loop is 1, colder_than_inner_loop is 2
> >>>
> >>>
> >>> 2) But there are also cases inner loop is colder than outer loop, like:
> >>>
> >>> loop 1's coldest outermost loop is 1, colder_than_inner_loop is NULL
> >>> loop 2's coldest outermost loop is 2, colder_than_inner_loop is NULL
> >>> loop 3's coldest outermost loop is 3, colder_than_inner_loop is NULL
> >>>
> >>>
> >>>>
> >>>> To optimize the common case but not avoiding iteration in the cases we care
> >>>> about we could instead cache the next outermost loop that is _not_ colder
> >>>> than loop.  So for your [ ... ] example above we'd have> hotter_than_inner_loop[loop] == outer (second_coldest_loop), where the
> >>>> candidate would then be 'second_coldest_loop' and we'd then iterate
> >>>> to hotter_than_inner_loop[hotter_than_inner_loop[loop]] to find the next
> >>>> cold candidate we can compare against?  For the common case we'd
> >>>> have hotter_than_inner_loop[looo] == NULL (no such loop) and we then
> >>>> simply pick 'outermost_loop'.
> >>>
> >>> Thanks.  It was difficult to understand, but finally I got to know what you
> >>> want to express :)
> >>>
> >>> We should cache the next loop that is *colder* than loop instead of '_not_ colder
> >>> than loop', and 'hotter_than_inner_loop' should be 'colder_than_inner_loop',
> >>> then it makes sense if the coldest loop is outside of outermost loop, continue to
> >>> find a colder loop between outermost loop and current loop in
> >>> colder_than_inner_loop[loop->num]?  Hope I understood you correctly...
> >>
> >> Heh, looking at the patch - I don't know.
> >>
> >> To make the calls to bb_colder_than_loop_preheader more obvious can you
> >> change that function to
> >>
> >> +bool bb_colder_than_loop_preheader (basic_block bb, loop *loop)
> >> +{
> >> +  return bb->count < loop_preheader_edge (loop)->src->count;
> >>
> >> ?
> >>
> >> +      if (colder_than_inner_loop[loop->num] != NULL
> >> +         && loop_depth (outermost_loop)
> >> +              < loop_depth (colder_than_inner_loop[loop->num]))
> >> +       return colder_than_inner_loop[loop->num];
> >>
> >> that is
> >>
> >> +  3) If COLDEST_LOOP is outside of OUTERMOST_LOOP, check whether there is a
> >> +  colder loop between OUTERMOST_LOOP and loop in pre-computed
> >> +  COLDER_THAN_INNER_LOOP, return it, otherwise return OUTERMOST_LOOP.  */
> >>
> >> since we index colder_than_inner_loop by loop->num as well this doesn't find the
> >> coldest outer loop of 'loop' within the nest up to outermost_loop,
> >> correct?  For the
> >> "common" case of each successive outer loop being colder than the immediate
> >> inner loop doesn't that result in only hoisting one level in case the
> >> coldest outer loop
> >> is outside of the outermost loop we can legally hoist to?
> >>
> >> My suggestion to cache the next outer "hotter" loop was to make the search for
> >> the optimal hoisting position O(1) for the common case (there is no
> >> hotter outer loop),
> >> allowing to do
> >>
> >>    hot_loop = next_hot_loop[loop->num];
> >>    if (!hot_loop || loop_depth (hot_loop) < loop_depth (outermost_loop))
> >>      return outermost_loop;
> >>
> >> and do the linear search in the other uncommon case (eventually the cache
> >> can be exploited to optimize that search as well).
> >>
> >> The cache should guarantee that for loops between next_hot_loop[loop->num]
> >> and loop each loop is colder than its inner loop, so the immediate inner loop
> >> of 'hot_loop' is the coldest place in the sub-nest.
> >>
> >> Basically the cache should make the common case O(1) (well, I _do_ hope
> >> that is the common case ;))
> >>
> >> Btw, can you make the cache array a vec<class loop> * please so we get
> >> index checking?
> >>
> >
> > Thanks, updated with your comments to use hotter_than_inner_loop instead of
> > colder_than_inner_loop, it works.
> >
> > For typical case like below:
> >
> > for (int j = 0; j < m; j++) // Loop 1
> >   if (__builtin_expect (y, 0))
> >    for (int j2 = 0; j2 < 256; j2++) // Loop 2
> >       for (int i = 0; i < m; i++) // Loop 3
> >         for (int s = 0; s < m; s++) // Loop 4
> >           for (int s1 = 0; s1 < m; s1++) // Loop 5
> >             if (__builtin_expect (y, 0))
> >               for (int s2 = 0; s2 < m; s2++) // Loop 6
> >
> >
> > The output is:
> >
> > loop 1's coldest_outermost_loop is 1, hotter_than_inner_loop is NULL
> > loop 2's coldest_outermost_loop is 2, hotter_than_inner_loop is 1
> > loop 3's coldest_outermost_loop is 2, hotter_than_inner_loop is NULL
> > loop 4's coldest_outermost_loop is 2, hotter_than_inner_loop is NULL
> > loop 5's coldest_outermost_loop is 2, hotter_than_inner_loop is NULL
> > loop 6's coldest_outermost_loop is 2, hotter_than_inner_loop is 5
> >
> >
> >
> > v8-0002-Don-t-move-cold-code-out-of-loop-by-checking-bb-c.patch
> >
> >
> > From: Xiong Hu Luo <luoxhu@linux.ibm.com>
> >
> > v8 changes:
> > 1. Use hotter_than_inner_loop instead of colder to store a hotter loop
> > nearest to loop.
> > 2. Update the logic in fill_coldest_and_hotter_out_loop and
> > get_coldest_out_loop to make common case O(1).
> > 3. Update function argument bb_colder_than_loop_preheader.
> > 4. Make cached array to vec<class *loop> for index checking.
> >
> > v7 changes:
> > 1. Refine get_coldest_out_loop to replace loop with checking
> > pre-computed coldest_outermost_loop and colder_than_inner_loop.
> > 2. Add function fill_cold_out_loop, compute coldest_outermost_loop and
> > colder_than_inner_loop recursively without loop.
> >
> > v6 changes:
> > 1. Add function fill_coldest_out_loop to pre compute the coldest
> > outermost loop for each loop.
> > 2. Rename find_coldest_out_loop to get_coldest_out_loop.
> > 3. Add testcase ssa-lim-22.c to differentiate with ssa-lim-19.c.
> >
> > v5 changes:
> > 1. Refine comments for new functions.
> > 2. Use basic_block instead of count in bb_colder_than_loop_preheader
> > to align with function name.
> > 3. Refine with simpler implementation for get_coldest_out_loop and
> > ref_in_loop_hot_body::operator for better understanding.
> >
> > v4 changes:
> > 1. Sort out profile_count comparision to function bb_cold_than_loop_preheader.
> > 2. Update ref_in_loop_hot_body::operator () to find cold_loop before compare.
> > 3. Split RTL invariant motion part out.
> > 4. Remove aux changes.
> >
> > v3 changes:
> > 1. Handle max_loop in determine_max_movement instead of outermost_invariant_loop.
> > 2. Remove unnecessary changes.
> > 3. Add for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body) in can_sm_ref_p.
> > 4. "gsi_next (&bsi);" in move_computations_worker is kept since it caused
> > infinite loop when implementing v1 and the iteration is missed to be
> > updated actually.
> >
> > v1: https://gcc.gnu.org/pipermail/gcc-patches/2021-August/576488.html
> > v2: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/579086.html
> > v3: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/580211.html
> > v4: https://gcc.gnu.org/pipermail/gcc-patches/2021-October/581231.html
> > v5: https://gcc.gnu.org/pipermail/gcc-patches/2021-October/581961.html
> >
> > There was a patch trying to avoid move cold block out of loop:
> >
> > https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html
> >
> > Richard suggested to "never hoist anything from a bb with lower execution
> > frequency to a bb with higher one in LIM invariantness_dom_walker
> > before_dom_children".
> >
> > In gimple LIM analysis, add get_coldest_out_loop to move invariants to
> > expected target loop, if profile count of the loop bb is colder
> > than target loop preheader, it won't be hoisted out of loop.
> > Likely for store motion, if all locations of the REF in loop is cold,
> > don't do store motion of it.
> >
> > SPEC2017 performance evaluation shows 1% performance improvement for
> > intrate GEOMEAN and no obvious regression for others.  Especially,
> > 500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
> > largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
> > on P8LE.
> >
> > gcc/ChangeLog:
> >
> >       * tree-ssa-loop-im.c (bb_colder_than_loop_preheader): New
> >       function.
> >       (get_coldest_out_loop): New function.
> >       (determine_max_movement): Use get_coldest_out_loop.
> >       (move_computations_worker): Adjust and fix iteration udpate.
> >       (class ref_in_loop_hot_body): New functor.
> >       (ref_in_loop_hot_body::operator): New.
> >       (can_sm_ref_p): Use for_all_locs_in_loop.
> >       (fill_coldest_and_hotter_out_loop): New.
> >       (tree_ssa_lim_finalize): Free coldest_outermost_loop and
> >       hotter_than_inner_loop.
> >       (loop_invariant_motion_in_fun): Call fill_coldet_and_hotter_out_loop.
> >
> > gcc/testsuite/ChangeLog:
> >
> >       * gcc.dg/tree-ssa/recip-3.c: Adjust.
> >       * gcc.dg/tree-ssa/ssa-lim-19.c: New test.
> >       * gcc.dg/tree-ssa/ssa-lim-20.c: New test.
> >       * gcc.dg/tree-ssa/ssa-lim-21.c: New test.
> >       * gcc.dg/tree-ssa/ssa-lim-22.c: New test.
> >       * gcc.dg/tree-ssa/ssa-lim-23.c: New test.
> > ---
> >  gcc/tree-ssa-loop-im.c                     | 150 ++++++++++++++++++++-
> >  gcc/testsuite/gcc.dg/tree-ssa/recip-3.c    |   2 +-
> >  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c |  29 ++++
> >  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c |  25 ++++
> >  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c |  35 +++++
> >  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c |  32 +++++
> >  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-23.c |  21 +++
> >  7 files changed, 291 insertions(+), 3 deletions(-)
> >  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
> >  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
> >  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
> >  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-22.c
> >  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-23.c
> >
> > diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
> > index 4b187c2cdaf..0bcc3bca91b 100644
> > --- a/gcc/tree-ssa-loop-im.c
> > +++ b/gcc/tree-ssa-loop-im.c
> > @@ -146,6 +146,11 @@ public:
> >  enum dep_kind { lim_raw, sm_war, sm_waw };
> >  enum dep_state { dep_unknown, dep_independent, dep_dependent };
> >
> > +/* coldest outermost loop for given loop.  */
> > +vec<class loop *> coldest_outermost_loop;
> > +/* hotter outer loop nearest to given loop.  */
> > +vec<class loop *> hotter_than_inner_loop;
> > +
> >  /* Populate the loop dependence cache of REF for LOOP, KIND with STATE.  */
> >
> >  static void
> > @@ -417,6 +422,61 @@ movement_possibility (gimple *stmt)
> >    return ret;
> >  }
> >
> > +/* Compare the profile count inequality of bb and loop's preheader, it is
> > +   three-state as stated in profile-count.h, FALSE is returned if inequality
> > +   cannot be decided.  */
> > +bool
> > +bb_colder_than_loop_preheader (basic_block bb, class loop *loop)
> > +{
> > +  gcc_assert (bb && loop);
> > +  return bb->count < loop_preheader_edge (loop)->src->count;
> > +}
> > +
> > +/* Check coldest loop between OUTERMOST_LOOP and LOOP by comparing profile
> > +   count.
> > +  It does three steps check:
> > +  1) Check whether CURR_BB is cold in it's own loop_father, if it is cold, just
> > +  return NULL which means it should not be moved out at all;
> > +  2) CURR_BB is NOT cold, check if pre-computed COLDEST_LOOP is outside of
> > +  OUTERMOST_LOOP, if it is inside of OUTERMOST_LOOP, return the COLDEST_LOOP;
> > +  3) If COLDEST_LOOP is outside of OUTERMOST_LOOP, check whether there is a
> > +  hotter loop between OUTERMOST_LOOP and loop in pre-computed
> > +  HOTTER_THAN_INNER_LOOP, return it's nested inner loop, otherwise return
> > +  OUTERMOST_LOOP.  */
> > +
> > +static class loop *
> > +get_coldest_out_loop (class loop *outermost_loop, class loop *loop,
> > +                   basic_block curr_bb)
> > +{
> > +  gcc_assert (outermost_loop == loop
> > +           || flow_loop_nested_p (outermost_loop, loop));
> > +
> > +  /* If bb_colder_than_loop_preheader returns false due to three-state
> > +    comparision, OUTERMOST_LOOP is returned finally to preserve the behavior.
> > +    Otherwise, return the coldest loop between OUTERMOST_LOOP and LOOP.  */
> > +  if (curr_bb && bb_colder_than_loop_preheader (curr_bb, loop))
> > +    return NULL;
> > +
> > +  class loop *coldest_loop = coldest_outermost_loop[loop->num];
> > +  if (loop_depth (coldest_loop) < loop_depth (outermost_loop))
> > +    {
> > +      class loop *hotter_loop = hotter_than_inner_loop[loop->num];
> > +      if (!hotter_loop
> > +       || loop_depth (hotter_loop) < loop_depth (outermost_loop))
> > +     return outermost_loop;
> > +
> > +      /*  hotter_loop is between OUTERMOST_LOOP and LOOP like:
> > +     [loop tree root, ..., coldest_loop, ..., outermost_loop, ...,
> > +     hotter_loop, second_coldest_loop, ..., loop]
> > +     return second_coldest_loop to be the hoist target.  */
> > +      class loop *aloop;
> > +      for (aloop = hotter_loop->inner; aloop; aloop = aloop->next)
> > +     if (flow_loop_nested_p (aloop, loop))
>
> should be:
>
>         if (aloop == loop || flow_loop_nested_p (aloop, loop))

OK with that fixed.

Are necessary prerequesites committed to avoid regressions?
I guess we need to keep a watchful eye and eventually revert
(or gate with a --param disabled by default) the new behavior if
severe regressions are discovered.

Thanks and sorry for the repeated delays.
Richard.

>
> > +       return aloop;
> > +    }
> > +  return coldest_loop;
> > +}
> > +
>
>
> --
> Thanks,
> Xionghu

Re: [PATCH v8 2/2] Don't move cold code out of loop by checking bb count

[Xionghu Luo]

On 2021/12/7 20:17, Richard Biener wrote:
>>> +  class loop *coldest_loop = coldest_outermost_loop[loop->num];
>>> +  if (loop_depth (coldest_loop) < loop_depth (outermost_loop))
>>> +    {
>>> +      class loop *hotter_loop = hotter_than_inner_loop[loop->num];
>>> +      if (!hotter_loop
>>> +       || loop_depth (hotter_loop) < loop_depth (outermost_loop))
>>> +     return outermost_loop;
>>> +
>>> +      /*  hotter_loop is between OUTERMOST_LOOP and LOOP like:
>>> +     [loop tree root, ..., coldest_loop, ..., outermost_loop, ...,
>>> +     hotter_loop, second_coldest_loop, ..., loop]
>>> +     return second_coldest_loop to be the hoist target.  */
>>> +      class loop *aloop;
>>> +      for (aloop = hotter_loop->inner; aloop; aloop = aloop->next)
>>> +     if (flow_loop_nested_p (aloop, loop))
>> should be:
>>
>>         if (aloop == loop || flow_loop_nested_p (aloop, loop))
> OK with that fixed.
> 
> Are necessary prerequesites committed to avoid regressions?
> I guess we need to keep a watchful eye and eventually revert
> (or gate with a --param disabled by default) the new behavior if
> severe regressions are discovered.
> 
> Thanks and sorry for the repeated delays.
> Richard.
> 

Thanks for your review, I learned quite a lot and gained very useful
comments & help through the period :)  There are still 3 patches required
to avoid regression or so, I've reorganized them and sent it out.

https://gcc.gnu.org/pipermail/gcc-patches/2021-December/586371.html


In addition, cooked the patch to add option for disable/enable it.
Is it OK to merge it to current patch?


[PATCH] Add option -fhoist-to-cold-loop


gcc/ChangeLog:

	* common.opt: New.
	* loop-invariant.c (find_invariants_bb):
	* tree-ssa-loop-im.c (get_coldest_out_loop):
	(can_sm_ref_p):
	(loop_invariant_motion_in_fun):
---
 gcc/common.opt         |  4 ++++
 gcc/loop-invariant.c   |  2 +-
 gcc/tree-ssa-loop-im.c | 33 ++++++++++++++++++++++-----------
 3 files changed, 27 insertions(+), 12 deletions(-)

diff --git a/gcc/common.opt b/gcc/common.opt
index b921f5e3b25..62b82bd8b95 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -1171,6 +1171,10 @@ fcode-hoisting
 Common Var(flag_code_hoisting) Optimization
 Enable code hoisting.
 
+fhoist-to-cold-loop
+Common Var(flag_hoist_to_cold_loop) Init(1) Optimization
+Enable hoisting code to cold loop.
+
 fcombine-stack-adjustments
 Common Var(flag_combine_stack_adjustments) Optimization
 Looks for opportunities to reduce stack adjustments and stack references.
diff --git a/gcc/loop-invariant.c b/gcc/loop-invariant.c
index 5c3be7bf0eb..75b9dd47cd7 100644
--- a/gcc/loop-invariant.c
+++ b/gcc/loop-invariant.c
@@ -1189,7 +1189,7 @@ find_invariants_bb (class loop *loop, basic_block bb, bool always_reached,
   rtx_insn *insn;
   basic_block preheader = loop_preheader_edge (loop)->src;
 
-  if (preheader->count > bb->count)
+  if (flag_hoist_to_cold_loop && preheader->count > bb->count)
     return;
 
   FOR_BB_INSNS (bb, insn)
diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
index 565ee62d3f7..d745f66851b 100644
--- a/gcc/tree-ssa-loop-im.c
+++ b/gcc/tree-ssa-loop-im.c
@@ -450,6 +450,9 @@ static class loop *
 get_coldest_out_loop (class loop *outermost_loop, class loop *loop,
 		      basic_block curr_bb)
 {
+  if (!flag_hoist_to_cold_loop)
+    return outermost_loop;
+
   gcc_assert (outermost_loop == loop
 	      || flow_loop_nested_p (outermost_loop, loop));
 
@@ -3031,8 +3034,9 @@ can_sm_ref_p (class loop *loop, im_mem_ref *ref)
   /* Verify whether the candidate is hot for LOOP.  Only do store motion if the
     candidate's profile count is hot.  Statement in cold BB shouldn't be moved
     out of it's loop_father.  */
-  if (!for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body (loop)))
-    return false;
+  if (flag_hoist_to_cold_loop)
+    if (!for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body (loop)))
+      return false;
 
   return true;
 }
@@ -3373,8 +3377,11 @@ tree_ssa_lim_finalize (void)
 
   free (bb_loop_postorder);
 
-  coldest_outermost_loop.release ();
-  hotter_than_inner_loop.release ();
+  if (flag_hoist_to_cold_loop)
+    {
+      coldest_outermost_loop.release ();
+      hotter_than_inner_loop.release ();
+    }
 }
 
 /* Moves invariants from loops.  Only "expensive" invariants are moved out --
@@ -3396,13 +3403,17 @@ loop_invariant_motion_in_fun (function *fun, bool store_motion)
 
   /* Pre-compute coldest outermost loop and nearest hotter loop of each loop.
    */
-  class loop *loop;
-  coldest_outermost_loop.create (number_of_loops (cfun));
-  coldest_outermost_loop.safe_grow_cleared (number_of_loops (cfun));
-  hotter_than_inner_loop.create (number_of_loops (cfun));
-  hotter_than_inner_loop.safe_grow_cleared (number_of_loops (cfun));
-  for (loop = current_loops->tree_root->inner; loop != NULL; loop = loop->next)
-    fill_coldest_and_hotter_out_loop (loop, NULL, loop);
+  if (flag_hoist_to_cold_loop)
+    {
+      class loop *loop;
+      coldest_outermost_loop.create (number_of_loops (cfun));
+      coldest_outermost_loop.safe_grow_cleared (number_of_loops (cfun));
+      hotter_than_inner_loop.create (number_of_loops (cfun));
+      hotter_than_inner_loop.safe_grow_cleared (number_of_loops (cfun));
+      for (loop = current_loops->tree_root->inner; loop != NULL;
+	   loop = loop->next)
+	fill_coldest_and_hotter_out_loop (loop, NULL, loop);
+    }
 
   int *rpo = XNEWVEC (int, last_basic_block_for_fn (fun));
   int n = pre_and_rev_post_order_compute_fn (fun, NULL, rpo, false);
-- 
2.27.0.90.geebb51ba8c

Re: [PATCH v8 2/2] Don't move cold code out of loop by checking bb count

[Richard Biener]

On Wed, Dec 8, 2021 at 7:32 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>
>
>
> On 2021/12/7 20:17, Richard Biener wrote:
> >>> +  class loop *coldest_loop = coldest_outermost_loop[loop->num];
> >>> +  if (loop_depth (coldest_loop) < loop_depth (outermost_loop))
> >>> +    {
> >>> +      class loop *hotter_loop = hotter_than_inner_loop[loop->num];
> >>> +      if (!hotter_loop
> >>> +       || loop_depth (hotter_loop) < loop_depth (outermost_loop))
> >>> +     return outermost_loop;
> >>> +
> >>> +      /*  hotter_loop is between OUTERMOST_LOOP and LOOP like:
> >>> +     [loop tree root, ..., coldest_loop, ..., outermost_loop, ...,
> >>> +     hotter_loop, second_coldest_loop, ..., loop]
> >>> +     return second_coldest_loop to be the hoist target.  */
> >>> +      class loop *aloop;
> >>> +      for (aloop = hotter_loop->inner; aloop; aloop = aloop->next)
> >>> +     if (flow_loop_nested_p (aloop, loop))
> >> should be:
> >>
> >>         if (aloop == loop || flow_loop_nested_p (aloop, loop))
> > OK with that fixed.
> >
> > Are necessary prerequesites committed to avoid regressions?
> > I guess we need to keep a watchful eye and eventually revert
> > (or gate with a --param disabled by default) the new behavior if
> > severe regressions are discovered.
> >
> > Thanks and sorry for the repeated delays.
> > Richard.
> >
>
> Thanks for your review, I learned quite a lot and gained very useful
> comments & help through the period :)  There are still 3 patches required
> to avoid regression or so, I've reorganized them and sent it out.
>
> https://gcc.gnu.org/pipermail/gcc-patches/2021-December/586371.html
>
>
> In addition, cooked the patch to add option for disable/enable it.
> Is it OK to merge it to current patch?

Hmm, let's go without this flag for now, we can add something like this
when we see a testcase where that makes sense (and where profile info
is not wrecked by other bugs).

Adding a --param would be a no brainer, but for new user-facing options
we should be more careful.

Richard.

>
> [PATCH] Add option -fhoist-to-cold-loop
>
>
> gcc/ChangeLog:
>
>         * common.opt: New.
>         * loop-invariant.c (find_invariants_bb):
>         * tree-ssa-loop-im.c (get_coldest_out_loop):
>         (can_sm_ref_p):
>         (loop_invariant_motion_in_fun):
> ---
>  gcc/common.opt         |  4 ++++
>  gcc/loop-invariant.c   |  2 +-
>  gcc/tree-ssa-loop-im.c | 33 ++++++++++++++++++++++-----------
>  3 files changed, 27 insertions(+), 12 deletions(-)
>
> diff --git a/gcc/common.opt b/gcc/common.opt
> index b921f5e3b25..62b82bd8b95 100644
> --- a/gcc/common.opt
> +++ b/gcc/common.opt
> @@ -1171,6 +1171,10 @@ fcode-hoisting
>  Common Var(flag_code_hoisting) Optimization
>  Enable code hoisting.
>
> +fhoist-to-cold-loop
> +Common Var(flag_hoist_to_cold_loop) Init(1) Optimization
> +Enable hoisting code to cold loop.
> +
>  fcombine-stack-adjustments
>  Common Var(flag_combine_stack_adjustments) Optimization
>  Looks for opportunities to reduce stack adjustments and stack references.
> diff --git a/gcc/loop-invariant.c b/gcc/loop-invariant.c
> index 5c3be7bf0eb..75b9dd47cd7 100644
> --- a/gcc/loop-invariant.c
> +++ b/gcc/loop-invariant.c
> @@ -1189,7 +1189,7 @@ find_invariants_bb (class loop *loop, basic_block bb, bool always_reached,
>    rtx_insn *insn;
>    basic_block preheader = loop_preheader_edge (loop)->src;
>
> -  if (preheader->count > bb->count)
> +  if (flag_hoist_to_cold_loop && preheader->count > bb->count)
>      return;
>
>    FOR_BB_INSNS (bb, insn)
> diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
> index 565ee62d3f7..d745f66851b 100644
> --- a/gcc/tree-ssa-loop-im.c
> +++ b/gcc/tree-ssa-loop-im.c
> @@ -450,6 +450,9 @@ static class loop *
>  get_coldest_out_loop (class loop *outermost_loop, class loop *loop,
>                       basic_block curr_bb)
>  {
> +  if (!flag_hoist_to_cold_loop)
> +    return outermost_loop;
> +
>    gcc_assert (outermost_loop == loop
>               || flow_loop_nested_p (outermost_loop, loop));
>
> @@ -3031,8 +3034,9 @@ can_sm_ref_p (class loop *loop, im_mem_ref *ref)
>    /* Verify whether the candidate is hot for LOOP.  Only do store motion if the
>      candidate's profile count is hot.  Statement in cold BB shouldn't be moved
>      out of it's loop_father.  */
> -  if (!for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body (loop)))
> -    return false;
> +  if (flag_hoist_to_cold_loop)
> +    if (!for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body (loop)))
> +      return false;
>
>    return true;
>  }
> @@ -3373,8 +3377,11 @@ tree_ssa_lim_finalize (void)
>
>    free (bb_loop_postorder);
>
> -  coldest_outermost_loop.release ();
> -  hotter_than_inner_loop.release ();
> +  if (flag_hoist_to_cold_loop)
> +    {
> +      coldest_outermost_loop.release ();
> +      hotter_than_inner_loop.release ();
> +    }
>  }
>
>  /* Moves invariants from loops.  Only "expensive" invariants are moved out --
> @@ -3396,13 +3403,17 @@ loop_invariant_motion_in_fun (function *fun, bool store_motion)
>
>    /* Pre-compute coldest outermost loop and nearest hotter loop of each loop.
>     */
> -  class loop *loop;
> -  coldest_outermost_loop.create (number_of_loops (cfun));
> -  coldest_outermost_loop.safe_grow_cleared (number_of_loops (cfun));
> -  hotter_than_inner_loop.create (number_of_loops (cfun));
> -  hotter_than_inner_loop.safe_grow_cleared (number_of_loops (cfun));
> -  for (loop = current_loops->tree_root->inner; loop != NULL; loop = loop->next)
> -    fill_coldest_and_hotter_out_loop (loop, NULL, loop);
> +  if (flag_hoist_to_cold_loop)
> +    {
> +      class loop *loop;
> +      coldest_outermost_loop.create (number_of_loops (cfun));
> +      coldest_outermost_loop.safe_grow_cleared (number_of_loops (cfun));
> +      hotter_than_inner_loop.create (number_of_loops (cfun));
> +      hotter_than_inner_loop.safe_grow_cleared (number_of_loops (cfun));
> +      for (loop = current_loops->tree_root->inner; loop != NULL;
> +          loop = loop->next)
> +       fill_coldest_and_hotter_out_loop (loop, NULL, loop);
> +    }
>
>    int *rpo = XNEWVEC (int, last_basic_block_for_fn (fun));
>    int n = pre_and_rev_post_order_compute_fn (fun, NULL, rpo, false);
> --
> 2.27.0.90.geebb51ba8c
>
>

Re: [PATCH v8 2/2] Don't move cold code out of loop by checking bb count

[Xionghu Luo]

On 2021/12/20 15:29, Richard Biener wrote:
> On Wed, Dec 8, 2021 at 7:32 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>>
>>
>>
>> On 2021/12/7 20:17, Richard Biener wrote:
>>>>> +  class loop *coldest_loop = coldest_outermost_loop[loop->num];
>>>>> +  if (loop_depth (coldest_loop) < loop_depth (outermost_loop))
>>>>> +    {
>>>>> +      class loop *hotter_loop = hotter_than_inner_loop[loop->num];
>>>>> +      if (!hotter_loop
>>>>> +       || loop_depth (hotter_loop) < loop_depth (outermost_loop))
>>>>> +     return outermost_loop;
>>>>> +
>>>>> +      /*  hotter_loop is between OUTERMOST_LOOP and LOOP like:
>>>>> +     [loop tree root, ..., coldest_loop, ..., outermost_loop, ...,
>>>>> +     hotter_loop, second_coldest_loop, ..., loop]
>>>>> +     return second_coldest_loop to be the hoist target.  */
>>>>> +      class loop *aloop;
>>>>> +      for (aloop = hotter_loop->inner; aloop; aloop = aloop->next)
>>>>> +     if (flow_loop_nested_p (aloop, loop))
>>>> should be:
>>>>
>>>>         if (aloop == loop || flow_loop_nested_p (aloop, loop))
>>> OK with that fixed.
>>>
>>> Are necessary prerequesites committed to avoid regressions?
>>> I guess we need to keep a watchful eye and eventually revert
>>> (or gate with a --param disabled by default) the new behavior if
>>> severe regressions are discovered.
>>>
>>> Thanks and sorry for the repeated delays.
>>> Richard.
>>>
>>
>> Thanks for your review, I learned quite a lot and gained very useful
>> comments & help through the period :)  There are still 3 patches required
>> to avoid regression or so, I've reorganized them and sent it out.
>>
>> https://gcc.gnu.org/pipermail/gcc-patches/2021-December/586371.html
>>
>>
>> In addition, cooked the patch to add option for disable/enable it.
>> Is it OK to merge it to current patch?
> 
> Hmm, let's go without this flag for now, we can add something like this
> when we see a testcase where that makes sense (and where profile info
> is not wrecked by other bugs).
> 
> Adding a --param would be a no brainer, but for new user-facing options
> we should be more careful.
> 

Thanks.  Committed to r12-6087.


-- 
Thanks,
Xionghu

Re: [RFC] Don't move cold code out of loop by checking bb count

[Jan Hubicka]

> Hi,
> 
> On 2021/9/28 20:09, Richard Biener wrote:
> > On Fri, Sep 24, 2021 at 8:29 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
> >>
> >> Update the patch to v3, not sure whether you prefer the paste style
> >> and continue to link the previous thread as Segher dislikes this...
> >>
> >>
> >> [PATCH v3] Don't move cold code out of loop by checking bb count
> >>
> >>
> >> Changes:
> >> 1. Handle max_loop in determine_max_movement instead of
> >> outermost_invariant_loop.
> >> 2. Remove unnecessary changes.
> >> 3. Add for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body) in can_sm_ref_p.
> >> 4. "gsi_next (&bsi);" in move_computations_worker is kept since it caused
> >> infinite loop when implementing v1 and the iteration is missed to be
> >> updated actually.
> >>
> >> v1: https://gcc.gnu.org/pipermail/gcc-patches/2021-August/576488.html
> >> v2: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/579086.html
> >>
> >> There was a patch trying to avoid move cold block out of loop:
> >>
> >> https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html
> >>
> >> Richard suggested to "never hoist anything from a bb with lower execution
> >> frequency to a bb with higher one in LIM invariantness_dom_walker
> >> before_dom_children".
> >>
> >> In gimple LIM analysis, add find_coldest_out_loop to move invariants to
> >> expected target loop, if profile count of the loop bb is colder
> >> than target loop preheader, it won't be hoisted out of loop.
> >> Likely for store motion, if all locations of the REF in loop is cold,
> >> don't do store motion of it.
> >>
> >> SPEC2017 performance evaluation shows 1% performance improvement for
> >> intrate GEOMEAN and no obvious regression for others.  Especially,
> >> 500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
> >> largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
> >> on P8LE.
> >>
> >> gcc/ChangeLog:
> >>
> >>         * loop-invariant.c (find_invariants_bb): Check profile count
> >>         before motion.
> >>         (find_invariants_body): Add argument.
> >>         * tree-ssa-loop-im.c (find_coldest_out_loop): New function.
> >>         (determine_max_movement): Use find_coldest_out_loop.
> >>         (move_computations_worker): Adjust and fix iteration udpate.
> >>         (execute_sm_exit): Check pointer validness.
> >>         (class ref_in_loop_hot_body): New functor.
> >>         (ref_in_loop_hot_body::operator): New.
> >>         (can_sm_ref_p): Use for_all_locs_in_loop.
> >>
> >> gcc/testsuite/ChangeLog:
> >>
> >>         * gcc.dg/tree-ssa/recip-3.c: Adjust.
> >>         * gcc.dg/tree-ssa/ssa-lim-18.c: New test.
> >>         * gcc.dg/tree-ssa/ssa-lim-19.c: New test.
> >>         * gcc.dg/tree-ssa/ssa-lim-20.c: New test.
> >> ---
> >>  gcc/loop-invariant.c                       | 10 ++--
> >>  gcc/tree-ssa-loop-im.c                     | 61 ++++++++++++++++++++--
> >>  gcc/testsuite/gcc.dg/tree-ssa/recip-3.c    |  2 +-
> >>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c | 20 +++++++
> >>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c | 27 ++++++++++
> >>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c | 25 +++++++++
> >>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c | 28 ++++++++++
> >>  7 files changed, 165 insertions(+), 8 deletions(-)
> >>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
> >>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
> >>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
> >>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
> >>
> >> diff --git a/gcc/loop-invariant.c b/gcc/loop-invariant.c
> >> index fca0c2b24be..5c3be7bf0eb 100644
> >> --- a/gcc/loop-invariant.c
> >> +++ b/gcc/loop-invariant.c
> >> @@ -1183,9 +1183,14 @@ find_invariants_insn (rtx_insn *insn, bool always_reached, bool always_executed)
> >>     call.  */
> >>
> >>  static void
> >> -find_invariants_bb (basic_block bb, bool always_reached, bool always_executed)
> >> +find_invariants_bb (class loop *loop, basic_block bb, bool always_reached,
> >> +                   bool always_executed)
> >>  {
> >>    rtx_insn *insn;
> >> +  basic_block preheader = loop_preheader_edge (loop)->src;
> >> +
> >> +  if (preheader->count > bb->count)
> >> +    return;
> >>
> >>    FOR_BB_INSNS (bb, insn)
> >>      {
> >> @@ -1214,8 +1219,7 @@ find_invariants_body (class loop *loop, basic_block *body,
> >>    unsigned i;
> >>
> >>    for (i = 0; i < loop->num_nodes; i++)
> >> -    find_invariants_bb (body[i],
> >> -                       bitmap_bit_p (always_reached, i),
> >> +    find_invariants_bb (loop, body[i], bitmap_bit_p (always_reached, i),
> >>                         bitmap_bit_p (always_executed, i));
> >>  }
> >>
> >> diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
> >> index 4b187c2cdaf..655fab03442 100644
> >> --- a/gcc/tree-ssa-loop-im.c
> >> +++ b/gcc/tree-ssa-loop-im.c
> >> @@ -417,6 +417,28 @@ movement_possibility (gimple *stmt)
> >>    return ret;
> >>  }
> >>
> >> +/* Find coldest loop between outmost_loop and loop by comapring profile count.  */
> >> +
> >> +static class loop *
> >> +find_coldest_out_loop (class loop *outmost_loop, class loop *loop,
> >> +                      basic_block curr_bb)
> >> +{
> >> +  class loop *cold_loop, *min_loop;
> >> +  cold_loop = min_loop = outmost_loop;
> >> +  profile_count min_count = loop_preheader_edge (min_loop)->src->count;
> >> +
> >> +  if (curr_bb && curr_bb->count < loop_preheader_edge (loop)->src->count)
> > 
> > Honza - can you comment on whether we should compare BB counts this way?
> > 
> > I would suspect that for, say,
> > 
> >   for (...)
> >      if (a)
> >        X;
> >      else
> >        Y;
> > 
> > that the counts for X and Y will be less than that of the preheader of the loop
> > only when the loop is estimated to run once.  That is, should we really compare
> > the to the preheader count or maybe better to the _header_ count which
> > would keep the number of iterations out of the equation?
> 
> I quickly tried to replace all the loop_preheader_edge (loop)->src with
> loop_preheader_edge (loop)->dest, it will cause many failures in
> gcc.dg/tree-ssa/ssa-lim-*.c, I didn't go deep to investigate, but it seems
> reasonable to compare the bb count with preheader count as both gimple lim
> and RTL loop-invariant move instructions to *preheader* instead of *header*
> after analysis?

I am not quite sure I understand what you shoot for.  But if you have
loop invariant inside a loop nest and you get range of loops in the nest
where you want to move it, you want to pick chepaer preheader count,
since the statement is going to be executed there.

For
> >   for (...)
> >      if (a)
> >        X;
> >      else
> >        Y;

You may have frequency of X less then preheader i.e. when probability
that a is true is lower than the expected iteration count.

If I understand correctly, you want to compare sum of counts of all
BBs where invariant evaulates currently to the minimal count of
preheader where you can move it.

If you have

for A
  for B
    for C
      invariant_computation

Usually you want to move it:

invariant_computation
for A
  for B
    for C

However if for B usually iterates 0 times, it may happen that preheader
of for C is executed less often then preheaders of for A/B and you want:

for A
  for B
    invariant_computation
    for C
> 
> > 
> > If we look at maybe_hot_count_p that's a quite sophisticated thing to
> > compare a count to the "IPA hot", here we're comparing two counts
> > within a function where it actually matters whether we use a<b or
> > !(a>=b) since 'unordered' is mapped to false (but there's no ordered_p
> > function).
> > 
> > Xionghu, you error on the side of not hoisting for unordered counts here
> > 
> >> +    return NULL;
> >> +
> >> +  while (min_loop != loop)
> >> +    {
> >> +      min_loop = superloop_at_depth (loop, loop_depth (min_loop) + 1);
> >> +      if (loop_preheader_edge (min_loop)->src->count < min_count)
> > 
> > but in the other direction here and on the side of not hoisting
> > in ref_in_loop_hot_body.
> > 
> > The three-state relational operator overloads are probably not the
> > very best idea...
> > (see profile-count.h for them)

In first version of the patch I had
 count1.known_le (count2)
which however made the code to look quite ugly and eventually I
convinced myself that three-state comparators are less pain than hard to
read conditionals...

But i guess we can ensapsulate them when it makes code easier to read. I
would be OK with having known_XY comparator variants in profile-count.h

Honza

Re: [RFC] Don't move cold code out of loop by checking bb count

[Xionghu Luo]

On 2021/10/27 20:54, Jan Hubicka wrote:
>> Hi,
>>
>> On 2021/9/28 20:09, Richard Biener wrote:
>>> On Fri, Sep 24, 2021 at 8:29 AM Xionghu Luo <luoxhu@linux.ibm.com> wrote:
>>>>
>>>> Update the patch to v3, not sure whether you prefer the paste style
>>>> and continue to link the previous thread as Segher dislikes this...
>>>>
>>>>
>>>> [PATCH v3] Don't move cold code out of loop by checking bb count
>>>>
>>>>
>>>> Changes:
>>>> 1. Handle max_loop in determine_max_movement instead of
>>>> outermost_invariant_loop.
>>>> 2. Remove unnecessary changes.
>>>> 3. Add for_all_locs_in_loop (loop, ref, ref_in_loop_hot_body) in can_sm_ref_p.
>>>> 4. "gsi_next (&bsi);" in move_computations_worker is kept since it caused
>>>> infinite loop when implementing v1 and the iteration is missed to be
>>>> updated actually.
>>>>
>>>> v1: https://gcc.gnu.org/pipermail/gcc-patches/2021-August/576488.html
>>>> v2: https://gcc.gnu.org/pipermail/gcc-patches/2021-September/579086.html
>>>>
>>>> There was a patch trying to avoid move cold block out of loop:
>>>>
>>>> https://gcc.gnu.org/pipermail/gcc/2014-November/215551.html
>>>>
>>>> Richard suggested to "never hoist anything from a bb with lower execution
>>>> frequency to a bb with higher one in LIM invariantness_dom_walker
>>>> before_dom_children".
>>>>
>>>> In gimple LIM analysis, add find_coldest_out_loop to move invariants to
>>>> expected target loop, if profile count of the loop bb is colder
>>>> than target loop preheader, it won't be hoisted out of loop.
>>>> Likely for store motion, if all locations of the REF in loop is cold,
>>>> don't do store motion of it.
>>>>
>>>> SPEC2017 performance evaluation shows 1% performance improvement for
>>>> intrate GEOMEAN and no obvious regression for others.  Especially,
>>>> 500.perlbench_r +7.52% (Perf shows function S_regtry of perlbench is
>>>> largely improved.), and 548.exchange2_r+1.98%, 526.blender_r +1.00%
>>>> on P8LE.
>>>>
>>>> gcc/ChangeLog:
>>>>
>>>>         * loop-invariant.c (find_invariants_bb): Check profile count
>>>>         before motion.
>>>>         (find_invariants_body): Add argument.
>>>>         * tree-ssa-loop-im.c (find_coldest_out_loop): New function.
>>>>         (determine_max_movement): Use find_coldest_out_loop.
>>>>         (move_computations_worker): Adjust and fix iteration udpate.
>>>>         (execute_sm_exit): Check pointer validness.
>>>>         (class ref_in_loop_hot_body): New functor.
>>>>         (ref_in_loop_hot_body::operator): New.
>>>>         (can_sm_ref_p): Use for_all_locs_in_loop.
>>>>
>>>> gcc/testsuite/ChangeLog:
>>>>
>>>>         * gcc.dg/tree-ssa/recip-3.c: Adjust.
>>>>         * gcc.dg/tree-ssa/ssa-lim-18.c: New test.
>>>>         * gcc.dg/tree-ssa/ssa-lim-19.c: New test.
>>>>         * gcc.dg/tree-ssa/ssa-lim-20.c: New test.
>>>> ---
>>>>  gcc/loop-invariant.c                       | 10 ++--
>>>>  gcc/tree-ssa-loop-im.c                     | 61 ++++++++++++++++++++--
>>>>  gcc/testsuite/gcc.dg/tree-ssa/recip-3.c    |  2 +-
>>>>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c | 20 +++++++
>>>>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c | 27 ++++++++++
>>>>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c | 25 +++++++++
>>>>  gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c | 28 ++++++++++
>>>>  7 files changed, 165 insertions(+), 8 deletions(-)
>>>>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-18.c
>>>>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-19.c
>>>>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-20.c
>>>>  create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/ssa-lim-21.c
>>>>
>>>> diff --git a/gcc/loop-invariant.c b/gcc/loop-invariant.c
>>>> index fca0c2b24be..5c3be7bf0eb 100644
>>>> --- a/gcc/loop-invariant.c
>>>> +++ b/gcc/loop-invariant.c
>>>> @@ -1183,9 +1183,14 @@ find_invariants_insn (rtx_insn *insn, bool always_reached, bool always_executed)
>>>>     call.  */
>>>>
>>>>  static void
>>>> -find_invariants_bb (basic_block bb, bool always_reached, bool always_executed)
>>>> +find_invariants_bb (class loop *loop, basic_block bb, bool always_reached,
>>>> +                   bool always_executed)
>>>>  {
>>>>    rtx_insn *insn;
>>>> +  basic_block preheader = loop_preheader_edge (loop)->src;
>>>> +
>>>> +  if (preheader->count > bb->count)
>>>> +    return;
>>>>
>>>>    FOR_BB_INSNS (bb, insn)
>>>>      {
>>>> @@ -1214,8 +1219,7 @@ find_invariants_body (class loop *loop, basic_block *body,
>>>>    unsigned i;
>>>>
>>>>    for (i = 0; i < loop->num_nodes; i++)
>>>> -    find_invariants_bb (body[i],
>>>> -                       bitmap_bit_p (always_reached, i),
>>>> +    find_invariants_bb (loop, body[i], bitmap_bit_p (always_reached, i),
>>>>                         bitmap_bit_p (always_executed, i));
>>>>  }
>>>>
>>>> diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
>>>> index 4b187c2cdaf..655fab03442 100644
>>>> --- a/gcc/tree-ssa-loop-im.c
>>>> +++ b/gcc/tree-ssa-loop-im.c
>>>> @@ -417,6 +417,28 @@ movement_possibility (gimple *stmt)
>>>>    return ret;
>>>>  }
>>>>
>>>> +/* Find coldest loop between outmost_loop and loop by comapring profile count.  */
>>>> +
>>>> +static class loop *
>>>> +find_coldest_out_loop (class loop *outmost_loop, class loop *loop,
>>>> +                      basic_block curr_bb)
>>>> +{
>>>> +  class loop *cold_loop, *min_loop;
>>>> +  cold_loop = min_loop = outmost_loop;
>>>> +  profile_count min_count = loop_preheader_edge (min_loop)->src->count;
>>>> +
>>>> +  if (curr_bb && curr_bb->count < loop_preheader_edge (loop)->src->count)
>>>
>>> Honza - can you comment on whether we should compare BB counts this way?
>>>
>>> I would suspect that for, say,
>>>
>>>   for (...)
>>>      if (a)
>>>        X;
>>>      else
>>>        Y;
>>>
>>> that the counts for X and Y will be less than that of the preheader of the loop
>>> only when the loop is estimated to run once.  That is, should we really compare
>>> the to the preheader count or maybe better to the _header_ count which
>>> would keep the number of iterations out of the equation?
>>
>> I quickly tried to replace all the loop_preheader_edge (loop)->src with
>> loop_preheader_edge (loop)->dest, it will cause many failures in
>> gcc.dg/tree-ssa/ssa-lim-*.c, I didn't go deep to investigate, but it seems
>> reasonable to compare the bb count with preheader count as both gimple lim
>> and RTL loop-invariant move instructions to *preheader* instead of *header*
>> after analysis?
> 
> I am not quite sure I understand what you shoot for.  But if you have
> loop invariant inside a loop nest and you get range of loops in the nest
> where you want to move it, you want to pick chepaer preheader count,
> since the statement is going to be executed there.
> 
> For
>>>   for (...)
>>>      if (a)
>>>        X;
>>>      else
>>>        Y;
> 
> You may have frequency of X less then preheader i.e. when probability
> that a is true is lower than the expected iteration count.
> 
> If I understand correctly, you want to compare sum of counts of all
> BBs where invariant evaulates currently to the minimal count of
> preheader where you can move it.
> 
> If you have
> 
> for A
>   for B
>     for C
>       invariant_computation
> 
> Usually you want to move it:
> 
> invariant_computation
> for A
>   for B
>     for C
> 
> However if for B usually iterates 0 times, it may happen that preheader
> of for C is executed less often then preheaders of for A/B and you want:
> 
> for A
>   for B
>     invariant_computation
>     for C

Thanks, this is what I am trying to do in both gimple lim and RTL loop-invariant motion. 

In gimple lim, the new added function find_coldest_out_loop[1] will check whether 
invariant_computation is hotter than C_preheader, if true, find a coldest
preheader from outermost nested loop, if B is the coldest, reset the outmost_loop
to B, which could avoid hoist cold statement to hot loops to reduce execution counts.
Gimple only change could improve 500.perlbench_r and 548.exchange2_r a bit[2].

RTL patch need only small check like below, it could improve performance
500.perlbench_r ~8% [2]for at least Power and aarch64.


[1] https://gcc.gnu.org/pipermail/gcc-patches/2021-October/581961.html
[2] https://gcc.gnu.org/pipermail/gcc-patches/2021-September/580110.html


This is the other patch Richard and I expecting your review :)


From 468e0b252a6b4a8b648c4a49850ed337ab5e03e1 Mon Sep 17 00:00:00 2001
From: Xiong Hu Luo <luoxhu@linux.ibm.com>
Date: Fri, 8 Oct 2021 22:05:39 -0500
Subject: [PATCH v4 2/2] loop-invariant: Don't move cold bb instructions to preheader in RTL

gcc/ChangeLog:

	* loop-invariant.c (find_invariants_bb): Check profile count
	before motion.
	(find_invariants_body): Add argument.
---
 gcc/loop-invariant.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/gcc/loop-invariant.c b/gcc/loop-invariant.c
index fca0c2b24be..5c3be7bf0eb 100644
--- a/gcc/loop-invariant.c
+++ b/gcc/loop-invariant.c
@@ -1183,9 +1183,14 @@ find_invariants_insn (rtx_insn *insn, bool always_reached, bool always_executed)
    call.  */
 
 static void
-find_invariants_bb (basic_block bb, bool always_reached, bool always_executed)
+find_invariants_bb (class loop *loop, basic_block bb, bool always_reached,
+		    bool always_executed)
 {
   rtx_insn *insn;
+  basic_block preheader = loop_preheader_edge (loop)->src;
+
+  if (preheader->count > bb->count)
+    return;
 
   FOR_BB_INSNS (bb, insn)
     {
@@ -1214,8 +1219,7 @@ find_invariants_body (class loop *loop, basic_block *body,
   unsigned i;
 
   for (i = 0; i < loop->num_nodes; i++)
-    find_invariants_bb (body[i],
-			bitmap_bit_p (always_reached, i),
+    find_invariants_bb (loop, body[i], bitmap_bit_p (always_reached, i),
 			bitmap_bit_p (always_executed, i));
 }
 
-- 
2.27.0.90.geebb51ba8c



>>
>>>
>>> If we look at maybe_hot_count_p that's a quite sophisticated thing to
>>> compare a count to the "IPA hot", here we're comparing two counts
>>> within a function where it actually matters whether we use a<b or
>>> !(a>=b) since 'unordered' is mapped to false (but there's no ordered_p
>>> function).
>>>
>>> Xionghu, you error on the side of not hoisting for unordered counts here
>>>
>>>> +    return NULL;
>>>> +
>>>> +  while (min_loop != loop)
>>>> +    {
>>>> +      min_loop = superloop_at_depth (loop, loop_depth (min_loop) + 1);
>>>> +      if (loop_preheader_edge (min_loop)->src->count < min_count)
>>>
>>> but in the other direction here and on the side of not hoisting
>>> in ref_in_loop_hot_body.
>>>
>>> The three-state relational operator overloads are probably not the
>>> very best idea...
>>> (see profile-count.h for them)
> 
> In first version of the patch I had
>  count1.known_le (count2)
> which however made the code to look quite ugly and eventually I
> convinced myself that three-state comparators are less pain than hard to
> read conditionals...
> 
> But i guess we can ensapsulate them when it makes code easier to read. I
> would be OK with having known_XY comparator variants in profile-count.h

I did add a new function to ensapsulate it in tree-ssa-loop-im.c, we could
replace it with known_le if needed:


+/* Compare the profile count inequality of bb and preheader, it is three-state
+   as stated in profile-count.h, FALSE is returned if inequality cannot be
+   decided.  */
+bool bb_colder_than_loop_preheader (basic_block bb, basic_block preheader)
+{
+  gcc_assert (bb && preheader);
+  return bb->count < preheader->count;
+}


> 
> Honza
> 

-- 
Thanks,
Xionghu