* [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition @ 2013-08-01 16:32 Teresa Johnson 2013-08-02 11:22 ` Bernhard Reutner-Fischer 0 siblings, 1 reply; 62+ messages in thread From: Teresa Johnson @ 2013-08-01 16:32 UTC (permalink / raw) To: gcc-patches; +Cc: Steven Bosscher, Jan Hubicka, Jeff Law [-- Attachment #1: Type: text/plain, Size: 14587 bytes --] Patch 3 of 3 split out from the patch I sent in May that fixes problems with -freorder-blocks-and-partition, with changes/fixes discussed in that thread. See http://gcc.gnu.org/ml/gcc-patches/2013-05/threads.html#00388 for context. This patch sanitizes the partitioning to address issues such as edge weight insanities that sometimes occur due to upstream optimizations, and ensures that hot blocks are not dominated by cold blocks. This needs to be resanitized after certain cfg optimizations that may cause hot blocks previously reached via both hot and cold paths to only be reached by cold paths. The verification code in sanitize_dominator_hotness was contributed by Steven Bosscher. Bootstrapped and tested on x86-64-unknown-linux-gnu. Also ensured that a profiledbootstrap passed with -freorder-blocks-and-partition enabled (and with the dwarf version changed to 2 to work around PR57451). Ok for trunk? (I also included the patch as an attachment since my mailer invariably messes up the formatting in the pasted version.) Thanks, Teresa 2013-08-01 Teresa Johnson <tejohnson@google.com> Steven Bosscher <steven@gcc.gnu.org> * cfgrtl.c (fixup_bb_partition): New routine. (commit_edge_insertions): Invoke fixup_partitions. (find_partition_fixes): New routine. (fixup_partitions): Ditto. (verify_hot_cold_block_grouping): Update comments. (rtl_verify_edges): Invoke find_partition_fixes. (rtl_verify_bb_pointers): Update comments. (rtl_verify_bb_layout): Ditto. * basic-block.h (fixup_partitions): Declare. * cfgcleanup.c (try_optimize_cfg): Invoke fixup_partitions. * bb-reorder.c (sanitize_dominator_hotness): New function. (find_rarely_executed_basic_blocks_and_crossing_edges): Invoke sanitize_dominator_hotness. Index: cfgrtl.c =================================================================== --- cfgrtl.c (revision 201281) +++ cfgrtl.c (working copy) @@ -1341,6 +1341,34 @@ fixup_partition_crossing (edge e) } } +/* Called when block BB has been reassigned to a different partition, + to ensure that the region crossing attributes are updated. */ + +static void +fixup_bb_partition (basic_block bb) +{ + edge e; + edge_iterator ei; + + /* Now need to make bb's pred edges non-region crossing. */ + FOR_EACH_EDGE (e, ei, bb->preds) + { + fixup_partition_crossing (e); + } + + /* Possibly need to make bb's successor edges region crossing, + or remove stale region crossing. */ + FOR_EACH_EDGE (e, ei, bb->succs) + { + if ((e->flags & EDGE_FALLTHRU) + && BB_PARTITION (bb) != BB_PARTITION (e->dest) + && e->dest != EXIT_BLOCK_PTR) + force_nonfallthru (e); + else + fixup_partition_crossing (e); + } +} + /* Attempt to change code to redirect edge E to TARGET. Don't do that on expense of adding new instructions or reordering basic blocks. @@ -1979,6 +2007,14 @@ commit_edge_insertions (void) { basic_block bb; + /* Optimization passes that invoke this routine can cause hot blocks + previously reached by both hot and cold blocks to become dominated only + by cold blocks. This will cause the verification below to fail, + and lead to now cold code in the hot section. In some cases this + may only be visible after newly unreachable blocks are deleted, + which will be done by fixup_partitions. */ + fixup_partitions (); + #ifdef ENABLE_CHECKING verify_flow_info (); #endif @@ -2173,6 +2209,101 @@ get_last_bb_insn (basic_block bb) return end; } +/* Sanity check partition hotness to ensure that basic blocks in + the cold partition don't dominate basic blocks in the hot partition. + If FLAG_ONLY is true, report violations as errors. Otherwise + re-mark the dominated blocks as cold, since this is run after + cfg optimizations that may make hot blocks previously reached + by both hot and cold blocks now only reachable along cold paths. */ + +vec<basic_block> +find_partition_fixes (bool flag_only) +{ + basic_block bb; + vec<basic_block> bbs_in_cold_partition = vNULL; + vec<basic_block> bbs_to_fix = vNULL; + + if (!crtl->has_bb_partition) + return vNULL; + + FOR_EACH_BB (bb) + if ((BB_PARTITION (bb) == BB_COLD_PARTITION)) + bbs_in_cold_partition.safe_push (bb); + + if (bbs_in_cold_partition.is_empty ()) + return vNULL; + + bool dom_calculated_here = !dom_info_available_p (CDI_DOMINATORS); + + if (dom_calculated_here) + calculate_dominance_info (CDI_DOMINATORS); + + while (! bbs_in_cold_partition.is_empty ()) + { + bb = bbs_in_cold_partition.pop (); + /* Any blocks dominated by a block in the cold section + must also be cold. */ + basic_block son; + for (son = first_dom_son (CDI_DOMINATORS, bb); + son; + son = next_dom_son (CDI_DOMINATORS, son)) + { + /* If son is not yet cold, then mark it cold here and + enqueue it for further processing. */ + if ((BB_PARTITION (son) != BB_COLD_PARTITION)) + { + if (flag_only) + error ("non-cold basic block %d dominated " + "by a block in the cold partition", son->index); + else + BB_SET_PARTITION (son, BB_COLD_PARTITION); + bbs_to_fix.safe_push (son); + bbs_in_cold_partition.safe_push (son); + } + } + } + + if (dom_calculated_here) + free_dominance_info (CDI_DOMINATORS); + + return bbs_to_fix; +} + +/* Perform cleanup on the hot/cold bb partitioning after optimization + passes that modify the cfg. */ + +void +fixup_partitions (void) +{ + basic_block bb; + + if (!crtl->has_bb_partition) + return; + + /* Delete any blocks that became unreachable and weren't + already cleaned up, for example during edge forwarding + and convert_jumps_to_returns. This will expose more + opportunities for fixing the partition boundaries here. + Also, the calculation of the dominance graph during verification + will assert if there are unreachable nodes. */ + delete_unreachable_blocks (); + + /* If there are partitions, do a sanity check on them: A basic block in + a cold partition cannot dominate a basic block in a hot partition. + Fixup any that now violate this requirement, as a result of edge + forwarding and unreachable block deletion. */ + vec<basic_block> bbs_to_fix = find_partition_fixes (false); + + /* Do the partition fixup after all necessary blocks have been converted to + cold, so that we only update the region crossings the minimum number of + places, which can require forcing edges to be non fallthru. */ + while (! bbs_to_fix.is_empty ()) + { + bb = bbs_to_fix.pop (); + fixup_bb_partition (bb); + } +} + /* Verify, in the basic block chain, that there is at most one switch between hot/cold partitions. This condition will not be true until after reorder_basic_blocks is called. */ @@ -2219,7 +2350,8 @@ verify_hot_cold_block_grouping (void) /* Perform several checks on the edges out of each block, such as the consistency of the branch probabilities, the correctness of hot/cold partition crossing edges, and the number of expected - successor edges. */ + successor edges. Also verify that the dominance relationship + between hot/cold blocks is sane. */ static int rtl_verify_edges (void) @@ -2382,6 +2514,14 @@ rtl_verify_edges (void) } } + /* If there are partitions, do a sanity check on them: A basic block in + a cold partition cannot dominate a basic block in a hot partition. */ + if (crtl->has_bb_partition && !err) + { + vec<basic_block> bbs_to_fix = find_partition_fixes (true); + err = !bbs_to_fix.is_empty (); + } + /* Clean up. */ return err; } @@ -2515,7 +2655,7 @@ rtl_verify_bb_pointers (void) and NOTE_INSN_BASIC_BLOCK - verify that no fall_thru edge crosses hot/cold partition boundaries - verify that there are no pending RTL branch predictions - - verify that there is a single hot/cold partition boundary after bbro + - verify that hot blocks are not dominated by cold blocks In future it can be extended check a lot of other stuff as well (reachability of basic blocks, life information, etc. etc.). */ @@ -2761,7 +2901,8 @@ rtl_verify_bb_layout (void) - check that all insns are in the basic blocks (except the switch handling code, barriers and notes) - check that all returns are followed by barriers - - check that all fallthru edge points to the adjacent blocks. */ + - check that all fallthru edge points to the adjacent blocks + - verify that there is a single hot/cold partition boundary after bbro */ static int rtl_verify_flow_info (void) Index: basic-block.h =================================================================== --- basic-block.h (revision 201281) +++ basic-block.h (working copy) @@ -797,6 +797,7 @@ extern bool contains_no_active_insn_p (const_basic extern bool forwarder_block_p (const_basic_block); extern bool can_fallthru (basic_block, basic_block); extern void emit_barrier_after_bb (basic_block bb); +extern void fixup_partitions (void); /* In cfgbuild.c. */ extern void find_many_sub_basic_blocks (sbitmap); Index: cfgcleanup.c =================================================================== --- cfgcleanup.c (revision 201281) +++ cfgcleanup.c (working copy) @@ -2807,10 +2807,21 @@ try_optimize_cfg (int mode) df_analyze (); } + if (changed) + { + /* Edge forwarding in particular can cause hot blocks previously + reached by both hot and cold blocks to become dominated only + by cold blocks. This will cause the verification below to fail, + and lead to now cold code in the hot section. This is not easy + to detect and fix during edge forwarding, and in some cases + is only visible after newly unreachable blocks are deleted, + which will be done in fixup_partitions. */ + fixup_partitions (); + #ifdef ENABLE_CHECKING - if (changed) - verify_flow_info (); + verify_flow_info (); #endif + } changed_overall |= changed; first_pass = false; Index: bb-reorder.c =================================================================== --- bb-reorder.c (revision 201281) +++ bb-reorder.c (working copy) @@ -1444,6 +1444,55 @@ fix_up_crossing_landing_pad (eh_landing_pad old_lp ei_next (&ei); } + +/* Ensure that no cold bbs dominate hot bbs along the dominance or + post-dominance DIR, for example as a result of edge weight insanities. + Returns the updated value of COLD_BB_COUNT and adds newly-hot bbs + to BBS_IN_HOT_PARTITION. */ + +static unsigned int +sanitize_dominator_hotness (enum cdi_direction dir, unsigned int cold_bb_count, + vec<basic_block> *bbs_in_hot_partition) +{ + if (!cold_bb_count) + return 0; + + bool dom_calculated_here = !dom_info_available_p (dir); + + if (dom_calculated_here) + calculate_dominance_info (dir); + + /* Keep examining hot bbs until we have either checked them all, or + re-marked all cold bbs as hot. */ + vec<basic_block> hot_bbs_to_check = bbs_in_hot_partition->copy (); + while (! hot_bbs_to_check.is_empty () + && cold_bb_count) + { + basic_block bb = hot_bbs_to_check.pop (); + basic_block dom_bb = get_immediate_dominator (dir, bb); + + /* If bb's immediate dominator is also hot then it is ok. */ + if (BB_PARTITION (dom_bb) != BB_COLD_PARTITION) + continue; + + /* We have a hot bb with an immediate dominator that is cold. + The dominator needs to be re-marked hot. */ + BB_SET_PARTITION (dom_bb, BB_HOT_PARTITION); + cold_bb_count--; + + /* Now we need to examine newly-hot dom_bb to see if it is also + dominated by a cold bb. */ + bbs_in_hot_partition->safe_push (dom_bb); + hot_bbs_to_check.safe_push (dom_bb); + } + + if (dom_calculated_here) + free_dominance_info (dir); + + return cold_bb_count; +} + + /* Find the basic blocks that are rarely executed and need to be moved to a separate section of the .o file (to cut down on paging and improve cache locality). Return a vector of all edges that cross. */ @@ -1455,16 +1504,42 @@ find_rarely_executed_basic_blocks_and_crossing_edg basic_block bb; edge e; edge_iterator ei; + unsigned int cold_bb_count = 0; + vec<basic_block> bbs_in_hot_partition = vNULL; /* Mark which partition (hot/cold) each basic block belongs in. */ FOR_EACH_BB (bb) { if (probably_never_executed_bb_p (cfun, bb)) - BB_SET_PARTITION (bb, BB_COLD_PARTITION); + { + BB_SET_PARTITION (bb, BB_COLD_PARTITION); + cold_bb_count++; + } else - BB_SET_PARTITION (bb, BB_HOT_PARTITION); + { + BB_SET_PARTITION (bb, BB_HOT_PARTITION); + bbs_in_hot_partition.safe_push (bb); + } } + /* Ensure that no cold bbs dominate hot bbs. This could happen as a result of + several different possibilities. One is that there are edge weight insanities + due to optimization phases that do not properly update basic block profile + counts. The second is that the entry of the function may not be hot, because + it is entered fewer times than the number of profile training runs, but there + is a loop inside the function that causes blocks within the function to be + above the threshold for hotness. Then do the same along the post-dominator + tree (which could have additional changes required after fixing up + dominators). */ + if (cold_bb_count) + cold_bb_count = sanitize_dominator_hotness (CDI_DOMINATORS, + cold_bb_count, + &bbs_in_hot_partition); + if (cold_bb_count) + cold_bb_count = sanitize_dominator_hotness (CDI_POST_DOMINATORS, + cold_bb_count, + &bbs_in_hot_partition); + /* The format of .gcc_except_table does not allow landing pads to be in a different partition as the throw. Fix this by either moving or duplicating the landing pads. */ -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 [-- Attachment #2: patch.diff --] [-- Type: application/octet-stream, Size: 14336 bytes --] Patch 3 of 3 split out from the patch I sent in May that fixes problems with -freorder-blocks-and-partition, with changes/fixes discussed in that thread. See http://gcc.gnu.org/ml/gcc-patches/2013-05/threads.html#00388 for context. This patch sanitizes the partitioning to address issues such as edge weight insanities that sometimes occur due to upstream optimizations, and ensures that hot blocks are not dominated by cold blocks. This needs to be resanitized after certain cfg optimizations that may cause hot blocks previously reached via both hot and cold paths to only be reached by cold paths. The verification code in sanitize_dominator_hotness was contributed by Steven Bosscher. Bootstrapped and tested on x86-64-unknown-linux-gnu. Also ensured that a profiledbootstrap passed with -freorder-blocks-and-partition enabled (and with the dwarf version changed to 2 to work around PR57451). Ok for trunk? Thanks, Teresa 2013-08-01 Teresa Johnson <tejohnson@google.com> Steven Bosscher <steven@gcc.gnu.org> * cfgrtl.c (fixup_bb_partition): New routine. (commit_edge_insertions): Invoke fixup_partitions. (find_partition_fixes): New routine. (fixup_partitions): Ditto. (verify_hot_cold_block_grouping): Update comments. (rtl_verify_edges): Invoke find_partition_fixes. (rtl_verify_bb_pointers): Update comments. (rtl_verify_bb_layout): Ditto. * basic-block.h (fixup_partitions): Declare. * cfgcleanup.c (try_optimize_cfg): Invoke fixup_partitions. * bb-reorder.c (sanitize_dominator_hotness): New function. (find_rarely_executed_basic_blocks_and_crossing_edges): Invoke sanitize_dominator_hotness. Index: cfgrtl.c =================================================================== --- cfgrtl.c (revision 201281) +++ cfgrtl.c (working copy) @@ -1341,6 +1341,34 @@ fixup_partition_crossing (edge e) } } +/* Called when block BB has been reassigned to a different partition, + to ensure that the region crossing attributes are updated. */ + +static void +fixup_bb_partition (basic_block bb) +{ + edge e; + edge_iterator ei; + + /* Now need to make bb's pred edges non-region crossing. */ + FOR_EACH_EDGE (e, ei, bb->preds) + { + fixup_partition_crossing (e); + } + + /* Possibly need to make bb's successor edges region crossing, + or remove stale region crossing. */ + FOR_EACH_EDGE (e, ei, bb->succs) + { + if ((e->flags & EDGE_FALLTHRU) + && BB_PARTITION (bb) != BB_PARTITION (e->dest) + && e->dest != EXIT_BLOCK_PTR) + force_nonfallthru (e); + else + fixup_partition_crossing (e); + } +} + /* Attempt to change code to redirect edge E to TARGET. Don't do that on expense of adding new instructions or reordering basic blocks. @@ -1979,6 +2007,14 @@ commit_edge_insertions (void) { basic_block bb; + /* Optimization passes that invoke this routine can cause hot blocks + previously reached by both hot and cold blocks to become dominated only + by cold blocks. This will cause the verification below to fail, + and lead to now cold code in the hot section. In some cases this + may only be visible after newly unreachable blocks are deleted, + which will be done by fixup_partitions. */ + fixup_partitions (); + #ifdef ENABLE_CHECKING verify_flow_info (); #endif @@ -2173,6 +2209,101 @@ get_last_bb_insn (basic_block bb) return end; } +/* Sanity check partition hotness to ensure that basic blocks in + the cold partition don't dominate basic blocks in the hot partition. + If FLAG_ONLY is true, report violations as errors. Otherwise + re-mark the dominated blocks as cold, since this is run after + cfg optimizations that may make hot blocks previously reached + by both hot and cold blocks now only reachable along cold paths. */ + +vec<basic_block> +find_partition_fixes (bool flag_only) +{ + basic_block bb; + vec<basic_block> bbs_in_cold_partition = vNULL; + vec<basic_block> bbs_to_fix = vNULL; + + if (!crtl->has_bb_partition) + return vNULL; + + FOR_EACH_BB (bb) + if ((BB_PARTITION (bb) == BB_COLD_PARTITION)) + bbs_in_cold_partition.safe_push (bb); + + if (bbs_in_cold_partition.is_empty ()) + return vNULL; + + bool dom_calculated_here = !dom_info_available_p (CDI_DOMINATORS); + + if (dom_calculated_here) + calculate_dominance_info (CDI_DOMINATORS); + + while (! bbs_in_cold_partition.is_empty ()) + { + bb = bbs_in_cold_partition.pop (); + /* Any blocks dominated by a block in the cold section + must also be cold. */ + basic_block son; + for (son = first_dom_son (CDI_DOMINATORS, bb); + son; + son = next_dom_son (CDI_DOMINATORS, son)) + { + /* If son is not yet cold, then mark it cold here and + enqueue it for further processing. */ + if ((BB_PARTITION (son) != BB_COLD_PARTITION)) + { + if (flag_only) + error ("non-cold basic block %d dominated " + "by a block in the cold partition", son->index); + else + BB_SET_PARTITION (son, BB_COLD_PARTITION); + bbs_to_fix.safe_push (son); + bbs_in_cold_partition.safe_push (son); + } + } + } + + if (dom_calculated_here) + free_dominance_info (CDI_DOMINATORS); + + return bbs_to_fix; +} + +/* Perform cleanup on the hot/cold bb partitioning after optimization + passes that modify the cfg. */ + +void +fixup_partitions (void) +{ + basic_block bb; + + if (!crtl->has_bb_partition) + return; + + /* Delete any blocks that became unreachable and weren't + already cleaned up, for example during edge forwarding + and convert_jumps_to_returns. This will expose more + opportunities for fixing the partition boundaries here. + Also, the calculation of the dominance graph during verification + will assert if there are unreachable nodes. */ + delete_unreachable_blocks (); + + /* If there are partitions, do a sanity check on them: A basic block in + a cold partition cannot dominate a basic block in a hot partition. + Fixup any that now violate this requirement, as a result of edge + forwarding and unreachable block deletion. */ + vec<basic_block> bbs_to_fix = find_partition_fixes (false); + + /* Do the partition fixup after all necessary blocks have been converted to + cold, so that we only update the region crossings the minimum number of + places, which can require forcing edges to be non fallthru. */ + while (! bbs_to_fix.is_empty ()) + { + bb = bbs_to_fix.pop (); + fixup_bb_partition (bb); + } +} + /* Verify, in the basic block chain, that there is at most one switch between hot/cold partitions. This condition will not be true until after reorder_basic_blocks is called. */ @@ -2219,7 +2350,8 @@ verify_hot_cold_block_grouping (void) /* Perform several checks on the edges out of each block, such as the consistency of the branch probabilities, the correctness of hot/cold partition crossing edges, and the number of expected - successor edges. */ + successor edges. Also verify that the dominance relationship + between hot/cold blocks is sane. */ static int rtl_verify_edges (void) @@ -2382,6 +2514,14 @@ rtl_verify_edges (void) } } + /* If there are partitions, do a sanity check on them: A basic block in + a cold partition cannot dominate a basic block in a hot partition. */ + if (crtl->has_bb_partition && !err) + { + vec<basic_block> bbs_to_fix = find_partition_fixes (true); + err = !bbs_to_fix.is_empty (); + } + /* Clean up. */ return err; } @@ -2515,7 +2655,7 @@ rtl_verify_bb_pointers (void) and NOTE_INSN_BASIC_BLOCK - verify that no fall_thru edge crosses hot/cold partition boundaries - verify that there are no pending RTL branch predictions - - verify that there is a single hot/cold partition boundary after bbro + - verify that hot blocks are not dominated by cold blocks In future it can be extended check a lot of other stuff as well (reachability of basic blocks, life information, etc. etc.). */ @@ -2761,7 +2901,8 @@ rtl_verify_bb_layout (void) - check that all insns are in the basic blocks (except the switch handling code, barriers and notes) - check that all returns are followed by barriers - - check that all fallthru edge points to the adjacent blocks. */ + - check that all fallthru edge points to the adjacent blocks + - verify that there is a single hot/cold partition boundary after bbro */ static int rtl_verify_flow_info (void) Index: basic-block.h =================================================================== --- basic-block.h (revision 201281) +++ basic-block.h (working copy) @@ -797,6 +797,7 @@ extern bool contains_no_active_insn_p (const_basic extern bool forwarder_block_p (const_basic_block); extern bool can_fallthru (basic_block, basic_block); extern void emit_barrier_after_bb (basic_block bb); +extern void fixup_partitions (void); /* In cfgbuild.c. */ extern void find_many_sub_basic_blocks (sbitmap); Index: cfgcleanup.c =================================================================== --- cfgcleanup.c (revision 201281) +++ cfgcleanup.c (working copy) @@ -2807,10 +2807,21 @@ try_optimize_cfg (int mode) df_analyze (); } + if (changed) + { + /* Edge forwarding in particular can cause hot blocks previously + reached by both hot and cold blocks to become dominated only + by cold blocks. This will cause the verification below to fail, + and lead to now cold code in the hot section. This is not easy + to detect and fix during edge forwarding, and in some cases + is only visible after newly unreachable blocks are deleted, + which will be done in fixup_partitions. */ + fixup_partitions (); + #ifdef ENABLE_CHECKING - if (changed) - verify_flow_info (); + verify_flow_info (); #endif + } changed_overall |= changed; first_pass = false; Index: bb-reorder.c =================================================================== --- bb-reorder.c (revision 201281) +++ bb-reorder.c (working copy) @@ -1444,6 +1444,55 @@ fix_up_crossing_landing_pad (eh_landing_pad old_lp ei_next (&ei); } + +/* Ensure that no cold bbs dominate hot bbs along the dominance or + post-dominance DIR, for example as a result of edge weight insanities. + Returns the updated value of COLD_BB_COUNT and adds newly-hot bbs + to BBS_IN_HOT_PARTITION. */ + +static unsigned int +sanitize_dominator_hotness (enum cdi_direction dir, unsigned int cold_bb_count, + vec<basic_block> *bbs_in_hot_partition) +{ + if (!cold_bb_count) + return 0; + + bool dom_calculated_here = !dom_info_available_p (dir); + + if (dom_calculated_here) + calculate_dominance_info (dir); + + /* Keep examining hot bbs until we have either checked them all, or + re-marked all cold bbs as hot. */ + vec<basic_block> hot_bbs_to_check = bbs_in_hot_partition->copy (); + while (! hot_bbs_to_check.is_empty () + && cold_bb_count) + { + basic_block bb = hot_bbs_to_check.pop (); + basic_block dom_bb = get_immediate_dominator (dir, bb); + + /* If bb's immediate dominator is also hot then it is ok. */ + if (BB_PARTITION (dom_bb) != BB_COLD_PARTITION) + continue; + + /* We have a hot bb with an immediate dominator that is cold. + The dominator needs to be re-marked hot. */ + BB_SET_PARTITION (dom_bb, BB_HOT_PARTITION); + cold_bb_count--; + + /* Now we need to examine newly-hot dom_bb to see if it is also + dominated by a cold bb. */ + bbs_in_hot_partition->safe_push (dom_bb); + hot_bbs_to_check.safe_push (dom_bb); + } + + if (dom_calculated_here) + free_dominance_info (dir); + + return cold_bb_count; +} + + /* Find the basic blocks that are rarely executed and need to be moved to a separate section of the .o file (to cut down on paging and improve cache locality). Return a vector of all edges that cross. */ @@ -1455,16 +1504,42 @@ find_rarely_executed_basic_blocks_and_crossing_edg basic_block bb; edge e; edge_iterator ei; + unsigned int cold_bb_count = 0; + vec<basic_block> bbs_in_hot_partition = vNULL; /* Mark which partition (hot/cold) each basic block belongs in. */ FOR_EACH_BB (bb) { if (probably_never_executed_bb_p (cfun, bb)) - BB_SET_PARTITION (bb, BB_COLD_PARTITION); + { + BB_SET_PARTITION (bb, BB_COLD_PARTITION); + cold_bb_count++; + } else - BB_SET_PARTITION (bb, BB_HOT_PARTITION); + { + BB_SET_PARTITION (bb, BB_HOT_PARTITION); + bbs_in_hot_partition.safe_push (bb); + } } + /* Ensure that no cold bbs dominate hot bbs. This could happen as a result of + several different possibilities. One is that there are edge weight insanities + due to optimization phases that do not properly update basic block profile + counts. The second is that the entry of the function may not be hot, because + it is entered fewer times than the number of profile training runs, but there + is a loop inside the function that causes blocks within the function to be + above the threshold for hotness. Then do the same along the post-dominator + tree (which could have additional changes required after fixing up + dominators). */ + if (cold_bb_count) + cold_bb_count = sanitize_dominator_hotness (CDI_DOMINATORS, + cold_bb_count, + &bbs_in_hot_partition); + if (cold_bb_count) + cold_bb_count = sanitize_dominator_hotness (CDI_POST_DOMINATORS, + cold_bb_count, + &bbs_in_hot_partition); + /* The format of .gcc_except_table does not allow landing pads to be in a different partition as the throw. Fix this by either moving or duplicating the landing pads. */ ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-01 16:32 [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition Teresa Johnson @ 2013-08-02 11:22 ` Bernhard Reutner-Fischer 2013-08-02 14:51 ` Teresa Johnson 0 siblings, 1 reply; 62+ messages in thread From: Bernhard Reutner-Fischer @ 2013-08-02 11:22 UTC (permalink / raw) To: Teresa Johnson; +Cc: gcc-patches, Steven Bosscher, Jan Hubicka, Jeff Law On 1 August 2013 18:32, Teresa Johnson <tejohnson@google.com> wrote: > Patch 3 of 3 split out from the patch I sent in May that fixes problems with > -freorder-blocks-and-partition, with changes/fixes discussed in that thread. > > See http://gcc.gnu.org/ml/gcc-patches/2013-05/threads.html#00388 for context. > > This patch sanitizes the partitioning to address issues such as edge > weight insanities that sometimes occur due to upstream optimizations, > and ensures that hot blocks are not dominated by cold blocks. This > needs to be resanitized after certain cfg optimizations that may > cause hot blocks previously reached via both hot and cold paths to > only be reached by cold paths. > > The verification code in sanitize_dominator_hotness was contributed by > Steven Bosscher. > > Bootstrapped and tested on x86-64-unknown-linux-gnu. Also ensured that > a profiledbootstrap passed with -freorder-blocks-and-partition enabled > (and with the dwarf version changed to 2 to work around PR57451). > > Ok for trunk? > > (I also included the patch as an attachment since my mailer invariably > messes up the formatting in the pasted version.) > > Thanks, > Teresa > > 2013-08-01 Teresa Johnson <tejohnson@google.com> > Steven Bosscher <steven@gcc.gnu.org> > > * cfgrtl.c (fixup_bb_partition): New routine. > (commit_edge_insertions): Invoke fixup_partitions. > (find_partition_fixes): New routine. > (fixup_partitions): Ditto. > (verify_hot_cold_block_grouping): Update comments. > (rtl_verify_edges): Invoke find_partition_fixes. > (rtl_verify_bb_pointers): Update comments. > (rtl_verify_bb_layout): Ditto. > * basic-block.h (fixup_partitions): Declare. > * cfgcleanup.c (try_optimize_cfg): Invoke fixup_partitions. > * bb-reorder.c (sanitize_dominator_hotness): New function. > (find_rarely_executed_basic_blocks_and_crossing_edges): Invoke > sanitize_dominator_hotness. > > Index: cfgrtl.c > =================================================================== > --- cfgrtl.c (revision 201281) > +++ cfgrtl.c (working copy) > @@ -1341,6 +1341,34 @@ fixup_partition_crossing (edge e) > } > } > > +/* Called when block BB has been reassigned to a different partition, > + to ensure that the region crossing attributes are updated. */ > + > +static void > +fixup_bb_partition (basic_block bb) > +{ > + edge e; > + edge_iterator ei; > + > + /* Now need to make bb's pred edges non-region crossing. */ > + FOR_EACH_EDGE (e, ei, bb->preds) > + { > + fixup_partition_crossing (e); > + } > + > + /* Possibly need to make bb's successor edges region crossing, > + or remove stale region crossing. */ > + FOR_EACH_EDGE (e, ei, bb->succs) > + { > + if ((e->flags & EDGE_FALLTHRU) > + && BB_PARTITION (bb) != BB_PARTITION (e->dest) > + && e->dest != EXIT_BLOCK_PTR) > + force_nonfallthru (e); > + else > + fixup_partition_crossing (e); > + } > +} > + > /* Attempt to change code to redirect edge E to TARGET. Don't do that on > expense of adding new instructions or reordering basic blocks. > > @@ -1979,6 +2007,14 @@ commit_edge_insertions (void) > { > basic_block bb; > > + /* Optimization passes that invoke this routine can cause hot blocks > + previously reached by both hot and cold blocks to become dominated only > + by cold blocks. This will cause the verification below to fail, > + and lead to now cold code in the hot section. In some cases this > + may only be visible after newly unreachable blocks are deleted, > + which will be done by fixup_partitions. */ > + fixup_partitions (); > + > #ifdef ENABLE_CHECKING > verify_flow_info (); > #endif > @@ -2173,6 +2209,101 @@ get_last_bb_insn (basic_block bb) > return end; > } > > +/* Sanity check partition hotness to ensure that basic blocks in > + the cold partition don't dominate basic blocks in the hot partition. > + If FLAG_ONLY is true, report violations as errors. Otherwise > + re-mark the dominated blocks as cold, since this is run after > + cfg optimizations that may make hot blocks previously reached > + by both hot and cold blocks now only reachable along cold paths. */ > + > +vec<basic_block> > +find_partition_fixes (bool flag_only) > +{ > + basic_block bb; > + vec<basic_block> bbs_in_cold_partition = vNULL; > + vec<basic_block> bbs_to_fix = vNULL; > + > + if (!crtl->has_bb_partition) > + return vNULL; I'd push this early return into the callers instead, at most turn it into a gcc_checking_assert to be safe. Both callers, fixup_partitions and rtl_verify_edges, look at ctrl->has_bb_partition already before calling this, so the above should be dead already. Did my mailer somehow swallow the static from find_partition_fixes? > + > + FOR_EACH_BB (bb) > + if ((BB_PARTITION (bb) == BB_COLD_PARTITION)) > + bbs_in_cold_partition.safe_push (bb); > + > + if (bbs_in_cold_partition.is_empty ()) > + return vNULL; > + > + bool dom_calculated_here = !dom_info_available_p (CDI_DOMINATORS); > + > + if (dom_calculated_here) > + calculate_dominance_info (CDI_DOMINATORS); > + > + while (! bbs_in_cold_partition.is_empty ()) > + { > + bb = bbs_in_cold_partition.pop (); > + /* Any blocks dominated by a block in the cold section > + must also be cold. */ > + basic_block son; > + for (son = first_dom_son (CDI_DOMINATORS, bb); > + son; > + son = next_dom_son (CDI_DOMINATORS, son)) > + { > + /* If son is not yet cold, then mark it cold here and > + enqueue it for further processing. */ > + if ((BB_PARTITION (son) != BB_COLD_PARTITION)) > + { > + if (flag_only) > + error ("non-cold basic block %d dominated " > + "by a block in the cold partition", son->index); > + else > + BB_SET_PARTITION (son, BB_COLD_PARTITION); > + bbs_to_fix.safe_push (son); > + bbs_in_cold_partition.safe_push (son); > + } > + } > + } > + > + if (dom_calculated_here) > + free_dominance_info (CDI_DOMINATORS); > + > + return bbs_to_fix; > +} > + > +/* Perform cleanup on the hot/cold bb partitioning after optimization > + passes that modify the cfg. */ > + > +void > +fixup_partitions (void) > +{ > + basic_block bb; > + > + if (!crtl->has_bb_partition) > + return; > + > + /* Delete any blocks that became unreachable and weren't > + already cleaned up, for example during edge forwarding > + and convert_jumps_to_returns. This will expose more > + opportunities for fixing the partition boundaries here. > + Also, the calculation of the dominance graph during verification > + will assert if there are unreachable nodes. */ > + delete_unreachable_blocks (); > + > + /* If there are partitions, do a sanity check on them: A basic block in > + a cold partition cannot dominate a basic block in a hot partition. > + Fixup any that now violate this requirement, as a result of edge > + forwarding and unreachable block deletion. */ > + vec<basic_block> bbs_to_fix = find_partition_fixes (false); > + > + /* Do the partition fixup after all necessary blocks have been converted to > + cold, so that we only update the region crossings the minimum number of > + places, which can require forcing edges to be non fallthru. */ > + while (! bbs_to_fix.is_empty ()) > + { > + bb = bbs_to_fix.pop (); > + fixup_bb_partition (bb); > + } > +} > + > /* Verify, in the basic block chain, that there is at most one switch > between hot/cold partitions. This condition will not be true until > after reorder_basic_blocks is called. */ > @@ -2219,7 +2350,8 @@ verify_hot_cold_block_grouping (void) > /* Perform several checks on the edges out of each block, such as > the consistency of the branch probabilities, the correctness > of hot/cold partition crossing edges, and the number of expected > - successor edges. */ > + successor edges. Also verify that the dominance relationship > + between hot/cold blocks is sane. */ > > static int > rtl_verify_edges (void) > @@ -2382,6 +2514,14 @@ rtl_verify_edges (void) > } > } > > + /* If there are partitions, do a sanity check on them: A basic block in > + a cold partition cannot dominate a basic block in a hot partition. */ > + if (crtl->has_bb_partition && !err) > + { > + vec<basic_block> bbs_to_fix = find_partition_fixes (true); > + err = !bbs_to_fix.is_empty (); > + } > + > /* Clean up. */ > return err; > } > @@ -2515,7 +2655,7 @@ rtl_verify_bb_pointers (void) > and NOTE_INSN_BASIC_BLOCK > - verify that no fall_thru edge crosses hot/cold partition boundaries > - verify that there are no pending RTL branch predictions > - - verify that there is a single hot/cold partition boundary after bbro > + - verify that hot blocks are not dominated by cold blocks > > In future it can be extended check a lot of other stuff as well > (reachability of basic blocks, life information, etc. etc.). */ > @@ -2761,7 +2901,8 @@ rtl_verify_bb_layout (void) > - check that all insns are in the basic blocks > (except the switch handling code, barriers and notes) > - check that all returns are followed by barriers > - - check that all fallthru edge points to the adjacent blocks. */ > + - check that all fallthru edge points to the adjacent blocks > + - verify that there is a single hot/cold partition boundary after bbro */ > > static int > rtl_verify_flow_info (void) > Index: basic-block.h > =================================================================== > --- basic-block.h (revision 201281) > +++ basic-block.h (working copy) > @@ -797,6 +797,7 @@ extern bool contains_no_active_insn_p (const_basic > extern bool forwarder_block_p (const_basic_block); > extern bool can_fallthru (basic_block, basic_block); > extern void emit_barrier_after_bb (basic_block bb); > +extern void fixup_partitions (void); > > /* In cfgbuild.c. */ > extern void find_many_sub_basic_blocks (sbitmap); > Index: cfgcleanup.c > =================================================================== > --- cfgcleanup.c (revision 201281) > +++ cfgcleanup.c (working copy) > @@ -2807,10 +2807,21 @@ try_optimize_cfg (int mode) > df_analyze (); > } > > + if (changed) > + { > + /* Edge forwarding in particular can cause hot blocks previously > + reached by both hot and cold blocks to become dominated only > + by cold blocks. This will cause the verification > below to fail, > + and lead to now cold code in the hot section. This is not easy > + to detect and fix during edge forwarding, and in some cases > + is only visible after newly unreachable blocks are deleted, > + which will be done in fixup_partitions. */ > + fixup_partitions (); > + > #ifdef ENABLE_CHECKING > - if (changed) > - verify_flow_info (); > + verify_flow_info (); > #endif > + } > > changed_overall |= changed; > first_pass = false; > Index: bb-reorder.c > =================================================================== > --- bb-reorder.c (revision 201281) > +++ bb-reorder.c (working copy) > @@ -1444,6 +1444,55 @@ fix_up_crossing_landing_pad (eh_landing_pad old_lp > ei_next (&ei); > } > > + > +/* Ensure that no cold bbs dominate hot bbs along the dominance or > + post-dominance DIR, for example as a result of edge weight insanities. > + Returns the updated value of COLD_BB_COUNT and adds newly-hot bbs > + to BBS_IN_HOT_PARTITION. */ > + > +static unsigned int > +sanitize_dominator_hotness (enum cdi_direction dir, unsigned int cold_bb_count, > + vec<basic_block> *bbs_in_hot_partition) > +{ > + if (!cold_bb_count) > + return 0; Same pattern as above. Callers do not invoke us if !cold_bb_count so the above check is dead code. Again, remove or checking assert? > + > + bool dom_calculated_here = !dom_info_available_p (dir); > + > + if (dom_calculated_here) > + calculate_dominance_info (dir); > + > + /* Keep examining hot bbs until we have either checked them all, or > + re-marked all cold bbs as hot. */ > + vec<basic_block> hot_bbs_to_check = bbs_in_hot_partition->copy (); > + while (! hot_bbs_to_check.is_empty () > + && cold_bb_count) The comment says "or", which sounds plausible, but the code says "and"? > + { > + basic_block bb = hot_bbs_to_check.pop (); > + basic_block dom_bb = get_immediate_dominator (dir, bb); > + > + /* If bb's immediate dominator is also hot then it is ok. */ > + if (BB_PARTITION (dom_bb) != BB_COLD_PARTITION) Why not follow the comment here and == BB_HOT_PARTITION instead, for clarity? > + continue; > + > + /* We have a hot bb with an immediate dominator that is cold. > + The dominator needs to be re-marked hot. */ > + BB_SET_PARTITION (dom_bb, BB_HOT_PARTITION); > + cold_bb_count--; > + > + /* Now we need to examine newly-hot dom_bb to see if it is also > + dominated by a cold bb. */ > + bbs_in_hot_partition->safe_push (dom_bb); > + hot_bbs_to_check.safe_push (dom_bb); > + } > + > + if (dom_calculated_here) > + free_dominance_info (dir); > + > + return cold_bb_count; > +} > + > + > /* Find the basic blocks that are rarely executed and need to be moved to > a separate section of the .o file (to cut down on paging and improve > cache locality). Return a vector of all edges that cross. */ > @@ -1455,16 +1504,42 @@ find_rarely_executed_basic_blocks_and_crossing_edg > basic_block bb; > edge e; > edge_iterator ei; > + unsigned int cold_bb_count = 0; > + vec<basic_block> bbs_in_hot_partition = vNULL; > > /* Mark which partition (hot/cold) each basic block belongs in. */ > FOR_EACH_BB (bb) > { > if (probably_never_executed_bb_p (cfun, bb)) > - BB_SET_PARTITION (bb, BB_COLD_PARTITION); > + { > + BB_SET_PARTITION (bb, BB_COLD_PARTITION); > + cold_bb_count++; > + } > else > - BB_SET_PARTITION (bb, BB_HOT_PARTITION); > + { > + BB_SET_PARTITION (bb, BB_HOT_PARTITION); > + bbs_in_hot_partition.safe_push (bb); > + } > } > > + /* Ensure that no cold bbs dominate hot bbs. This could happen as a result of > + several different possibilities. One is that there are edge > weight insanities > + due to optimization phases that do not properly update basic block profile > + counts. The second is that the entry of the function may not be > hot, because > + it is entered fewer times than the number of profile training > runs, but there > + is a loop inside the function that causes blocks within the function to be > + above the threshold for hotness. Then do the same along the post-dominator > + tree (which could have additional changes required after fixing up > + dominators). */ > + if (cold_bb_count) > + cold_bb_count = sanitize_dominator_hotness (CDI_DOMINATORS, > + cold_bb_count, > + &bbs_in_hot_partition); > + if (cold_bb_count) > + cold_bb_count = sanitize_dominator_hotness (CDI_POST_DOMINATORS, > + cold_bb_count, > + &bbs_in_hot_partition); I take it this last store to cold_bb_count is eliminated anyway. Thanks, > + > /* The format of .gcc_except_table does not allow landing pads to > be in a different partition as the throw. Fix this by either > moving or duplicating the landing pads. */ > > > -- > Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-02 11:22 ` Bernhard Reutner-Fischer @ 2013-08-02 14:51 ` Teresa Johnson 2013-08-02 15:05 ` Jan Hubicka 0 siblings, 1 reply; 62+ messages in thread From: Teresa Johnson @ 2013-08-02 14:51 UTC (permalink / raw) To: Bernhard Reutner-Fischer Cc: gcc-patches, Steven Bosscher, Jan Hubicka, Jeff Law On Fri, Aug 2, 2013 at 4:22 AM, Bernhard Reutner-Fischer <rep.dot.nop@gmail.com> wrote: > On 1 August 2013 18:32, Teresa Johnson <tejohnson@google.com> wrote: >> Patch 3 of 3 split out from the patch I sent in May that fixes problems with >> -freorder-blocks-and-partition, with changes/fixes discussed in that thread. >> >> See http://gcc.gnu.org/ml/gcc-patches/2013-05/threads.html#00388 for context. >> >> This patch sanitizes the partitioning to address issues such as edge >> weight insanities that sometimes occur due to upstream optimizations, >> and ensures that hot blocks are not dominated by cold blocks. This >> needs to be resanitized after certain cfg optimizations that may >> cause hot blocks previously reached via both hot and cold paths to >> only be reached by cold paths. >> >> The verification code in sanitize_dominator_hotness was contributed by >> Steven Bosscher. >> >> Bootstrapped and tested on x86-64-unknown-linux-gnu. Also ensured that >> a profiledbootstrap passed with -freorder-blocks-and-partition enabled >> (and with the dwarf version changed to 2 to work around PR57451). >> >> Ok for trunk? >> >> (I also included the patch as an attachment since my mailer invariably >> messes up the formatting in the pasted version.) >> >> Thanks, >> Teresa >> >> 2013-08-01 Teresa Johnson <tejohnson@google.com> >> Steven Bosscher <steven@gcc.gnu.org> >> >> * cfgrtl.c (fixup_bb_partition): New routine. >> (commit_edge_insertions): Invoke fixup_partitions. >> (find_partition_fixes): New routine. >> (fixup_partitions): Ditto. >> (verify_hot_cold_block_grouping): Update comments. >> (rtl_verify_edges): Invoke find_partition_fixes. >> (rtl_verify_bb_pointers): Update comments. >> (rtl_verify_bb_layout): Ditto. >> * basic-block.h (fixup_partitions): Declare. >> * cfgcleanup.c (try_optimize_cfg): Invoke fixup_partitions. >> * bb-reorder.c (sanitize_dominator_hotness): New function. >> (find_rarely_executed_basic_blocks_and_crossing_edges): Invoke >> sanitize_dominator_hotness. >> >> Index: cfgrtl.c >> =================================================================== >> --- cfgrtl.c (revision 201281) >> +++ cfgrtl.c (working copy) >> @@ -1341,6 +1341,34 @@ fixup_partition_crossing (edge e) >> } >> } >> >> +/* Called when block BB has been reassigned to a different partition, >> + to ensure that the region crossing attributes are updated. */ >> + >> +static void >> +fixup_bb_partition (basic_block bb) >> +{ >> + edge e; >> + edge_iterator ei; >> + >> + /* Now need to make bb's pred edges non-region crossing. */ >> + FOR_EACH_EDGE (e, ei, bb->preds) >> + { >> + fixup_partition_crossing (e); >> + } >> + >> + /* Possibly need to make bb's successor edges region crossing, >> + or remove stale region crossing. */ >> + FOR_EACH_EDGE (e, ei, bb->succs) >> + { >> + if ((e->flags & EDGE_FALLTHRU) >> + && BB_PARTITION (bb) != BB_PARTITION (e->dest) >> + && e->dest != EXIT_BLOCK_PTR) >> + force_nonfallthru (e); >> + else >> + fixup_partition_crossing (e); >> + } >> +} >> + >> /* Attempt to change code to redirect edge E to TARGET. Don't do that on >> expense of adding new instructions or reordering basic blocks. >> >> @@ -1979,6 +2007,14 @@ commit_edge_insertions (void) >> { >> basic_block bb; >> >> + /* Optimization passes that invoke this routine can cause hot blocks >> + previously reached by both hot and cold blocks to become dominated only >> + by cold blocks. This will cause the verification below to fail, >> + and lead to now cold code in the hot section. In some cases this >> + may only be visible after newly unreachable blocks are deleted, >> + which will be done by fixup_partitions. */ >> + fixup_partitions (); >> + >> #ifdef ENABLE_CHECKING >> verify_flow_info (); >> #endif >> @@ -2173,6 +2209,101 @@ get_last_bb_insn (basic_block bb) >> return end; >> } >> >> +/* Sanity check partition hotness to ensure that basic blocks in >> + the cold partition don't dominate basic blocks in the hot partition. >> + If FLAG_ONLY is true, report violations as errors. Otherwise >> + re-mark the dominated blocks as cold, since this is run after >> + cfg optimizations that may make hot blocks previously reached >> + by both hot and cold blocks now only reachable along cold paths. */ >> + >> +vec<basic_block> >> +find_partition_fixes (bool flag_only) >> +{ >> + basic_block bb; >> + vec<basic_block> bbs_in_cold_partition = vNULL; >> + vec<basic_block> bbs_to_fix = vNULL; >> + >> + if (!crtl->has_bb_partition) >> + return vNULL; > > I'd push this early return into the callers instead, at most turn it into a > gcc_checking_assert to be safe. > > Both callers, fixup_partitions and rtl_verify_edges, look at > ctrl->has_bb_partition already before calling this, so the above should > be dead already. Right, I was being paranoid - changed to a gcc_checking_assert. > > Did my mailer somehow swallow the static from find_partition_fixes? Oops, I missed that - added the static. > >> + >> + FOR_EACH_BB (bb) >> + if ((BB_PARTITION (bb) == BB_COLD_PARTITION)) >> + bbs_in_cold_partition.safe_push (bb); >> + >> + if (bbs_in_cold_partition.is_empty ()) >> + return vNULL; >> + >> + bool dom_calculated_here = !dom_info_available_p (CDI_DOMINATORS); >> + >> + if (dom_calculated_here) >> + calculate_dominance_info (CDI_DOMINATORS); >> + >> + while (! bbs_in_cold_partition.is_empty ()) >> + { >> + bb = bbs_in_cold_partition.pop (); >> + /* Any blocks dominated by a block in the cold section >> + must also be cold. */ >> + basic_block son; >> + for (son = first_dom_son (CDI_DOMINATORS, bb); >> + son; >> + son = next_dom_son (CDI_DOMINATORS, son)) >> + { >> + /* If son is not yet cold, then mark it cold here and >> + enqueue it for further processing. */ >> + if ((BB_PARTITION (son) != BB_COLD_PARTITION)) >> + { >> + if (flag_only) >> + error ("non-cold basic block %d dominated " >> + "by a block in the cold partition", son->index); >> + else >> + BB_SET_PARTITION (son, BB_COLD_PARTITION); >> + bbs_to_fix.safe_push (son); >> + bbs_in_cold_partition.safe_push (son); >> + } >> + } >> + } >> + >> + if (dom_calculated_here) >> + free_dominance_info (CDI_DOMINATORS); >> + >> + return bbs_to_fix; >> +} >> + >> +/* Perform cleanup on the hot/cold bb partitioning after optimization >> + passes that modify the cfg. */ >> + >> +void >> +fixup_partitions (void) >> +{ >> + basic_block bb; >> + >> + if (!crtl->has_bb_partition) >> + return; >> + >> + /* Delete any blocks that became unreachable and weren't >> + already cleaned up, for example during edge forwarding >> + and convert_jumps_to_returns. This will expose more >> + opportunities for fixing the partition boundaries here. >> + Also, the calculation of the dominance graph during verification >> + will assert if there are unreachable nodes. */ >> + delete_unreachable_blocks (); >> + >> + /* If there are partitions, do a sanity check on them: A basic block in >> + a cold partition cannot dominate a basic block in a hot partition. >> + Fixup any that now violate this requirement, as a result of edge >> + forwarding and unreachable block deletion. */ >> + vec<basic_block> bbs_to_fix = find_partition_fixes (false); >> + >> + /* Do the partition fixup after all necessary blocks have been converted to >> + cold, so that we only update the region crossings the minimum number of >> + places, which can require forcing edges to be non fallthru. */ >> + while (! bbs_to_fix.is_empty ()) >> + { >> + bb = bbs_to_fix.pop (); >> + fixup_bb_partition (bb); >> + } >> +} >> + >> /* Verify, in the basic block chain, that there is at most one switch >> between hot/cold partitions. This condition will not be true until >> after reorder_basic_blocks is called. */ >> @@ -2219,7 +2350,8 @@ verify_hot_cold_block_grouping (void) >> /* Perform several checks on the edges out of each block, such as >> the consistency of the branch probabilities, the correctness >> of hot/cold partition crossing edges, and the number of expected >> - successor edges. */ >> + successor edges. Also verify that the dominance relationship >> + between hot/cold blocks is sane. */ >> >> static int >> rtl_verify_edges (void) >> @@ -2382,6 +2514,14 @@ rtl_verify_edges (void) >> } >> } >> >> + /* If there are partitions, do a sanity check on them: A basic block in >> + a cold partition cannot dominate a basic block in a hot partition. */ >> + if (crtl->has_bb_partition && !err) >> + { >> + vec<basic_block> bbs_to_fix = find_partition_fixes (true); >> + err = !bbs_to_fix.is_empty (); >> + } >> + >> /* Clean up. */ >> return err; >> } >> @@ -2515,7 +2655,7 @@ rtl_verify_bb_pointers (void) >> and NOTE_INSN_BASIC_BLOCK >> - verify that no fall_thru edge crosses hot/cold partition boundaries >> - verify that there are no pending RTL branch predictions >> - - verify that there is a single hot/cold partition boundary after bbro >> + - verify that hot blocks are not dominated by cold blocks >> >> In future it can be extended check a lot of other stuff as well >> (reachability of basic blocks, life information, etc. etc.). */ >> @@ -2761,7 +2901,8 @@ rtl_verify_bb_layout (void) >> - check that all insns are in the basic blocks >> (except the switch handling code, barriers and notes) >> - check that all returns are followed by barriers >> - - check that all fallthru edge points to the adjacent blocks. */ >> + - check that all fallthru edge points to the adjacent blocks >> + - verify that there is a single hot/cold partition boundary after bbro */ >> >> static int >> rtl_verify_flow_info (void) >> Index: basic-block.h >> =================================================================== >> --- basic-block.h (revision 201281) >> +++ basic-block.h (working copy) >> @@ -797,6 +797,7 @@ extern bool contains_no_active_insn_p (const_basic >> extern bool forwarder_block_p (const_basic_block); >> extern bool can_fallthru (basic_block, basic_block); >> extern void emit_barrier_after_bb (basic_block bb); >> +extern void fixup_partitions (void); >> >> /* In cfgbuild.c. */ >> extern void find_many_sub_basic_blocks (sbitmap); >> Index: cfgcleanup.c >> =================================================================== >> --- cfgcleanup.c (revision 201281) >> +++ cfgcleanup.c (working copy) >> @@ -2807,10 +2807,21 @@ try_optimize_cfg (int mode) >> df_analyze (); >> } >> >> + if (changed) >> + { >> + /* Edge forwarding in particular can cause hot blocks previously >> + reached by both hot and cold blocks to become dominated only >> + by cold blocks. This will cause the verification >> below to fail, >> + and lead to now cold code in the hot section. This is not easy >> + to detect and fix during edge forwarding, and in some cases >> + is only visible after newly unreachable blocks are deleted, >> + which will be done in fixup_partitions. */ >> + fixup_partitions (); >> + >> #ifdef ENABLE_CHECKING >> - if (changed) >> - verify_flow_info (); >> + verify_flow_info (); >> #endif >> + } >> >> changed_overall |= changed; >> first_pass = false; >> Index: bb-reorder.c >> =================================================================== >> --- bb-reorder.c (revision 201281) >> +++ bb-reorder.c (working copy) >> @@ -1444,6 +1444,55 @@ fix_up_crossing_landing_pad (eh_landing_pad old_lp >> ei_next (&ei); >> } >> >> + >> +/* Ensure that no cold bbs dominate hot bbs along the dominance or >> + post-dominance DIR, for example as a result of edge weight insanities. >> + Returns the updated value of COLD_BB_COUNT and adds newly-hot bbs >> + to BBS_IN_HOT_PARTITION. */ >> + >> +static unsigned int >> +sanitize_dominator_hotness (enum cdi_direction dir, unsigned int cold_bb_count, >> + vec<basic_block> *bbs_in_hot_partition) >> +{ >> + if (!cold_bb_count) >> + return 0; > > Same pattern as above. Callers do not invoke us if !cold_bb_count so the above > check is dead code. Again, remove or checking assert? Changed to checking assert. >> + >> + bool dom_calculated_here = !dom_info_available_p (dir); >> + >> + if (dom_calculated_here) >> + calculate_dominance_info (dir); >> + >> + /* Keep examining hot bbs until we have either checked them all, or >> + re-marked all cold bbs as hot. */ >> + vec<basic_block> hot_bbs_to_check = bbs_in_hot_partition->copy (); >> + while (! hot_bbs_to_check.is_empty () >> + && cold_bb_count) > > The comment says "or", which sounds plausible, but the code says "and"? The comment is describing the conditions for exiting the loop, which is why it was an "or". I changed the comment to describe the conditions for staying in the loop to make it more consistent/clearer: /* Keep examining hot bbs while we still have some left to check and there are remaining cold bbs. */ >> + { >> + basic_block bb = hot_bbs_to_check.pop (); >> + basic_block dom_bb = get_immediate_dominator (dir, bb); >> + >> + /* If bb's immediate dominator is also hot then it is ok. */ >> + if (BB_PARTITION (dom_bb) != BB_COLD_PARTITION) > > Why not follow the comment here and == BB_HOT_PARTITION instead, for clarity? The entry/exit blocks are unpartitioned, and we want to skip those. I have clarified that in the comment: /* If bb's immediate dominator is also hot (or unpartitioned, e.g. the entry block) then it is ok. If it is cold, it needs to be adjusted. */ > >> + continue; >> + >> + /* We have a hot bb with an immediate dominator that is cold. >> + The dominator needs to be re-marked hot. */ >> + BB_SET_PARTITION (dom_bb, BB_HOT_PARTITION); >> + cold_bb_count--; >> + >> + /* Now we need to examine newly-hot dom_bb to see if it is also >> + dominated by a cold bb. */ >> + bbs_in_hot_partition->safe_push (dom_bb); >> + hot_bbs_to_check.safe_push (dom_bb); >> + } >> + >> + if (dom_calculated_here) >> + free_dominance_info (dir); >> + >> + return cold_bb_count; >> +} >> + >> + >> /* Find the basic blocks that are rarely executed and need to be moved to >> a separate section of the .o file (to cut down on paging and improve >> cache locality). Return a vector of all edges that cross. */ >> @@ -1455,16 +1504,42 @@ find_rarely_executed_basic_blocks_and_crossing_edg >> basic_block bb; >> edge e; >> edge_iterator ei; >> + unsigned int cold_bb_count = 0; >> + vec<basic_block> bbs_in_hot_partition = vNULL; >> >> /* Mark which partition (hot/cold) each basic block belongs in. */ >> FOR_EACH_BB (bb) >> { >> if (probably_never_executed_bb_p (cfun, bb)) >> - BB_SET_PARTITION (bb, BB_COLD_PARTITION); >> + { >> + BB_SET_PARTITION (bb, BB_COLD_PARTITION); >> + cold_bb_count++; >> + } >> else >> - BB_SET_PARTITION (bb, BB_HOT_PARTITION); >> + { >> + BB_SET_PARTITION (bb, BB_HOT_PARTITION); >> + bbs_in_hot_partition.safe_push (bb); >> + } >> } >> >> + /* Ensure that no cold bbs dominate hot bbs. This could happen as a result of >> + several different possibilities. One is that there are edge >> weight insanities >> + due to optimization phases that do not properly update basic block profile >> + counts. The second is that the entry of the function may not be >> hot, because >> + it is entered fewer times than the number of profile training >> runs, but there >> + is a loop inside the function that causes blocks within the function to be >> + above the threshold for hotness. Then do the same along the post-dominator >> + tree (which could have additional changes required after fixing up >> + dominators). */ >> + if (cold_bb_count) >> + cold_bb_count = sanitize_dominator_hotness (CDI_DOMINATORS, >> + cold_bb_count, >> + &bbs_in_hot_partition); >> + if (cold_bb_count) >> + cold_bb_count = sanitize_dominator_hotness (CDI_POST_DOMINATORS, >> + cold_bb_count, >> + &bbs_in_hot_partition); > > I take it this last store to cold_bb_count is eliminated anyway. Yep, and I went ahead and removed the dead store. Thanks for the review. New patch below. Teresa 2013-08-01 Teresa Johnson <tejohnson@google.com> Steven Bosscher <steven@gcc.gnu.org> * cfgrtl.c (fixup_bb_partition): New routine. (commit_edge_insertions): Invoke fixup_partitions. (find_partition_fixes): New routine. (fixup_partitions): Ditto. (verify_hot_cold_block_grouping): Update comments. (rtl_verify_edges): Invoke find_partition_fixes. (rtl_verify_bb_pointers): Update comments. (rtl_verify_bb_layout): Ditto. * basic-block.h (fixup_partitions): Declare. * cfgcleanup.c (try_optimize_cfg): Invoke fixup_partitions. * bb-reorder.c (sanitize_dominator_hotness): New function. (find_rarely_executed_basic_blocks_and_crossing_edges): Invoke sanitize_dominator_hotness. Index: cfgrtl.c =================================================================== --- cfgrtl.c (revision 201281) +++ cfgrtl.c (working copy) @@ -1341,6 +1341,34 @@ fixup_partition_crossing (edge e) } } +/* Called when block BB has been reassigned to a different partition, + to ensure that the region crossing attributes are updated. */ + +static void +fixup_bb_partition (basic_block bb) +{ + edge e; + edge_iterator ei; + + /* Now need to make bb's pred edges non-region crossing. */ + FOR_EACH_EDGE (e, ei, bb->preds) + { + fixup_partition_crossing (e); + } + + /* Possibly need to make bb's successor edges region crossing, + or remove stale region crossing. */ + FOR_EACH_EDGE (e, ei, bb->succs) + { + if ((e->flags & EDGE_FALLTHRU) + && BB_PARTITION (bb) != BB_PARTITION (e->dest) + && e->dest != EXIT_BLOCK_PTR) + force_nonfallthru (e); + else + fixup_partition_crossing (e); + } +} + /* Attempt to change code to redirect edge E to TARGET. Don't do that on expense of adding new instructions or reordering basic blocks. @@ -1979,6 +2007,14 @@ commit_edge_insertions (void) { basic_block bb; + /* Optimization passes that invoke this routine can cause hot blocks + previously reached by both hot and cold blocks to become dominated only + by cold blocks. This will cause the verification below to fail, + and lead to now cold code in the hot section. In some cases this + may only be visible after newly unreachable blocks are deleted, + which will be done by fixup_partitions. */ + fixup_partitions (); + #ifdef ENABLE_CHECKING verify_flow_info (); #endif @@ -2173,6 +2209,101 @@ get_last_bb_insn (basic_block bb) return end; } +/* Sanity check partition hotness to ensure that basic blocks in + the cold partition don't dominate basic blocks in the hot partition. + If FLAG_ONLY is true, report violations as errors. Otherwise + re-mark the dominated blocks as cold, since this is run after + cfg optimizations that may make hot blocks previously reached + by both hot and cold blocks now only reachable along cold paths. */ + +static vec<basic_block> +find_partition_fixes (bool flag_only) +{ + basic_block bb; + vec<basic_block> bbs_in_cold_partition = vNULL; + vec<basic_block> bbs_to_fix = vNULL; + + /* Callers check this. */ + gcc_checking_assert (crtl->has_bb_partition); + + FOR_EACH_BB (bb) + if ((BB_PARTITION (bb) == BB_COLD_PARTITION)) + bbs_in_cold_partition.safe_push (bb); + + if (bbs_in_cold_partition.is_empty ()) + return vNULL; + + bool dom_calculated_here = !dom_info_available_p (CDI_DOMINATORS); + + if (dom_calculated_here) + calculate_dominance_info (CDI_DOMINATORS); + + while (! bbs_in_cold_partition.is_empty ()) + { + bb = bbs_in_cold_partition.pop (); + /* Any blocks dominated by a block in the cold section + must also be cold. */ + basic_block son; + for (son = first_dom_son (CDI_DOMINATORS, bb); + son; + son = next_dom_son (CDI_DOMINATORS, son)) + { + /* If son is not yet cold, then mark it cold here and + enqueue it for further processing. */ + if ((BB_PARTITION (son) != BB_COLD_PARTITION)) + { + if (flag_only) + error ("non-cold basic block %d dominated " + "by a block in the cold partition", son->index); + else + BB_SET_PARTITION (son, BB_COLD_PARTITION); + bbs_to_fix.safe_push (son); + bbs_in_cold_partition.safe_push (son); + } + } + } + + if (dom_calculated_here) + free_dominance_info (CDI_DOMINATORS); + + return bbs_to_fix; +} + +/* Perform cleanup on the hot/cold bb partitioning after optimization + passes that modify the cfg. */ + +void +fixup_partitions (void) +{ + basic_block bb; + + if (!crtl->has_bb_partition) + return; + + /* Delete any blocks that became unreachable and weren't + already cleaned up, for example during edge forwarding + and convert_jumps_to_returns. This will expose more + opportunities for fixing the partition boundaries here. + Also, the calculation of the dominance graph during verification + will assert if there are unreachable nodes. */ + delete_unreachable_blocks (); + + /* If there are partitions, do a sanity check on them: A basic block in + a cold partition cannot dominate a basic block in a hot partition. + Fixup any that now violate this requirement, as a result of edge + forwarding and unreachable block deletion. */ + vec<basic_block> bbs_to_fix = find_partition_fixes (false); + + /* Do the partition fixup after all necessary blocks have been converted to + cold, so that we only update the region crossings the minimum number of + places, which can require forcing edges to be non fallthru. */ + while (! bbs_to_fix.is_empty ()) + { + bb = bbs_to_fix.pop (); + fixup_bb_partition (bb); + } +} + /* Verify, in the basic block chain, that there is at most one switch between hot/cold partitions. This condition will not be true until after reorder_basic_blocks is called. */ @@ -2219,7 +2350,8 @@ verify_hot_cold_block_grouping (void) /* Perform several checks on the edges out of each block, such as the consistency of the branch probabilities, the correctness of hot/cold partition crossing edges, and the number of expected - successor edges. */ + successor edges. Also verify that the dominance relationship + between hot/cold blocks is sane. */ static int rtl_verify_edges (void) @@ -2382,6 +2514,14 @@ rtl_verify_edges (void) } } + /* If there are partitions, do a sanity check on them: A basic block in + a cold partition cannot dominate a basic block in a hot partition. */ + if (crtl->has_bb_partition && !err) + { + vec<basic_block> bbs_to_fix = find_partition_fixes (true); + err = !bbs_to_fix.is_empty (); + } + /* Clean up. */ return err; } @@ -2515,7 +2655,7 @@ rtl_verify_bb_pointers (void) and NOTE_INSN_BASIC_BLOCK - verify that no fall_thru edge crosses hot/cold partition boundaries - verify that there are no pending RTL branch predictions - - verify that there is a single hot/cold partition boundary after bbro + - verify that hot blocks are not dominated by cold blocks In future it can be extended check a lot of other stuff as well (reachability of basic blocks, life information, etc. etc.). */ @@ -2761,7 +2901,8 @@ rtl_verify_bb_layout (void) - check that all insns are in the basic blocks (except the switch handling code, barriers and notes) - check that all returns are followed by barriers - - check that all fallthru edge points to the adjacent blocks. */ + - check that all fallthru edge points to the adjacent blocks + - verify that there is a single hot/cold partition boundary after bbro */ static int rtl_verify_flow_info (void) Index: basic-block.h =================================================================== --- basic-block.h (revision 201281) +++ basic-block.h (working copy) @@ -797,6 +797,7 @@ extern bool contains_no_active_insn_p (const_basic extern bool forwarder_block_p (const_basic_block); extern bool can_fallthru (basic_block, basic_block); extern void emit_barrier_after_bb (basic_block bb); +extern void fixup_partitions (void); /* In cfgbuild.c. */ extern void find_many_sub_basic_blocks (sbitmap); Index: cfgcleanup.c =================================================================== --- cfgcleanup.c (revision 201281) +++ cfgcleanup.c (working copy) @@ -2807,10 +2807,21 @@ try_optimize_cfg (int mode) df_analyze (); } + if (changed) + { + /* Edge forwarding in particular can cause hot blocks previously + reached by both hot and cold blocks to become dominated only + by cold blocks. This will cause the verification below to fail, + and lead to now cold code in the hot section. This is not easy + to detect and fix during edge forwarding, and in some cases + is only visible after newly unreachable blocks are deleted, + which will be done in fixup_partitions. */ + fixup_partitions (); + #ifdef ENABLE_CHECKING - if (changed) - verify_flow_info (); + verify_flow_info (); #endif + } changed_overall |= changed; first_pass = false; Index: bb-reorder.c =================================================================== --- bb-reorder.c (revision 201281) +++ bb-reorder.c (working copy) @@ -1444,6 +1444,57 @@ fix_up_crossing_landing_pad (eh_landing_pad old_lp ei_next (&ei); } + +/* Ensure that no cold bbs dominate hot bbs along the dominance or + post-dominance DIR, for example as a result of edge weight insanities. + Returns the updated value of COLD_BB_COUNT and adds newly-hot bbs + to BBS_IN_HOT_PARTITION. */ + +static unsigned int +sanitize_dominator_hotness (enum cdi_direction dir, unsigned int cold_bb_count, + vec<basic_block> *bbs_in_hot_partition) +{ + /* Callers check this. */ + gcc_checking_assert (cold_bb_count); + + bool dom_calculated_here = !dom_info_available_p (dir); + + if (dom_calculated_here) + calculate_dominance_info (dir); + + /* Keep examining hot bbs while we still have some left to check + and there are remaining cold bbs. */ + vec<basic_block> hot_bbs_to_check = bbs_in_hot_partition->copy (); + while (! hot_bbs_to_check.is_empty () + && cold_bb_count) + { + basic_block bb = hot_bbs_to_check.pop (); + basic_block dom_bb = get_immediate_dominator (dir, bb); + + /* If bb's immediate dominator is also hot (or unpartitioned, + e.g. the entry block) then it is ok. If it is cold, it + needs to be adjusted. */ + if (BB_PARTITION (dom_bb) != BB_COLD_PARTITION) + continue; + + /* We have a hot bb with an immediate dominator that is cold. + The dominator needs to be re-marked hot. */ + BB_SET_PARTITION (dom_bb, BB_HOT_PARTITION); + cold_bb_count--; + + /* Now we need to examine newly-hot dom_bb to see if it is also + dominated by a cold bb. */ + bbs_in_hot_partition->safe_push (dom_bb); + hot_bbs_to_check.safe_push (dom_bb); + } + + if (dom_calculated_here) + free_dominance_info (dir); + + return cold_bb_count; +} + + /* Find the basic blocks that are rarely executed and need to be moved to a separate section of the .o file (to cut down on paging and improve cache locality). Return a vector of all edges that cross. */ @@ -1455,16 +1506,42 @@ find_rarely_executed_basic_blocks_and_crossing_edg basic_block bb; edge e; edge_iterator ei; + unsigned int cold_bb_count = 0; + vec<basic_block> bbs_in_hot_partition = vNULL; /* Mark which partition (hot/cold) each basic block belongs in. */ FOR_EACH_BB (bb) { if (probably_never_executed_bb_p (cfun, bb)) - BB_SET_PARTITION (bb, BB_COLD_PARTITION); + { + BB_SET_PARTITION (bb, BB_COLD_PARTITION); + cold_bb_count++; + } else - BB_SET_PARTITION (bb, BB_HOT_PARTITION); + { + BB_SET_PARTITION (bb, BB_HOT_PARTITION); + bbs_in_hot_partition.safe_push (bb); + } } + /* Ensure that no cold bbs dominate hot bbs. This could happen as a result of + several different possibilities. One is that there are edge weight insanities + due to optimization phases that do not properly update basic block profile + counts. The second is that the entry of the function may not be hot, because + it is entered fewer times than the number of profile training runs, but there + is a loop inside the function that causes blocks within the function to be + above the threshold for hotness. Then do the same along the post-dominator + tree (which could have additional changes required after fixing up + dominators). */ + if (cold_bb_count) + cold_bb_count = sanitize_dominator_hotness (CDI_DOMINATORS, + cold_bb_count, + &bbs_in_hot_partition); + if (cold_bb_count) + sanitize_dominator_hotness (CDI_POST_DOMINATORS, + cold_bb_count, + &bbs_in_hot_partition); + /* The format of .gcc_except_table does not allow landing pads to be in a different partition as the throw. Fix this by either moving or duplicating the landing pads. */ -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-02 14:51 ` Teresa Johnson @ 2013-08-02 15:05 ` Jan Hubicka 2013-08-02 23:05 ` Steven Bosscher 2013-08-03 4:48 ` Teresa Johnson 0 siblings, 2 replies; 62+ messages in thread From: Jan Hubicka @ 2013-08-02 15:05 UTC (permalink / raw) To: Teresa Johnson Cc: Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jan Hubicka, Jeff Law > > 2013-08-01 Teresa Johnson <tejohnson@google.com> > Steven Bosscher <steven@gcc.gnu.org> > > * cfgrtl.c (fixup_bb_partition): New routine. > (commit_edge_insertions): Invoke fixup_partitions. > (find_partition_fixes): New routine. > (fixup_partitions): Ditto. > (verify_hot_cold_block_grouping): Update comments. > (rtl_verify_edges): Invoke find_partition_fixes. > (rtl_verify_bb_pointers): Update comments. > (rtl_verify_bb_layout): Ditto. > * basic-block.h (fixup_partitions): Declare. > * cfgcleanup.c (try_optimize_cfg): Invoke fixup_partitions. > * bb-reorder.c (sanitize_dominator_hotness): New function. > (find_rarely_executed_basic_blocks_and_crossing_edges): Invoke > sanitize_dominator_hotness. > > Index: cfgrtl.c > =================================================================== > --- cfgrtl.c (revision 201281) > +++ cfgrtl.c (working copy) > @@ -1341,6 +1341,34 @@ fixup_partition_crossing (edge e) > } > } > > +/* Called when block BB has been reassigned to a different partition, > + to ensure that the region crossing attributes are updated. */ > + > +static void > +fixup_bb_partition (basic_block bb) > +{ > + edge e; > + edge_iterator ei; > + > + /* Now need to make bb's pred edges non-region crossing. */ > + FOR_EACH_EDGE (e, ei, bb->preds) > + { > + fixup_partition_crossing (e); > + } > + > + /* Possibly need to make bb's successor edges region crossing, > + or remove stale region crossing. */ > + FOR_EACH_EDGE (e, ei, bb->succs) > + { > + if ((e->flags & EDGE_FALLTHRU) > + && BB_PARTITION (bb) != BB_PARTITION (e->dest) > + && e->dest != EXIT_BLOCK_PTR) > + force_nonfallthru (e); > + else > + fixup_partition_crossing (e); > + } > +} Is there particular reason why preds can not be fallhtrus and why force_nonfallthru edge does not need partition crossing fixup? (if so, perhpas it could be mentioned in the description, if not, I think force_nonfallthru path has to check if new BB was introduced and do the right thing on the edge. > +/* Sanity check partition hotness to ensure that basic blocks in > + the cold partition don't dominate basic blocks in the hot partition. > + If FLAG_ONLY is true, report violations as errors. Otherwise > + re-mark the dominated blocks as cold, since this is run after > + cfg optimizations that may make hot blocks previously reached > + by both hot and cold blocks now only reachable along cold paths. */ With profile, I suppose we can have cold blocks dominating hot blocks when the hot blocks is in loop whose trip count is high enough. Indeed for partitioning reasons it does not make sense to push those into different section. I also wonder, if we finally get the pass stable, can we enable it by default and offline probably cold blocks w/o profile? Primarily blocks reachable only by EH + blocks leading to a crash or throw(). For C++ those should be common enough to make a difference... Honza ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-02 15:05 ` Jan Hubicka @ 2013-08-02 23:05 ` Steven Bosscher 2013-08-03 4:53 ` Teresa Johnson 2013-08-03 4:48 ` Teresa Johnson 1 sibling, 1 reply; 62+ messages in thread From: Steven Bosscher @ 2013-08-02 23:05 UTC (permalink / raw) To: Jan Hubicka Cc: Teresa Johnson, Bernhard Reutner-Fischer, gcc-patches, Jeff Law On Fri, Aug 2, 2013 at 5:05 PM, Jan Hubicka <hubicka@ucw.cz> wrote: >> +/* Called when block BB has been reassigned to a different partition, >> + to ensure that the region crossing attributes are updated. */ >> + >> +static void >> +fixup_bb_partition (basic_block bb) >> +{ >> + edge e; >> + edge_iterator ei; >> + >> + /* Now need to make bb's pred edges non-region crossing. */ >> + FOR_EACH_EDGE (e, ei, bb->preds) >> + { >> + fixup_partition_crossing (e); >> + } >> + >> + /* Possibly need to make bb's successor edges region crossing, >> + or remove stale region crossing. */ >> + FOR_EACH_EDGE (e, ei, bb->succs) >> + { >> + if ((e->flags & EDGE_FALLTHRU) >> + && BB_PARTITION (bb) != BB_PARTITION (e->dest) >> + && e->dest != EXIT_BLOCK_PTR) >> + force_nonfallthru (e); >> + else >> + fixup_partition_crossing (e); >> + } >> +} > > Is there particular reason why preds can not be fallhtrus Yes, by definition a crossing edge cannot fall through. There is always a control transfer from one section to another. >> +/* Sanity check partition hotness to ensure that basic blocks in >> + the cold partition don't dominate basic blocks in the hot partition. >> + If FLAG_ONLY is true, report violations as errors. Otherwise >> + re-mark the dominated blocks as cold, since this is run after >> + cfg optimizations that may make hot blocks previously reached >> + by both hot and cold blocks now only reachable along cold paths. */ > > With profile, I suppose we can have cold blocks dominating hot blocks when the > hot blocks is in loop whose trip count is high enough. That is the common case, actually. > Indeed for partitioning > reasons it does not make sense to push those into different section. The partitioning algrorithm makes sure this doesn't happen. The hottest path from the entry block to a hot basic block is always part of the hot partition. > I also wonder, if we finally get the pass stable, can we enable it by default > and offline probably cold blocks w/o profile? That is the general idea behind all this work, obviously ;-) > Primarily blocks reachable only > by EH + blocks leading to a crash or throw(). For C++ those should be common > enough to make a difference... Yup, and IIRC Theresa posted some numbers that showed this. Ciao! Steven ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-02 23:05 ` Steven Bosscher @ 2013-08-03 4:53 ` Teresa Johnson 0 siblings, 0 replies; 62+ messages in thread From: Teresa Johnson @ 2013-08-03 4:53 UTC (permalink / raw) To: Steven Bosscher Cc: Jan Hubicka, Bernhard Reutner-Fischer, gcc-patches, Jeff Law On Fri, Aug 2, 2013 at 4:04 PM, Steven Bosscher <stevenb.gcc@gmail.com> wrote: > On Fri, Aug 2, 2013 at 5:05 PM, Jan Hubicka <hubicka@ucw.cz> wrote: >>> +/* Called when block BB has been reassigned to a different partition, >>> + to ensure that the region crossing attributes are updated. */ >>> + >>> +static void >>> +fixup_bb_partition (basic_block bb) >>> +{ >>> + edge e; >>> + edge_iterator ei; >>> + >>> + /* Now need to make bb's pred edges non-region crossing. */ >>> + FOR_EACH_EDGE (e, ei, bb->preds) >>> + { >>> + fixup_partition_crossing (e); >>> + } >>> + >>> + /* Possibly need to make bb's successor edges region crossing, >>> + or remove stale region crossing. */ >>> + FOR_EACH_EDGE (e, ei, bb->succs) >>> + { >>> + if ((e->flags & EDGE_FALLTHRU) >>> + && BB_PARTITION (bb) != BB_PARTITION (e->dest) >>> + && e->dest != EXIT_BLOCK_PTR) >>> + force_nonfallthru (e); >>> + else >>> + fixup_partition_crossing (e); >>> + } >>> +} >> >> Is there particular reason why preds can not be fallhtrus > > Yes, by definition a crossing edge cannot fall through. There is > always a control transfer from one section to another. > > >>> +/* Sanity check partition hotness to ensure that basic blocks in >>> + the cold partition don't dominate basic blocks in the hot partition. >>> + If FLAG_ONLY is true, report violations as errors. Otherwise >>> + re-mark the dominated blocks as cold, since this is run after >>> + cfg optimizations that may make hot blocks previously reached >>> + by both hot and cold blocks now only reachable along cold paths. */ >> >> With profile, I suppose we can have cold blocks dominating hot blocks when the >> hot blocks is in loop whose trip count is high enough. > > That is the common case, actually. > >> Indeed for partitioning >> reasons it does not make sense to push those into different section. > > The partitioning algrorithm makes sure this doesn't happen. The > hottest path from the entry block to a hot basic block is always part > of the hot partition. Well, at least with this patch that will be true. The trunk version just partitions based on the bb's count without regard to paths. Thanks, Teresa > > >> I also wonder, if we finally get the pass stable, can we enable it by default >> and offline probably cold blocks w/o profile? > > That is the general idea behind all this work, obviously ;-) > >> Primarily blocks reachable only >> by EH + blocks leading to a crash or throw(). For C++ those should be common >> enough to make a difference... > > Yup, and IIRC Theresa posted some numbers that showed this. > > Ciao! > Steven -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-02 15:05 ` Jan Hubicka 2013-08-02 23:05 ` Steven Bosscher @ 2013-08-03 4:48 ` Teresa Johnson 2013-08-05 13:36 ` Teresa Johnson 1 sibling, 1 reply; 62+ messages in thread From: Teresa Johnson @ 2013-08-03 4:48 UTC (permalink / raw) To: Jan Hubicka Cc: Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law On Fri, Aug 2, 2013 at 8:05 AM, Jan Hubicka <hubicka@ucw.cz> wrote: >> >> 2013-08-01 Teresa Johnson <tejohnson@google.com> >> Steven Bosscher <steven@gcc.gnu.org> >> >> * cfgrtl.c (fixup_bb_partition): New routine. >> (commit_edge_insertions): Invoke fixup_partitions. >> (find_partition_fixes): New routine. >> (fixup_partitions): Ditto. >> (verify_hot_cold_block_grouping): Update comments. >> (rtl_verify_edges): Invoke find_partition_fixes. >> (rtl_verify_bb_pointers): Update comments. >> (rtl_verify_bb_layout): Ditto. >> * basic-block.h (fixup_partitions): Declare. >> * cfgcleanup.c (try_optimize_cfg): Invoke fixup_partitions. >> * bb-reorder.c (sanitize_dominator_hotness): New function. >> (find_rarely_executed_basic_blocks_and_crossing_edges): Invoke >> sanitize_dominator_hotness. >> >> Index: cfgrtl.c >> =================================================================== >> --- cfgrtl.c (revision 201281) >> +++ cfgrtl.c (working copy) >> @@ -1341,6 +1341,34 @@ fixup_partition_crossing (edge e) >> } >> } >> >> +/* Called when block BB has been reassigned to a different partition, >> + to ensure that the region crossing attributes are updated. */ >> + >> +static void >> +fixup_bb_partition (basic_block bb) >> +{ >> + edge e; >> + edge_iterator ei; >> + >> + /* Now need to make bb's pred edges non-region crossing. */ >> + FOR_EACH_EDGE (e, ei, bb->preds) >> + { >> + fixup_partition_crossing (e); >> + } >> + >> + /* Possibly need to make bb's successor edges region crossing, >> + or remove stale region crossing. */ >> + FOR_EACH_EDGE (e, ei, bb->succs) >> + { >> + if ((e->flags & EDGE_FALLTHRU) >> + && BB_PARTITION (bb) != BB_PARTITION (e->dest) >> + && e->dest != EXIT_BLOCK_PTR) >> + force_nonfallthru (e); >> + else >> + fixup_partition_crossing (e); >> + } >> +} > > Is there particular reason why preds can not be fallhtrus and why > force_nonfallthru edge does not need partition crossing fixup? > (if so, perhpas it could be mentioned in the description, if not, > I think force_nonfallthru path has to check if new BB was introduced > and do the right thing on the edge. I need to clarify the comments in this routine, because without the context of how this is called it isn't clear. This routine is only called when we detect a hot bb that is now dominated by a cold bb and needs to become cold. Therefore, its preds will no longer be region crossing (any non-dominating blocks that were previously hot would have been marked cold in the caller for the same reason, so we will not end up adjusting the region crossing-ness or fallthrough-ness of those pred edges). Any that were region crossing before but aren't any longer could not have been fall through (as Steven noted, you can't have a fall through across a partition boundary). I will add some better comments here. Regarding the call to force_nonfallthru, that routine calls fixup_partition_crossing as needed, and I will update the comment to reflect that too. > >> +/* Sanity check partition hotness to ensure that basic blocks in >> + the cold partition don't dominate basic blocks in the hot partition. >> + If FLAG_ONLY is true, report violations as errors. Otherwise >> + re-mark the dominated blocks as cold, since this is run after >> + cfg optimizations that may make hot blocks previously reached >> + by both hot and cold blocks now only reachable along cold paths. */ > > With profile, I suppose we can have cold blocks dominating hot blocks when the > hot blocks is in loop whose trip count is high enough. Indeed for partitioning > reasons it does not make sense to push those into different section. > > I also wonder, if we finally get the pass stable, can we enable it by default > and offline probably cold blocks w/o profile? Primarily blocks reachable only > by EH + blocks leading to a crash or throw(). For C++ those should be common > enough to make a difference... Yep, as soon as PR57451 is fixed, which I hope to get to next week, then I am going to send a patch to turn this on by default, at least with profile feedback, which is where I've been doing performance tuning. But you are right that there are some cases where it should be beneficial without profile data as well. Thanks, Teresa > > Honza -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-03 4:48 ` Teresa Johnson @ 2013-08-05 13:36 ` Teresa Johnson 2013-08-05 14:11 ` Jan Hubicka [not found] ` <20130808222332.GA31755@kam.mff.cuni.cz> 0 siblings, 2 replies; 62+ messages in thread From: Teresa Johnson @ 2013-08-05 13:36 UTC (permalink / raw) To: Jan Hubicka Cc: Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law On Fri, Aug 2, 2013 at 9:48 PM, Teresa Johnson <tejohnson@google.com> wrote: > On Fri, Aug 2, 2013 at 8:05 AM, Jan Hubicka <hubicka@ucw.cz> wrote: >>> >>> 2013-08-01 Teresa Johnson <tejohnson@google.com> >>> Steven Bosscher <steven@gcc.gnu.org> >>> >>> * cfgrtl.c (fixup_bb_partition): New routine. >>> (commit_edge_insertions): Invoke fixup_partitions. >>> (find_partition_fixes): New routine. >>> (fixup_partitions): Ditto. >>> (verify_hot_cold_block_grouping): Update comments. >>> (rtl_verify_edges): Invoke find_partition_fixes. >>> (rtl_verify_bb_pointers): Update comments. >>> (rtl_verify_bb_layout): Ditto. >>> * basic-block.h (fixup_partitions): Declare. >>> * cfgcleanup.c (try_optimize_cfg): Invoke fixup_partitions. >>> * bb-reorder.c (sanitize_dominator_hotness): New function. >>> (find_rarely_executed_basic_blocks_and_crossing_edges): Invoke >>> sanitize_dominator_hotness. >>> >>> Index: cfgrtl.c >>> =================================================================== >>> --- cfgrtl.c (revision 201281) >>> +++ cfgrtl.c (working copy) >>> @@ -1341,6 +1341,34 @@ fixup_partition_crossing (edge e) >>> } >>> } >>> >>> +/* Called when block BB has been reassigned to a different partition, >>> + to ensure that the region crossing attributes are updated. */ >>> + >>> +static void >>> +fixup_bb_partition (basic_block bb) >>> +{ >>> + edge e; >>> + edge_iterator ei; >>> + >>> + /* Now need to make bb's pred edges non-region crossing. */ >>> + FOR_EACH_EDGE (e, ei, bb->preds) >>> + { >>> + fixup_partition_crossing (e); >>> + } >>> + >>> + /* Possibly need to make bb's successor edges region crossing, >>> + or remove stale region crossing. */ >>> + FOR_EACH_EDGE (e, ei, bb->succs) >>> + { >>> + if ((e->flags & EDGE_FALLTHRU) >>> + && BB_PARTITION (bb) != BB_PARTITION (e->dest) >>> + && e->dest != EXIT_BLOCK_PTR) >>> + force_nonfallthru (e); >>> + else >>> + fixup_partition_crossing (e); >>> + } >>> +} >> >> Is there particular reason why preds can not be fallhtrus and why >> force_nonfallthru edge does not need partition crossing fixup? >> (if so, perhpas it could be mentioned in the description, if not, >> I think force_nonfallthru path has to check if new BB was introduced >> and do the right thing on the edge. > > I need to clarify the comments in this routine, because without the > context of how this is called it isn't clear. This routine is only > called when we detect a hot bb that is now dominated by a cold bb and > needs to become cold. Therefore, its preds will no longer be region > crossing (any non-dominating blocks that were previously hot would > have been marked cold in the caller for the same reason, so we will > not end up adjusting the region crossing-ness or fallthrough-ness of > those pred edges). Any that were region crossing before but aren't any > longer could not have been fall through (as Steven noted, you can't > have a fall through across a partition boundary). I will add some > better comments here. > > Regarding the call to force_nonfallthru, that routine calls > fixup_partition_crossing as needed, and I will update the comment to > reflect that too. Patch with updated comments below. Ok for trunk? Thanks, Teresa 2013-08-05 Teresa Johnson <tejohnson@google.com> Steven Bosscher <steven@gcc.gnu.org> * cfgrtl.c (fixup_bb_partition): New routine. (commit_edge_insertions): Invoke fixup_partitions. (find_partition_fixes): New routine. (fixup_partitions): Ditto. (verify_hot_cold_block_grouping): Update comments. (rtl_verify_edges): Invoke find_partition_fixes. (rtl_verify_bb_pointers): Update comments. (rtl_verify_bb_layout): Ditto. * basic-block.h (fixup_partitions): Declare. * cfgcleanup.c (try_optimize_cfg): Invoke fixup_partitions. * bb-reorder.c (sanitize_dominator_hotness): New function. (find_rarely_executed_basic_blocks_and_crossing_edges): Invoke sanitize_dominator_hotness. Index: cfgrtl.c =================================================================== --- cfgrtl.c (revision 201461) +++ cfgrtl.c (working copy) @@ -1341,6 +1341,43 @@ fixup_partition_crossing (edge e) } } +/* Called when block BB has been reassigned to the cold partition, + because it is now dominated by another cold block, + to ensure that the region crossing attributes are updated. */ + +static void +fixup_new_cold_bb (basic_block bb) +{ + edge e; + edge_iterator ei; + + /* This is called when a hot bb is found to now be dominated + by a cold bb and therefore needs to become cold. Therefore, + its preds will no longer be region crossing. Any non-dominating + preds that were previously hot would also have become cold + in the caller for the same region. Any preds that were previously + region-crossing will be adjusted in fixup_partition_crossing. */ + FOR_EACH_EDGE (e, ei, bb->preds) + { + fixup_partition_crossing (e); + } + + /* Possibly need to make bb's successor edges region crossing, + or remove stale region crossing. */ + FOR_EACH_EDGE (e, ei, bb->succs) + { + /* We can't have fall-through edges across partition boundaries. + Note that force_nonfallthru will do any necessary partition + boundary fixup by calling fixup_partition_crossing itself. */ + if ((e->flags & EDGE_FALLTHRU) + && BB_PARTITION (bb) != BB_PARTITION (e->dest) + && e->dest != EXIT_BLOCK_PTR) + force_nonfallthru (e); + else + fixup_partition_crossing (e); + } +} + /* Attempt to change code to redirect edge E to TARGET. Don't do that on expense of adding new instructions or reordering basic blocks. @@ -1979,6 +2016,14 @@ commit_edge_insertions (void) { basic_block bb; + /* Optimization passes that invoke this routine can cause hot blocks + previously reached by both hot and cold blocks to become dominated only + by cold blocks. This will cause the verification below to fail, + and lead to now cold code in the hot section. In some cases this + may only be visible after newly unreachable blocks are deleted, + which will be done by fixup_partitions. */ + fixup_partitions (); + #ifdef ENABLE_CHECKING verify_flow_info (); #endif @@ -2173,6 +2218,101 @@ get_last_bb_insn (basic_block bb) return end; } +/* Sanity check partition hotness to ensure that basic blocks in + the cold partition don't dominate basic blocks in the hot partition. + If FLAG_ONLY is true, report violations as errors. Otherwise + re-mark the dominated blocks as cold, since this is run after + cfg optimizations that may make hot blocks previously reached + by both hot and cold blocks now only reachable along cold paths. */ + +static vec<basic_block> +find_partition_fixes (bool flag_only) +{ + basic_block bb; + vec<basic_block> bbs_in_cold_partition = vNULL; + vec<basic_block> bbs_to_fix = vNULL; + + /* Callers check this. */ + gcc_checking_assert (crtl->has_bb_partition); + + FOR_EACH_BB (bb) + if ((BB_PARTITION (bb) == BB_COLD_PARTITION)) + bbs_in_cold_partition.safe_push (bb); + + if (bbs_in_cold_partition.is_empty ()) + return vNULL; + + bool dom_calculated_here = !dom_info_available_p (CDI_DOMINATORS); + + if (dom_calculated_here) + calculate_dominance_info (CDI_DOMINATORS); + + while (! bbs_in_cold_partition.is_empty ()) + { + bb = bbs_in_cold_partition.pop (); + /* Any blocks dominated by a block in the cold section + must also be cold. */ + basic_block son; + for (son = first_dom_son (CDI_DOMINATORS, bb); + son; + son = next_dom_son (CDI_DOMINATORS, son)) + { + /* If son is not yet cold, then mark it cold here and + enqueue it for further processing. */ + if ((BB_PARTITION (son) != BB_COLD_PARTITION)) + { + if (flag_only) + error ("non-cold basic block %d dominated " + "by a block in the cold partition", son->index); + else + BB_SET_PARTITION (son, BB_COLD_PARTITION); + bbs_to_fix.safe_push (son); + bbs_in_cold_partition.safe_push (son); + } + } + } + + if (dom_calculated_here) + free_dominance_info (CDI_DOMINATORS); + + return bbs_to_fix; +} + +/* Perform cleanup on the hot/cold bb partitioning after optimization + passes that modify the cfg. */ + +void +fixup_partitions (void) +{ + basic_block bb; + + if (!crtl->has_bb_partition) + return; + + /* Delete any blocks that became unreachable and weren't + already cleaned up, for example during edge forwarding + and convert_jumps_to_returns. This will expose more + opportunities for fixing the partition boundaries here. + Also, the calculation of the dominance graph during verification + will assert if there are unreachable nodes. */ + delete_unreachable_blocks (); + + /* If there are partitions, do a sanity check on them: A basic block in + a cold partition cannot dominate a basic block in a hot partition. + Fixup any that now violate this requirement, as a result of edge + forwarding and unreachable block deletion. */ + vec<basic_block> bbs_to_fix = find_partition_fixes (false); + + /* Do the partition fixup after all necessary blocks have been converted to + cold, so that we only update the region crossings the minimum number of + places, which can require forcing edges to be non fallthru. */ + while (! bbs_to_fix.is_empty ()) + { + bb = bbs_to_fix.pop (); + fixup_new_cold_bb (bb); + } +} + /* Verify, in the basic block chain, that there is at most one switch between hot/cold partitions. This condition will not be true until after reorder_basic_blocks is called. */ @@ -2219,7 +2359,8 @@ verify_hot_cold_block_grouping (void) /* Perform several checks on the edges out of each block, such as the consistency of the branch probabilities, the correctness of hot/cold partition crossing edges, and the number of expected - successor edges. */ + successor edges. Also verify that the dominance relationship + between hot/cold blocks is sane. */ static int rtl_verify_edges (void) @@ -2382,6 +2523,14 @@ rtl_verify_edges (void) } } + /* If there are partitions, do a sanity check on them: A basic block in + a cold partition cannot dominate a basic block in a hot partition. */ + if (crtl->has_bb_partition && !err) + { + vec<basic_block> bbs_to_fix = find_partition_fixes (true); + err = !bbs_to_fix.is_empty (); + } + /* Clean up. */ return err; } @@ -2515,7 +2664,7 @@ rtl_verify_bb_pointers (void) and NOTE_INSN_BASIC_BLOCK - verify that no fall_thru edge crosses hot/cold partition boundaries - verify that there are no pending RTL branch predictions - - verify that there is a single hot/cold partition boundary after bbro + - verify that hot blocks are not dominated by cold blocks In future it can be extended check a lot of other stuff as well (reachability of basic blocks, life information, etc. etc.). */ @@ -2761,7 +2910,8 @@ rtl_verify_bb_layout (void) - check that all insns are in the basic blocks (except the switch handling code, barriers and notes) - check that all returns are followed by barriers - - check that all fallthru edge points to the adjacent blocks. */ + - check that all fallthru edge points to the adjacent blocks + - verify that there is a single hot/cold partition boundary after bbro */ static int rtl_verify_flow_info (void) Index: basic-block.h =================================================================== --- basic-block.h (revision 201461) +++ basic-block.h (working copy) @@ -797,6 +797,7 @@ extern bool contains_no_active_insn_p (const_basic extern bool forwarder_block_p (const_basic_block); extern bool can_fallthru (basic_block, basic_block); extern void emit_barrier_after_bb (basic_block bb); +extern void fixup_partitions (void); /* In cfgbuild.c. */ extern void find_many_sub_basic_blocks (sbitmap); Index: cfgcleanup.c =================================================================== --- cfgcleanup.c (revision 201461) +++ cfgcleanup.c (working copy) @@ -2807,10 +2807,21 @@ try_optimize_cfg (int mode) df_analyze (); } + if (changed) + { + /* Edge forwarding in particular can cause hot blocks previously + reached by both hot and cold blocks to become dominated only + by cold blocks. This will cause the verification below to fail, + and lead to now cold code in the hot section. This is not easy + to detect and fix during edge forwarding, and in some cases + is only visible after newly unreachable blocks are deleted, + which will be done in fixup_partitions. */ + fixup_partitions (); + #ifdef ENABLE_CHECKING - if (changed) - verify_flow_info (); + verify_flow_info (); #endif + } changed_overall |= changed; first_pass = false; Index: bb-reorder.c =================================================================== --- bb-reorder.c (revision 201461) +++ bb-reorder.c (working copy) @@ -1444,6 +1444,57 @@ fix_up_crossing_landing_pad (eh_landing_pad old_lp ei_next (&ei); } + +/* Ensure that no cold bbs dominate hot bbs along the dominance or + post-dominance DIR, for example as a result of edge weight insanities. + Returns the updated value of COLD_BB_COUNT and adds newly-hot bbs + to BBS_IN_HOT_PARTITION. */ + +static unsigned int +sanitize_dominator_hotness (enum cdi_direction dir, unsigned int cold_bb_count, + vec<basic_block> *bbs_in_hot_partition) +{ + /* Callers check this. */ + gcc_checking_assert (cold_bb_count); + + bool dom_calculated_here = !dom_info_available_p (dir); + + if (dom_calculated_here) + calculate_dominance_info (dir); + + /* Keep examining hot bbs while we still have some left to check + and there are remaining cold bbs. */ + vec<basic_block> hot_bbs_to_check = bbs_in_hot_partition->copy (); + while (! hot_bbs_to_check.is_empty () + && cold_bb_count) + { + basic_block bb = hot_bbs_to_check.pop (); + basic_block dom_bb = get_immediate_dominator (dir, bb); + + /* If bb's immediate dominator is also hot (or unpartitioned, + e.g. the entry block) then it is ok. If it is cold, it + needs to be adjusted. */ + if (BB_PARTITION (dom_bb) != BB_COLD_PARTITION) + continue; + + /* We have a hot bb with an immediate dominator that is cold. + The dominator needs to be re-marked hot. */ + BB_SET_PARTITION (dom_bb, BB_HOT_PARTITION); + cold_bb_count--; + + /* Now we need to examine newly-hot dom_bb to see if it is also + dominated by a cold bb. */ + bbs_in_hot_partition->safe_push (dom_bb); + hot_bbs_to_check.safe_push (dom_bb); + } + + if (dom_calculated_here) + free_dominance_info (dir); + + return cold_bb_count; +} + + /* Find the basic blocks that are rarely executed and need to be moved to a separate section of the .o file (to cut down on paging and improve cache locality). Return a vector of all edges that cross. */ @@ -1455,16 +1506,42 @@ find_rarely_executed_basic_blocks_and_crossing_edg basic_block bb; edge e; edge_iterator ei; + unsigned int cold_bb_count = 0; + vec<basic_block> bbs_in_hot_partition = vNULL; /* Mark which partition (hot/cold) each basic block belongs in. */ FOR_EACH_BB (bb) { if (probably_never_executed_bb_p (cfun, bb)) - BB_SET_PARTITION (bb, BB_COLD_PARTITION); + { + BB_SET_PARTITION (bb, BB_COLD_PARTITION); + cold_bb_count++; + } else - BB_SET_PARTITION (bb, BB_HOT_PARTITION); + { + BB_SET_PARTITION (bb, BB_HOT_PARTITION); + bbs_in_hot_partition.safe_push (bb); + } } + /* Ensure that no cold bbs dominate hot bbs. This could happen as a result of + several different possibilities. One is that there are edge weight insanities + due to optimization phases that do not properly update basic block profile + counts. The second is that the entry of the function may not be hot, because + it is entered fewer times than the number of profile training runs, but there + is a loop inside the function that causes blocks within the function to be + above the threshold for hotness. Then do the same along the post-dominator + tree (which could have additional changes required after fixing up + dominators). */ + if (cold_bb_count) + cold_bb_count = sanitize_dominator_hotness (CDI_DOMINATORS, + cold_bb_count, + &bbs_in_hot_partition); + if (cold_bb_count) + sanitize_dominator_hotness (CDI_POST_DOMINATORS, + cold_bb_count, + &bbs_in_hot_partition); + /* The format of .gcc_except_table does not allow landing pads to be in a different partition as the throw. Fix this by either moving or duplicating the landing pads. */ > >> >>> +/* Sanity check partition hotness to ensure that basic blocks in >>> + the cold partition don't dominate basic blocks in the hot partition. >>> + If FLAG_ONLY is true, report violations as errors. Otherwise >>> + re-mark the dominated blocks as cold, since this is run after >>> + cfg optimizations that may make hot blocks previously reached >>> + by both hot and cold blocks now only reachable along cold paths. */ >> >> With profile, I suppose we can have cold blocks dominating hot blocks when the >> hot blocks is in loop whose trip count is high enough. Indeed for partitioning >> reasons it does not make sense to push those into different section. >> >> I also wonder, if we finally get the pass stable, can we enable it by default >> and offline probably cold blocks w/o profile? Primarily blocks reachable only >> by EH + blocks leading to a crash or throw(). For C++ those should be common >> enough to make a difference... > > Yep, as soon as PR57451 is fixed, which I hope to get to next week, > then I am going to send a patch to turn this on by default, at least > with profile feedback, which is where I've been doing performance > tuning. But you are right that there are some cases where it should be > beneficial without profile data as well. > > Thanks, > Teresa > >> >> Honza > > > > -- > Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-05 13:36 ` Teresa Johnson @ 2013-08-05 14:11 ` Jan Hubicka 2013-08-05 14:57 ` Teresa Johnson [not found] ` <20130808222332.GA31755@kam.mff.cuni.cz> 1 sibling, 1 reply; 62+ messages in thread From: Jan Hubicka @ 2013-08-05 14:11 UTC (permalink / raw) To: Teresa Johnson Cc: Jan Hubicka, Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law The patch looks OK to me in general (I can not approve it). Still have one question... > + > +/* Ensure that no cold bbs dominate hot bbs along the dominance or > + post-dominance DIR, for example as a result of edge weight insanities. > + Returns the updated value of COLD_BB_COUNT and adds newly-hot bbs > + to BBS_IN_HOT_PARTITION. */ > + > +static unsigned int > +sanitize_dominator_hotness (enum cdi_direction dir, unsigned int cold_bb_count, > + vec<basic_block> *bbs_in_hot_partition) > +{ > + /* Callers check this. */ > + gcc_checking_assert (cold_bb_count); > + > + bool dom_calculated_here = !dom_info_available_p (dir); > + > + if (dom_calculated_here) > + calculate_dominance_info (dir); > + > + /* Keep examining hot bbs while we still have some left to check > + and there are remaining cold bbs. */ > + vec<basic_block> hot_bbs_to_check = bbs_in_hot_partition->copy (); > + while (! hot_bbs_to_check.is_empty () > + && cold_bb_count) > + { > + basic_block bb = hot_bbs_to_check.pop (); > + basic_block dom_bb = get_immediate_dominator (dir, bb); > + > + /* If bb's immediate dominator is also hot (or unpartitioned, > + e.g. the entry block) then it is ok. If it is cold, it > + needs to be adjusted. */ > + if (BB_PARTITION (dom_bb) != BB_COLD_PARTITION) > + continue; What will hapepn on if (t) something else something else for (i=0;i<1000000;i++) something else2 I would expect if/something and something else to be cold by profile feedback. Your dominator code will bring the if into hot partition but both paths of conditional will be cold, so the number of crossings will actually grow. If we want to have at least some path to hot blocks in the hot region, I suspect we could walk back from hot regions to entry and keep those in hot regions rather than relying on the dominator tree... But I am sure such things can be dealt with incrementally. Honza ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-05 14:11 ` Jan Hubicka @ 2013-08-05 14:57 ` Teresa Johnson 2013-08-06 3:01 ` Teresa Johnson 0 siblings, 1 reply; 62+ messages in thread From: Teresa Johnson @ 2013-08-05 14:57 UTC (permalink / raw) To: Jan Hubicka Cc: Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law On Mon, Aug 5, 2013 at 7:11 AM, Jan Hubicka <hubicka@ucw.cz> wrote: > The patch looks OK to me in general (I can not approve it). > Still have one question... >> + >> +/* Ensure that no cold bbs dominate hot bbs along the dominance or >> + post-dominance DIR, for example as a result of edge weight insanities. >> + Returns the updated value of COLD_BB_COUNT and adds newly-hot bbs >> + to BBS_IN_HOT_PARTITION. */ >> + >> +static unsigned int >> +sanitize_dominator_hotness (enum cdi_direction dir, unsigned int cold_bb_count, >> + vec<basic_block> *bbs_in_hot_partition) >> +{ >> + /* Callers check this. */ >> + gcc_checking_assert (cold_bb_count); >> + >> + bool dom_calculated_here = !dom_info_available_p (dir); >> + >> + if (dom_calculated_here) >> + calculate_dominance_info (dir); >> + >> + /* Keep examining hot bbs while we still have some left to check >> + and there are remaining cold bbs. */ >> + vec<basic_block> hot_bbs_to_check = bbs_in_hot_partition->copy (); >> + while (! hot_bbs_to_check.is_empty () >> + && cold_bb_count) >> + { >> + basic_block bb = hot_bbs_to_check.pop (); >> + basic_block dom_bb = get_immediate_dominator (dir, bb); >> + >> + /* If bb's immediate dominator is also hot (or unpartitioned, >> + e.g. the entry block) then it is ok. If it is cold, it >> + needs to be adjusted. */ >> + if (BB_PARTITION (dom_bb) != BB_COLD_PARTITION) >> + continue; > > What will hapepn on > > if (t) > something > else > something else > for (i=0;i<1000000;i++) > something else2 > > I would expect if/something and something else to be cold by profile feedback. > Your dominator code will bring the if into hot partition but both paths > of conditional will be cold, so the number of crossings will actually grow. You are right, this case will not be handled well. > > If we want to have at least some path to hot blocks in the hot region, I suspect > we could walk back from hot regions to entry and keep those in hot regions rather > than relying on the dominator tree... > But I am sure such things can be dealt with incrementally. I am going to fix this and will resend the patch. Rather than look at the immediate dominator of each hot block, we need to ensure that at least one pred bb is hot. In your example, if that was a 50-50 branch, then IMO both preds should be marked hot. Thanks, Teresa > > Honza -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-05 14:57 ` Teresa Johnson @ 2013-08-06 3:01 ` Teresa Johnson 0 siblings, 0 replies; 62+ messages in thread From: Teresa Johnson @ 2013-08-06 3:01 UTC (permalink / raw) To: Jan Hubicka Cc: Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law On Mon, Aug 5, 2013 at 7:57 AM, Teresa Johnson <tejohnson@google.com> wrote: > On Mon, Aug 5, 2013 at 7:11 AM, Jan Hubicka <hubicka@ucw.cz> wrote: >> The patch looks OK to me in general (I can not approve it). >> Still have one question... >>> + >>> +/* Ensure that no cold bbs dominate hot bbs along the dominance or >>> + post-dominance DIR, for example as a result of edge weight insanities. >>> + Returns the updated value of COLD_BB_COUNT and adds newly-hot bbs >>> + to BBS_IN_HOT_PARTITION. */ >>> + >>> +static unsigned int >>> +sanitize_dominator_hotness (enum cdi_direction dir, unsigned int cold_bb_count, >>> + vec<basic_block> *bbs_in_hot_partition) >>> +{ >>> + /* Callers check this. */ >>> + gcc_checking_assert (cold_bb_count); >>> + >>> + bool dom_calculated_here = !dom_info_available_p (dir); >>> + >>> + if (dom_calculated_here) >>> + calculate_dominance_info (dir); >>> + >>> + /* Keep examining hot bbs while we still have some left to check >>> + and there are remaining cold bbs. */ >>> + vec<basic_block> hot_bbs_to_check = bbs_in_hot_partition->copy (); >>> + while (! hot_bbs_to_check.is_empty () >>> + && cold_bb_count) >>> + { >>> + basic_block bb = hot_bbs_to_check.pop (); >>> + basic_block dom_bb = get_immediate_dominator (dir, bb); >>> + >>> + /* If bb's immediate dominator is also hot (or unpartitioned, >>> + e.g. the entry block) then it is ok. If it is cold, it >>> + needs to be adjusted. */ >>> + if (BB_PARTITION (dom_bb) != BB_COLD_PARTITION) >>> + continue; >> >> What will hapepn on >> >> if (t) >> something >> else >> something else >> for (i=0;i<1000000;i++) >> something else2 >> >> I would expect if/something and something else to be cold by profile feedback. >> Your dominator code will bring the if into hot partition but both paths >> of conditional will be cold, so the number of crossings will actually grow. > > You are right, this case will not be handled well. > >> >> If we want to have at least some path to hot blocks in the hot region, I suspect >> we could walk back from hot regions to entry and keep those in hot regions rather >> than relying on the dominator tree... >> But I am sure such things can be dealt with incrementally. > > I am going to fix this and will resend the patch. Rather than look at > the immediate dominator of each hot block, we need to ensure that at > least one pred bb is hot. In your example, if that was a 50-50 branch, > then IMO both preds should be marked hot. New patch below that walks the preds of each hot bb instead of the dominators. Bootstrapped and tested on x86-64-unknown-linux-gnu. Also ensured that a profiledbootstrap passed with -freorder-blocks-and-partition enabled still works. Ok for trunk? Thanks, Teresa 2013-08-05 Teresa Johnson <tejohnson@google.com> Steven Bosscher <steven@gcc.gnu.org> * cfgrtl.c (fixup_new_cold_bb): New routine. (commit_edge_insertions): Invoke fixup_partitions. (find_partition_fixes): New routine. (fixup_partitions): Ditto. (verify_hot_cold_block_grouping): Update comments. (rtl_verify_edges): Invoke find_partition_fixes. (rtl_verify_bb_pointers): Update comments. (rtl_verify_bb_layout): Ditto. * basic-block.h (fixup_partitions): Declare. * cfgcleanup.c (try_optimize_cfg): Invoke fixup_partitions. * bb-reorder.c (sanitize_hot_paths): New function. (find_rarely_executed_basic_blocks_and_crossing_edges): Invoke sanitize_hot_paths. Index: cfgrtl.c =================================================================== --- cfgrtl.c (revision 201461) +++ cfgrtl.c (working copy) @@ -1341,6 +1341,43 @@ fixup_partition_crossing (edge e) } } +/* Called when block BB has been reassigned to the cold partition, + because it is now dominated by another cold block, + to ensure that the region crossing attributes are updated. */ + +static void +fixup_new_cold_bb (basic_block bb) +{ + edge e; + edge_iterator ei; + + /* This is called when a hot bb is found to now be dominated + by a cold bb and therefore needs to become cold. Therefore, + its preds will no longer be region crossing. Any non-dominating + preds that were previously hot would also have become cold + in the caller for the same region. Any preds that were previously + region-crossing will be adjusted in fixup_partition_crossing. */ + FOR_EACH_EDGE (e, ei, bb->preds) + { + fixup_partition_crossing (e); + } + + /* Possibly need to make bb's successor edges region crossing, + or remove stale region crossing. */ + FOR_EACH_EDGE (e, ei, bb->succs) + { + /* We can't have fall-through edges across partition boundaries. + Note that force_nonfallthru will do any necessary partition + boundary fixup by calling fixup_partition_crossing itself. */ + if ((e->flags & EDGE_FALLTHRU) + && BB_PARTITION (bb) != BB_PARTITION (e->dest) + && e->dest != EXIT_BLOCK_PTR) + force_nonfallthru (e); + else + fixup_partition_crossing (e); + } +} + /* Attempt to change code to redirect edge E to TARGET. Don't do that on expense of adding new instructions or reordering basic blocks. @@ -1979,6 +2016,14 @@ commit_edge_insertions (void) { basic_block bb; + /* Optimization passes that invoke this routine can cause hot blocks + previously reached by both hot and cold blocks to become dominated only + by cold blocks. This will cause the verification below to fail, + and lead to now cold code in the hot section. In some cases this + may only be visible after newly unreachable blocks are deleted, + which will be done by fixup_partitions. */ + fixup_partitions (); + #ifdef ENABLE_CHECKING verify_flow_info (); #endif @@ -2173,6 +2218,101 @@ get_last_bb_insn (basic_block bb) return end; } +/* Sanity check partition hotness to ensure that basic blocks in + the cold partition don't dominate basic blocks in the hot partition. + If FLAG_ONLY is true, report violations as errors. Otherwise + re-mark the dominated blocks as cold, since this is run after + cfg optimizations that may make hot blocks previously reached + by both hot and cold blocks now only reachable along cold paths. */ + +static vec<basic_block> +find_partition_fixes (bool flag_only) +{ + basic_block bb; + vec<basic_block> bbs_in_cold_partition = vNULL; + vec<basic_block> bbs_to_fix = vNULL; + + /* Callers check this. */ + gcc_checking_assert (crtl->has_bb_partition); + + FOR_EACH_BB (bb) + if ((BB_PARTITION (bb) == BB_COLD_PARTITION)) + bbs_in_cold_partition.safe_push (bb); + + if (bbs_in_cold_partition.is_empty ()) + return vNULL; + + bool dom_calculated_here = !dom_info_available_p (CDI_DOMINATORS); + + if (dom_calculated_here) + calculate_dominance_info (CDI_DOMINATORS); + + while (! bbs_in_cold_partition.is_empty ()) + { + bb = bbs_in_cold_partition.pop (); + /* Any blocks dominated by a block in the cold section + must also be cold. */ + basic_block son; + for (son = first_dom_son (CDI_DOMINATORS, bb); + son; + son = next_dom_son (CDI_DOMINATORS, son)) + { + /* If son is not yet cold, then mark it cold here and + enqueue it for further processing. */ + if ((BB_PARTITION (son) != BB_COLD_PARTITION)) + { + if (flag_only) + error ("non-cold basic block %d dominated " + "by a block in the cold partition (%d)", son->index, bb->index); + else + BB_SET_PARTITION (son, BB_COLD_PARTITION); + bbs_to_fix.safe_push (son); + bbs_in_cold_partition.safe_push (son); + } + } + } + + if (dom_calculated_here) + free_dominance_info (CDI_DOMINATORS); + + return bbs_to_fix; +} + +/* Perform cleanup on the hot/cold bb partitioning after optimization + passes that modify the cfg. */ + +void +fixup_partitions (void) +{ + basic_block bb; + + if (!crtl->has_bb_partition) + return; + + /* Delete any blocks that became unreachable and weren't + already cleaned up, for example during edge forwarding + and convert_jumps_to_returns. This will expose more + opportunities for fixing the partition boundaries here. + Also, the calculation of the dominance graph during verification + will assert if there are unreachable nodes. */ + delete_unreachable_blocks (); + + /* If there are partitions, do a sanity check on them: A basic block in + a cold partition cannot dominate a basic block in a hot partition. + Fixup any that now violate this requirement, as a result of edge + forwarding and unreachable block deletion. */ + vec<basic_block> bbs_to_fix = find_partition_fixes (false); + + /* Do the partition fixup after all necessary blocks have been converted to + cold, so that we only update the region crossings the minimum number of + places, which can require forcing edges to be non fallthru. */ + while (! bbs_to_fix.is_empty ()) + { + bb = bbs_to_fix.pop (); + fixup_new_cold_bb (bb); + } +} + /* Verify, in the basic block chain, that there is at most one switch between hot/cold partitions. This condition will not be true until after reorder_basic_blocks is called. */ @@ -2219,7 +2359,8 @@ verify_hot_cold_block_grouping (void) /* Perform several checks on the edges out of each block, such as the consistency of the branch probabilities, the correctness of hot/cold partition crossing edges, and the number of expected - successor edges. */ + successor edges. Also verify that the dominance relationship + between hot/cold blocks is sane. */ static int rtl_verify_edges (void) @@ -2382,6 +2523,14 @@ rtl_verify_edges (void) } } + /* If there are partitions, do a sanity check on them: A basic block in + a cold partition cannot dominate a basic block in a hot partition. */ + if (crtl->has_bb_partition && !err) + { + vec<basic_block> bbs_to_fix = find_partition_fixes (true); + err = !bbs_to_fix.is_empty (); + } + /* Clean up. */ return err; } @@ -2515,7 +2664,7 @@ rtl_verify_bb_pointers (void) and NOTE_INSN_BASIC_BLOCK - verify that no fall_thru edge crosses hot/cold partition boundaries - verify that there are no pending RTL branch predictions - - verify that there is a single hot/cold partition boundary after bbro + - verify that hot blocks are not dominated by cold blocks In future it can be extended check a lot of other stuff as well (reachability of basic blocks, life information, etc. etc.). */ @@ -2761,7 +2910,8 @@ rtl_verify_bb_layout (void) - check that all insns are in the basic blocks (except the switch handling code, barriers and notes) - check that all returns are followed by barriers - - check that all fallthru edge points to the adjacent blocks. */ + - check that all fallthru edge points to the adjacent blocks + - verify that there is a single hot/cold partition boundary after bbro */ static int rtl_verify_flow_info (void) Index: basic-block.h =================================================================== --- basic-block.h (revision 201461) +++ basic-block.h (working copy) @@ -797,6 +797,7 @@ extern bool contains_no_active_insn_p (const_basic extern bool forwarder_block_p (const_basic_block); extern bool can_fallthru (basic_block, basic_block); extern void emit_barrier_after_bb (basic_block bb); +extern void fixup_partitions (void); /* In cfgbuild.c. */ extern void find_many_sub_basic_blocks (sbitmap); Index: cfgcleanup.c =================================================================== --- cfgcleanup.c (revision 201461) +++ cfgcleanup.c (working copy) @@ -2807,10 +2807,21 @@ try_optimize_cfg (int mode) df_analyze (); } + if (changed) + { + /* Edge forwarding in particular can cause hot blocks previously + reached by both hot and cold blocks to become dominated only + by cold blocks. This will cause the verification below to fail, + and lead to now cold code in the hot section. This is not easy + to detect and fix during edge forwarding, and in some cases + is only visible after newly unreachable blocks are deleted, + which will be done in fixup_partitions. */ + fixup_partitions (); + #ifdef ENABLE_CHECKING - if (changed) - verify_flow_info (); + verify_flow_info (); #endif + } changed_overall |= changed; first_pass = false; Index: bb-reorder.c =================================================================== --- bb-reorder.c (revision 201461) +++ bb-reorder.c (working copy) @@ -1444,27 +1444,134 @@ fix_up_crossing_landing_pad (eh_landing_pad old_lp ei_next (&ei); } + +/* Ensure that all hot bbs are included in a hot path through the + procedure. This is done by calling this function twice, once + with WALK_UP true (to look for paths from the entry to hot bbs) and + once with WALK_UP false (to look for paths from hot bbs to the exit). + Returns the updated value of COLD_BB_COUNT and adds newly-hot bbs + to BBS_IN_HOT_PARTITION. */ + +static unsigned int +sanitize_hot_paths (bool walk_up, unsigned int cold_bb_count, + vec<basic_block> *bbs_in_hot_partition) +{ + /* Callers check this. */ + gcc_checking_assert (cold_bb_count); + + /* Keep examining hot bbs while we still have some left to check + and there are remaining cold bbs. */ + vec<basic_block> hot_bbs_to_check = bbs_in_hot_partition->copy (); + while (! hot_bbs_to_check.is_empty () + && cold_bb_count) + { + basic_block bb = hot_bbs_to_check.pop (); + vec<edge, va_gc> *edges = walk_up ? bb->preds : bb->succs; + edge e; + edge_iterator ei; + int highest_probability = 0; + bool found = false; + + /* Walk the preds/succs and check if there is at least one already + marked hot. Keep track of the most frequent pred/succ so that we + can mark it hot if we don't find one. */ + FOR_EACH_EDGE (e, ei, edges) + { + basic_block reach_bb = walk_up ? e->src : e->dest; + + if (e->flags & EDGE_DFS_BACK) + continue; + + if (BB_PARTITION (reach_bb) != BB_COLD_PARTITION) + { + found = true; + break; + } + if (e->probability > highest_probability) + highest_probability = e->probability; + } + + /* If bb is reached by (or reaches, in the case of !WALK_UP) another hot + block (or unpartitioned, e.g. the entry block) then it is ok. If not, + then the most frequent pred (or succ) needs to be adjusted. In the + case where multiple preds/succs have the same probability (e.g. a + 50-50 branch), then both will be adjusted. */ + if (found) + continue; + + FOR_EACH_EDGE (e, ei, edges) + { + if (e->flags & EDGE_DFS_BACK) + continue; + if (e->probability < highest_probability) + continue; + + basic_block reach_bb = walk_up ? e->src : e->dest; + + /* We have a hot bb with an immediate dominator that is cold. + The dominator needs to be re-marked hot. */ + BB_SET_PARTITION (reach_bb, BB_HOT_PARTITION); + cold_bb_count--; + + /* Now we need to examine newly-hot reach_bb to see if it is also + dominated by a cold bb. */ + bbs_in_hot_partition->safe_push (reach_bb); + hot_bbs_to_check.safe_push (reach_bb); + } + } + + return cold_bb_count; +} + + /* Find the basic blocks that are rarely executed and need to be moved to a separate section of the .o file (to cut down on paging and improve cache locality). Return a vector of all edges that cross. */ -static vec<edge> +static vec<edge> find_rarely_executed_basic_blocks_and_crossing_edges (void) { vec<edge> crossing_edges = vNULL; basic_block bb; edge e; edge_iterator ei; + unsigned int cold_bb_count = 0; + vec<basic_block> bbs_in_hot_partition = vNULL; /* Mark which partition (hot/cold) each basic block belongs in. */ FOR_EACH_BB (bb) { if (probably_never_executed_bb_p (cfun, bb)) - BB_SET_PARTITION (bb, BB_COLD_PARTITION); + { + BB_SET_PARTITION (bb, BB_COLD_PARTITION); + cold_bb_count++; + } else - BB_SET_PARTITION (bb, BB_HOT_PARTITION); + { + BB_SET_PARTITION (bb, BB_HOT_PARTITION); + bbs_in_hot_partition.safe_push (bb); + } } + /* Ensure that hot bbs are included along a hot path from the entry to exit. + Several different possibilities may include cold bbs along all paths + to/from a hot bb. One is that there are edge weight insanities + due to optimization phases that do not properly update basic block profile + counts. The second is that the entry of the function may not be hot, because + it is entered fewer times than the number of profile training runs, but there + is a loop inside the function that causes blocks within the function to be + above the threshold for hotness. This is fixed by walking up from hot bbs + to the entry block, and then down from hot bbs to the exit, performing + partitioning fixups as necessary. */ + if (cold_bb_count) + { + mark_dfs_back_edges (); + cold_bb_count = sanitize_hot_paths (true, cold_bb_count, + &bbs_in_hot_partition); + if (cold_bb_count) + sanitize_hot_paths (false, cold_bb_count, &bbs_in_hot_partition); + } + /* The format of .gcc_except_table does not allow landing pads to be in a different partition as the throw. Fix this by either moving or duplicating the landing pads. */ > > Thanks, > Teresa > >> >> Honza > > > > -- > Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
[parent not found: <20130808222332.GA31755@kam.mff.cuni.cz>]
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition [not found] ` <20130808222332.GA31755@kam.mff.cuni.cz> @ 2013-08-08 23:04 ` Teresa Johnson 2013-08-09 9:58 ` Jan Hubicka 0 siblings, 1 reply; 62+ messages in thread From: Teresa Johnson @ 2013-08-08 23:04 UTC (permalink / raw) To: Jan Hubicka Cc: Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, marxin.liska On Thu, Aug 8, 2013 at 3:23 PM, Jan Hubicka <hubicka@ucw.cz> wrote: > Hi, > Martin Liska was kind enough to generate disk seeking graph of gimp statrup with his function reordering. > His code simply measures time of firest execution of a function and orders functions in the given order. > The functions stay in the subsections (unlikely/startup/exit/hot/normal) that are then glued together > in this order. > > I am attaching disk seeking with and without -freorder-blocks-and-partition (with your patch). > > In 2.pdf you can see two increasing sequences in the text segment. If I am not mistaken the bottom > one comes for hot and the top one for normal section. The big unused part on bottom is unlikely > section since most of gimp is not trained. 2.pdf is reordered with Martin's technique? > > Now 1.pdf is with -freorder-blocks-and-partition and your patch. You can see there is third sequence > near bottom of the text seciton. that is beggining of unlikely section, so it tracks jumps where we > fall into cold section of function. 1.pdf is generated using the usual FDO + -freorder-blocks-and-partition (i.e. not using Martin's technique)? > > It still seems rather bad (i.e. good part of unlikely section is actually used). I think the dominator > based approach is not going to work too reliably (I can "fix" my testcase to contain multiple nested > conditionals and then the heuristic about predecestors won't help). Yes, this doesn't look good. Did you use the latest version of my patch that doesn't walk the dominators? Do you know how many training runs are done for this benchmark? I think a lot of the issues that you pointed out with the hot loop preceded by non-looping conditional code as in your earlier example, or multiple nested conditionals, comes from the fact that the cold cutoff is not 0, but some number less than the number of training runs. Perhaps the cutoff for splitting should be 0. Then the main issue that needs to be corrected is profile insanities, not code that is executed once (since that would not be marked cold). The only other issue that I can think of here is that the training data was not representative and didn't execute these blocks. > > What about simply walking the CFG from entry through all edges with non-0 counts and making all reachable > blocks hot + forcingly make any hot blocks not reachable this way reachable? Is this different than what I currently have + changing the cold cutoff to 0? In that case any blocks reachable through non-0 edges should be non-0 and marked hot, and the current patch forces the most frequent paths to all hot blocks to be hot. Thanks, Teresa > I think we are really looking primarily for dead parts of the functions (sanity checks/error handling) > that should not be visited by train run. We can then see how to make the heuristic more aggressive? > > Honza -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-08 23:04 ` Teresa Johnson @ 2013-08-09 9:58 ` Jan Hubicka 2013-08-09 14:38 ` Teresa Johnson 0 siblings, 1 reply; 62+ messages in thread From: Jan Hubicka @ 2013-08-09 9:58 UTC (permalink / raw) To: Teresa Johnson Cc: Jan Hubicka, Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, marxin.liska > On Thu, Aug 8, 2013 at 3:23 PM, Jan Hubicka <hubicka@ucw.cz> wrote: > > Hi, > > Martin Liska was kind enough to generate disk seeking graph of gimp statrup with his function reordering. > > His code simply measures time of firest execution of a function and orders functions in the given order. > > The functions stay in the subsections (unlikely/startup/exit/hot/normal) that are then glued together > > in this order. > > > > I am attaching disk seeking with and without -freorder-blocks-and-partition (with your patch). > > > > In 2.pdf you can see two increasing sequences in the text segment. If I am not mistaken the bottom > > one comes for hot and the top one for normal section. The big unused part on bottom is unlikely > > section since most of gimp is not trained. > > 2.pdf is reordered with Martin's technique? > > > > > Now 1.pdf is with -freorder-blocks-and-partition and your patch. You can see there is third sequence > > near bottom of the text seciton. that is beggining of unlikely section, so it tracks jumps where we > > fall into cold section of function. > > 1.pdf is generated using the usual FDO + > -freorder-blocks-and-partition (i.e. not using Martin's technique)? 2.pdf is Martin's reordering (that works ortoghonally to what we already have - it just orders the functions inside idividual subsections. This make the subsections more visible than without his patch). 1.pdf is Marting's rerodering + yours patch (I asked him to double check it is the latest verision) + -freorder-blocks-and-partition. He simply trains and measures the gimp startup, nothing else, so there should not be problem with representativity of the data. > > > > > It still seems rather bad (i.e. good part of unlikely section is actually used). I think the dominator > > based approach is not going to work too reliably (I can "fix" my testcase to contain multiple nested > > conditionals and then the heuristic about predecestors won't help). > > Yes, this doesn't look good. Did you use the latest version of my > patch that doesn't walk the dominators? > > Do you know how many training runs are done for this benchmark? I > think a lot of the issues that you pointed out with the hot loop > preceded by non-looping conditional code as in your earlier example, > or multiple nested conditionals, comes from the fact that the cold > cutoff is not 0, but some number less than the number of training > runs. Perhaps the cutoff for splitting should be 0. Then the main > issue that needs to be corrected is profile insanities, not code that > is executed once (since that would not be marked cold). Hmm, compute_function_frequency uses probably_never_executed_bb_p that requires the count of basic block to be less than number of train runs. In Martin's setup that will be 0. This is the same as what -frerorder-block-of-partition does? > > The only other issue that I can think of here is that the training > data was not representative and didn't execute these blocks. > > > > > What about simply walking the CFG from entry through all edges with non-0 counts and making all reachable > > blocks hot + forcingly make any hot blocks not reachable this way reachable? > > Is this different than what I currently have + changing the cold > cutoff to 0? In that case any blocks reachable through non-0 edges > should be non-0 and marked hot, and the current patch forces the most > frequent paths to all hot blocks to be hot. Do we sanity check that the cold partition does not contain any blocks of count 0? It may be that the profile is broken enough to make partitioning not work. I can think of inlining where the count gets scaled all way down to 0. Perhaps count scaling code can be modified to never round towards 0 for block executing non-0 times... Honza > > Thanks, > Teresa > > > I think we are really looking primarily for dead parts of the functions (sanity checks/error handling) > > that should not be visited by train run. We can then see how to make the heuristic more aggressive? > > > > Honza > > > > -- > Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-09 9:58 ` Jan Hubicka @ 2013-08-09 14:38 ` Teresa Johnson 2013-08-09 15:28 ` Jan Hubicka 0 siblings, 1 reply; 62+ messages in thread From: Teresa Johnson @ 2013-08-09 14:38 UTC (permalink / raw) To: Jan Hubicka Cc: Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, marxin.liska, Sriraman Tallam On Fri, Aug 9, 2013 at 2:58 AM, Jan Hubicka <hubicka@ucw.cz> wrote: >> On Thu, Aug 8, 2013 at 3:23 PM, Jan Hubicka <hubicka@ucw.cz> wrote: >> > Hi, >> > Martin Liska was kind enough to generate disk seeking graph of gimp statrup with his function reordering. >> > His code simply measures time of firest execution of a function and orders functions in the given order. >> > The functions stay in the subsections (unlikely/startup/exit/hot/normal) that are then glued together >> > in this order. >> > >> > I am attaching disk seeking with and without -freorder-blocks-and-partition (with your patch). >> > >> > In 2.pdf you can see two increasing sequences in the text segment. If I am not mistaken the bottom >> > one comes for hot and the top one for normal section. The big unused part on bottom is unlikely >> > section since most of gimp is not trained. >> >> 2.pdf is reordered with Martin's technique? >> >> > >> > Now 1.pdf is with -freorder-blocks-and-partition and your patch. You can see there is third sequence >> > near bottom of the text seciton. that is beggining of unlikely section, so it tracks jumps where we >> > fall into cold section of function. >> >> 1.pdf is generated using the usual FDO + >> -freorder-blocks-and-partition (i.e. not using Martin's technique)? > > 2.pdf is Martin's reordering (that works ortoghonally to what we already have - > it just orders the functions inside idividual subsections. This make the > subsections more visible than without his patch). > 1.pdf is Marting's rerodering + yours patch (I asked him to double check it is > the latest verision) + -freorder-blocks-and-partition. > > He simply trains and measures the gimp startup, nothing else, so there should not > be problem with representativity of the data. Ok, so a single training run, and it is essentially the same as what is being used to create the graph after optimization. >> >> > >> > It still seems rather bad (i.e. good part of unlikely section is actually used). I think the dominator >> > based approach is not going to work too reliably (I can "fix" my testcase to contain multiple nested >> > conditionals and then the heuristic about predecestors won't help). >> >> Yes, this doesn't look good. Did you use the latest version of my >> patch that doesn't walk the dominators? >> >> Do you know how many training runs are done for this benchmark? I >> think a lot of the issues that you pointed out with the hot loop >> preceded by non-looping conditional code as in your earlier example, >> or multiple nested conditionals, comes from the fact that the cold >> cutoff is not 0, but some number less than the number of training >> runs. Perhaps the cutoff for splitting should be 0. Then the main >> issue that needs to be corrected is profile insanities, not code that >> is executed once (since that would not be marked cold). > > Hmm, compute_function_frequency uses probably_never_executed_bb_p that requires > the count of basic block to be less than number of train runs. In Martin's > setup that will be 0. > > This is the same as what -frerorder-block-of-partition does? Right, it simply puts blocks that are probably_never_executed_bb_p into the cold section. But it sounds like that is not an issue here since Martin is doing a single training run so the cutoff is essentially 0. >> >> The only other issue that I can think of here is that the training >> data was not representative and didn't execute these blocks. >> >> > >> > What about simply walking the CFG from entry through all edges with non-0 counts and making all reachable >> > blocks hot + forcingly make any hot blocks not reachable this way reachable? >> >> Is this different than what I currently have + changing the cold >> cutoff to 0? In that case any blocks reachable through non-0 edges >> should be non-0 and marked hot, and the current patch forces the most >> frequent paths to all hot blocks to be hot. > > Do we sanity check that the cold partition does not contain any blocks of > count 0? It may be that the profile is broken enough to make partitioning > not work. Do you mean sanity check that the cold partition does not contain any blocks of count > 0? (they should all be zero) I don't think that sanity check is there, but I can try adding that. The issue with such a sanity check may be due to the later fixup I have in this patch (fixup_partitions). It is invoked after certain optimizations on the cfg that may make hot blocks previously reached by both hot and cold edges only reachable by cold blocks. These blocks are remarked cold. If the profile data hasn't been updated correctly it is possible that they would still have a non-0 count, although they are essentially cold after the cfg transformation. But certainly such a sanity check should always succeed after the original partitioning. > I can think of inlining where the count gets scaled all way down to 0. Perhaps > count scaling code can be modified to never round towards 0 for block executing > non-0 times... This reminds me of why this situation could happen. When I have been testing this on the google branch I found situations where COMDAT routines have 0 profile counts (if I remember correctly, this happens when profile-gen binary has call to out-of-line copy of COMDAT in module A, linker chooses the out-of-line copy from module B, therefore the profile data for COMDAT in module A is 0). When the COMDAT gets inlined, the 0 counts on its bbs are scaled to 0, even though the callsite is non-zero. I have a patch that I was planning to send as a follow-up that handles this case by propagating the callsite bb's count to the inlined code when it has 0 counts, scaling by the edge frequencies. I can either include that patch in this one, or send it for review separately right now. Do you want to give it a try with this one to see if it addresses the issue? Also, can you send me reproduction instructions for gimp? I don't think I need Martin's patch, but which version of gimp and what is the equivalent way for me to train it? I have some scripts to generate a similar type of instruction heat map graph that I have been using to tune partitioning and function reordering. Essentially it uses linux perf to sample on instructions_retired and then munge the data in several ways to produce various stats and graphs. One thing that has been useful has been to combine the perf data with nm output to determine which cold functions are being executed at runtime. However, for this to tell me which split cold bbs are being executed I need to use a patch that Sri sent for review several months back that gives the split cold section its own name: http://gcc.gnu.org/ml/gcc-patches/2013-04/msg01571.html Steven had some follow up comments that Sri hasn't had a chance to address yet: http://gcc.gnu.org/ml/gcc-patches/2013-05/msg00798.html (cc'ing Sri as we should probably revive this patch soon to address gdb and other issues with detecting split functions properly) Thanks! Teresa > > Honza >> >> Thanks, >> Teresa >> >> > I think we are really looking primarily for dead parts of the functions (sanity checks/error handling) >> > that should not be visited by train run. We can then see how to make the heuristic more aggressive? >> > >> > Honza >> >> >> >> -- >> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-09 14:38 ` Teresa Johnson @ 2013-08-09 15:28 ` Jan Hubicka 2013-08-09 15:54 ` Martin Liška 2013-08-09 21:02 ` Teresa Johnson 0 siblings, 2 replies; 62+ messages in thread From: Jan Hubicka @ 2013-08-09 15:28 UTC (permalink / raw) To: Teresa Johnson Cc: Jan Hubicka, Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, marxin.liska, Sriraman Tallam > > Do we sanity check that the cold partition does not contain any blocks of > > count 0? It may be that the profile is broken enough to make partitioning > > not work. > > Do you mean sanity check that the cold partition does not contain any > blocks of count > 0? (they should all be zero) I don't think that > sanity check is there, but I can try adding that. Thanks, lets start with this - I suppose we need to figure out if 1) the reachable blocks goes to cold section because partitioning decides so even if they have non-0 count. 2) the reachable blocks goes to cold section because they have incorrectly updated count to 0 by someone 3) profiling gets some blocks wrong. > > The issue with such a sanity check may be due to the later fixup I > have in this patch (fixup_partitions). It is invoked after certain > optimizations on the cfg that may make hot blocks previously reached > by both hot and cold edges only reachable by cold blocks. These blocks > are remarked cold. If the profile data hasn't been updated correctly > it is possible that they would still have a non-0 count, although they > are essentially cold after the cfg transformation. Well, or the other posibility is that the edges was updated wrong and the blocks are really cold. We need to figure out if that happens commonly enough. I will try to think of some artificial testcases. > > But certainly such a sanity check should always succeed after the > original partitioning. > > > I can think of inlining where the count gets scaled all way down to 0. Perhaps > > count scaling code can be modified to never round towards 0 for block executing > > non-0 times... > > This reminds me of why this situation could happen. When I have been > testing this on the google branch I found situations where COMDAT > routines have 0 profile counts (if I remember correctly, this happens > when profile-gen binary has call to out-of-line copy of COMDAT in > module A, linker chooses the out-of-line copy from module B, therefore > the profile data for COMDAT in module A is 0). When the COMDAT gets > inlined, the 0 counts on its bbs are scaled to 0, even though the > callsite is non-zero. I have a patch that I was planning to send as a > follow-up that handles this case by propagating the callsite bb's > count to the inlined code when it has 0 counts, scaling by the edge > frequencies. I can either include that patch in this one, or send it > for review separately right now. Do you want to give it a try with > this one to see if it addresses the issue? This scenario should not happen with LTO setup: the LTO symbol tables contains code before early optimization and should be identical with profiling or without (modulo the new references and call from profile code). But this patch seems useful as a backup solution for non-LTO, so yes, please send it separately and I can try to double check that it really do not happen with LTO. (acutally LTO symtab may just chose COMDAT from module that has counts with it. It has all the info for it. I was thinkin about it few weeks back. It is bit hard to do - you need to verify that all references from the function are the same or linking might fail if you overwrite linker's decisiosns). > > Also, can you send me reproduction instructions for gimp? I don't > think I need Martin's patch, but which version of gimp and what is the > equivalent way for me to train it? I have some scripts to generate a > similar type of instruction heat map graph that I have been using to > tune partitioning and function reordering. Essentially it uses linux > perf to sample on instructions_retired and then munge the data in > several ways to produce various stats and graphs. One thing that has > been useful has been to combine the perf data with nm output to > determine which cold functions are being executed at runtime. Martin? > > However, for this to tell me which split cold bbs are being executed I > need to use a patch that Sri sent for review several months back that > gives the split cold section its own name: > http://gcc.gnu.org/ml/gcc-patches/2013-04/msg01571.html > Steven had some follow up comments that Sri hasn't had a chance to address yet: > http://gcc.gnu.org/ml/gcc-patches/2013-05/msg00798.html > (cc'ing Sri as we should probably revive this patch soon to address > gdb and other issues with detecting split functions properly) Intresting, I used linker script for this purposes, but that his GNU ld only... Honza > > Thanks! > Teresa > > > > > Honza > >> > >> Thanks, > >> Teresa > >> > >> > I think we are really looking primarily for dead parts of the functions (sanity checks/error handling) > >> > that should not be visited by train run. We can then see how to make the heuristic more aggressive? > >> > > >> > Honza > >> > >> > >> > >> -- > >> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 > > > > -- > Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-09 15:28 ` Jan Hubicka @ 2013-08-09 15:54 ` Martin Liška 2013-08-09 21:03 ` Teresa Johnson 2013-08-09 21:02 ` Teresa Johnson 1 sibling, 1 reply; 62+ messages in thread From: Martin Liška @ 2013-08-09 15:54 UTC (permalink / raw) To: Jan Hubicka Cc: Teresa Johnson, Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, Sriraman Tallam Hi On 9 August 2013 17:28, Jan Hubicka <hubicka@ucw.cz> wrote: >> > Do we sanity check that the cold partition does not contain any blocks of >> > count 0? It may be that the profile is broken enough to make partitioning >> > not work. >> >> Do you mean sanity check that the cold partition does not contain any >> blocks of count > 0? (they should all be zero) I don't think that >> sanity check is there, but I can try adding that. > > Thanks, lets start with this - I suppose we need to figure out if > 1) the reachable blocks goes to cold section because partitioning decides > so even if they have non-0 count. > 2) the reachable blocks goes to cold section because they have incorrectly > updated count to 0 by someone > 3) profiling gets some blocks wrong. >> >> The issue with such a sanity check may be due to the later fixup I >> have in this patch (fixup_partitions). It is invoked after certain >> optimizations on the cfg that may make hot blocks previously reached >> by both hot and cold edges only reachable by cold blocks. These blocks >> are remarked cold. If the profile data hasn't been updated correctly >> it is possible that they would still have a non-0 count, although they >> are essentially cold after the cfg transformation. > > Well, or the other posibility is that the edges was updated wrong > and the blocks are really cold. We need to figure out if that happens > commonly enough. > > I will try to think of some artificial testcases. > >> >> But certainly such a sanity check should always succeed after the >> original partitioning. >> >> > I can think of inlining where the count gets scaled all way down to 0. Perhaps >> > count scaling code can be modified to never round towards 0 for block executing >> > non-0 times... >> >> This reminds me of why this situation could happen. When I have been >> testing this on the google branch I found situations where COMDAT >> routines have 0 profile counts (if I remember correctly, this happens >> when profile-gen binary has call to out-of-line copy of COMDAT in >> module A, linker chooses the out-of-line copy from module B, therefore >> the profile data for COMDAT in module A is 0). When the COMDAT gets >> inlined, the 0 counts on its bbs are scaled to 0, even though the >> callsite is non-zero. I have a patch that I was planning to send as a >> follow-up that handles this case by propagating the callsite bb's >> count to the inlined code when it has 0 counts, scaling by the edge >> frequencies. I can either include that patch in this one, or send it >> for review separately right now. Do you want to give it a try with >> this one to see if it addresses the issue? > > This scenario should not happen with LTO setup: the LTO symbol tables contains > code before early optimization and should be identical with profiling or > without (modulo the new references and call from profile code). > > But this patch seems useful as a backup solution for non-LTO, so yes, please > send it separately and I can try to double check that it really do not happen > with LTO. > (acutally LTO symtab may just chose COMDAT from module that has counts with it. > It has all the info for it. I was thinkin about it few weeks back. It is > bit hard to do - you need to verify that all references from the function are > the same or linking might fail if you overwrite linker's decisiosns). >> >> Also, can you send me reproduction instructions for gimp? I don't >> think I need Martin's patch, but which version of gimp and what is the >> equivalent way for me to train it? I have some scripts to generate a >> similar type of instruction heat map graph that I have been using to >> tune partitioning and function reordering. Essentially it uses linux >> perf to sample on instructions_retired and then munge the data in >> several ways to produce various stats and graphs. One thing that has >> been useful has been to combine the perf data with nm output to >> determine which cold functions are being executed at runtime. > > Martin? I use gimp from git repository, commit: 88ecd59c3436d302b644a5d25c1938c0e7b60ae0 (from Fet 5 2013) Link: http://www.gimp.org/source/#gimp_from_git Martin >> >> However, for this to tell me which split cold bbs are being executed I >> need to use a patch that Sri sent for review several months back that >> gives the split cold section its own name: >> http://gcc.gnu.org/ml/gcc-patches/2013-04/msg01571.html >> Steven had some follow up comments that Sri hasn't had a chance to address yet: >> http://gcc.gnu.org/ml/gcc-patches/2013-05/msg00798.html >> (cc'ing Sri as we should probably revive this patch soon to address >> gdb and other issues with detecting split functions properly) > > Intresting, I used linker script for this purposes, but that his GNU ld only... > > Honza >> >> Thanks! >> Teresa >> >> > >> > Honza >> >> >> >> Thanks, >> >> Teresa >> >> >> >> > I think we are really looking primarily for dead parts of the functions (sanity checks/error handling) >> >> > that should not be visited by train run. We can then see how to make the heuristic more aggressive? >> >> > >> >> > Honza >> >> >> >> >> >> >> >> -- >> >> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 >> >> >> >> -- >> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-09 15:54 ` Martin Liška @ 2013-08-09 21:03 ` Teresa Johnson 0 siblings, 0 replies; 62+ messages in thread From: Teresa Johnson @ 2013-08-09 21:03 UTC (permalink / raw) To: Martin Liška Cc: Jan Hubicka, Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, Sriraman Tallam On Fri, Aug 9, 2013 at 8:54 AM, Martin Liška <marxin.liska@gmail.com> wrote: > Hi > > On 9 August 2013 17:28, Jan Hubicka <hubicka@ucw.cz> wrote: >>> > Do we sanity check that the cold partition does not contain any blocks of >>> > count 0? It may be that the profile is broken enough to make partitioning >>> > not work. >>> >>> Do you mean sanity check that the cold partition does not contain any >>> blocks of count > 0? (they should all be zero) I don't think that >>> sanity check is there, but I can try adding that. >> >> Thanks, lets start with this - I suppose we need to figure out if >> 1) the reachable blocks goes to cold section because partitioning decides >> so even if they have non-0 count. >> 2) the reachable blocks goes to cold section because they have incorrectly >> updated count to 0 by someone >> 3) profiling gets some blocks wrong. >>> >>> The issue with such a sanity check may be due to the later fixup I >>> have in this patch (fixup_partitions). It is invoked after certain >>> optimizations on the cfg that may make hot blocks previously reached >>> by both hot and cold edges only reachable by cold blocks. These blocks >>> are remarked cold. If the profile data hasn't been updated correctly >>> it is possible that they would still have a non-0 count, although they >>> are essentially cold after the cfg transformation. >> >> Well, or the other posibility is that the edges was updated wrong >> and the blocks are really cold. We need to figure out if that happens >> commonly enough. >> >> I will try to think of some artificial testcases. >> >>> >>> But certainly such a sanity check should always succeed after the >>> original partitioning. >>> >>> > I can think of inlining where the count gets scaled all way down to 0. Perhaps >>> > count scaling code can be modified to never round towards 0 for block executing >>> > non-0 times... >>> >>> This reminds me of why this situation could happen. When I have been >>> testing this on the google branch I found situations where COMDAT >>> routines have 0 profile counts (if I remember correctly, this happens >>> when profile-gen binary has call to out-of-line copy of COMDAT in >>> module A, linker chooses the out-of-line copy from module B, therefore >>> the profile data for COMDAT in module A is 0). When the COMDAT gets >>> inlined, the 0 counts on its bbs are scaled to 0, even though the >>> callsite is non-zero. I have a patch that I was planning to send as a >>> follow-up that handles this case by propagating the callsite bb's >>> count to the inlined code when it has 0 counts, scaling by the edge >>> frequencies. I can either include that patch in this one, or send it >>> for review separately right now. Do you want to give it a try with >>> this one to see if it addresses the issue? >> >> This scenario should not happen with LTO setup: the LTO symbol tables contains >> code before early optimization and should be identical with profiling or >> without (modulo the new references and call from profile code). >> >> But this patch seems useful as a backup solution for non-LTO, so yes, please >> send it separately and I can try to double check that it really do not happen >> with LTO. >> (acutally LTO symtab may just chose COMDAT from module that has counts with it. >> It has all the info for it. I was thinkin about it few weeks back. It is >> bit hard to do - you need to verify that all references from the function are >> the same or linking might fail if you overwrite linker's decisiosns). >>> >>> Also, can you send me reproduction instructions for gimp? I don't >>> think I need Martin's patch, but which version of gimp and what is the >>> equivalent way for me to train it? I have some scripts to generate a >>> similar type of instruction heat map graph that I have been using to >>> tune partitioning and function reordering. Essentially it uses linux >>> perf to sample on instructions_retired and then munge the data in >>> several ways to produce various stats and graphs. One thing that has >>> been useful has been to combine the perf data with nm output to >>> determine which cold functions are being executed at runtime. >> >> Martin? > > I use gimp from git repository, commit: > 88ecd59c3436d302b644a5d25c1938c0e7b60ae0 (from Fet 5 2013) > Link: http://www.gimp.org/source/#gimp_from_git Thanks. Were you building with LTO? And just -O2, or any other options I should use? Teresa > > Martin > >>> >>> However, for this to tell me which split cold bbs are being executed I >>> need to use a patch that Sri sent for review several months back that >>> gives the split cold section its own name: >>> http://gcc.gnu.org/ml/gcc-patches/2013-04/msg01571.html >>> Steven had some follow up comments that Sri hasn't had a chance to address yet: >>> http://gcc.gnu.org/ml/gcc-patches/2013-05/msg00798.html >>> (cc'ing Sri as we should probably revive this patch soon to address >>> gdb and other issues with detecting split functions properly) >> >> Intresting, I used linker script for this purposes, but that his GNU ld only... >> >> Honza >>> >>> Thanks! >>> Teresa >>> >>> > >>> > Honza >>> >> >>> >> Thanks, >>> >> Teresa >>> >> >>> >> > I think we are really looking primarily for dead parts of the functions (sanity checks/error handling) >>> >> > that should not be visited by train run. We can then see how to make the heuristic more aggressive? >>> >> > >>> >> > Honza >>> >> >>> >> >>> >> >>> >> -- >>> >> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 >>> >>> >>> >>> -- >>> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-09 15:28 ` Jan Hubicka 2013-08-09 15:54 ` Martin Liška @ 2013-08-09 21:02 ` Teresa Johnson 2013-08-09 22:43 ` Jan Hubicka ` (2 more replies) 1 sibling, 3 replies; 62+ messages in thread From: Teresa Johnson @ 2013-08-09 21:02 UTC (permalink / raw) To: Jan Hubicka Cc: Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, marxin.liska, Sriraman Tallam On Fri, Aug 9, 2013 at 8:28 AM, Jan Hubicka <hubicka@ucw.cz> wrote: >> > Do we sanity check that the cold partition does not contain any blocks of >> > count 0? It may be that the profile is broken enough to make partitioning >> > not work. >> >> Do you mean sanity check that the cold partition does not contain any >> blocks of count > 0? (they should all be zero) I don't think that >> sanity check is there, but I can try adding that. > > Thanks, lets start with this - I suppose we need to figure out if > 1) the reachable blocks goes to cold section because partitioning decides > so even if they have non-0 count. Right, this should be easy enough to check and should hopefully never happen. > 2) the reachable blocks goes to cold section because they have incorrectly > updated count to 0 by someone A sanity check should find this too. But it can happen now for various reasons like the comdat issue I described below. But it will be good to flag these and fix them. > 3) profiling gets some blocks wrong. This is the one that will be tough to fix, if the training run isn't representative. >> >> The issue with such a sanity check may be due to the later fixup I >> have in this patch (fixup_partitions). It is invoked after certain >> optimizations on the cfg that may make hot blocks previously reached >> by both hot and cold edges only reachable by cold blocks. These blocks >> are remarked cold. If the profile data hasn't been updated correctly >> it is possible that they would still have a non-0 count, although they >> are essentially cold after the cfg transformation. > > Well, or the other posibility is that the edges was updated wrong > and the blocks are really cold. We need to figure out if that happens > commonly enough. > > I will try to think of some artificial testcases. > >> >> But certainly such a sanity check should always succeed after the >> original partitioning. >> >> > I can think of inlining where the count gets scaled all way down to 0. Perhaps >> > count scaling code can be modified to never round towards 0 for block executing >> > non-0 times... >> >> This reminds me of why this situation could happen. When I have been >> testing this on the google branch I found situations where COMDAT >> routines have 0 profile counts (if I remember correctly, this happens >> when profile-gen binary has call to out-of-line copy of COMDAT in >> module A, linker chooses the out-of-line copy from module B, therefore >> the profile data for COMDAT in module A is 0). When the COMDAT gets >> inlined, the 0 counts on its bbs are scaled to 0, even though the >> callsite is non-zero. I have a patch that I was planning to send as a >> follow-up that handles this case by propagating the callsite bb's >> count to the inlined code when it has 0 counts, scaling by the edge >> frequencies. I can either include that patch in this one, or send it >> for review separately right now. Do you want to give it a try with >> this one to see if it addresses the issue? > > This scenario should not happen with LTO setup: the LTO symbol tables contains > code before early optimization and should be identical with profiling or > without (modulo the new references and call from profile code). > > But this patch seems useful as a backup solution for non-LTO, so yes, please > send it separately and I can try to double check that it really do not happen > with LTO. > (acutally LTO symtab may just chose COMDAT from module that has counts with it. > It has all the info for it. I was thinkin about it few weeks back. It is > bit hard to do - you need to verify that all references from the function are > the same or linking might fail if you overwrite linker's decisiosns). I see, yes LTO can deal with this better since it has global information. In non-LTO mode (including LIPO) we have the issue. I take it gimp is built with LTO and therefore shouldn't be hitting this comdat issue? Let me do a couple things: - port over my comdat inlining fix from the google branch to trunk and send it for review. If you or Martin could try it to see if it helps with function splitting to avoid the hits from the cold code that would be great - I'll add some new sanity checking to try to detect non-zero blocks in the cold section, or 0 blocks reached by non-zero edges and see if I can flush out any problems with my tests or a profiledbootstrap or gimp. - I'll try building and profiling gimp myself to see if I can reproduce the issue with code executing out of the cold section. Thanks, Teresa >> >> Also, can you send me reproduction instructions for gimp? I don't >> think I need Martin's patch, but which version of gimp and what is the >> equivalent way for me to train it? I have some scripts to generate a >> similar type of instruction heat map graph that I have been using to >> tune partitioning and function reordering. Essentially it uses linux >> perf to sample on instructions_retired and then munge the data in >> several ways to produce various stats and graphs. One thing that has >> been useful has been to combine the perf data with nm output to >> determine which cold functions are being executed at runtime. > > Martin? > >> >> However, for this to tell me which split cold bbs are being executed I >> need to use a patch that Sri sent for review several months back that >> gives the split cold section its own name: >> http://gcc.gnu.org/ml/gcc-patches/2013-04/msg01571.html >> Steven had some follow up comments that Sri hasn't had a chance to address yet: >> http://gcc.gnu.org/ml/gcc-patches/2013-05/msg00798.html >> (cc'ing Sri as we should probably revive this patch soon to address >> gdb and other issues with detecting split functions properly) > > Intresting, I used linker script for this purposes, but that his GNU ld only... > > Honza >> >> Thanks! >> Teresa >> >> > >> > Honza >> >> >> >> Thanks, >> >> Teresa >> >> >> >> > I think we are really looking primarily for dead parts of the functions (sanity checks/error handling) >> >> > that should not be visited by train run. We can then see how to make the heuristic more aggressive? >> >> > >> >> > Honza >> >> >> >> >> >> >> >> -- >> >> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 >> >> >> >> -- >> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-09 21:02 ` Teresa Johnson @ 2013-08-09 22:43 ` Jan Hubicka 2013-08-11 12:21 ` Jan Hubicka 2013-08-17 15:54 ` Teresa Johnson 2 siblings, 0 replies; 62+ messages in thread From: Jan Hubicka @ 2013-08-09 22:43 UTC (permalink / raw) To: Teresa Johnson Cc: Jan Hubicka, Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, marxin.liska, Sriraman Tallam > > I see, yes LTO can deal with this better since it has global > information. In non-LTO mode (including LIPO) we have the issue. Thinking about it, there is still one problem left: I usually suggest users to train with -fno-lto to avoid excessive linking time with instrumentation. This actually will bring differences in between compile and link decisions of linker. We can't even replace one body by another, since inlining done at training stage may distribute the profile across multiple comdat copies. I suppose only way is to extend lto-symtab to actually merge profiles. To merge CFG profiles it will need to read in the body. Martin is working on identical function merging, I suppose once we implement CFG profile merging for that purpose, we can use it here. I will try to look into this. > > I take it gimp is built with LTO and therefore shouldn't be hitting > this comdat issue? I think gimp is still mostly C program. (did not double check my last contribution to it is from 90s :) > > Let me do a couple things: > - port over my comdat inlining fix from the google branch to trunk and > send it for review. If you or Martin could try it to see if it helps > with function splitting to avoid the hits from the cold code that > would be great > - I'll add some new sanity checking to try to detect non-zero blocks > in the cold section, or 0 blocks reached by non-zero edges and see if > I can flush out any problems with my tests or a profiledbootstrap or > gimp. > - I'll try building and profiling gimp myself to see if I can > reproduce the issue with code executing out of the cold section. > > Thanks, > Teresa > > >> > >> Also, can you send me reproduction instructions for gimp? I don't > >> think I need Martin's patch, but which version of gimp and what is the > >> equivalent way for me to train it? I have some scripts to generate a > >> similar type of instruction heat map graph that I have been using to > >> tune partitioning and function reordering. Essentially it uses linux > >> perf to sample on instructions_retired and then munge the data in > >> several ways to produce various stats and graphs. One thing that has > >> been useful has been to combine the perf data with nm output to > >> determine which cold functions are being executed at runtime. > > > > Martin? > > > >> > >> However, for this to tell me which split cold bbs are being executed I > >> need to use a patch that Sri sent for review several months back that > >> gives the split cold section its own name: > >> http://gcc.gnu.org/ml/gcc-patches/2013-04/msg01571.html > >> Steven had some follow up comments that Sri hasn't had a chance to address yet: > >> http://gcc.gnu.org/ml/gcc-patches/2013-05/msg00798.html > >> (cc'ing Sri as we should probably revive this patch soon to address > >> gdb and other issues with detecting split functions properly) > > > > Intresting, I used linker script for this purposes, but that his GNU ld only... > > > > Honza > >> > >> Thanks! > >> Teresa > >> > >> > > >> > Honza > >> >> > >> >> Thanks, > >> >> Teresa > >> >> > >> >> > I think we are really looking primarily for dead parts of the functions (sanity checks/error handling) > >> >> > that should not be visited by train run. We can then see how to make the heuristic more aggressive? > >> >> > > >> >> > Honza > >> >> > >> >> > >> >> > >> >> -- > >> >> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 > >> > >> > >> > >> -- > >> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 > > > > -- > Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-09 21:02 ` Teresa Johnson 2013-08-09 22:43 ` Jan Hubicka @ 2013-08-11 12:21 ` Jan Hubicka 2013-08-11 13:25 ` Teresa Johnson 2013-08-17 15:54 ` Teresa Johnson 2 siblings, 1 reply; 62+ messages in thread From: Jan Hubicka @ 2013-08-11 12:21 UTC (permalink / raw) To: Teresa Johnson Cc: Jan Hubicka, Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, marxin.liska, Sriraman Tallam > > I see, yes LTO can deal with this better since it has global > information. In non-LTO mode (including LIPO) we have the issue. Either Martin or me will implement merging of the multiple copies at LTO link time. This is needed for Martin's code unification patch anyway. Theoretically gcov runtime can also have symbol names and cfg checksums of comdats in the static data and at exit produce buckets based on matching names+checksums+counter counts, merge all data into in each bucket to one representative by the existing merging routines and then memcpy them to all the oriignal copiles. This way all compilation units will receive same results. I am not very keen about making gcov runtime bigger and more complex than it needs to be, but having sane profile for comdats seems quite important. Perhaps, in GNU toolchain, ordered subsections can be used to make linker to produce ordered list of comdats, so the runtime won't need to do hashing + lookups. Honza > > I take it gimp is built with LTO and therefore shouldn't be hitting > this comdat issue? > > Let me do a couple things: > - port over my comdat inlining fix from the google branch to trunk and > send it for review. If you or Martin could try it to see if it helps > with function splitting to avoid the hits from the cold code that > would be great > - I'll add some new sanity checking to try to detect non-zero blocks > in the cold section, or 0 blocks reached by non-zero edges and see if > I can flush out any problems with my tests or a profiledbootstrap or > gimp. > - I'll try building and profiling gimp myself to see if I can > reproduce the issue with code executing out of the cold section. > > Thanks, > Teresa > > >> > >> Also, can you send me reproduction instructions for gimp? I don't > >> think I need Martin's patch, but which version of gimp and what is the > >> equivalent way for me to train it? I have some scripts to generate a > >> similar type of instruction heat map graph that I have been using to > >> tune partitioning and function reordering. Essentially it uses linux > >> perf to sample on instructions_retired and then munge the data in > >> several ways to produce various stats and graphs. One thing that has > >> been useful has been to combine the perf data with nm output to > >> determine which cold functions are being executed at runtime. > > > > Martin? > > > >> > >> However, for this to tell me which split cold bbs are being executed I > >> need to use a patch that Sri sent for review several months back that > >> gives the split cold section its own name: > >> http://gcc.gnu.org/ml/gcc-patches/2013-04/msg01571.html > >> Steven had some follow up comments that Sri hasn't had a chance to address yet: > >> http://gcc.gnu.org/ml/gcc-patches/2013-05/msg00798.html > >> (cc'ing Sri as we should probably revive this patch soon to address > >> gdb and other issues with detecting split functions properly) > > > > Intresting, I used linker script for this purposes, but that his GNU ld only... > > > > Honza > >> > >> Thanks! > >> Teresa > >> > >> > > >> > Honza > >> >> > >> >> Thanks, > >> >> Teresa > >> >> > >> >> > I think we are really looking primarily for dead parts of the functions (sanity checks/error handling) > >> >> > that should not be visited by train run. We can then see how to make the heuristic more aggressive? > >> >> > > >> >> > Honza > >> >> > >> >> > >> >> > >> >> -- > >> >> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 > >> > >> > >> > >> -- > >> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 > > > > -- > Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-11 12:21 ` Jan Hubicka @ 2013-08-11 13:25 ` Teresa Johnson 2013-08-11 15:55 ` Martin Liška 2013-08-11 21:05 ` Jan Hubicka 0 siblings, 2 replies; 62+ messages in thread From: Teresa Johnson @ 2013-08-11 13:25 UTC (permalink / raw) To: Jan Hubicka Cc: Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, marxin.liska, Sriraman Tallam, Rong Xu Cc'ing Rong since he is also working on trying to address the comdat profile issue. Rong, you may need to see an earlier message for more context: http://gcc.gnu.org/ml/gcc-patches/2013-08/msg00558.html Teresa On Sun, Aug 11, 2013 at 5:21 AM, Jan Hubicka <hubicka@ucw.cz> wrote: >> >> I see, yes LTO can deal with this better since it has global >> information. In non-LTO mode (including LIPO) we have the issue. > > Either Martin or me will implement merging of the multiple copies at > LTO link time. This is needed for Martin's code unification patch anyway. > > Theoretically gcov runtime can also have symbol names and cfg checksums of > comdats in the static data and at exit produce buckets based on matching > names+checksums+counter counts, merge all data into in each bucket to one > representative by the existing merging routines and then memcpy them to > all the oriignal copiles. This way all compilation units will receive same > results. > > I am not very keen about making gcov runtime bigger and more complex than it > needs to be, but having sane profile for comdats seems quite important. > Perhaps, in GNU toolchain, ordered subsections can be used to make linker to > produce ordered list of comdats, so the runtime won't need to do hashing + > lookups. > > Honza >> >> I take it gimp is built with LTO and therefore shouldn't be hitting >> this comdat issue? >> >> Let me do a couple things: >> - port over my comdat inlining fix from the google branch to trunk and >> send it for review. If you or Martin could try it to see if it helps >> with function splitting to avoid the hits from the cold code that >> would be great >> - I'll add some new sanity checking to try to detect non-zero blocks >> in the cold section, or 0 blocks reached by non-zero edges and see if >> I can flush out any problems with my tests or a profiledbootstrap or >> gimp. >> - I'll try building and profiling gimp myself to see if I can >> reproduce the issue with code executing out of the cold section. >> >> Thanks, >> Teresa >> >> >> >> >> Also, can you send me reproduction instructions for gimp? I don't >> >> think I need Martin's patch, but which version of gimp and what is the >> >> equivalent way for me to train it? I have some scripts to generate a >> >> similar type of instruction heat map graph that I have been using to >> >> tune partitioning and function reordering. Essentially it uses linux >> >> perf to sample on instructions_retired and then munge the data in >> >> several ways to produce various stats and graphs. One thing that has >> >> been useful has been to combine the perf data with nm output to >> >> determine which cold functions are being executed at runtime. >> > >> > Martin? >> > >> >> >> >> However, for this to tell me which split cold bbs are being executed I >> >> need to use a patch that Sri sent for review several months back that >> >> gives the split cold section its own name: >> >> http://gcc.gnu.org/ml/gcc-patches/2013-04/msg01571.html >> >> Steven had some follow up comments that Sri hasn't had a chance to address yet: >> >> http://gcc.gnu.org/ml/gcc-patches/2013-05/msg00798.html >> >> (cc'ing Sri as we should probably revive this patch soon to address >> >> gdb and other issues with detecting split functions properly) >> > >> > Intresting, I used linker script for this purposes, but that his GNU ld only... >> > >> > Honza >> >> >> >> Thanks! >> >> Teresa >> >> >> >> > >> >> > Honza >> >> >> >> >> >> Thanks, >> >> >> Teresa >> >> >> >> >> >> > I think we are really looking primarily for dead parts of the functions (sanity checks/error handling) >> >> >> > that should not be visited by train run. We can then see how to make the heuristic more aggressive? >> >> >> > >> >> >> > Honza >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 >> >> >> >> >> >> >> >> -- >> >> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 >> >> >> >> -- >> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-11 13:25 ` Teresa Johnson @ 2013-08-11 15:55 ` Martin Liška 2013-08-11 17:55 ` Jan Hubicka 2013-08-11 21:05 ` Jan Hubicka 1 sibling, 1 reply; 62+ messages in thread From: Martin Liška @ 2013-08-11 15:55 UTC (permalink / raw) To: Teresa Johnson Cc: Jan Hubicka, Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, Sriraman Tallam, Rong Xu [-- Attachment #1: Type: text/plain, Size: 4934 bytes --] Hello, I did a collection of systemtap graphs for GIMP. All these graphs were created with enabled LTO, profiling and -O2. 1) gimp-reordered.pdf - function are reordered according to my newly created profile that utilizes LTO infrastructure 2) gimp-no-top-level-reorder.pdf - (GCC rev. 201648) -fno-top-level-reorder 3) gimp-top-level-reorder.pdf - (GCC rev. 201648) -ftop-level-reorder Honza has an idea how to minimize hot text section and I will send new graphs for the proposed patch. Moreover, I will send graphs for Inkscape which is written in C++. Have a nice day, Martin On 11 August 2013 15:25, Teresa Johnson <tejohnson@google.com> wrote: > Cc'ing Rong since he is also working on trying to address the comdat > profile issue. Rong, you may need to see an earlier message for more > context: > http://gcc.gnu.org/ml/gcc-patches/2013-08/msg00558.html > > Teresa > > On Sun, Aug 11, 2013 at 5:21 AM, Jan Hubicka <hubicka@ucw.cz> wrote: >>> >>> I see, yes LTO can deal with this better since it has global >>> information. In non-LTO mode (including LIPO) we have the issue. >> >> Either Martin or me will implement merging of the multiple copies at >> LTO link time. This is needed for Martin's code unification patch anyway. >> >> Theoretically gcov runtime can also have symbol names and cfg checksums of >> comdats in the static data and at exit produce buckets based on matching >> names+checksums+counter counts, merge all data into in each bucket to one >> representative by the existing merging routines and then memcpy them to >> all the oriignal copiles. This way all compilation units will receive same >> results. >> >> I am not very keen about making gcov runtime bigger and more complex than it >> needs to be, but having sane profile for comdats seems quite important. >> Perhaps, in GNU toolchain, ordered subsections can be used to make linker to >> produce ordered list of comdats, so the runtime won't need to do hashing + >> lookups. >> >> Honza >>> >>> I take it gimp is built with LTO and therefore shouldn't be hitting >>> this comdat issue? >>> >>> Let me do a couple things: >>> - port over my comdat inlining fix from the google branch to trunk and >>> send it for review. If you or Martin could try it to see if it helps >>> with function splitting to avoid the hits from the cold code that >>> would be great >>> - I'll add some new sanity checking to try to detect non-zero blocks >>> in the cold section, or 0 blocks reached by non-zero edges and see if >>> I can flush out any problems with my tests or a profiledbootstrap or >>> gimp. >>> - I'll try building and profiling gimp myself to see if I can >>> reproduce the issue with code executing out of the cold section. >>> >>> Thanks, >>> Teresa >>> >>> >> >>> >> Also, can you send me reproduction instructions for gimp? I don't >>> >> think I need Martin's patch, but which version of gimp and what is the >>> >> equivalent way for me to train it? I have some scripts to generate a >>> >> similar type of instruction heat map graph that I have been using to >>> >> tune partitioning and function reordering. Essentially it uses linux >>> >> perf to sample on instructions_retired and then munge the data in >>> >> several ways to produce various stats and graphs. One thing that has >>> >> been useful has been to combine the perf data with nm output to >>> >> determine which cold functions are being executed at runtime. >>> > >>> > Martin? >>> > >>> >> >>> >> However, for this to tell me which split cold bbs are being executed I >>> >> need to use a patch that Sri sent for review several months back that >>> >> gives the split cold section its own name: >>> >> http://gcc.gnu.org/ml/gcc-patches/2013-04/msg01571.html >>> >> Steven had some follow up comments that Sri hasn't had a chance to address yet: >>> >> http://gcc.gnu.org/ml/gcc-patches/2013-05/msg00798.html >>> >> (cc'ing Sri as we should probably revive this patch soon to address >>> >> gdb and other issues with detecting split functions properly) >>> > >>> > Intresting, I used linker script for this purposes, but that his GNU ld only... >>> > >>> > Honza >>> >> >>> >> Thanks! >>> >> Teresa >>> >> >>> >> > >>> >> > Honza >>> >> >> >>> >> >> Thanks, >>> >> >> Teresa >>> >> >> >>> >> >> > I think we are really looking primarily for dead parts of the functions (sanity checks/error handling) >>> >> >> > that should not be visited by train run. We can then see how to make the heuristic more aggressive? >>> >> >> > >>> >> >> > Honza >>> >> >> >>> >> >> >>> >> >> >>> >> >> -- >>> >> >> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 >>> >> >>> >> >>> >> >>> >> -- >>> >> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 >>> >>> >>> >>> -- >>> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 > > > > -- > Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 [-- Attachment #2: gimp-graphs.tar.bz2 --] [-- Type: application/x-bzip2, Size: 105015 bytes --] ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-11 15:55 ` Martin Liška @ 2013-08-11 17:55 ` Jan Hubicka 0 siblings, 0 replies; 62+ messages in thread From: Jan Hubicka @ 2013-08-11 17:55 UTC (permalink / raw) To: Martin Liška Cc: Teresa Johnson, Jan Hubicka, Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, Sriraman Tallam, Rong Xu > Hello, > I did a collection of systemtap graphs for GIMP. > > All these graphs were created with enabled LTO, profiling and -O2. > > 1) gimp-reordered.pdf - function are reordered according to my newly > created profile that utilizes LTO infrastructure > 2) gimp-no-top-level-reorder.pdf - (GCC rev. 201648) -fno-top-level-reorder > 3) gimp-top-level-reorder.pdf - (GCC rev. 201648) -ftop-level-reorder Thanks for the graphs! gimp-top-level-reorder seems to be bogus (it shows accesses into dynstr only). To catch the -fno-reorder-blocks-partition problem, perhaps you can modify the Martin's linker script to make .text.unlikely section non-executable. This way it will crash application every time we jump into it. Honza > > Honza has an idea how to minimize hot text section and I will send new > graphs for the proposed patch. > Moreover, I will send graphs for Inkscape which is written in C++. > > Have a nice day, > Martin > > On 11 August 2013 15:25, Teresa Johnson <tejohnson@google.com> wrote: > > Cc'ing Rong since he is also working on trying to address the comdat > > profile issue. Rong, you may need to see an earlier message for more > > context: > > http://gcc.gnu.org/ml/gcc-patches/2013-08/msg00558.html > > > > Teresa > > > > On Sun, Aug 11, 2013 at 5:21 AM, Jan Hubicka <hubicka@ucw.cz> wrote: > >>> > >>> I see, yes LTO can deal with this better since it has global > >>> information. In non-LTO mode (including LIPO) we have the issue. > >> > >> Either Martin or me will implement merging of the multiple copies at > >> LTO link time. This is needed for Martin's code unification patch anyway. > >> > >> Theoretically gcov runtime can also have symbol names and cfg checksums of > >> comdats in the static data and at exit produce buckets based on matching > >> names+checksums+counter counts, merge all data into in each bucket to one > >> representative by the existing merging routines and then memcpy them to > >> all the oriignal copiles. This way all compilation units will receive same > >> results. > >> > >> I am not very keen about making gcov runtime bigger and more complex than it > >> needs to be, but having sane profile for comdats seems quite important. > >> Perhaps, in GNU toolchain, ordered subsections can be used to make linker to > >> produce ordered list of comdats, so the runtime won't need to do hashing + > >> lookups. > >> > >> Honza > >>> > >>> I take it gimp is built with LTO and therefore shouldn't be hitting > >>> this comdat issue? > >>> > >>> Let me do a couple things: > >>> - port over my comdat inlining fix from the google branch to trunk and > >>> send it for review. If you or Martin could try it to see if it helps > >>> with function splitting to avoid the hits from the cold code that > >>> would be great > >>> - I'll add some new sanity checking to try to detect non-zero blocks > >>> in the cold section, or 0 blocks reached by non-zero edges and see if > >>> I can flush out any problems with my tests or a profiledbootstrap or > >>> gimp. > >>> - I'll try building and profiling gimp myself to see if I can > >>> reproduce the issue with code executing out of the cold section. > >>> > >>> Thanks, > >>> Teresa > >>> > >>> >> > >>> >> Also, can you send me reproduction instructions for gimp? I don't > >>> >> think I need Martin's patch, but which version of gimp and what is the > >>> >> equivalent way for me to train it? I have some scripts to generate a > >>> >> similar type of instruction heat map graph that I have been using to > >>> >> tune partitioning and function reordering. Essentially it uses linux > >>> >> perf to sample on instructions_retired and then munge the data in > >>> >> several ways to produce various stats and graphs. One thing that has > >>> >> been useful has been to combine the perf data with nm output to > >>> >> determine which cold functions are being executed at runtime. > >>> > > >>> > Martin? > >>> > > >>> >> > >>> >> However, for this to tell me which split cold bbs are being executed I > >>> >> need to use a patch that Sri sent for review several months back that > >>> >> gives the split cold section its own name: > >>> >> http://gcc.gnu.org/ml/gcc-patches/2013-04/msg01571.html > >>> >> Steven had some follow up comments that Sri hasn't had a chance to address yet: > >>> >> http://gcc.gnu.org/ml/gcc-patches/2013-05/msg00798.html > >>> >> (cc'ing Sri as we should probably revive this patch soon to address > >>> >> gdb and other issues with detecting split functions properly) > >>> > > >>> > Intresting, I used linker script for this purposes, but that his GNU ld only... > >>> > > >>> > Honza > >>> >> > >>> >> Thanks! > >>> >> Teresa > >>> >> > >>> >> > > >>> >> > Honza > >>> >> >> > >>> >> >> Thanks, > >>> >> >> Teresa > >>> >> >> > >>> >> >> > I think we are really looking primarily for dead parts of the functions (sanity checks/error handling) > >>> >> >> > that should not be visited by train run. We can then see how to make the heuristic more aggressive? > >>> >> >> > > >>> >> >> > Honza > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> -- > >>> >> >> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 > >>> >> > >>> >> > >>> >> > >>> >> -- > >>> >> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 > >>> > >>> > >>> > >>> -- > >>> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 > > > > > > > > -- > > Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-11 13:25 ` Teresa Johnson 2013-08-11 15:55 ` Martin Liška @ 2013-08-11 21:05 ` Jan Hubicka 1 sibling, 0 replies; 62+ messages in thread From: Jan Hubicka @ 2013-08-11 21:05 UTC (permalink / raw) To: Teresa Johnson Cc: Jan Hubicka, Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, marxin.liska, Sriraman Tallam, Rong Xu Hi, thinking about it a bit more, I suppose easiest way is to 1) make separate sets of counters for each comdat and place them into comdat section named as DECL_COMDAT_GROUP (node) + cfg_checksum + individual_counter_counts. This will make linker to unify the sections for us. 2) extend API of libgcov initialization so multiple counters can be recorded per file. 3) at merging time, gcov needs to merge all comdat section counters into temporary memory, so multiple merging won't produce bad results 4) counter streaming will need to be updated to deal with separate comdat sections... 5) probably we will want to update histogram production to avoid counting same comdat many times (this can be done by adding an "processed" flag into the per-function sections I don't see any obvious problems with this plan, just that it is quite some work. If you had chance to implement something along these lines, I think it would help ;)) Honza > Cc'ing Rong since he is also working on trying to address the comdat > profile issue. Rong, you may need to see an earlier message for more > context: > http://gcc.gnu.org/ml/gcc-patches/2013-08/msg00558.html > > Teresa > > On Sun, Aug 11, 2013 at 5:21 AM, Jan Hubicka <hubicka@ucw.cz> wrote: > >> > >> I see, yes LTO can deal with this better since it has global > >> information. In non-LTO mode (including LIPO) we have the issue. > > > > Either Martin or me will implement merging of the multiple copies at > > LTO link time. This is needed for Martin's code unification patch anyway. > > > > Theoretically gcov runtime can also have symbol names and cfg checksums of > > comdats in the static data and at exit produce buckets based on matching > > names+checksums+counter counts, merge all data into in each bucket to one > > representative by the existing merging routines and then memcpy them to > > all the oriignal copiles. This way all compilation units will receive same > > results. > > > > I am not very keen about making gcov runtime bigger and more complex than it > > needs to be, but having sane profile for comdats seems quite important. > > Perhaps, in GNU toolchain, ordered subsections can be used to make linker to > > produce ordered list of comdats, so the runtime won't need to do hashing + > > lookups. > > > > Honza > >> > >> I take it gimp is built with LTO and therefore shouldn't be hitting > >> this comdat issue? > >> > >> Let me do a couple things: > >> - port over my comdat inlining fix from the google branch to trunk and > >> send it for review. If you or Martin could try it to see if it helps > >> with function splitting to avoid the hits from the cold code that > >> would be great > >> - I'll add some new sanity checking to try to detect non-zero blocks > >> in the cold section, or 0 blocks reached by non-zero edges and see if > >> I can flush out any problems with my tests or a profiledbootstrap or > >> gimp. > >> - I'll try building and profiling gimp myself to see if I can > >> reproduce the issue with code executing out of the cold section. > >> > >> Thanks, > >> Teresa > >> > >> >> > >> >> Also, can you send me reproduction instructions for gimp? I don't > >> >> think I need Martin's patch, but which version of gimp and what is the > >> >> equivalent way for me to train it? I have some scripts to generate a > >> >> similar type of instruction heat map graph that I have been using to > >> >> tune partitioning and function reordering. Essentially it uses linux > >> >> perf to sample on instructions_retired and then munge the data in > >> >> several ways to produce various stats and graphs. One thing that has > >> >> been useful has been to combine the perf data with nm output to > >> >> determine which cold functions are being executed at runtime. > >> > > >> > Martin? > >> > > >> >> > >> >> However, for this to tell me which split cold bbs are being executed I > >> >> need to use a patch that Sri sent for review several months back that > >> >> gives the split cold section its own name: > >> >> http://gcc.gnu.org/ml/gcc-patches/2013-04/msg01571.html > >> >> Steven had some follow up comments that Sri hasn't had a chance to address yet: > >> >> http://gcc.gnu.org/ml/gcc-patches/2013-05/msg00798.html > >> >> (cc'ing Sri as we should probably revive this patch soon to address > >> >> gdb and other issues with detecting split functions properly) > >> > > >> > Intresting, I used linker script for this purposes, but that his GNU ld only... > >> > > >> > Honza > >> >> > >> >> Thanks! > >> >> Teresa > >> >> > >> >> > > >> >> > Honza > >> >> >> > >> >> >> Thanks, > >> >> >> Teresa > >> >> >> > >> >> >> > I think we are really looking primarily for dead parts of the functions (sanity checks/error handling) > >> >> >> > that should not be visited by train run. We can then see how to make the heuristic more aggressive? > >> >> >> > > >> >> >> > Honza > >> >> >> > >> >> >> > >> >> >> > >> >> >> -- > >> >> >> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 > >> >> > >> >> > >> >> > >> >> -- > >> >> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 > >> > >> > >> > >> -- > >> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 > > > > -- > Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-09 21:02 ` Teresa Johnson 2013-08-09 22:43 ` Jan Hubicka 2013-08-11 12:21 ` Jan Hubicka @ 2013-08-17 15:54 ` Teresa Johnson 2013-08-17 21:02 ` Jan Hubicka 2 siblings, 1 reply; 62+ messages in thread From: Teresa Johnson @ 2013-08-17 15:54 UTC (permalink / raw) To: Jan Hubicka Cc: Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, marxin.liska, Sriraman Tallam On Fri, Aug 9, 2013 at 2:02 PM, Teresa Johnson <tejohnson@google.com> wrote: > On Fri, Aug 9, 2013 at 8:28 AM, Jan Hubicka <hubicka@ucw.cz> wrote: >>> > Do we sanity check that the cold partition does not contain any blocks of >>> > count 0? It may be that the profile is broken enough to make partitioning >>> > not work. >>> >>> Do you mean sanity check that the cold partition does not contain any >>> blocks of count > 0? (they should all be zero) I don't think that >>> sanity check is there, but I can try adding that. >> >> Thanks, lets start with this - I suppose we need to figure out if >> 1) the reachable blocks goes to cold section because partitioning decides >> so even if they have non-0 count. > > Right, this should be easy enough to check and should hopefully never happen. > >> 2) the reachable blocks goes to cold section because they have incorrectly >> updated count to 0 by someone > > A sanity check should find this too. But it can happen now for various > reasons like the comdat issue I described below. But it will be good > to flag these and fix them. > >> 3) profiling gets some blocks wrong. > > This is the one that will be tough to fix, if the training run isn't > representative. > >>> >>> The issue with such a sanity check may be due to the later fixup I >>> have in this patch (fixup_partitions). It is invoked after certain >>> optimizations on the cfg that may make hot blocks previously reached >>> by both hot and cold edges only reachable by cold blocks. These blocks >>> are remarked cold. If the profile data hasn't been updated correctly >>> it is possible that they would still have a non-0 count, although they >>> are essentially cold after the cfg transformation. >> >> Well, or the other posibility is that the edges was updated wrong >> and the blocks are really cold. We need to figure out if that happens >> commonly enough. >> >> I will try to think of some artificial testcases. >> >>> >>> But certainly such a sanity check should always succeed after the >>> original partitioning. >>> >>> > I can think of inlining where the count gets scaled all way down to 0. Perhaps >>> > count scaling code can be modified to never round towards 0 for block executing >>> > non-0 times... >>> >>> This reminds me of why this situation could happen. When I have been >>> testing this on the google branch I found situations where COMDAT >>> routines have 0 profile counts (if I remember correctly, this happens >>> when profile-gen binary has call to out-of-line copy of COMDAT in >>> module A, linker chooses the out-of-line copy from module B, therefore >>> the profile data for COMDAT in module A is 0). When the COMDAT gets >>> inlined, the 0 counts on its bbs are scaled to 0, even though the >>> callsite is non-zero. I have a patch that I was planning to send as a >>> follow-up that handles this case by propagating the callsite bb's >>> count to the inlined code when it has 0 counts, scaling by the edge >>> frequencies. I can either include that patch in this one, or send it >>> for review separately right now. Do you want to give it a try with >>> this one to see if it addresses the issue? >> >> This scenario should not happen with LTO setup: the LTO symbol tables contains >> code before early optimization and should be identical with profiling or >> without (modulo the new references and call from profile code). >> >> But this patch seems useful as a backup solution for non-LTO, so yes, please >> send it separately and I can try to double check that it really do not happen >> with LTO. >> (acutally LTO symtab may just chose COMDAT from module that has counts with it. >> It has all the info for it. I was thinkin about it few weeks back. It is >> bit hard to do - you need to verify that all references from the function are >> the same or linking might fail if you overwrite linker's decisiosns). > > I see, yes LTO can deal with this better since it has global > information. In non-LTO mode (including LIPO) we have the issue. > > I take it gimp is built with LTO and therefore shouldn't be hitting > this comdat issue? > > Let me do a couple things: Here is some status: > - port over my comdat inlining fix from the google branch to trunk and > send it for review. If you or Martin could try it to see if it helps > with function splitting to avoid the hits from the cold code that > would be great I have included the cleaned up patch below. I will send it to trunk for review, but included it below if you want to try it out before I send it pending a little more extensive testing I want to do after the cleanup (it passes bootstrap and some small test cases I checked manually). > - I'll add some new sanity checking to try to detect non-zero blocks > in the cold section, or 0 blocks reached by non-zero edges and see if > I can flush out any problems with my tests or a profiledbootstrap or > gimp. I added both of these and ran into issues due to profile maintenance. For example, there were non-zero blocks in the cold section because pro_and_epilogue split a simple return block that was previously reach by both hot and cold paths. The new return block that was then only reached via the cold path did not have its count properly updated to reflect this, and since with this patch, blocks dominated by cold blocks are remarked cold, we ended up with a non-zero count block in the cold section. And there were 0 count blocks reached by non-zero edges because copyprop did not clean up edge weights after removing some branches and blocks, leading to non-zero edge weights that had previously targeted a branch that was removed, now targeting a 0 count block that the removed branch always branched around. In any case, the good news is in that the cases I looked at, the splitting code is doing the right thing and these blocks that were marked cold really were cold. It would be great to fix the profile maintenance issues, but that in the meantime the above sanity checks are too aggressive. I think it makes sense to commit the current patch if possible, as it is making the splitting more sane. > - I'll try building and profiling gimp myself to see if I can > reproduce the issue with code executing out of the cold section. I have spent some time this week trying to get the latest gimp Martin pointed me to configured and built, but it took awhile to track down and configure/build all of the required versions of dependent packages. I'm still hitting some issues trying to get it compiled, so it may not yet be configured properly. I'll take a look again early next week. Teresa patch for updating counts based on estimated frequencies to address inlined comdats with 0 profile counts: 013-08-16 Teresa Johnson <tejohnson@google.com> * tree-inline.c (copy_bb): Compute count based on frequency. (copy_edges_for_bb): Ditto. (copy_cfg_body): Ditto. (copy_body): Pass down frequency. (expand_call_inline): Ditto. (tree_function_versioning): Ditto. * predict.c (init_and_estimate_bb_frequencies): New function. (rebuild_frequencies): Invoke init_and_estimate_bb_frequencies. * predict.h (init_and_estimate_bb_frequencies): Declare. * profile.c (branch_prob): Invoke init_and_estimate_bb_frequencies. * ipa-inline-transform.c (update_noncloned_frequencies): Scale edge counts. (clone_inlined_nodes): Compute edge count scale if needed. Index: tree-inline.c =================================================================== --- tree-inline.c (revision 201644) +++ tree-inline.c (working copy) @@ -1502,7 +1502,7 @@ remap_gimple_stmt (gimple stmt, copy_body_data *id static basic_block copy_bb (copy_body_data *id, basic_block bb, int frequency_scale, - gcov_type count_scale) + gcov_type count_scale, gcov_type freq_to_count_scale) { gimple_stmt_iterator gsi, copy_gsi, seq_gsi; basic_block copy_basic_block; @@ -1519,7 +1519,13 @@ copy_bb (copy_body_data *id, basic_block bb, int f basic_block_info automatically. */ copy_basic_block = create_basic_block (NULL, (void *) 0, (basic_block) prev->aux); - copy_basic_block->count = apply_scale (bb->count, count_scale); + copy_basic_block->count + = (count_scale + ? apply_scale (bb->count, count_scale) + /* When the callee bb counts were all zero (e.g. this was a COMDAT + that didn't get profile counts) then we compute the new bb counts + via the statically-estimated frequencies. */ + : RDIV ((gcov_type)bb->frequency * freq_to_count_scale, BB_FREQ_MAX)); /* We are going to rebuild frequencies from scratch. These values have just small importance to drive canonicalize_loop_headers. */ @@ -1888,7 +1894,8 @@ update_ssa_across_abnormal_edges (basic_block bb, debug stmts are left after a statement that must end the basic block. */ static bool -copy_edges_for_bb (basic_block bb, gcov_type count_scale, basic_block ret_bb, +copy_edges_for_bb (basic_block bb, gcov_type count_scale, + basic_block ret_bb, bool can_make_abnormal_goto) { basic_block new_bb = (basic_block) bb->aux; @@ -1912,7 +1919,14 @@ static bool && old_edge->dest->aux != EXIT_BLOCK_PTR) flags |= EDGE_FALLTHRU; new_edge = make_edge (new_bb, (basic_block) old_edge->dest->aux, flags); - new_edge->count = apply_scale (old_edge->count, count_scale); + basic_block new_src_bb = (basic_block) old_edge->src->aux; + new_edge->count + = (count_scale + ? apply_scale (old_edge->count, count_scale) + // The bb counts have already been scaled with freq_to_count_scale + // when that is non-zero, so just scale that new bb count by + // the edge probability. + : apply_probability (new_src_bb->count, old_edge->probability)); new_edge->probability = old_edge->probability; } @@ -2282,7 +2296,8 @@ redirect_all_calls (copy_body_data * id, basic_blo another function. Walks FN via CFG, returns new fndecl. */ static tree -copy_cfg_body (copy_body_data * id, gcov_type count, int frequency_scale, +copy_cfg_body (copy_body_data * id, gcov_type count, + int frequency, int frequency_scale, basic_block entry_block_map, basic_block exit_block_map, bitmap blocks_to_copy, basic_block new_entry) { @@ -2293,15 +2308,20 @@ static tree basic_block bb; tree new_fndecl = NULL; bool need_debug_cleanup = false; - gcov_type count_scale; + gcov_type count_scale = 0; + gcov_type freq_to_count_scale = 0; int last; int incoming_frequency = 0; gcov_type incoming_count = 0; - if (ENTRY_BLOCK_PTR_FOR_FUNCTION (src_cfun)->count) - count_scale - = GCOV_COMPUTE_SCALE (count, - ENTRY_BLOCK_PTR_FOR_FUNCTION (src_cfun)->count); + basic_block entry_bb = ENTRY_BLOCK_PTR_FOR_FUNCTION (src_cfun); + if (entry_bb->count) + count_scale = GCOV_COMPUTE_SCALE (count, entry_bb->count); + /* When the callee bb counts were all zero (e.g. this was a COMDAT + that didn't get profile counts) then we compute the new bb counts + via the statically-estimated frequencies. */ + else if (entry_bb->frequency) + freq_to_count_scale = RDIV (count * frequency, entry_bb->frequency); else count_scale = REG_BR_PROB_BASE; @@ -2323,7 +2343,13 @@ static tree incoming_frequency += EDGE_FREQUENCY (e); incoming_count += e->count; } - incoming_count = apply_scale (incoming_count, count_scale); + incoming_count + = (count_scale + ? apply_scale (incoming_count, count_scale) + /* When the callee bb counts were all zero (e.g. this was a COMDAT + that didn't get profile counts) then we compute the new bb counts + via the statically-estimated frequencies. */ + : RDIV (incoming_frequency * freq_to_count_scale, BB_FREQ_MAX)); incoming_frequency = apply_scale ((gcov_type)incoming_frequency, frequency_scale); ENTRY_BLOCK_PTR->count = incoming_count; @@ -2350,7 +2376,8 @@ static tree FOR_EACH_BB_FN (bb, cfun_to_copy) if (!blocks_to_copy || bitmap_bit_p (blocks_to_copy, bb->index)) { - basic_block new_bb = copy_bb (id, bb, frequency_scale, count_scale); + basic_block new_bb = copy_bb (id, bb, frequency_scale, count_scale, + freq_to_count_scale); bb->aux = new_bb; new_bb->aux = bb; new_bb->loop_father = entry_block_map->loop_father; @@ -2364,7 +2391,8 @@ static tree FOR_ALL_BB_FN (bb, cfun_to_copy) if (!blocks_to_copy || (bb->index > 0 && bitmap_bit_p (blocks_to_copy, bb->index))) - need_debug_cleanup |= copy_edges_for_bb (bb, count_scale, exit_block_map, + need_debug_cleanup |= copy_edges_for_bb (bb, count_scale, + exit_block_map, can_make_abormal_goto); if (new_entry) @@ -2562,7 +2590,8 @@ copy_tree_body (copy_body_data *id) another function. */ static tree -copy_body (copy_body_data *id, gcov_type count, int frequency_scale, +copy_body (copy_body_data *id, gcov_type count, int frequency, + int frequency_scale, basic_block entry_block_map, basic_block exit_block_map, bitmap blocks_to_copy, basic_block new_entry) { @@ -2571,7 +2600,8 @@ static tree /* If this body has a CFG, walk CFG and copy. */ gcc_assert (ENTRY_BLOCK_PTR_FOR_FUNCTION (DECL_STRUCT_FUNCTION (fndecl))); - body = copy_cfg_body (id, count, frequency_scale, entry_block_map, exit_block_map, + body = copy_cfg_body (id, count, frequency, frequency_scale, + entry_block_map, exit_block_map, blocks_to_copy, new_entry); copy_debug_stmts (id); @@ -4172,7 +4202,7 @@ expand_call_inline (basic_block bb, gimple stmt, c function in any way before this point, as this CALL_EXPR may be a self-referential call; if we're calling ourselves, we need to duplicate our body before altering anything. */ - copy_body (id, bb->count, + copy_body (id, bb->count, bb->frequency, GCOV_COMPUTE_SCALE (cg_edge->frequency, CGRAPH_FREQ_BASE), bb, return_block, NULL, NULL); @@ -5299,8 +5329,9 @@ tree_function_versioning (tree old_decl, tree new_ } /* Copy the Function's body. */ - copy_body (&id, old_entry_block->count, REG_BR_PROB_BASE, - ENTRY_BLOCK_PTR, EXIT_BLOCK_PTR, blocks_to_copy, new_entry); + copy_body (&id, old_entry_block->count, old_entry_block->frequency, + REG_BR_PROB_BASE, ENTRY_BLOCK_PTR, EXIT_BLOCK_PTR, + blocks_to_copy, new_entry); /* Renumber the lexical scoping (non-code) blocks consecutively. */ number_blocks (new_decl); Index: predict.c =================================================================== --- predict.c (revision 201644) +++ predict.c (working copy) @@ -2976,6 +2976,24 @@ make_pass_strip_predict_hints (gcc::context *ctxt) return new pass_strip_predict_hints (ctxt); } +/* Initialize loop edges and compute estimated bb frequencies when there + is no profile data available. */ + +void +init_and_estimate_bb_frequencies (void) +{ + if (profile_status == PROFILE_READ && counts_to_freqs ()) + return; + + loop_optimizer_init (0); + add_noreturn_fake_exit_edges (); + mark_irreducible_loops (); + connect_infinite_loops_to_exit (); + estimate_bb_frequencies (); + remove_fake_exit_edges (); + loop_optimizer_finalize (); +} + /* Rebuild function frequencies. Passes are in general expected to maintain profile by hand, however in some cases this is not possible: for example when inlining several functions with loops freuqencies might run @@ -2986,15 +3004,7 @@ rebuild_frequencies (void) { timevar_push (TV_REBUILD_FREQUENCIES); if (profile_status == PROFILE_GUESSED) - { - loop_optimizer_init (0); - add_noreturn_fake_exit_edges (); - mark_irreducible_loops (); - connect_infinite_loops_to_exit (); - estimate_bb_frequencies (); - remove_fake_exit_edges (); - loop_optimizer_finalize (); - } + init_and_estimate_bb_frequencies (); else if (profile_status == PROFILE_READ) counts_to_freqs (); else Index: predict.h =================================================================== --- predict.h (revision 201644) +++ predict.h (working copy) @@ -38,6 +38,7 @@ enum prediction extern void predict_insn_def (rtx, enum br_predictor, enum prediction); extern int counts_to_freqs (void); extern void estimate_bb_frequencies (void); +extern void init_and_estimate_bb_frequencies (void); extern const char *predictor_name (enum br_predictor); extern tree build_predict_expr (enum br_predictor, enum prediction); extern void tree_estimate_probability (void); Index: profile.c =================================================================== --- profile.c (revision 201644) +++ profile.c (working copy) @@ -1305,6 +1305,12 @@ branch_prob (void) values.release (); free_edge_list (el); + + /* Call after setting profile_status to PROFILE_READ, will then + invoke counts_to_freqs and if the sum of the counts is zero, will + estimate the frequencies. */ + init_and_estimate_bb_frequencies (); + coverage_end_function (lineno_checksum, cfg_checksum); } ^L Index: ipa-inline-transform.c =================================================================== --- ipa-inline-transform.c (revision 201644) +++ ipa-inline-transform.c (working copy) @@ -51,7 +51,7 @@ int nfunctions_inlined; static void update_noncloned_frequencies (struct cgraph_node *node, - int freq_scale) + gcov_type count_scale, int freq_scale) { struct cgraph_edge *e; @@ -60,14 +60,16 @@ update_noncloned_frequencies (struct cgraph_node * freq_scale = 1; for (e = node->callees; e; e = e->next_callee) { + e->count = apply_scale (e->count, count_scale); e->frequency = e->frequency * (gcov_type) freq_scale / CGRAPH_FREQ_BASE; if (e->frequency > CGRAPH_FREQ_MAX) e->frequency = CGRAPH_FREQ_MAX; if (!e->inline_failed) - update_noncloned_frequencies (e->callee, freq_scale); + update_noncloned_frequencies (e->callee, count_scale, freq_scale); } for (e = node->indirect_calls; e; e = e->next_callee) { + e->count = apply_scale (e->count, count_scale); e->frequency = e->frequency * (gcov_type) freq_scale / CGRAPH_FREQ_BASE; if (e->frequency > CGRAPH_FREQ_MAX) e->frequency = CGRAPH_FREQ_MAX; @@ -169,7 +171,13 @@ clone_inlined_nodes (struct cgraph_edge *e, bool d } duplicate = false; e->callee->symbol.externally_visible = false; - update_noncloned_frequencies (e->callee, e->frequency); + // In the case of a COMDAT, the callee's count may be from other + // modules, and we need to scale it for the current module's calls + // (e.g. e->count may be 0 despite e->callee->count > 0). + gcov_type count_scale = REG_BR_PROB_BASE; + if (e->callee->count > e->count) + count_scale = GCOV_COMPUTE_SCALE (e->count, e->callee->count); + update_noncloned_frequencies (e->callee, count_scale, e->frequency); } else { > > Thanks, > Teresa > >>> >>> Also, can you send me reproduction instructions for gimp? I don't >>> think I need Martin's patch, but which version of gimp and what is the >>> equivalent way for me to train it? I have some scripts to generate a >>> similar type of instruction heat map graph that I have been using to >>> tune partitioning and function reordering. Essentially it uses linux >>> perf to sample on instructions_retired and then munge the data in >>> several ways to produce various stats and graphs. One thing that has >>> been useful has been to combine the perf data with nm output to >>> determine which cold functions are being executed at runtime. >> >> Martin? >> >>> >>> However, for this to tell me which split cold bbs are being executed I >>> need to use a patch that Sri sent for review several months back that >>> gives the split cold section its own name: >>> http://gcc.gnu.org/ml/gcc-patches/2013-04/msg01571.html >>> Steven had some follow up comments that Sri hasn't had a chance to address yet: >>> http://gcc.gnu.org/ml/gcc-patches/2013-05/msg00798.html >>> (cc'ing Sri as we should probably revive this patch soon to address >>> gdb and other issues with detecting split functions properly) >> >> Intresting, I used linker script for this purposes, but that his GNU ld only... >> >> Honza >>> >>> Thanks! >>> Teresa >>> >>> > >>> > Honza >>> >> >>> >> Thanks, >>> >> Teresa >>> >> >>> >> > I think we are really looking primarily for dead parts of the functions (sanity checks/error handling) >>> >> > that should not be visited by train run. We can then see how to make the heuristic more aggressive? >>> >> > >>> >> > Honza >>> >> >>> >> >>> >> >>> >> -- >>> >> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 >>> >>> >>> >>> -- >>> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 > > > > -- > Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-17 15:54 ` Teresa Johnson @ 2013-08-17 21:02 ` Jan Hubicka 2013-08-19 13:51 ` Teresa Johnson 2013-08-19 15:34 ` Teresa Johnson 0 siblings, 2 replies; 62+ messages in thread From: Jan Hubicka @ 2013-08-17 21:02 UTC (permalink / raw) To: Teresa Johnson Cc: Jan Hubicka, Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, marxin.liska, Sriraman Tallam > > I added both of these and ran into issues due to profile maintenance. > For example, there were non-zero blocks in the cold section because > pro_and_epilogue split a simple return block that was previously reach > by both hot and cold paths. The new return block that was then only > reached via the cold path did not have its count properly updated to > reflect this, and since with this patch, blocks dominated by cold > blocks are remarked cold, we ended up with a non-zero count block in > the cold section. And there were 0 count blocks reached by non-zero > edges because copyprop did not clean up edge weights after removing > some branches and blocks, leading to non-zero edge weights that had > previously targeted a branch that was removed, now targeting a 0 count > block that the removed branch always branched around. I see, can you please send fixes for the problems you identified? Thanks for owrking on this! > > In any case, the good news is in that the cases I looked at, the > splitting code is doing the right thing and these blocks that were > marked cold really were cold. It would be great to fix the profile > maintenance issues, but that in the meantime the above sanity checks > are too aggressive. We can keep them and output info into dump file - it is what most of the profile sanity checking does anyway. Did you try to use Martin's linker script to turn text.unlikely section unexecutable? I think that way we will easily find what really causes us to use it during startup of trained application (just like Martin does for gimp). > > I think it makes sense to commit the current patch if possible, as it > is making the splitting more sane. My only concern about the patch is that I am not convinced the dominator based code has chance to work reliably enough so we won't see too many accesses into the cold section. We can commit it and work on better solution incrementally but it will probably mean replacing it later. If you think it makes things easier to work on it incrementally, I think the patch is OK. > > > - I'll try building and profiling gimp myself to see if I can > > reproduce the issue with code executing out of the cold section. > > I have spent some time this week trying to get the latest gimp Martin > pointed me to configured and built, but it took awhile to track down > and configure/build all of the required versions of dependent > packages. I'm still hitting some issues trying to get it compiled, so > it may not yet be configured properly. I'll take a look again early > next week. I do not think there is anything special about gimp. You can probably take any other bigger app, like GCC itself. With profiledbootstrap and linker script to lock unlikely section you should get ICEs where we jump into cold secton and should not. > > Teresa > > patch for updating counts based on estimated frequencies to address > inlined comdats with 0 profile counts: > > 013-08-16 Teresa Johnson <tejohnson@google.com> > > * tree-inline.c (copy_bb): Compute count based on frequency. > (copy_edges_for_bb): Ditto. > (copy_cfg_body): Ditto. > (copy_body): Pass down frequency. > (expand_call_inline): Ditto. > (tree_function_versioning): Ditto. > * predict.c (init_and_estimate_bb_frequencies): New function. > (rebuild_frequencies): Invoke init_and_estimate_bb_frequencies. > * predict.h (init_and_estimate_bb_frequencies): Declare. > * profile.c (branch_prob): Invoke init_and_estimate_bb_frequencies. > * ipa-inline-transform.c (update_noncloned_frequencies): Scale edge > counts. > (clone_inlined_nodes): Compute edge count scale if needed. I do not see why inliner needs to care about scaling more than it does right now. So you have init_and_estimate_bb_frequencies that force profile guessing on a given function body. In addition to that I thing you need something like freqs_to_counts that will compute counts based on freqs with given scale (actually you can do that as part of propagation before frequencies are scalled to the usual 0...FREQ_MAX scale and precision is lost). Because offline COMDAT functoin will be porduced for every COMDAT used, I think it is bad to porduce any COMDAT (or any reachable function via calls with non-0 count) that has empty profile (either because it got lost by COMDAT merging or because of reading mismatch). So I guess you can just check functions with 0 count and non-0 count callers and initialize their guessed profile. Some capping will probably be needed to not propagate insanely large numbers.. Since new direct calls can be discovered later, inline may want to do that again each time it inlines non-0 count call of COMDAT with 0 count... Honza > > Index: tree-inline.c > =================================================================== > --- tree-inline.c (revision 201644) > +++ tree-inline.c (working copy) > @@ -1502,7 +1502,7 @@ remap_gimple_stmt (gimple stmt, copy_body_data *id > > static basic_block > copy_bb (copy_body_data *id, basic_block bb, int frequency_scale, > - gcov_type count_scale) > + gcov_type count_scale, gcov_type freq_to_count_scale) > { > gimple_stmt_iterator gsi, copy_gsi, seq_gsi; > basic_block copy_basic_block; > @@ -1519,7 +1519,13 @@ copy_bb (copy_body_data *id, basic_block bb, int f > basic_block_info automatically. */ > copy_basic_block = create_basic_block (NULL, (void *) 0, > (basic_block) prev->aux); > - copy_basic_block->count = apply_scale (bb->count, count_scale); > + copy_basic_block->count > + = (count_scale > + ? apply_scale (bb->count, count_scale) > + /* When the callee bb counts were all zero (e.g. this was a COMDAT > + that didn't get profile counts) then we compute the new bb counts > + via the statically-estimated frequencies. */ > + : RDIV ((gcov_type)bb->frequency * freq_to_count_scale, BB_FREQ_MAX)); > > /* We are going to rebuild frequencies from scratch. These values > have just small importance to drive canonicalize_loop_headers. */ > @@ -1888,7 +1894,8 @@ update_ssa_across_abnormal_edges (basic_block bb, > debug stmts are left after a statement that must end the basic block. */ > > static bool > -copy_edges_for_bb (basic_block bb, gcov_type count_scale, basic_block ret_bb, > +copy_edges_for_bb (basic_block bb, gcov_type count_scale, > + basic_block ret_bb, > bool can_make_abnormal_goto) > { > basic_block new_bb = (basic_block) bb->aux; > @@ -1912,7 +1919,14 @@ static bool > && old_edge->dest->aux != EXIT_BLOCK_PTR) > flags |= EDGE_FALLTHRU; > new_edge = make_edge (new_bb, (basic_block) old_edge->dest->aux, flags); > - new_edge->count = apply_scale (old_edge->count, count_scale); > + basic_block new_src_bb = (basic_block) old_edge->src->aux; > + new_edge->count > + = (count_scale > + ? apply_scale (old_edge->count, count_scale) > + // The bb counts have already been scaled with > freq_to_count_scale > + // when that is non-zero, so just scale that new bb count by > + // the edge probability. > + : apply_probability (new_src_bb->count, old_edge->probability)); > new_edge->probability = old_edge->probability; > } > > @@ -2282,7 +2296,8 @@ redirect_all_calls (copy_body_data * id, basic_blo > another function. Walks FN via CFG, returns new fndecl. */ > > static tree > -copy_cfg_body (copy_body_data * id, gcov_type count, int frequency_scale, > +copy_cfg_body (copy_body_data * id, gcov_type count, > + int frequency, int frequency_scale, > basic_block entry_block_map, basic_block exit_block_map, > bitmap blocks_to_copy, basic_block new_entry) > { > @@ -2293,15 +2308,20 @@ static tree > basic_block bb; > tree new_fndecl = NULL; > bool need_debug_cleanup = false; > - gcov_type count_scale; > + gcov_type count_scale = 0; > + gcov_type freq_to_count_scale = 0; > int last; > int incoming_frequency = 0; > gcov_type incoming_count = 0; > > - if (ENTRY_BLOCK_PTR_FOR_FUNCTION (src_cfun)->count) > - count_scale > - = GCOV_COMPUTE_SCALE (count, > - ENTRY_BLOCK_PTR_FOR_FUNCTION (src_cfun)->count); > + basic_block entry_bb = ENTRY_BLOCK_PTR_FOR_FUNCTION (src_cfun); > + if (entry_bb->count) > + count_scale = GCOV_COMPUTE_SCALE (count, entry_bb->count); > + /* When the callee bb counts were all zero (e.g. this was a COMDAT > + that didn't get profile counts) then we compute the new bb counts > + via the statically-estimated frequencies. */ > + else if (entry_bb->frequency) > + freq_to_count_scale = RDIV (count * frequency, entry_bb->frequency); > else > count_scale = REG_BR_PROB_BASE; > > @@ -2323,7 +2343,13 @@ static tree > incoming_frequency += EDGE_FREQUENCY (e); > incoming_count += e->count; > } > - incoming_count = apply_scale (incoming_count, count_scale); > + incoming_count > + = (count_scale > + ? apply_scale (incoming_count, count_scale) > + /* When the callee bb counts were all zero (e.g. this was a COMDAT > + that didn't get profile counts) then we compute the > new bb counts > + via the statically-estimated frequencies. */ > + : RDIV (incoming_frequency * freq_to_count_scale, BB_FREQ_MAX)); > incoming_frequency > = apply_scale ((gcov_type)incoming_frequency, frequency_scale); > ENTRY_BLOCK_PTR->count = incoming_count; > @@ -2350,7 +2376,8 @@ static tree > FOR_EACH_BB_FN (bb, cfun_to_copy) > if (!blocks_to_copy || bitmap_bit_p (blocks_to_copy, bb->index)) > { > - basic_block new_bb = copy_bb (id, bb, frequency_scale, count_scale); > + basic_block new_bb = copy_bb (id, bb, frequency_scale, count_scale, > + freq_to_count_scale); > bb->aux = new_bb; > new_bb->aux = bb; > new_bb->loop_father = entry_block_map->loop_father; > @@ -2364,7 +2391,8 @@ static tree > FOR_ALL_BB_FN (bb, cfun_to_copy) > if (!blocks_to_copy > || (bb->index > 0 && bitmap_bit_p (blocks_to_copy, bb->index))) > - need_debug_cleanup |= copy_edges_for_bb (bb, count_scale, exit_block_map, > + need_debug_cleanup |= copy_edges_for_bb (bb, count_scale, > + exit_block_map, > can_make_abormal_goto); > > if (new_entry) > @@ -2562,7 +2590,8 @@ copy_tree_body (copy_body_data *id) > another function. */ > > static tree > -copy_body (copy_body_data *id, gcov_type count, int frequency_scale, > +copy_body (copy_body_data *id, gcov_type count, int frequency, > + int frequency_scale, > basic_block entry_block_map, basic_block exit_block_map, > bitmap blocks_to_copy, basic_block new_entry) > { > @@ -2571,7 +2600,8 @@ static tree > > /* If this body has a CFG, walk CFG and copy. */ > gcc_assert (ENTRY_BLOCK_PTR_FOR_FUNCTION (DECL_STRUCT_FUNCTION (fndecl))); > - body = copy_cfg_body (id, count, frequency_scale, entry_block_map, > exit_block_map, > + body = copy_cfg_body (id, count, frequency, frequency_scale, > + entry_block_map, exit_block_map, > blocks_to_copy, new_entry); > copy_debug_stmts (id); > > @@ -4172,7 +4202,7 @@ expand_call_inline (basic_block bb, gimple stmt, c > function in any way before this point, as this CALL_EXPR may be > a self-referential call; if we're calling ourselves, we need to > duplicate our body before altering anything. */ > - copy_body (id, bb->count, > + copy_body (id, bb->count, bb->frequency, > GCOV_COMPUTE_SCALE (cg_edge->frequency, CGRAPH_FREQ_BASE), > bb, return_block, NULL, NULL); > > @@ -5299,8 +5329,9 @@ tree_function_versioning (tree old_decl, tree new_ > } > > /* Copy the Function's body. */ > - copy_body (&id, old_entry_block->count, REG_BR_PROB_BASE, > - ENTRY_BLOCK_PTR, EXIT_BLOCK_PTR, blocks_to_copy, new_entry); > + copy_body (&id, old_entry_block->count, old_entry_block->frequency, > + REG_BR_PROB_BASE, ENTRY_BLOCK_PTR, EXIT_BLOCK_PTR, > + blocks_to_copy, new_entry); > > /* Renumber the lexical scoping (non-code) blocks consecutively. */ > number_blocks (new_decl); > Index: predict.c > =================================================================== > --- predict.c (revision 201644) > +++ predict.c (working copy) > @@ -2976,6 +2976,24 @@ make_pass_strip_predict_hints (gcc::context *ctxt) > return new pass_strip_predict_hints (ctxt); > } > > +/* Initialize loop edges and compute estimated bb frequencies when there > + is no profile data available. */ > + > +void > +init_and_estimate_bb_frequencies (void) > +{ > + if (profile_status == PROFILE_READ && counts_to_freqs ()) > + return; > + > + loop_optimizer_init (0); > + add_noreturn_fake_exit_edges (); > + mark_irreducible_loops (); > + connect_infinite_loops_to_exit (); > + estimate_bb_frequencies (); > + remove_fake_exit_edges (); > + loop_optimizer_finalize (); > +} > + > /* Rebuild function frequencies. Passes are in general expected to > maintain profile by hand, however in some cases this is not possible: > for example when inlining several functions with loops freuqencies might run > @@ -2986,15 +3004,7 @@ rebuild_frequencies (void) > { > timevar_push (TV_REBUILD_FREQUENCIES); > if (profile_status == PROFILE_GUESSED) > - { > - loop_optimizer_init (0); > - add_noreturn_fake_exit_edges (); > - mark_irreducible_loops (); > - connect_infinite_loops_to_exit (); > - estimate_bb_frequencies (); > - remove_fake_exit_edges (); > - loop_optimizer_finalize (); > - } > + init_and_estimate_bb_frequencies (); > else if (profile_status == PROFILE_READ) > counts_to_freqs (); > else > Index: predict.h > =================================================================== > --- predict.h (revision 201644) > +++ predict.h (working copy) > @@ -38,6 +38,7 @@ enum prediction > extern void predict_insn_def (rtx, enum br_predictor, enum prediction); > extern int counts_to_freqs (void); > extern void estimate_bb_frequencies (void); > +extern void init_and_estimate_bb_frequencies (void); > extern const char *predictor_name (enum br_predictor); > extern tree build_predict_expr (enum br_predictor, enum prediction); > extern void tree_estimate_probability (void); > Index: profile.c > =================================================================== > --- profile.c (revision 201644) > +++ profile.c (working copy) > @@ -1305,6 +1305,12 @@ branch_prob (void) > > values.release (); > free_edge_list (el); > + > + /* Call after setting profile_status to PROFILE_READ, will then > + invoke counts_to_freqs and if the sum of the counts is zero, will > + estimate the frequencies. */ > + init_and_estimate_bb_frequencies (); > + > coverage_end_function (lineno_checksum, cfg_checksum); > } > ^L > Index: ipa-inline-transform.c > =================================================================== > --- ipa-inline-transform.c (revision 201644) > +++ ipa-inline-transform.c (working copy) > @@ -51,7 +51,7 @@ int nfunctions_inlined; > > static void > update_noncloned_frequencies (struct cgraph_node *node, > - int freq_scale) > + gcov_type count_scale, int freq_scale) > { > struct cgraph_edge *e; > > @@ -60,14 +60,16 @@ update_noncloned_frequencies (struct cgraph_node * > freq_scale = 1; > for (e = node->callees; e; e = e->next_callee) > { > + e->count = apply_scale (e->count, count_scale); > e->frequency = e->frequency * (gcov_type) freq_scale / CGRAPH_FREQ_BASE; > if (e->frequency > CGRAPH_FREQ_MAX) > e->frequency = CGRAPH_FREQ_MAX; > if (!e->inline_failed) > - update_noncloned_frequencies (e->callee, freq_scale); > + update_noncloned_frequencies (e->callee, count_scale, freq_scale); > } > for (e = node->indirect_calls; e; e = e->next_callee) > { > + e->count = apply_scale (e->count, count_scale); > e->frequency = e->frequency * (gcov_type) freq_scale / CGRAPH_FREQ_BASE; > if (e->frequency > CGRAPH_FREQ_MAX) > e->frequency = CGRAPH_FREQ_MAX; > @@ -169,7 +171,13 @@ clone_inlined_nodes (struct cgraph_edge *e, bool d > } > duplicate = false; > e->callee->symbol.externally_visible = false; > - update_noncloned_frequencies (e->callee, e->frequency); > + // In the case of a COMDAT, the callee's count may be from other > + // modules, and we need to scale it for the current module's calls > + // (e.g. e->count may be 0 despite e->callee->count > 0). > + gcov_type count_scale = REG_BR_PROB_BASE; > + if (e->callee->count > e->count) > + count_scale = GCOV_COMPUTE_SCALE (e->count, e->callee->count); > + update_noncloned_frequencies (e->callee, count_scale, e->frequency); > } > else > { ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-17 21:02 ` Jan Hubicka @ 2013-08-19 13:51 ` Teresa Johnson 2013-08-19 15:16 ` Jan Hubicka 2013-08-19 15:34 ` Teresa Johnson 1 sibling, 1 reply; 62+ messages in thread From: Teresa Johnson @ 2013-08-19 13:51 UTC (permalink / raw) To: Jan Hubicka Cc: Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, marxin.liska, Sriraman Tallam On Sat, Aug 17, 2013 at 1:44 PM, Jan Hubicka <hubicka@ucw.cz> wrote: >> >> I added both of these and ran into issues due to profile maintenance. >> For example, there were non-zero blocks in the cold section because >> pro_and_epilogue split a simple return block that was previously reach >> by both hot and cold paths. The new return block that was then only >> reached via the cold path did not have its count properly updated to >> reflect this, and since with this patch, blocks dominated by cold >> blocks are remarked cold, we ended up with a non-zero count block in >> the cold section. And there were 0 count blocks reached by non-zero >> edges because copyprop did not clean up edge weights after removing >> some branches and blocks, leading to non-zero edge weights that had >> previously targeted a branch that was removed, now targeting a 0 count >> block that the removed branch always branched around. > > I see, can you please send fixes for the problems you identified? > Thanks for owrking on this! I don't have fixes at this point - I just identified the phase and transformation from looking at the dump. But I'll try to fix them soon while I'm working on performance tuning for splitting. I have a feeling there are probably a bunch of places where the profile isn't getting updated properly, unfortunately. >> >> In any case, the good news is in that the cases I looked at, the >> splitting code is doing the right thing and these blocks that were >> marked cold really were cold. It would be great to fix the profile >> maintenance issues, but that in the meantime the above sanity checks >> are too aggressive. > > We can keep them and output info into dump file - it is what most of > the profile sanity checking does anyway. Ok, I will add that. > > Did you try to use Martin's linker script to turn text.unlikely > section unexecutable? I think that way we will easily find what really > causes us to use it during startup of trained application (just like > Martin does for gimp). I haven't - where can I get that script? >> >> I think it makes sense to commit the current patch if possible, as it >> is making the splitting more sane. > > My only concern about the patch is that I am not convinced the dominator > based code has chance to work reliably enough so we won't see too many > accesses into the cold section. Remember it isn't using dominance anymore. The latest patch was instead ensuring the most frequent path between hot blocks and the entry/exit are marked hot. That should be better than the dominance approach used in the earlier version. > We can commit it and work on better solution incrementally but it will > probably mean replacing it later. If you think it makes things easier > to work on it incrementally, I think the patch is OK. Yes, I think this is a big step forward from what is there now for splitting, which does the splitting purely based on bb count in isolation. I don't have a much better solution in mind yet. >> >> > - I'll try building and profiling gimp myself to see if I can >> > reproduce the issue with code executing out of the cold section. >> >> I have spent some time this week trying to get the latest gimp Martin >> pointed me to configured and built, but it took awhile to track down >> and configure/build all of the required versions of dependent >> packages. I'm still hitting some issues trying to get it compiled, so >> it may not yet be configured properly. I'll take a look again early >> next week. > > I do not think there is anything special about gimp. You can probably > take any other bigger app, like GCC itself. With profiledbootstrap > and linker script to lock unlikely section you should get ICEs where > we jump into cold secton and should not. Ok, please point me to the linker script and I will try gcc profiledbootstrap as well. I wanted to try gimp if possible as I haven't seen this much jumping to the cold section in some of the internal apps I tried. Thanks, Teresa >> >> Teresa >> >> patch for updating counts based on estimated frequencies to address >> inlined comdats with 0 profile counts: >> >> 013-08-16 Teresa Johnson <tejohnson@google.com> >> >> * tree-inline.c (copy_bb): Compute count based on frequency. >> (copy_edges_for_bb): Ditto. >> (copy_cfg_body): Ditto. >> (copy_body): Pass down frequency. >> (expand_call_inline): Ditto. >> (tree_function_versioning): Ditto. >> * predict.c (init_and_estimate_bb_frequencies): New function. >> (rebuild_frequencies): Invoke init_and_estimate_bb_frequencies. >> * predict.h (init_and_estimate_bb_frequencies): Declare. >> * profile.c (branch_prob): Invoke init_and_estimate_bb_frequencies. >> * ipa-inline-transform.c (update_noncloned_frequencies): Scale edge >> counts. >> (clone_inlined_nodes): Compute edge count scale if needed. > > I do not see why inliner needs to care about scaling more than it does right > now. So you have init_and_estimate_bb_frequencies that force profile guessing > on a given function body. In addition to that I thing you need something like > freqs_to_counts that will compute counts based on freqs with given scale > (actually you can do that as part of propagation before frequencies are scalled > to the usual 0...FREQ_MAX scale and precision is lost). > > Because offline COMDAT functoin will be porduced for every COMDAT used, I think > it is bad to porduce any COMDAT (or any reachable function via calls with non-0 > count) that has empty profile (either because it got lost by COMDAT merging > or because of reading mismatch). > > So I guess you can just check functions with 0 count and non-0 count callers > and initialize their guessed profile. > Some capping will probably be needed to not propagate insanely large numbers.. > > Since new direct calls can be discovered later, inline may want to do that > again each time it inlines non-0 count call of COMDAT with 0 count... > > Honza >> >> Index: tree-inline.c >> =================================================================== >> --- tree-inline.c (revision 201644) >> +++ tree-inline.c (working copy) >> @@ -1502,7 +1502,7 @@ remap_gimple_stmt (gimple stmt, copy_body_data *id >> >> static basic_block >> copy_bb (copy_body_data *id, basic_block bb, int frequency_scale, >> - gcov_type count_scale) >> + gcov_type count_scale, gcov_type freq_to_count_scale) >> { >> gimple_stmt_iterator gsi, copy_gsi, seq_gsi; >> basic_block copy_basic_block; >> @@ -1519,7 +1519,13 @@ copy_bb (copy_body_data *id, basic_block bb, int f >> basic_block_info automatically. */ >> copy_basic_block = create_basic_block (NULL, (void *) 0, >> (basic_block) prev->aux); >> - copy_basic_block->count = apply_scale (bb->count, count_scale); >> + copy_basic_block->count >> + = (count_scale >> + ? apply_scale (bb->count, count_scale) >> + /* When the callee bb counts were all zero (e.g. this was a COMDAT >> + that didn't get profile counts) then we compute the new bb counts >> + via the statically-estimated frequencies. */ >> + : RDIV ((gcov_type)bb->frequency * freq_to_count_scale, BB_FREQ_MAX)); >> >> /* We are going to rebuild frequencies from scratch. These values >> have just small importance to drive canonicalize_loop_headers. */ >> @@ -1888,7 +1894,8 @@ update_ssa_across_abnormal_edges (basic_block bb, >> debug stmts are left after a statement that must end the basic block. */ >> >> static bool >> -copy_edges_for_bb (basic_block bb, gcov_type count_scale, basic_block ret_bb, >> +copy_edges_for_bb (basic_block bb, gcov_type count_scale, >> + basic_block ret_bb, >> bool can_make_abnormal_goto) >> { >> basic_block new_bb = (basic_block) bb->aux; >> @@ -1912,7 +1919,14 @@ static bool >> && old_edge->dest->aux != EXIT_BLOCK_PTR) >> flags |= EDGE_FALLTHRU; >> new_edge = make_edge (new_bb, (basic_block) old_edge->dest->aux, flags); >> - new_edge->count = apply_scale (old_edge->count, count_scale); >> + basic_block new_src_bb = (basic_block) old_edge->src->aux; >> + new_edge->count >> + = (count_scale >> + ? apply_scale (old_edge->count, count_scale) >> + // The bb counts have already been scaled with >> freq_to_count_scale >> + // when that is non-zero, so just scale that new bb count by >> + // the edge probability. >> + : apply_probability (new_src_bb->count, old_edge->probability)); >> new_edge->probability = old_edge->probability; >> } >> >> @@ -2282,7 +2296,8 @@ redirect_all_calls (copy_body_data * id, basic_blo >> another function. Walks FN via CFG, returns new fndecl. */ >> >> static tree >> -copy_cfg_body (copy_body_data * id, gcov_type count, int frequency_scale, >> +copy_cfg_body (copy_body_data * id, gcov_type count, >> + int frequency, int frequency_scale, >> basic_block entry_block_map, basic_block exit_block_map, >> bitmap blocks_to_copy, basic_block new_entry) >> { >> @@ -2293,15 +2308,20 @@ static tree >> basic_block bb; >> tree new_fndecl = NULL; >> bool need_debug_cleanup = false; >> - gcov_type count_scale; >> + gcov_type count_scale = 0; >> + gcov_type freq_to_count_scale = 0; >> int last; >> int incoming_frequency = 0; >> gcov_type incoming_count = 0; >> >> - if (ENTRY_BLOCK_PTR_FOR_FUNCTION (src_cfun)->count) >> - count_scale >> - = GCOV_COMPUTE_SCALE (count, >> - ENTRY_BLOCK_PTR_FOR_FUNCTION (src_cfun)->count); >> + basic_block entry_bb = ENTRY_BLOCK_PTR_FOR_FUNCTION (src_cfun); >> + if (entry_bb->count) >> + count_scale = GCOV_COMPUTE_SCALE (count, entry_bb->count); >> + /* When the callee bb counts were all zero (e.g. this was a COMDAT >> + that didn't get profile counts) then we compute the new bb counts >> + via the statically-estimated frequencies. */ >> + else if (entry_bb->frequency) >> + freq_to_count_scale = RDIV (count * frequency, entry_bb->frequency); >> else >> count_scale = REG_BR_PROB_BASE; >> >> @@ -2323,7 +2343,13 @@ static tree >> incoming_frequency += EDGE_FREQUENCY (e); >> incoming_count += e->count; >> } >> - incoming_count = apply_scale (incoming_count, count_scale); >> + incoming_count >> + = (count_scale >> + ? apply_scale (incoming_count, count_scale) >> + /* When the callee bb counts were all zero (e.g. this was a COMDAT >> + that didn't get profile counts) then we compute the >> new bb counts >> + via the statically-estimated frequencies. */ >> + : RDIV (incoming_frequency * freq_to_count_scale, BB_FREQ_MAX)); >> incoming_frequency >> = apply_scale ((gcov_type)incoming_frequency, frequency_scale); >> ENTRY_BLOCK_PTR->count = incoming_count; >> @@ -2350,7 +2376,8 @@ static tree >> FOR_EACH_BB_FN (bb, cfun_to_copy) >> if (!blocks_to_copy || bitmap_bit_p (blocks_to_copy, bb->index)) >> { >> - basic_block new_bb = copy_bb (id, bb, frequency_scale, count_scale); >> + basic_block new_bb = copy_bb (id, bb, frequency_scale, count_scale, >> + freq_to_count_scale); >> bb->aux = new_bb; >> new_bb->aux = bb; >> new_bb->loop_father = entry_block_map->loop_father; >> @@ -2364,7 +2391,8 @@ static tree >> FOR_ALL_BB_FN (bb, cfun_to_copy) >> if (!blocks_to_copy >> || (bb->index > 0 && bitmap_bit_p (blocks_to_copy, bb->index))) >> - need_debug_cleanup |= copy_edges_for_bb (bb, count_scale, exit_block_map, >> + need_debug_cleanup |= copy_edges_for_bb (bb, count_scale, >> + exit_block_map, >> can_make_abormal_goto); >> >> if (new_entry) >> @@ -2562,7 +2590,8 @@ copy_tree_body (copy_body_data *id) >> another function. */ >> >> static tree >> -copy_body (copy_body_data *id, gcov_type count, int frequency_scale, >> +copy_body (copy_body_data *id, gcov_type count, int frequency, >> + int frequency_scale, >> basic_block entry_block_map, basic_block exit_block_map, >> bitmap blocks_to_copy, basic_block new_entry) >> { >> @@ -2571,7 +2600,8 @@ static tree >> >> /* If this body has a CFG, walk CFG and copy. */ >> gcc_assert (ENTRY_BLOCK_PTR_FOR_FUNCTION (DECL_STRUCT_FUNCTION (fndecl))); >> - body = copy_cfg_body (id, count, frequency_scale, entry_block_map, >> exit_block_map, >> + body = copy_cfg_body (id, count, frequency, frequency_scale, >> + entry_block_map, exit_block_map, >> blocks_to_copy, new_entry); >> copy_debug_stmts (id); >> >> @@ -4172,7 +4202,7 @@ expand_call_inline (basic_block bb, gimple stmt, c >> function in any way before this point, as this CALL_EXPR may be >> a self-referential call; if we're calling ourselves, we need to >> duplicate our body before altering anything. */ >> - copy_body (id, bb->count, >> + copy_body (id, bb->count, bb->frequency, >> GCOV_COMPUTE_SCALE (cg_edge->frequency, CGRAPH_FREQ_BASE), >> bb, return_block, NULL, NULL); >> >> @@ -5299,8 +5329,9 @@ tree_function_versioning (tree old_decl, tree new_ >> } >> >> /* Copy the Function's body. */ >> - copy_body (&id, old_entry_block->count, REG_BR_PROB_BASE, >> - ENTRY_BLOCK_PTR, EXIT_BLOCK_PTR, blocks_to_copy, new_entry); >> + copy_body (&id, old_entry_block->count, old_entry_block->frequency, >> + REG_BR_PROB_BASE, ENTRY_BLOCK_PTR, EXIT_BLOCK_PTR, >> + blocks_to_copy, new_entry); >> >> /* Renumber the lexical scoping (non-code) blocks consecutively. */ >> number_blocks (new_decl); >> Index: predict.c >> =================================================================== >> --- predict.c (revision 201644) >> +++ predict.c (working copy) >> @@ -2976,6 +2976,24 @@ make_pass_strip_predict_hints (gcc::context *ctxt) >> return new pass_strip_predict_hints (ctxt); >> } >> >> +/* Initialize loop edges and compute estimated bb frequencies when there >> + is no profile data available. */ >> + >> +void >> +init_and_estimate_bb_frequencies (void) >> +{ >> + if (profile_status == PROFILE_READ && counts_to_freqs ()) >> + return; >> + >> + loop_optimizer_init (0); >> + add_noreturn_fake_exit_edges (); >> + mark_irreducible_loops (); >> + connect_infinite_loops_to_exit (); >> + estimate_bb_frequencies (); >> + remove_fake_exit_edges (); >> + loop_optimizer_finalize (); >> +} >> + >> /* Rebuild function frequencies. Passes are in general expected to >> maintain profile by hand, however in some cases this is not possible: >> for example when inlining several functions with loops freuqencies might run >> @@ -2986,15 +3004,7 @@ rebuild_frequencies (void) >> { >> timevar_push (TV_REBUILD_FREQUENCIES); >> if (profile_status == PROFILE_GUESSED) >> - { >> - loop_optimizer_init (0); >> - add_noreturn_fake_exit_edges (); >> - mark_irreducible_loops (); >> - connect_infinite_loops_to_exit (); >> - estimate_bb_frequencies (); >> - remove_fake_exit_edges (); >> - loop_optimizer_finalize (); >> - } >> + init_and_estimate_bb_frequencies (); >> else if (profile_status == PROFILE_READ) >> counts_to_freqs (); >> else >> Index: predict.h >> =================================================================== >> --- predict.h (revision 201644) >> +++ predict.h (working copy) >> @@ -38,6 +38,7 @@ enum prediction >> extern void predict_insn_def (rtx, enum br_predictor, enum prediction); >> extern int counts_to_freqs (void); >> extern void estimate_bb_frequencies (void); >> +extern void init_and_estimate_bb_frequencies (void); >> extern const char *predictor_name (enum br_predictor); >> extern tree build_predict_expr (enum br_predictor, enum prediction); >> extern void tree_estimate_probability (void); >> Index: profile.c >> =================================================================== >> --- profile.c (revision 201644) >> +++ profile.c (working copy) >> @@ -1305,6 +1305,12 @@ branch_prob (void) >> >> values.release (); >> free_edge_list (el); >> + >> + /* Call after setting profile_status to PROFILE_READ, will then >> + invoke counts_to_freqs and if the sum of the counts is zero, will >> + estimate the frequencies. */ >> + init_and_estimate_bb_frequencies (); >> + >> coverage_end_function (lineno_checksum, cfg_checksum); >> } >> ^L >> Index: ipa-inline-transform.c >> =================================================================== >> --- ipa-inline-transform.c (revision 201644) >> +++ ipa-inline-transform.c (working copy) >> @@ -51,7 +51,7 @@ int nfunctions_inlined; >> >> static void >> update_noncloned_frequencies (struct cgraph_node *node, >> - int freq_scale) >> + gcov_type count_scale, int freq_scale) >> { >> struct cgraph_edge *e; >> >> @@ -60,14 +60,16 @@ update_noncloned_frequencies (struct cgraph_node * >> freq_scale = 1; >> for (e = node->callees; e; e = e->next_callee) >> { >> + e->count = apply_scale (e->count, count_scale); >> e->frequency = e->frequency * (gcov_type) freq_scale / CGRAPH_FREQ_BASE; >> if (e->frequency > CGRAPH_FREQ_MAX) >> e->frequency = CGRAPH_FREQ_MAX; >> if (!e->inline_failed) >> - update_noncloned_frequencies (e->callee, freq_scale); >> + update_noncloned_frequencies (e->callee, count_scale, freq_scale); >> } >> for (e = node->indirect_calls; e; e = e->next_callee) >> { >> + e->count = apply_scale (e->count, count_scale); >> e->frequency = e->frequency * (gcov_type) freq_scale / CGRAPH_FREQ_BASE; >> if (e->frequency > CGRAPH_FREQ_MAX) >> e->frequency = CGRAPH_FREQ_MAX; >> @@ -169,7 +171,13 @@ clone_inlined_nodes (struct cgraph_edge *e, bool d >> } >> duplicate = false; >> e->callee->symbol.externally_visible = false; >> - update_noncloned_frequencies (e->callee, e->frequency); >> + // In the case of a COMDAT, the callee's count may be from other >> + // modules, and we need to scale it for the current module's calls >> + // (e.g. e->count may be 0 despite e->callee->count > 0). >> + gcov_type count_scale = REG_BR_PROB_BASE; >> + if (e->callee->count > e->count) >> + count_scale = GCOV_COMPUTE_SCALE (e->count, e->callee->count); >> + update_noncloned_frequencies (e->callee, count_scale, e->frequency); >> } >> else >> { -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-19 13:51 ` Teresa Johnson @ 2013-08-19 15:16 ` Jan Hubicka 2013-08-19 17:48 ` Teresa Johnson 0 siblings, 1 reply; 62+ messages in thread From: Jan Hubicka @ 2013-08-19 15:16 UTC (permalink / raw) To: Teresa Johnson Cc: Jan Hubicka, Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, marxin.liska, Sriraman Tallam > Remember it isn't using dominance anymore. The latest patch was > instead ensuring the most frequent path between hot blocks and the > entry/exit are marked hot. That should be better than the dominance > approach used in the earlier version. Indeed, that looks more resonable approach. Can you point me to the last version of patch? Last one I remember still walked dominators... > > > We can commit it and work on better solution incrementally but it will > > probably mean replacing it later. If you think it makes things easier > > to work on it incrementally, I think the patch is OK. > > Yes, I think this is a big step forward from what is there now for > splitting, which does the splitting purely based on bb count in > isolation. I don't have a much better solution in mind yet. > > >> > >> > - I'll try building and profiling gimp myself to see if I can > >> > reproduce the issue with code executing out of the cold section. > >> > >> I have spent some time this week trying to get the latest gimp Martin > >> pointed me to configured and built, but it took awhile to track down > >> and configure/build all of the required versions of dependent > >> packages. I'm still hitting some issues trying to get it compiled, so > >> it may not yet be configured properly. I'll take a look again early > >> next week. > > > > I do not think there is anything special about gimp. You can probably > > take any other bigger app, like GCC itself. With profiledbootstrap > > and linker script to lock unlikely section you should get ICEs where > > we jump into cold secton and should not. > > Ok, please point me to the linker script and I will try gcc > profiledbootstrap as well. I wanted to try gimp if possible as I > haven't seen this much jumping to the cold section in some of the > internal apps I tried. You can also discuss with Martin the systemtap script to plot disk accesses during the startup. It is very handy for analyzing the code layout issues It may be interesting to get similar script taking traces from valgrind and ploting the most frequent calls in the final layout ;) Honza ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-19 15:16 ` Jan Hubicka @ 2013-08-19 17:48 ` Teresa Johnson 2013-08-19 19:56 ` Martin Liška 2013-08-27 18:12 ` Teresa Johnson 0 siblings, 2 replies; 62+ messages in thread From: Teresa Johnson @ 2013-08-19 17:48 UTC (permalink / raw) To: Jan Hubicka Cc: Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, marxin.liska, Sriraman Tallam On Mon, Aug 19, 2013 at 8:09 AM, Jan Hubicka <hubicka@ucw.cz> wrote: >> Remember it isn't using dominance anymore. The latest patch was >> instead ensuring the most frequent path between hot blocks and the >> entry/exit are marked hot. That should be better than the dominance >> approach used in the earlier version. > > Indeed, that looks more resonable approach. > Can you point me to the last version of patch? Last one I remember still > walked dominators... I've included the latest patch below. I still use dominators in the post-cfg-optimization fixup (fixup_partitions), but not in the partition sanitizing done during the partitioning itself (sanitize_hot_paths). The former is looking for hot bbs newly dominated by cold bbs after cfg transformations. >> >> > We can commit it and work on better solution incrementally but it will >> > probably mean replacing it later. If you think it makes things easier >> > to work on it incrementally, I think the patch is OK. >> >> Yes, I think this is a big step forward from what is there now for >> splitting, which does the splitting purely based on bb count in >> isolation. I don't have a much better solution in mind yet. >> >> >> >> >> > - I'll try building and profiling gimp myself to see if I can >> >> > reproduce the issue with code executing out of the cold section. >> >> >> >> I have spent some time this week trying to get the latest gimp Martin >> >> pointed me to configured and built, but it took awhile to track down >> >> and configure/build all of the required versions of dependent >> >> packages. I'm still hitting some issues trying to get it compiled, so >> >> it may not yet be configured properly. I'll take a look again early >> >> next week. >> > >> > I do not think there is anything special about gimp. You can probably >> > take any other bigger app, like GCC itself. With profiledbootstrap >> > and linker script to lock unlikely section you should get ICEs where >> > we jump into cold secton and should not. >> >> Ok, please point me to the linker script and I will try gcc >> profiledbootstrap as well. I wanted to try gimp if possible as I >> haven't seen this much jumping to the cold section in some of the >> internal apps I tried. > > You can also discuss with Martin the systemtap script to plot disk accesses > during the startup. It is very handy for analyzing the code layout issues Ok. I am using linux perf to collect this info (fed through some scripts that munge and plot the data). > > It may be interesting to get similar script taking traces from valgrind > and ploting the most frequent calls in the final layout ;) I think linux perf -g to get a callgraph should give similar data. Teresa > > Honza 2013-08-05 Teresa Johnson <tejohnson@google.com> Steven Bosscher <steven@gcc.gnu.org> * cfgrtl.c (fixup_new_cold_bb): New routine. (commit_edge_insertions): Invoke fixup_partitions. (find_partition_fixes): New routine. (fixup_partitions): Ditto. (verify_hot_cold_block_grouping): Update comments. (rtl_verify_edges): Invoke find_partition_fixes. (rtl_verify_bb_pointers): Update comments. (rtl_verify_bb_layout): Ditto. * basic-block.h (fixup_partitions): Declare. * cfgcleanup.c (try_optimize_cfg): Invoke fixup_partitions. * bb-reorder.c (sanitize_hot_paths): New function. (find_rarely_executed_basic_blocks_and_crossing_edges): Invoke sanitize_hot_paths. Index: cfgrtl.c =================================================================== --- cfgrtl.c (revision 201461) +++ cfgrtl.c (working copy) @@ -1341,6 +1341,43 @@ fixup_partition_crossing (edge e) } } +/* Called when block BB has been reassigned to the cold partition, + because it is now dominated by another cold block, + to ensure that the region crossing attributes are updated. */ + +static void +fixup_new_cold_bb (basic_block bb) +{ + edge e; + edge_iterator ei; + + /* This is called when a hot bb is found to now be dominated + by a cold bb and therefore needs to become cold. Therefore, + its preds will no longer be region crossing. Any non-dominating + preds that were previously hot would also have become cold + in the caller for the same region. Any preds that were previously + region-crossing will be adjusted in fixup_partition_crossing. */ + FOR_EACH_EDGE (e, ei, bb->preds) + { + fixup_partition_crossing (e); + } + + /* Possibly need to make bb's successor edges region crossing, + or remove stale region crossing. */ + FOR_EACH_EDGE (e, ei, bb->succs) + { + /* We can't have fall-through edges across partition boundaries. + Note that force_nonfallthru will do any necessary partition + boundary fixup by calling fixup_partition_crossing itself. */ + if ((e->flags & EDGE_FALLTHRU) + && BB_PARTITION (bb) != BB_PARTITION (e->dest) + && e->dest != EXIT_BLOCK_PTR) + force_nonfallthru (e); + else + fixup_partition_crossing (e); + } +} + /* Attempt to change code to redirect edge E to TARGET. Don't do that on expense of adding new instructions or reordering basic blocks. @@ -1979,6 +2016,14 @@ commit_edge_insertions (void) { basic_block bb; + /* Optimization passes that invoke this routine can cause hot blocks + previously reached by both hot and cold blocks to become dominated only + by cold blocks. This will cause the verification below to fail, + and lead to now cold code in the hot section. In some cases this + may only be visible after newly unreachable blocks are deleted, + which will be done by fixup_partitions. */ + fixup_partitions (); + #ifdef ENABLE_CHECKING verify_flow_info (); #endif @@ -2173,6 +2218,101 @@ get_last_bb_insn (basic_block bb) return end; } +/* Sanity check partition hotness to ensure that basic blocks in + the cold partition don't dominate basic blocks in the hot partition. + If FLAG_ONLY is true, report violations as errors. Otherwise + re-mark the dominated blocks as cold, since this is run after + cfg optimizations that may make hot blocks previously reached + by both hot and cold blocks now only reachable along cold paths. */ + +static vec<basic_block> +find_partition_fixes (bool flag_only) +{ + basic_block bb; + vec<basic_block> bbs_in_cold_partition = vNULL; + vec<basic_block> bbs_to_fix = vNULL; + + /* Callers check this. */ + gcc_checking_assert (crtl->has_bb_partition); + + FOR_EACH_BB (bb) + if ((BB_PARTITION (bb) == BB_COLD_PARTITION)) + bbs_in_cold_partition.safe_push (bb); + + if (bbs_in_cold_partition.is_empty ()) + return vNULL; + + bool dom_calculated_here = !dom_info_available_p (CDI_DOMINATORS); + + if (dom_calculated_here) + calculate_dominance_info (CDI_DOMINATORS); + + while (! bbs_in_cold_partition.is_empty ()) + { + bb = bbs_in_cold_partition.pop (); + /* Any blocks dominated by a block in the cold section + must also be cold. */ + basic_block son; + for (son = first_dom_son (CDI_DOMINATORS, bb); + son; + son = next_dom_son (CDI_DOMINATORS, son)) + { + /* If son is not yet cold, then mark it cold here and + enqueue it for further processing. */ + if ((BB_PARTITION (son) != BB_COLD_PARTITION)) + { + if (flag_only) + error ("non-cold basic block %d dominated " + "by a block in the cold partition (%d)", son->index, bb->index); + else + BB_SET_PARTITION (son, BB_COLD_PARTITION); + bbs_to_fix.safe_push (son); + bbs_in_cold_partition.safe_push (son); + } + } + } + + if (dom_calculated_here) + free_dominance_info (CDI_DOMINATORS); + + return bbs_to_fix; +} + +/* Perform cleanup on the hot/cold bb partitioning after optimization + passes that modify the cfg. */ + +void +fixup_partitions (void) +{ + basic_block bb; + + if (!crtl->has_bb_partition) + return; + + /* Delete any blocks that became unreachable and weren't + already cleaned up, for example during edge forwarding + and convert_jumps_to_returns. This will expose more + opportunities for fixing the partition boundaries here. + Also, the calculation of the dominance graph during verification + will assert if there are unreachable nodes. */ + delete_unreachable_blocks (); + + /* If there are partitions, do a sanity check on them: A basic block in + a cold partition cannot dominate a basic block in a hot partition. + Fixup any that now violate this requirement, as a result of edge + forwarding and unreachable block deletion. */ + vec<basic_block> bbs_to_fix = find_partition_fixes (false); + + /* Do the partition fixup after all necessary blocks have been converted to + cold, so that we only update the region crossings the minimum number of + places, which can require forcing edges to be non fallthru. */ + while (! bbs_to_fix.is_empty ()) + { + bb = bbs_to_fix.pop (); + fixup_new_cold_bb (bb); + } +} + /* Verify, in the basic block chain, that there is at most one switch between hot/cold partitions. This condition will not be true until after reorder_basic_blocks is called. */ @@ -2219,7 +2359,8 @@ verify_hot_cold_block_grouping (void) /* Perform several checks on the edges out of each block, such as the consistency of the branch probabilities, the correctness of hot/cold partition crossing edges, and the number of expected - successor edges. */ + successor edges. Also verify that the dominance relationship + between hot/cold blocks is sane. */ static int rtl_verify_edges (void) @@ -2382,6 +2523,14 @@ rtl_verify_edges (void) } } + /* If there are partitions, do a sanity check on them: A basic block in + a cold partition cannot dominate a basic block in a hot partition. */ + if (crtl->has_bb_partition && !err) + { + vec<basic_block> bbs_to_fix = find_partition_fixes (true); + err = !bbs_to_fix.is_empty (); + } + /* Clean up. */ return err; } @@ -2515,7 +2664,7 @@ rtl_verify_bb_pointers (void) and NOTE_INSN_BASIC_BLOCK - verify that no fall_thru edge crosses hot/cold partition boundaries - verify that there are no pending RTL branch predictions - - verify that there is a single hot/cold partition boundary after bbro + - verify that hot blocks are not dominated by cold blocks In future it can be extended check a lot of other stuff as well (reachability of basic blocks, life information, etc. etc.). */ @@ -2761,7 +2910,8 @@ rtl_verify_bb_layout (void) - check that all insns are in the basic blocks (except the switch handling code, barriers and notes) - check that all returns are followed by barriers - - check that all fallthru edge points to the adjacent blocks. */ + - check that all fallthru edge points to the adjacent blocks + - verify that there is a single hot/cold partition boundary after bbro */ static int rtl_verify_flow_info (void) Index: basic-block.h =================================================================== --- basic-block.h (revision 201461) +++ basic-block.h (working copy) @@ -797,6 +797,7 @@ extern bool contains_no_active_insn_p (const_basic extern bool forwarder_block_p (const_basic_block); extern bool can_fallthru (basic_block, basic_block); extern void emit_barrier_after_bb (basic_block bb); +extern void fixup_partitions (void); /* In cfgbuild.c. */ extern void find_many_sub_basic_blocks (sbitmap); Index: cfgcleanup.c =================================================================== --- cfgcleanup.c (revision 201461) +++ cfgcleanup.c (working copy) @@ -2807,10 +2807,21 @@ try_optimize_cfg (int mode) df_analyze (); } + if (changed) + { + /* Edge forwarding in particular can cause hot blocks previously + reached by both hot and cold blocks to become dominated only + by cold blocks. This will cause the verification below to fail, + and lead to now cold code in the hot section. This is not easy + to detect and fix during edge forwarding, and in some cases + is only visible after newly unreachable blocks are deleted, + which will be done in fixup_partitions. */ + fixup_partitions (); + #ifdef ENABLE_CHECKING - if (changed) - verify_flow_info (); + verify_flow_info (); #endif + } changed_overall |= changed; first_pass = false; Index: bb-reorder.c =================================================================== --- bb-reorder.c (revision 201461) +++ bb-reorder.c (working copy) @@ -1444,27 +1444,134 @@ fix_up_crossing_landing_pad (eh_landing_pad old_lp ei_next (&ei); } + +/* Ensure that all hot bbs are included in a hot path through the + procedure. This is done by calling this function twice, once + with WALK_UP true (to look for paths from the entry to hot bbs) and + once with WALK_UP false (to look for paths from hot bbs to the exit). + Returns the updated value of COLD_BB_COUNT and adds newly-hot bbs + to BBS_IN_HOT_PARTITION. */ + +static unsigned int +sanitize_hot_paths (bool walk_up, unsigned int cold_bb_count, + vec<basic_block> *bbs_in_hot_partition) +{ + /* Callers check this. */ + gcc_checking_assert (cold_bb_count); + + /* Keep examining hot bbs while we still have some left to check + and there are remaining cold bbs. */ + vec<basic_block> hot_bbs_to_check = bbs_in_hot_partition->copy (); + while (! hot_bbs_to_check.is_empty () + && cold_bb_count) + { + basic_block bb = hot_bbs_to_check.pop (); + vec<edge, va_gc> *edges = walk_up ? bb->preds : bb->succs; + edge e; + edge_iterator ei; + int highest_probability = 0; + bool found = false; + + /* Walk the preds/succs and check if there is at least one already + marked hot. Keep track of the most frequent pred/succ so that we + can mark it hot if we don't find one. */ + FOR_EACH_EDGE (e, ei, edges) + { + basic_block reach_bb = walk_up ? e->src : e->dest; + + if (e->flags & EDGE_DFS_BACK) + continue; + + if (BB_PARTITION (reach_bb) != BB_COLD_PARTITION) + { + found = true; + break; + } + if (e->probability > highest_probability) + highest_probability = e->probability; + } + + /* If bb is reached by (or reaches, in the case of !WALK_UP) another hot + block (or unpartitioned, e.g. the entry block) then it is ok. If not, + then the most frequent pred (or succ) needs to be adjusted. In the + case where multiple preds/succs have the same probability (e.g. a + 50-50 branch), then both will be adjusted. */ + if (found) + continue; + + FOR_EACH_EDGE (e, ei, edges) + { + if (e->flags & EDGE_DFS_BACK) + continue; + if (e->probability < highest_probability) + continue; + + basic_block reach_bb = walk_up ? e->src : e->dest; + + /* We have a hot bb with an immediate dominator that is cold. + The dominator needs to be re-marked hot. */ + BB_SET_PARTITION (reach_bb, BB_HOT_PARTITION); + cold_bb_count--; + + /* Now we need to examine newly-hot reach_bb to see if it is also + dominated by a cold bb. */ + bbs_in_hot_partition->safe_push (reach_bb); + hot_bbs_to_check.safe_push (reach_bb); + } + } + + return cold_bb_count; +} + + /* Find the basic blocks that are rarely executed and need to be moved to a separate section of the .o file (to cut down on paging and improve cache locality). Return a vector of all edges that cross. */ -static vec<edge> +static vec<edge> find_rarely_executed_basic_blocks_and_crossing_edges (void) { vec<edge> crossing_edges = vNULL; basic_block bb; edge e; edge_iterator ei; + unsigned int cold_bb_count = 0; + vec<basic_block> bbs_in_hot_partition = vNULL; /* Mark which partition (hot/cold) each basic block belongs in. */ FOR_EACH_BB (bb) { if (probably_never_executed_bb_p (cfun, bb)) - BB_SET_PARTITION (bb, BB_COLD_PARTITION); + { + BB_SET_PARTITION (bb, BB_COLD_PARTITION); + cold_bb_count++; + } else - BB_SET_PARTITION (bb, BB_HOT_PARTITION); + { + BB_SET_PARTITION (bb, BB_HOT_PARTITION); + bbs_in_hot_partition.safe_push (bb); + } } + /* Ensure that hot bbs are included along a hot path from the entry to exit. + Several different possibilities may include cold bbs along all paths + to/from a hot bb. One is that there are edge weight insanities + due to optimization phases that do not properly update basic block profile + counts. The second is that the entry of the function may not be hot, because + it is entered fewer times than the number of profile training runs, but there + is a loop inside the function that causes blocks within the function to be + above the threshold for hotness. This is fixed by walking up from hot bbs + to the entry block, and then down from hot bbs to the exit, performing + partitioning fixups as necessary. */ + if (cold_bb_count) + { + mark_dfs_back_edges (); + cold_bb_count = sanitize_hot_paths (true, cold_bb_count, + &bbs_in_hot_partition); + if (cold_bb_count) + sanitize_hot_paths (false, cold_bb_count, &bbs_in_hot_partition); + } + /* The format of .gcc_except_table does not allow landing pads to be in a different partition as the throw. Fix this by either moving or duplicating the landing pads. */ -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-19 17:48 ` Teresa Johnson @ 2013-08-19 19:56 ` Martin Liška 2013-08-27 18:12 ` Teresa Johnson 1 sibling, 0 replies; 62+ messages in thread From: Martin Liška @ 2013-08-19 19:56 UTC (permalink / raw) To: Teresa Johnson Cc: Jan Hubicka, Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, Sriraman Tallam [-- Attachment #1: Type: text/plain, Size: 19977 bytes --] Dear Teresa, On 19 August 2013 19:47, Teresa Johnson <tejohnson@google.com> wrote: > On Mon, Aug 19, 2013 at 8:09 AM, Jan Hubicka <hubicka@ucw.cz> wrote: >>> Remember it isn't using dominance anymore. The latest patch was >>> instead ensuring the most frequent path between hot blocks and the >>> entry/exit are marked hot. That should be better than the dominance >>> approach used in the earlier version. >> >> Indeed, that looks more resonable approach. >> Can you point me to the last version of patch? Last one I remember still >> walked dominators... > > I've included the latest patch below. I still use dominators in the > post-cfg-optimization fixup (fixup_partitions), but not in the > partition sanitizing done during the partitioning itself > (sanitize_hot_paths). The former is looking for hot bbs newly > dominated by cold bbs after cfg transformations. > >>> >>> > We can commit it and work on better solution incrementally but it will >>> > probably mean replacing it later. If you think it makes things easier >>> > to work on it incrementally, I think the patch is OK. >>> >>> Yes, I think this is a big step forward from what is there now for >>> splitting, which does the splitting purely based on bb count in >>> isolation. I don't have a much better solution in mind yet. >>> >>> >> >>> >> > - I'll try building and profiling gimp myself to see if I can >>> >> > reproduce the issue with code executing out of the cold section. >>> >> >>> >> I have spent some time this week trying to get the latest gimp Martin >>> >> pointed me to configured and built, but it took awhile to track down >>> >> and configure/build all of the required versions of dependent >>> >> packages. I'm still hitting some issues trying to get it compiled, so >>> >> it may not yet be configured properly. I'll take a look again early >>> >> next week. >>> > >>> > I do not think there is anything special about gimp. You can probably >>> > take any other bigger app, like GCC itself. With profiledbootstrap >>> > and linker script to lock unlikely section you should get ICEs where >>> > we jump into cold secton and should not. >>> >>> Ok, please point me to the linker script and I will try gcc >>> profiledbootstrap as well. I wanted to try gimp if possible as I >>> haven't seen this much jumping to the cold section in some of the >>> internal apps I tried. >> >> You can also discuss with Martin the systemtap script to plot disk accesses >> during the startup. It is very handy for analyzing the code layout issues I send you as an attachment linker script that preserves .text.unlikely, .text.exit, .text.startup and .text.hot sections. I tries to modify memory access flags for .text.unlikely section to read-only, but I am not so familiar with linker scripting. Maybe, you can define with MEMORY command the read-only memory area for .text.unlikely. I am using for my graphing disabled read-ahead function in the Linux kernel with systemtap script and the results are presented with matplotlib library. If you are more interested in this tool-chain, write me and I can send you these scripts. I was also using valgrind to trace functions, so I will try to get locations of called functions to identify which functions are called in corresponding sections. Martin > Ok. I am using linux perf to collect this info (fed through some > scripts that munge and plot the data). > >> >> It may be interesting to get similar script taking traces from valgrind >> and ploting the most frequent calls in the final layout ;) > > I think linux perf -g to get a callgraph should give similar data. > > Teresa > >> >> Honza > > 2013-08-05 Teresa Johnson <tejohnson@google.com> > Steven Bosscher <steven@gcc.gnu.org> > > * cfgrtl.c (fixup_new_cold_bb): New routine. > (commit_edge_insertions): Invoke fixup_partitions. > (find_partition_fixes): New routine. > (fixup_partitions): Ditto. > (verify_hot_cold_block_grouping): Update comments. > (rtl_verify_edges): Invoke find_partition_fixes. > (rtl_verify_bb_pointers): Update comments. > (rtl_verify_bb_layout): Ditto. > * basic-block.h (fixup_partitions): Declare. > * cfgcleanup.c (try_optimize_cfg): Invoke fixup_partitions. > * bb-reorder.c (sanitize_hot_paths): New function. > (find_rarely_executed_basic_blocks_and_crossing_edges): Invoke > sanitize_hot_paths. > > Index: cfgrtl.c > =================================================================== > --- cfgrtl.c (revision 201461) > +++ cfgrtl.c (working copy) > @@ -1341,6 +1341,43 @@ fixup_partition_crossing (edge e) > } > } > > +/* Called when block BB has been reassigned to the cold partition, > + because it is now dominated by another cold block, > + to ensure that the region crossing attributes are updated. */ > + > +static void > +fixup_new_cold_bb (basic_block bb) > +{ > + edge e; > + edge_iterator ei; > + > + /* This is called when a hot bb is found to now be dominated > + by a cold bb and therefore needs to become cold. Therefore, > + its preds will no longer be region crossing. Any non-dominating > + preds that were previously hot would also have become cold > + in the caller for the same region. Any preds that were previously > + region-crossing will be adjusted in fixup_partition_crossing. */ > + FOR_EACH_EDGE (e, ei, bb->preds) > + { > + fixup_partition_crossing (e); > + } > + > + /* Possibly need to make bb's successor edges region crossing, > + or remove stale region crossing. */ > + FOR_EACH_EDGE (e, ei, bb->succs) > + { > + /* We can't have fall-through edges across partition boundaries. > + Note that force_nonfallthru will do any necessary partition > + boundary fixup by calling fixup_partition_crossing itself. */ > + if ((e->flags & EDGE_FALLTHRU) > + && BB_PARTITION (bb) != BB_PARTITION (e->dest) > + && e->dest != EXIT_BLOCK_PTR) > + force_nonfallthru (e); > + else > + fixup_partition_crossing (e); > + } > +} > + > /* Attempt to change code to redirect edge E to TARGET. Don't do that on > expense of adding new instructions or reordering basic blocks. > > @@ -1979,6 +2016,14 @@ commit_edge_insertions (void) > { > basic_block bb; > > + /* Optimization passes that invoke this routine can cause hot blocks > + previously reached by both hot and cold blocks to become dominated only > + by cold blocks. This will cause the verification below to fail, > + and lead to now cold code in the hot section. In some cases this > + may only be visible after newly unreachable blocks are deleted, > + which will be done by fixup_partitions. */ > + fixup_partitions (); > + > #ifdef ENABLE_CHECKING > verify_flow_info (); > #endif > @@ -2173,6 +2218,101 @@ get_last_bb_insn (basic_block bb) > return end; > } > > +/* Sanity check partition hotness to ensure that basic blocks in > + the cold partition don't dominate basic blocks in the hot partition. > + If FLAG_ONLY is true, report violations as errors. Otherwise > + re-mark the dominated blocks as cold, since this is run after > + cfg optimizations that may make hot blocks previously reached > + by both hot and cold blocks now only reachable along cold paths. */ > + > +static vec<basic_block> > +find_partition_fixes (bool flag_only) > +{ > + basic_block bb; > + vec<basic_block> bbs_in_cold_partition = vNULL; > + vec<basic_block> bbs_to_fix = vNULL; > + > + /* Callers check this. */ > + gcc_checking_assert (crtl->has_bb_partition); > + > + FOR_EACH_BB (bb) > + if ((BB_PARTITION (bb) == BB_COLD_PARTITION)) > + bbs_in_cold_partition.safe_push (bb); > + > + if (bbs_in_cold_partition.is_empty ()) > + return vNULL; > + > + bool dom_calculated_here = !dom_info_available_p (CDI_DOMINATORS); > + > + if (dom_calculated_here) > + calculate_dominance_info (CDI_DOMINATORS); > + > + while (! bbs_in_cold_partition.is_empty ()) > + { > + bb = bbs_in_cold_partition.pop (); > + /* Any blocks dominated by a block in the cold section > + must also be cold. */ > + basic_block son; > + for (son = first_dom_son (CDI_DOMINATORS, bb); > + son; > + son = next_dom_son (CDI_DOMINATORS, son)) > + { > + /* If son is not yet cold, then mark it cold here and > + enqueue it for further processing. */ > + if ((BB_PARTITION (son) != BB_COLD_PARTITION)) > + { > + if (flag_only) > + error ("non-cold basic block %d dominated " > + "by a block in the cold partition (%d)", > son->index, bb->index); > + else > + BB_SET_PARTITION (son, BB_COLD_PARTITION); > + bbs_to_fix.safe_push (son); > + bbs_in_cold_partition.safe_push (son); > + } > + } > + } > + > + if (dom_calculated_here) > + free_dominance_info (CDI_DOMINATORS); > + > + return bbs_to_fix; > +} > + > +/* Perform cleanup on the hot/cold bb partitioning after optimization > + passes that modify the cfg. */ > + > +void > +fixup_partitions (void) > +{ > + basic_block bb; > + > + if (!crtl->has_bb_partition) > + return; > + > + /* Delete any blocks that became unreachable and weren't > + already cleaned up, for example during edge forwarding > + and convert_jumps_to_returns. This will expose more > + opportunities for fixing the partition boundaries here. > + Also, the calculation of the dominance graph during verification > + will assert if there are unreachable nodes. */ > + delete_unreachable_blocks (); > + > + /* If there are partitions, do a sanity check on them: A basic block in > + a cold partition cannot dominate a basic block in a hot partition. > + Fixup any that now violate this requirement, as a result of edge > + forwarding and unreachable block deletion. */ > + vec<basic_block> bbs_to_fix = find_partition_fixes (false); > + > + /* Do the partition fixup after all necessary blocks have been converted to > + cold, so that we only update the region crossings the minimum number of > + places, which can require forcing edges to be non fallthru. */ > + while (! bbs_to_fix.is_empty ()) > + { > + bb = bbs_to_fix.pop (); > + fixup_new_cold_bb (bb); > + } > +} > + > /* Verify, in the basic block chain, that there is at most one switch > between hot/cold partitions. This condition will not be true until > after reorder_basic_blocks is called. */ > @@ -2219,7 +2359,8 @@ verify_hot_cold_block_grouping (void) > /* Perform several checks on the edges out of each block, such as > the consistency of the branch probabilities, the correctness > of hot/cold partition crossing edges, and the number of expected > - successor edges. */ > + successor edges. Also verify that the dominance relationship > + between hot/cold blocks is sane. */ > > static int > rtl_verify_edges (void) > @@ -2382,6 +2523,14 @@ rtl_verify_edges (void) > } > } > > + /* If there are partitions, do a sanity check on them: A basic block in > + a cold partition cannot dominate a basic block in a hot partition. */ > + if (crtl->has_bb_partition && !err) > + { > + vec<basic_block> bbs_to_fix = find_partition_fixes (true); > + err = !bbs_to_fix.is_empty (); > + } > + > /* Clean up. */ > return err; > } > @@ -2515,7 +2664,7 @@ rtl_verify_bb_pointers (void) > and NOTE_INSN_BASIC_BLOCK > - verify that no fall_thru edge crosses hot/cold partition boundaries > - verify that there are no pending RTL branch predictions > - - verify that there is a single hot/cold partition boundary after bbro > + - verify that hot blocks are not dominated by cold blocks > > In future it can be extended check a lot of other stuff as well > (reachability of basic blocks, life information, etc. etc.). */ > @@ -2761,7 +2910,8 @@ rtl_verify_bb_layout (void) > - check that all insns are in the basic blocks > (except the switch handling code, barriers and notes) > - check that all returns are followed by barriers > - - check that all fallthru edge points to the adjacent blocks. */ > + - check that all fallthru edge points to the adjacent blocks > + - verify that there is a single hot/cold partition boundary after bbro */ > > static int > rtl_verify_flow_info (void) > Index: basic-block.h > =================================================================== > --- basic-block.h (revision 201461) > +++ basic-block.h (working copy) > @@ -797,6 +797,7 @@ extern bool contains_no_active_insn_p (const_basic > extern bool forwarder_block_p (const_basic_block); > extern bool can_fallthru (basic_block, basic_block); > extern void emit_barrier_after_bb (basic_block bb); > +extern void fixup_partitions (void); > > /* In cfgbuild.c. */ > extern void find_many_sub_basic_blocks (sbitmap); > Index: cfgcleanup.c > =================================================================== > --- cfgcleanup.c (revision 201461) > +++ cfgcleanup.c (working copy) > @@ -2807,10 +2807,21 @@ try_optimize_cfg (int mode) > df_analyze (); > } > > + if (changed) > + { > + /* Edge forwarding in particular can cause hot blocks previously > + reached by both hot and cold blocks to become dominated only > + by cold blocks. This will cause the verification > below to fail, > + and lead to now cold code in the hot section. This is not easy > + to detect and fix during edge forwarding, and in some cases > + is only visible after newly unreachable blocks are deleted, > + which will be done in fixup_partitions. */ > + fixup_partitions (); > + > #ifdef ENABLE_CHECKING > - if (changed) > - verify_flow_info (); > + verify_flow_info (); > #endif > + } > > changed_overall |= changed; > first_pass = false; > Index: bb-reorder.c > =================================================================== > --- bb-reorder.c (revision 201461) > +++ bb-reorder.c (working copy) > @@ -1444,27 +1444,134 @@ fix_up_crossing_landing_pad (eh_landing_pad old_lp > ei_next (&ei); > } > > + > +/* Ensure that all hot bbs are included in a hot path through the > + procedure. This is done by calling this function twice, once > + with WALK_UP true (to look for paths from the entry to hot bbs) and > + once with WALK_UP false (to look for paths from hot bbs to the exit). > + Returns the updated value of COLD_BB_COUNT and adds newly-hot bbs > + to BBS_IN_HOT_PARTITION. */ > + > +static unsigned int > +sanitize_hot_paths (bool walk_up, unsigned int cold_bb_count, > + vec<basic_block> *bbs_in_hot_partition) > +{ > + /* Callers check this. */ > + gcc_checking_assert (cold_bb_count); > + > + /* Keep examining hot bbs while we still have some left to check > + and there are remaining cold bbs. */ > + vec<basic_block> hot_bbs_to_check = bbs_in_hot_partition->copy (); > + while (! hot_bbs_to_check.is_empty () > + && cold_bb_count) > + { > + basic_block bb = hot_bbs_to_check.pop (); > + vec<edge, va_gc> *edges = walk_up ? bb->preds : bb->succs; > + edge e; > + edge_iterator ei; > + int highest_probability = 0; > + bool found = false; > + > + /* Walk the preds/succs and check if there is at least one already > + marked hot. Keep track of the most frequent pred/succ so that we > + can mark it hot if we don't find one. */ > + FOR_EACH_EDGE (e, ei, edges) > + { > + basic_block reach_bb = walk_up ? e->src : e->dest; > + > + if (e->flags & EDGE_DFS_BACK) > + continue; > + > + if (BB_PARTITION (reach_bb) != BB_COLD_PARTITION) > + { > + found = true; > + break; > + } > + if (e->probability > highest_probability) > + highest_probability = e->probability; > + } > + > + /* If bb is reached by (or reaches, in the case of !WALK_UP) another hot > + block (or unpartitioned, e.g. the entry block) then it is ok. If not, > + then the most frequent pred (or succ) needs to be adjusted. In the > + case where multiple preds/succs have the same probability (e.g. a > + 50-50 branch), then both will be adjusted. */ > + if (found) > + continue; > + > + FOR_EACH_EDGE (e, ei, edges) > + { > + if (e->flags & EDGE_DFS_BACK) > + continue; > + if (e->probability < highest_probability) > + continue; > + > + basic_block reach_bb = walk_up ? e->src : e->dest; > + > + /* We have a hot bb with an immediate dominator that is cold. > + The dominator needs to be re-marked hot. */ > + BB_SET_PARTITION (reach_bb, BB_HOT_PARTITION); > + cold_bb_count--; > + > + /* Now we need to examine newly-hot reach_bb to see if it is also > + dominated by a cold bb. */ > + bbs_in_hot_partition->safe_push (reach_bb); > + hot_bbs_to_check.safe_push (reach_bb); > + } > + } > + > + return cold_bb_count; > +} > + > + > /* Find the basic blocks that are rarely executed and need to be moved to > a separate section of the .o file (to cut down on paging and improve > cache locality). Return a vector of all edges that cross. */ > > -static vec<edge> > +static vec<edge> > find_rarely_executed_basic_blocks_and_crossing_edges (void) > { > vec<edge> crossing_edges = vNULL; > basic_block bb; > edge e; > edge_iterator ei; > + unsigned int cold_bb_count = 0; > + vec<basic_block> bbs_in_hot_partition = vNULL; > > /* Mark which partition (hot/cold) each basic block belongs in. */ > FOR_EACH_BB (bb) > { > if (probably_never_executed_bb_p (cfun, bb)) > - BB_SET_PARTITION (bb, BB_COLD_PARTITION); > + { > + BB_SET_PARTITION (bb, BB_COLD_PARTITION); > + cold_bb_count++; > + } > else > - BB_SET_PARTITION (bb, BB_HOT_PARTITION); > + { > + BB_SET_PARTITION (bb, BB_HOT_PARTITION); > + bbs_in_hot_partition.safe_push (bb); > + } > } > > + /* Ensure that hot bbs are included along a hot path from the entry to exit. > + Several different possibilities may include cold bbs along all paths > + to/from a hot bb. One is that there are edge weight insanities > + due to optimization phases that do not properly update basic block profile > + counts. The second is that the entry of the function may not be > hot, because > + it is entered fewer times than the number of profile training > runs, but there > + is a loop inside the function that causes blocks within the function to be > + above the threshold for hotness. This is fixed by walking up from hot bbs > + to the entry block, and then down from hot bbs to the exit, performing > + partitioning fixups as necessary. */ > + if (cold_bb_count) > + { > + mark_dfs_back_edges (); > + cold_bb_count = sanitize_hot_paths (true, cold_bb_count, > + &bbs_in_hot_partition); > + if (cold_bb_count) > + sanitize_hot_paths (false, cold_bb_count, &bbs_in_hot_partition); > + } > + > /* The format of .gcc_except_table does not allow landing pads to > be in a different partition as the throw. Fix this by either > moving or duplicating the landing pads. */ > > > -- > Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 [-- Attachment #2: ld.script --] [-- Type: application/octet-stream, Size: 8644 bytes --] /* Script for -z combreloc: combine and sort reloc sections */ OUTPUT_FORMAT("elf64-x86-64", "elf64-x86-64", "elf64-x86-64") OUTPUT_ARCH(i386:x86-64) ENTRY(_start) SEARCH_DIR("/home/marxin/binutils-bin/x86_64-unknown-linux-gnu/lib64"); SEARCH_DIR("/home/marxin/binutils-bin/lib64"); SEARCH_DIR("/usr/local/lib64"); SEARCH_DIR("/lib64"); SEARCH_DIR("/usr/lib64"); SEARCH_DIR("/home/marxin/binutils-bin/x86_64-unknown-linux-gnu/lib"); SEARCH_DIR("/home/marxin/binutils-bin/lib"); SEARCH_DIR("/usr/local/lib"); SEARCH_DIR("/lib"); SEARCH_DIR("/usr/lib"); SECTIONS { /* Read-only sections, merged into text segment: */ PROVIDE (__executable_start = SEGMENT_START("text-segment", 0x400000)); . = SEGMENT_START("text-segment", 0x400000) + SIZEOF_HEADERS; .interp : { *(.interp) } .note.gnu.build-id : { *(.note.gnu.build-id) } .hash : { *(.hash) } .gnu.hash : { *(.gnu.hash) } .dynsym : { *(.dynsym) } .dynstr : { *(.dynstr) } .gnu.version : { *(.gnu.version) } .gnu.version_d : { *(.gnu.version_d) } .gnu.version_r : { *(.gnu.version_r) } .rela.dyn : { *(.rela.init) *(.rela.text .rela.text.* .rela.gnu.linkonce.t.*) *(.rela.fini) *(.rela.rodata .rela.rodata.* .rela.gnu.linkonce.r.*) *(.rela.data .rela.data.* .rela.gnu.linkonce.d.*) *(.rela.tdata .rela.tdata.* .rela.gnu.linkonce.td.*) *(.rela.tbss .rela.tbss.* .rela.gnu.linkonce.tb.*) *(.rela.ctors) *(.rela.dtors) *(.rela.got) *(.rela.bss .rela.bss.* .rela.gnu.linkonce.b.*) *(.rela.ldata .rela.ldata.* .rela.gnu.linkonce.l.*) *(.rela.lbss .rela.lbss.* .rela.gnu.linkonce.lb.*) *(.rela.lrodata .rela.lrodata.* .rela.gnu.linkonce.lr.*) *(.rela.ifunc) } .rela.plt : { *(.rela.plt) PROVIDE_HIDDEN (__rela_iplt_start = .); *(.rela.iplt) PROVIDE_HIDDEN (__rela_iplt_end = .); } .init : { KEEP (*(SORT_NONE(.init))) } .plt : { *(.plt) *(.iplt) } .text.unlikely : { *(.text.unlikely .text.*_unlikely .text.unlikely.*) } .text.exit : { *(.text.exit .text.exit.*) } .text.startup : { *(.text.startup .text.startup.*) } .text.hot : { *(.text.hot .text.hot.*) } .text : { *(.text .stub .text.* .gnu.linkonce.t.*) /* .gnu.warning sections are handled specially by elf32.em. */ *(.gnu.warning) } .fini : { KEEP (*(SORT_NONE(.fini))) } PROVIDE (__etext = .); PROVIDE (_etext = .); PROVIDE (etext = .); .rodata : { *(.rodata .rodata.* .gnu.linkonce.r.*) } .rodata1 : { *(.rodata1) } .eh_frame_hdr : { *(.eh_frame_hdr) } .eh_frame : ONLY_IF_RO { KEEP (*(.eh_frame)) } .gcc_except_table : ONLY_IF_RO { *(.gcc_except_table .gcc_except_table.*) } /* These sections are generated by the Sun/Oracle C++ compiler. */ .exception_ranges : ONLY_IF_RO { *(.exception_ranges .exception_ranges*) } /* Adjust the address for the data segment. We want to adjust up to the same address within the page on the next page up. */ . = ALIGN (CONSTANT (MAXPAGESIZE)) - ((CONSTANT (MAXPAGESIZE) - .) & (CONSTANT (MAXPAGESIZE) - 1)); . = DATA_SEGMENT_ALIGN (CONSTANT (MAXPAGESIZE), CONSTANT (COMMONPAGESIZE)); /* Exception handling */ .eh_frame : ONLY_IF_RW { KEEP (*(.eh_frame)) } .gcc_except_table : ONLY_IF_RW { *(.gcc_except_table .gcc_except_table.*) } .exception_ranges : ONLY_IF_RW { *(.exception_ranges .exception_ranges*) } /* Thread Local Storage sections */ .tdata : { *(.tdata .tdata.* .gnu.linkonce.td.*) } .tbss : { *(.tbss .tbss.* .gnu.linkonce.tb.*) *(.tcommon) } .preinit_array : { PROVIDE_HIDDEN (__preinit_array_start = .); KEEP (*(.preinit_array)) PROVIDE_HIDDEN (__preinit_array_end = .); } .init_array : { PROVIDE_HIDDEN (__init_array_start = .); KEEP (*(SORT_BY_INIT_PRIORITY(.init_array.*) SORT_BY_INIT_PRIORITY(.ctors.*))) KEEP (*(.init_array EXCLUDE_FILE (*crtbegin.o *crtbegin?.o *crtend.o *crtend?.o ) .ctors)) PROVIDE_HIDDEN (__init_array_end = .); } .fini_array : { PROVIDE_HIDDEN (__fini_array_start = .); KEEP (*(SORT_BY_INIT_PRIORITY(.fini_array.*) SORT_BY_INIT_PRIORITY(.dtors.*))) KEEP (*(.fini_array EXCLUDE_FILE (*crtbegin.o *crtbegin?.o *crtend.o *crtend?.o ) .dtors)) PROVIDE_HIDDEN (__fini_array_end = .); } .ctors : { /* gcc uses crtbegin.o to find the start of the constructors, so we make sure it is first. Because this is a wildcard, it doesn't matter if the user does not actually link against crtbegin.o; the linker won't look for a file to match a wildcard. The wildcard also means that it doesn't matter which directory crtbegin.o is in. */ KEEP (*crtbegin.o(.ctors)) KEEP (*crtbegin?.o(.ctors)) /* We don't want to include the .ctor section from the crtend.o file until after the sorted ctors. The .ctor section from the crtend file contains the end of ctors marker and it must be last */ KEEP (*(EXCLUDE_FILE (*crtend.o *crtend?.o ) .ctors)) KEEP (*(SORT(.ctors.*))) KEEP (*(.ctors)) } .dtors : { KEEP (*crtbegin.o(.dtors)) KEEP (*crtbegin?.o(.dtors)) KEEP (*(EXCLUDE_FILE (*crtend.o *crtend?.o ) .dtors)) KEEP (*(SORT(.dtors.*))) KEEP (*(.dtors)) } .jcr : { KEEP (*(.jcr)) } .data.rel.ro : { *(.data.rel.ro.local* .gnu.linkonce.d.rel.ro.local.*) *(.data.rel.ro .data.rel.ro.* .gnu.linkonce.d.rel.ro.*) } .dynamic : { *(.dynamic) } .got : { *(.got) *(.igot) } . = DATA_SEGMENT_RELRO_END (SIZEOF (.got.plt) >= 24 ? 24 : 0, .); .got.plt : { *(.got.plt) *(.igot.plt) } .data : { *(.data .data.* .gnu.linkonce.d.*) SORT(CONSTRUCTORS) } .data1 : { *(.data1) } _edata = .; PROVIDE (edata = .); . = .; __bss_start = .; .bss : { *(.dynbss) *(.bss .bss.* .gnu.linkonce.b.*) *(COMMON) /* Align here to ensure that the .bss section occupies space up to _end. Align after .bss to ensure correct alignment even if the .bss section disappears because there are no input sections. FIXME: Why do we need it? When there is no .bss section, we don't pad the .data section. */ . = ALIGN(. != 0 ? 64 / 8 : 1); } .lbss : { *(.dynlbss) *(.lbss .lbss.* .gnu.linkonce.lb.*) *(LARGE_COMMON) } . = ALIGN(64 / 8); . = SEGMENT_START("ldata-segment", .); .lrodata ALIGN(CONSTANT (MAXPAGESIZE)) + (. & (CONSTANT (MAXPAGESIZE) - 1)) : { *(.lrodata .lrodata.* .gnu.linkonce.lr.*) } .ldata ALIGN(CONSTANT (MAXPAGESIZE)) + (. & (CONSTANT (MAXPAGESIZE) - 1)) : { *(.ldata .ldata.* .gnu.linkonce.l.*) . = ALIGN(. != 0 ? 64 / 8 : 1); } . = ALIGN(64 / 8); _end = .; PROVIDE (end = .); . = DATA_SEGMENT_END (.); /* Stabs debugging sections. */ .stab 0 : { *(.stab) } .stabstr 0 : { *(.stabstr) } .stab.excl 0 : { *(.stab.excl) } .stab.exclstr 0 : { *(.stab.exclstr) } .stab.index 0 : { *(.stab.index) } .stab.indexstr 0 : { *(.stab.indexstr) } .comment 0 : { *(.comment) } /* DWARF debug sections. Symbols in the DWARF debugging sections are relative to the beginning of the section so we begin them at 0. */ /* DWARF 1 */ .debug 0 : { *(.debug) } .line 0 : { *(.line) } /* GNU DWARF 1 extensions */ .debug_srcinfo 0 : { *(.debug_srcinfo) } .debug_sfnames 0 : { *(.debug_sfnames) } /* DWARF 1.1 and DWARF 2 */ .debug_aranges 0 : { *(.debug_aranges) } .debug_pubnames 0 : { *(.debug_pubnames) } /* DWARF 2 */ .debug_info 0 : { *(.debug_info .gnu.linkonce.wi.*) } .debug_abbrev 0 : { *(.debug_abbrev) } .debug_line 0 : { *(.debug_line .debug_line.* .debug_line_end ) } .debug_frame 0 : { *(.debug_frame) } .debug_str 0 : { *(.debug_str) } .debug_loc 0 : { *(.debug_loc) } .debug_macinfo 0 : { *(.debug_macinfo) } /* SGI/MIPS DWARF 2 extensions */ .debug_weaknames 0 : { *(.debug_weaknames) } .debug_funcnames 0 : { *(.debug_funcnames) } .debug_typenames 0 : { *(.debug_typenames) } .debug_varnames 0 : { *(.debug_varnames) } /* DWARF 3 */ .debug_pubtypes 0 : { *(.debug_pubtypes) } .debug_ranges 0 : { *(.debug_ranges) } /* DWARF Extension. */ .debug_macro 0 : { *(.debug_macro) } .gnu.attributes 0 : { KEEP (*(.gnu.attributes)) } /DISCARD/ : { *(.note.GNU-stack) *(.gnu_debuglink) *(.gnu.lto_*) } } ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-19 17:48 ` Teresa Johnson 2013-08-19 19:56 ` Martin Liška @ 2013-08-27 18:12 ` Teresa Johnson 2013-08-28 16:59 ` Jan Hubicka 2013-08-31 16:20 ` Jan Hubicka 1 sibling, 2 replies; 62+ messages in thread From: Teresa Johnson @ 2013-08-27 18:12 UTC (permalink / raw) To: Jan Hubicka Cc: Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, marxin.liska, Sriraman Tallam On Mon, Aug 19, 2013 at 10:47 AM, Teresa Johnson <tejohnson@google.com> wrote: > On Mon, Aug 19, 2013 at 8:09 AM, Jan Hubicka <hubicka@ucw.cz> wrote: >>> Remember it isn't using dominance anymore. The latest patch was >>> instead ensuring the most frequent path between hot blocks and the >>> entry/exit are marked hot. That should be better than the dominance >>> approach used in the earlier version. >> >> Indeed, that looks more resonable approach. >> Can you point me to the last version of patch? Last one I remember still >> walked dominators... > > I've included the latest patch below. I still use dominators in the > post-cfg-optimization fixup (fixup_partitions), but not in the > partition sanitizing done during the partitioning itself > (sanitize_hot_paths). The former is looking for hot bbs newly > dominated by cold bbs after cfg transformations. > >>> >>> > We can commit it and work on better solution incrementally but it will >>> > probably mean replacing it later. If you think it makes things easier >>> > to work on it incrementally, I think the patch is OK. >>> >>> Yes, I think this is a big step forward from what is there now for >>> splitting, which does the splitting purely based on bb count in >>> isolation. I don't have a much better solution in mind yet. >>> Ping on this patch. Honza, did the latest version I sent last week look ok to you? I've included below a new version that adds the partitioning insanity warnings we discussed (emitted to the dump only since as I noted there are various optimization passes that provoke this due to not fixing up profile data). (Honza - I am also going to move the discussion we started in this thread on the COMDAT missing profile handling patch to a different thread. Should have an update on that shortly.) Latest patch below retested with bootstrap and regression testing on x86-64-unknown-linux-gnu, and with a profiledbootstrap and -freorder-blocks-and-partition forced on. Ok for trunk? Thanks, Teresa 2013-08-27 Teresa Johnson <tejohnson@google.com> Steven Bosscher <steven@gcc.gnu.org> * cfgrtl.c (fixup_new_cold_bb): New routine. (commit_edge_insertions): Invoke fixup_partitions. (find_partition_fixes): New routine. (fixup_partitions): Ditto. (verify_hot_cold_block_grouping): Update comments. (rtl_verify_edges): Invoke find_partition_fixes. (rtl_verify_bb_pointers): Update comments. (rtl_verify_bb_layout): Ditto. * basic-block.h (probably_never_executed_edge_p): Declare. (fixup_partitions): Ditto. * cfgcleanup.c (try_optimize_cfg): Invoke fixup_partitions. * bb-reorder.c (sanitize_hot_paths): New function. (find_rarely_executed_basic_blocks_and_crossing_edges): Invoke sanitize_hot_paths. * predict.c (probably_never_executed_edge_p): New routine. * cfg.c (check_bb_profile): Add partition insanity warnings. Index: cfgrtl.c =================================================================== --- cfgrtl.c (revision 202021) +++ cfgrtl.c (working copy) @@ -1358,6 +1358,43 @@ fixup_partition_crossing (edge e) } } +/* Called when block BB has been reassigned to the cold partition, + because it is now dominated by another cold block, + to ensure that the region crossing attributes are updated. */ + +static void +fixup_new_cold_bb (basic_block bb) +{ + edge e; + edge_iterator ei; + + /* This is called when a hot bb is found to now be dominated + by a cold bb and therefore needs to become cold. Therefore, + its preds will no longer be region crossing. Any non-dominating + preds that were previously hot would also have become cold + in the caller for the same region. Any preds that were previously + region-crossing will be adjusted in fixup_partition_crossing. */ + FOR_EACH_EDGE (e, ei, bb->preds) + { + fixup_partition_crossing (e); + } + + /* Possibly need to make bb's successor edges region crossing, + or remove stale region crossing. */ + FOR_EACH_EDGE (e, ei, bb->succs) + { + /* We can't have fall-through edges across partition boundaries. + Note that force_nonfallthru will do any necessary partition + boundary fixup by calling fixup_partition_crossing itself. */ + if ((e->flags & EDGE_FALLTHRU) + && BB_PARTITION (bb) != BB_PARTITION (e->dest) + && e->dest != EXIT_BLOCK_PTR) + force_nonfallthru (e); + else + fixup_partition_crossing (e); + } +} + /* Attempt to change code to redirect edge E to TARGET. Don't do that on expense of adding new instructions or reordering basic blocks. @@ -1996,6 +2033,14 @@ commit_edge_insertions (void) { basic_block bb; + /* Optimization passes that invoke this routine can cause hot blocks + previously reached by both hot and cold blocks to become dominated only + by cold blocks. This will cause the verification below to fail, + and lead to now cold code in the hot section. In some cases this + may only be visible after newly unreachable blocks are deleted, + which will be done by fixup_partitions. */ + fixup_partitions (); + #ifdef ENABLE_CHECKING verify_flow_info (); #endif @@ -2190,6 +2235,101 @@ get_last_bb_insn (basic_block bb) return end; } +/* Sanity check partition hotness to ensure that basic blocks in + the cold partition don't dominate basic blocks in the hot partition. + If FLAG_ONLY is true, report violations as errors. Otherwise + re-mark the dominated blocks as cold, since this is run after + cfg optimizations that may make hot blocks previously reached + by both hot and cold blocks now only reachable along cold paths. */ + +static vec<basic_block> +find_partition_fixes (bool flag_only) +{ + basic_block bb; + vec<basic_block> bbs_in_cold_partition = vNULL; + vec<basic_block> bbs_to_fix = vNULL; + + /* Callers check this. */ + gcc_checking_assert (crtl->has_bb_partition); + + FOR_EACH_BB (bb) + if ((BB_PARTITION (bb) == BB_COLD_PARTITION)) + bbs_in_cold_partition.safe_push (bb); + + if (bbs_in_cold_partition.is_empty ()) + return vNULL; + + bool dom_calculated_here = !dom_info_available_p (CDI_DOMINATORS); + + if (dom_calculated_here) + calculate_dominance_info (CDI_DOMINATORS); + + while (! bbs_in_cold_partition.is_empty ()) + { + bb = bbs_in_cold_partition.pop (); + /* Any blocks dominated by a block in the cold section + must also be cold. */ + basic_block son; + for (son = first_dom_son (CDI_DOMINATORS, bb); + son; + son = next_dom_son (CDI_DOMINATORS, son)) + { + /* If son is not yet cold, then mark it cold here and + enqueue it for further processing. */ + if ((BB_PARTITION (son) != BB_COLD_PARTITION)) + { + if (flag_only) + error ("non-cold basic block %d dominated " + "by a block in the cold partition (%d)", son->index, bb->index); + else + BB_SET_PARTITION (son, BB_COLD_PARTITION); + bbs_to_fix.safe_push (son); + bbs_in_cold_partition.safe_push (son); + } + } + } + + if (dom_calculated_here) + free_dominance_info (CDI_DOMINATORS); + + return bbs_to_fix; +} + +/* Perform cleanup on the hot/cold bb partitioning after optimization + passes that modify the cfg. */ + +void +fixup_partitions (void) +{ + basic_block bb; + + if (!crtl->has_bb_partition) + return; + + /* Delete any blocks that became unreachable and weren't + already cleaned up, for example during edge forwarding + and convert_jumps_to_returns. This will expose more + opportunities for fixing the partition boundaries here. + Also, the calculation of the dominance graph during verification + will assert if there are unreachable nodes. */ + delete_unreachable_blocks (); + + /* If there are partitions, do a sanity check on them: A basic block in + a cold partition cannot dominate a basic block in a hot partition. + Fixup any that now violate this requirement, as a result of edge + forwarding and unreachable block deletion. */ + vec<basic_block> bbs_to_fix = find_partition_fixes (false); + + /* Do the partition fixup after all necessary blocks have been converted to + cold, so that we only update the region crossings the minimum number of + places, which can require forcing edges to be non fallthru. */ + while (! bbs_to_fix.is_empty ()) + { + bb = bbs_to_fix.pop (); + fixup_new_cold_bb (bb); + } +} + /* Verify, in the basic block chain, that there is at most one switch between hot/cold partitions. This condition will not be true until after reorder_basic_blocks is called. */ @@ -2236,7 +2376,8 @@ verify_hot_cold_block_grouping (void) /* Perform several checks on the edges out of each block, such as the consistency of the branch probabilities, the correctness of hot/cold partition crossing edges, and the number of expected - successor edges. */ + successor edges. Also verify that the dominance relationship + between hot/cold blocks is sane. */ static int rtl_verify_edges (void) @@ -2399,6 +2540,14 @@ rtl_verify_edges (void) } } + /* If there are partitions, do a sanity check on them: A basic block in + a cold partition cannot dominate a basic block in a hot partition. */ + if (crtl->has_bb_partition && !err) + { + vec<basic_block> bbs_to_fix = find_partition_fixes (true); + err = !bbs_to_fix.is_empty (); + } + /* Clean up. */ return err; } @@ -2532,7 +2681,7 @@ rtl_verify_bb_pointers (void) and NOTE_INSN_BASIC_BLOCK - verify that no fall_thru edge crosses hot/cold partition boundaries - verify that there are no pending RTL branch predictions - - verify that there is a single hot/cold partition boundary after bbro + - verify that hot blocks are not dominated by cold blocks In future it can be extended check a lot of other stuff as well (reachability of basic blocks, life information, etc. etc.). */ @@ -2778,7 +2927,8 @@ rtl_verify_bb_layout (void) - check that all insns are in the basic blocks (except the switch handling code, barriers and notes) - check that all returns are followed by barriers - - check that all fallthru edge points to the adjacent blocks. */ + - check that all fallthru edge points to the adjacent blocks + - verify that there is a single hot/cold partition boundary after bbro */ static int rtl_verify_flow_info (void) Index: basic-block.h =================================================================== --- basic-block.h (revision 202021) +++ basic-block.h (working copy) @@ -726,6 +726,7 @@ extern void compute_available (sbitmap *, sbitmap extern bool maybe_hot_bb_p (struct function *, const_basic_block); extern bool maybe_hot_edge_p (edge); extern bool probably_never_executed_bb_p (struct function *, const_basic_block); +extern bool probably_never_executed_edge_p (struct function *, edge); extern bool optimize_bb_for_size_p (const_basic_block); extern bool optimize_bb_for_speed_p (const_basic_block); extern bool optimize_edge_for_size_p (edge); @@ -797,6 +798,7 @@ extern bool contains_no_active_insn_p (const_basic extern bool forwarder_block_p (const_basic_block); extern bool can_fallthru (basic_block, basic_block); extern void emit_barrier_after_bb (basic_block bb); +extern void fixup_partitions (void); /* In cfgbuild.c. */ extern void find_many_sub_basic_blocks (sbitmap); Index: cfgcleanup.c =================================================================== --- cfgcleanup.c (revision 202021) +++ cfgcleanup.c (working copy) @@ -2807,10 +2807,21 @@ try_optimize_cfg (int mode) df_analyze (); } + if (changed) + { + /* Edge forwarding in particular can cause hot blocks previously + reached by both hot and cold blocks to become dominated only + by cold blocks. This will cause the verification below to fail, + and lead to now cold code in the hot section. This is not easy + to detect and fix during edge forwarding, and in some cases + is only visible after newly unreachable blocks are deleted, + which will be done in fixup_partitions. */ + fixup_partitions (); + #ifdef ENABLE_CHECKING - if (changed) - verify_flow_info (); + verify_flow_info (); #endif + } changed_overall |= changed; first_pass = false; Index: bb-reorder.c =================================================================== --- bb-reorder.c (revision 202021) +++ bb-reorder.c (working copy) @@ -1444,27 +1444,134 @@ fix_up_crossing_landing_pad (eh_landing_pad old_lp ei_next (&ei); } + +/* Ensure that all hot bbs are included in a hot path through the + procedure. This is done by calling this function twice, once + with WALK_UP true (to look for paths from the entry to hot bbs) and + once with WALK_UP false (to look for paths from hot bbs to the exit). + Returns the updated value of COLD_BB_COUNT and adds newly-hot bbs + to BBS_IN_HOT_PARTITION. */ + +static unsigned int +sanitize_hot_paths (bool walk_up, unsigned int cold_bb_count, + vec<basic_block> *bbs_in_hot_partition) +{ + /* Callers check this. */ + gcc_checking_assert (cold_bb_count); + + /* Keep examining hot bbs while we still have some left to check + and there are remaining cold bbs. */ + vec<basic_block> hot_bbs_to_check = bbs_in_hot_partition->copy (); + while (! hot_bbs_to_check.is_empty () + && cold_bb_count) + { + basic_block bb = hot_bbs_to_check.pop (); + vec<edge, va_gc> *edges = walk_up ? bb->preds : bb->succs; + edge e; + edge_iterator ei; + int highest_probability = 0; + bool found = false; + + /* Walk the preds/succs and check if there is at least one already + marked hot. Keep track of the most frequent pred/succ so that we + can mark it hot if we don't find one. */ + FOR_EACH_EDGE (e, ei, edges) + { + basic_block reach_bb = walk_up ? e->src : e->dest; + + if (e->flags & EDGE_DFS_BACK) + continue; + + if (BB_PARTITION (reach_bb) != BB_COLD_PARTITION) + { + found = true; + break; + } + if (e->probability > highest_probability) + highest_probability = e->probability; + } + + /* If bb is reached by (or reaches, in the case of !WALK_UP) another hot + block (or unpartitioned, e.g. the entry block) then it is ok. If not, + then the most frequent pred (or succ) needs to be adjusted. In the + case where multiple preds/succs have the same probability (e.g. a + 50-50 branch), then both will be adjusted. */ + if (found) + continue; + + FOR_EACH_EDGE (e, ei, edges) + { + if (e->flags & EDGE_DFS_BACK) + continue; + if (e->probability < highest_probability) + continue; + + basic_block reach_bb = walk_up ? e->src : e->dest; + + /* We have a hot bb with an immediate dominator that is cold. + The dominator needs to be re-marked hot. */ + BB_SET_PARTITION (reach_bb, BB_HOT_PARTITION); + cold_bb_count--; + + /* Now we need to examine newly-hot reach_bb to see if it is also + dominated by a cold bb. */ + bbs_in_hot_partition->safe_push (reach_bb); + hot_bbs_to_check.safe_push (reach_bb); + } + } + + return cold_bb_count; +} + + /* Find the basic blocks that are rarely executed and need to be moved to a separate section of the .o file (to cut down on paging and improve cache locality). Return a vector of all edges that cross. */ -static vec<edge> +static vec<edge> find_rarely_executed_basic_blocks_and_crossing_edges (void) { vec<edge> crossing_edges = vNULL; basic_block bb; edge e; edge_iterator ei; + unsigned int cold_bb_count = 0; + vec<basic_block> bbs_in_hot_partition = vNULL; /* Mark which partition (hot/cold) each basic block belongs in. */ FOR_EACH_BB (bb) { if (probably_never_executed_bb_p (cfun, bb)) - BB_SET_PARTITION (bb, BB_COLD_PARTITION); + { + BB_SET_PARTITION (bb, BB_COLD_PARTITION); + cold_bb_count++; + } else - BB_SET_PARTITION (bb, BB_HOT_PARTITION); + { + BB_SET_PARTITION (bb, BB_HOT_PARTITION); + bbs_in_hot_partition.safe_push (bb); + } } + /* Ensure that hot bbs are included along a hot path from the entry to exit. + Several different possibilities may include cold bbs along all paths + to/from a hot bb. One is that there are edge weight insanities + due to optimization phases that do not properly update basic block profile + counts. The second is that the entry of the function may not be hot, because + it is entered fewer times than the number of profile training runs, but there + is a loop inside the function that causes blocks within the function to be + above the threshold for hotness. This is fixed by walking up from hot bbs + to the entry block, and then down from hot bbs to the exit, performing + partitioning fixups as necessary. */ + if (cold_bb_count) + { + mark_dfs_back_edges (); + cold_bb_count = sanitize_hot_paths (true, cold_bb_count, + &bbs_in_hot_partition); + if (cold_bb_count) + sanitize_hot_paths (false, cold_bb_count, &bbs_in_hot_partition); + } + /* The format of .gcc_except_table does not allow landing pads to be in a different partition as the throw. Fix this by either moving or duplicating the landing pads. */ Index: predict.c =================================================================== --- predict.c (revision 202021) +++ predict.c (working copy) @@ -241,6 +241,22 @@ probably_never_executed_bb_p (struct function *fun return false; } + +/* Return true in case edge E is probably never executed. */ + +bool +probably_never_executed_edge_p (struct function *fun, edge e) +{ + gcc_checking_assert (fun); + if (profile_info && flag_branch_probabilities) + return ((e->count + profile_info->runs / 2) / profile_info->runs) == 0; + if ((!profile_info || !flag_branch_probabilities) + && (cgraph_get_node (fun->decl)->frequency + == NODE_FREQUENCY_UNLIKELY_EXECUTED)) + return true; + return false; +} + /* Return true if NODE should be optimized for size. */ bool Index: cfg.c =================================================================== --- cfg.c (revision 202021) +++ cfg.c (working copy) @@ -446,6 +446,21 @@ check_bb_profile (basic_block bb, FILE * file, int (flags & TDF_COMMENT) ? ";; " : "", s_indent, (int) lsum, (int) bb->count); } + if (BB_PARTITION (bb) == BB_COLD_PARTITION) + { + /* Warn about inconsistencies in the partitioning that are + currently caused by profile insanities created via optimization. */ + if (!probably_never_executed_bb_p (fun, bb)) + fprintf (file, "%s%sBlock in cold partition with hot count\n", + (flags & TDF_COMMENT) ? ";; " : "", s_indent); + FOR_EACH_EDGE (e, ei, bb->preds) + { + if (!probably_never_executed_edge_p (fun, e)) + fprintf (file, + "%s%sBlock in cold partition with incoming hot edge\n", + (flags & TDF_COMMENT) ? ";; " : "", s_indent); + } + } } ^L void -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-27 18:12 ` Teresa Johnson @ 2013-08-28 16:59 ` Jan Hubicka 2013-08-28 18:35 ` Teresa Johnson 2013-08-31 16:20 ` Jan Hubicka 1 sibling, 1 reply; 62+ messages in thread From: Jan Hubicka @ 2013-08-28 16:59 UTC (permalink / raw) To: Teresa Johnson Cc: Jan Hubicka, Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, marxin.liska, Sriraman Tallam Hi, with Martin we did bit of progress on analyzing the problems. We now have COMDAT profile merging for FDO and we also noticed that forks can make your basic blocks appear never executed even though they are executed every run: the fork is accounted as 3 independnet runs of the program. First run is until fork, the second 2 runs are both variant. I have patch to track this. Moreover vforks seems to produce repeated merging of results. These two factors seems to help to riddle enought the firefox profiles so it took us weeks to understand what happens. > + if (changed) > + { > + /* Edge forwarding in particular can cause hot blocks previously > + reached by both hot and cold blocks to become dominated only > + by cold blocks. This will cause the verification > below to fail, > + and lead to now cold code in the hot section. This is not easy > + to detect and fix during edge forwarding, and in some cases > + is only visible after newly unreachable blocks are deleted, > + which will be done in fixup_partitions. */ > + fixup_partitions (); Is it really necessary to run this from internal loop of the cfgcleanup? It seems you will play back and forth game where edge forwarding will remove your fallthru and you will be re-adding it? I would wait for cfgcleanup to finish its job (I don't really think it needs to be iterative) and then do the fixup possibly cleaning up when the blocks was repoisitoined (I suppose it is only case where the code above introduces new cfgcleanup oppurtunities). > + /* Walk the preds/succs and check if there is at least one already > + marked hot. Keep track of the most frequent pred/succ so that we > + can mark it hot if we don't find one. */ > + FOR_EACH_EDGE (e, ei, edges) > + { > + basic_block reach_bb = walk_up ? e->src : e->dest; > + > + if (e->flags & EDGE_DFS_BACK) > + continue; > + > + if (BB_PARTITION (reach_bb) != BB_COLD_PARTITION) > + { > + found = true; > + break; > + } > + if (e->probability > highest_probability) > + highest_probability = e->probability; When doing predecestor walk if you have two predecestors, one executing 100000 times and getting to the block with probability 1%, you want to chose it over block executing once and getting to you with probability 100%. You probably want to look for most likely predecestor here. You need to look for highest e->count and if they are all 0 for highest EDGE_FREQUENCY and for maximal probability only if all fails? > + } > + > + /* If bb is reached by (or reaches, in the case of !WALK_UP) another hot > + block (or unpartitioned, e.g. the entry block) then it is ok. If not, > + then the most frequent pred (or succ) needs to be adjusted. In the > + case where multiple preds/succs have the same probability (e.g. a > + 50-50 branch), then both will be adjusted. */ > + if (found) > + continue; > + > + FOR_EACH_EDGE (e, ei, edges) > + { > + if (e->flags & EDGE_DFS_BACK) > + continue; > + if (e->probability < highest_probability) > + continue; Again for predecestor walk you need to wind down the bit crazy logic described above. > Index: predict.c > =================================================================== > --- predict.c (revision 202021) > +++ predict.c (working copy) > @@ -241,6 +241,22 @@ probably_never_executed_bb_p (struct function *fun > return false; > } > > + > +/* Return true in case edge E is probably never executed. */ > + > +bool > +probably_never_executed_edge_p (struct function *fun, edge e) > +{ > + gcc_checking_assert (fun); > + if (profile_info && flag_branch_probabilities) > + return ((e->count + profile_info->runs / 2) / profile_info->runs) == 0; > + if ((!profile_info || !flag_branch_probabilities) > + && (cgraph_get_node (fun->decl)->frequency > + == NODE_FREQUENCY_UNLIKELY_EXECUTED)) > + return true; > + return false; Instead of duplicating the conditional, break out the tests into probably_never_executed_count_p, like we have for maybe_hot_count_p. It would be nice to extend this to work w/o profile: probably_never_executed_edge_p can return true for EH edges, setjmp edges and we can then walk bodies ignoring EH/setjmp flagging blocks that are probably never executed enabling the partitioning to do a job w/o profile. Otherwise the patch look OK to me. Thanks for working on this. Do we have agreement on C++ way of mixing declarations and code? Honza ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-28 16:59 ` Jan Hubicka @ 2013-08-28 18:35 ` Teresa Johnson 2013-08-30 7:17 ` Teresa Johnson 0 siblings, 1 reply; 62+ messages in thread From: Teresa Johnson @ 2013-08-28 18:35 UTC (permalink / raw) To: Jan Hubicka Cc: Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, marxin.liska, Sriraman Tallam, Rong Xu On Wed, Aug 28, 2013 at 9:58 AM, Jan Hubicka <hubicka@ucw.cz> wrote: > Hi, > with Martin we did bit of progress on analyzing the problems. We now have > COMDAT profile merging for FDO Great! Is this the LTO merging you were talking about in an earlier message, or the gcov runtime fix (that would presumably not be lto-specific)? > and we also noticed that forks can make your > basic blocks appear never executed even though they are executed every run: > the fork is accounted as 3 independnet runs of the program. First run > is until fork, the second 2 runs are both variant. > > I have patch to track this. Moreover vforks seems to produce repeated > merging of results. Aha, does this explain the gimp issue as well? > > These two factors seems to help to riddle enought the firefox profiles > so it took us weeks to understand what happens. >> + if (changed) >> + { >> + /* Edge forwarding in particular can cause hot blocks previously >> + reached by both hot and cold blocks to become dominated only >> + by cold blocks. This will cause the verification >> below to fail, >> + and lead to now cold code in the hot section. This is not easy >> + to detect and fix during edge forwarding, and in some cases >> + is only visible after newly unreachable blocks are deleted, >> + which will be done in fixup_partitions. */ >> + fixup_partitions (); > > Is it really necessary to run this from internal loop of the cfgcleanup? The reason I added it here is that just below there is a call to verify_flow_info, and that will fail with the new verification. > It seems > you will play back and forth game where edge forwarding will remove your fallthru > and you will be re-adding it? fixup_partitions will not add new fall through edges. (It may invoke force_nonfallthru to do the opposite.) So there shouldn't be any ping-ponging effect. > > I would wait for cfgcleanup to finish its job (I don't really think it needs to be > iterative) and then do the fixup possibly cleaning up when the blocks was repoisitoined > (I suppose it is only case where the code above introduces new cfgcleanup oppurtunities). As noted above, I can't do this due to the call to verify_flow_info for each iteration. One option is to move both down outside the loop. > >> + /* Walk the preds/succs and check if there is at least one already >> + marked hot. Keep track of the most frequent pred/succ so that we >> + can mark it hot if we don't find one. */ >> + FOR_EACH_EDGE (e, ei, edges) >> + { >> + basic_block reach_bb = walk_up ? e->src : e->dest; >> + >> + if (e->flags & EDGE_DFS_BACK) >> + continue; >> + >> + if (BB_PARTITION (reach_bb) != BB_COLD_PARTITION) >> + { >> + found = true; >> + break; >> + } >> + if (e->probability > highest_probability) >> + highest_probability = e->probability; > > When doing predecestor walk if you have two predecestors, one executing 100000 > times and getting to the block with probability 1%, you want to chose it over > block executing once and getting to you with probability 100%. > > You probably want to look for most likely predecestor here. You need to look > for highest e->count and if they are all 0 for highest EDGE_FREQUENCY and for > maximal probability only if all fails? Yes, thanks, let me do that. > >> + } >> + >> + /* If bb is reached by (or reaches, in the case of !WALK_UP) another hot >> + block (or unpartitioned, e.g. the entry block) then it is ok. If not, >> + then the most frequent pred (or succ) needs to be adjusted. In the >> + case where multiple preds/succs have the same probability (e.g. a >> + 50-50 branch), then both will be adjusted. */ >> + if (found) >> + continue; >> + >> + FOR_EACH_EDGE (e, ei, edges) >> + { >> + if (e->flags & EDGE_DFS_BACK) >> + continue; >> + if (e->probability < highest_probability) >> + continue; > > Again for predecestor walk you need to wind down the bit crazy logic described above. >> Index: predict.c >> =================================================================== >> --- predict.c (revision 202021) >> +++ predict.c (working copy) >> @@ -241,6 +241,22 @@ probably_never_executed_bb_p (struct function *fun >> return false; >> } >> >> + >> +/* Return true in case edge E is probably never executed. */ >> + >> +bool >> +probably_never_executed_edge_p (struct function *fun, edge e) >> +{ >> + gcc_checking_assert (fun); >> + if (profile_info && flag_branch_probabilities) >> + return ((e->count + profile_info->runs / 2) / profile_info->runs) == 0; >> + if ((!profile_info || !flag_branch_probabilities) >> + && (cgraph_get_node (fun->decl)->frequency >> + == NODE_FREQUENCY_UNLIKELY_EXECUTED)) >> + return true; >> + return false; > Instead of duplicating the conditional, break out the tests into > probably_never_executed_count_p, like we have for maybe_hot_count_p. ok > > It would be nice to extend this to work w/o profile: probably_never_executed_edge_p > can return true for EH edges, setjmp edges and we can then walk bodies ignoring > EH/setjmp flagging blocks that are probably never executed enabling the partitioning > to do a job w/o profile. agreed. although I would prefer to leave that for a follow-on patch so it can be tuned a bit. > > Otherwise the patch look OK to me. Thanks for working on this. Do we have agreement on C++ way of mixing > declarations and code? According to http://gcc.gnu.org/wiki/CppConventions: "In new code variables which are used in a small scope should be defined at the point of first use, rather than at the top of the function. Variables which are used throughout the function may be defined at the top of the function, as in C." I think I am following that, but let me know if you see something that needs to be fixed. Thanks, Teresa > > Honza -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-28 18:35 ` Teresa Johnson @ 2013-08-30 7:17 ` Teresa Johnson 2013-08-30 9:16 ` Jan Hubicka 0 siblings, 1 reply; 62+ messages in thread From: Teresa Johnson @ 2013-08-30 7:17 UTC (permalink / raw) To: Jan Hubicka Cc: Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, marxin.liska, Sriraman Tallam, Rong Xu On Wed, Aug 28, 2013 at 11:20 AM, Teresa Johnson <tejohnson@google.com> wrote: > On Wed, Aug 28, 2013 at 9:58 AM, Jan Hubicka <hubicka@ucw.cz> wrote: >> Hi, >> with Martin we did bit of progress on analyzing the problems. We now have >> COMDAT profile merging for FDO > > Great! Is this the LTO merging you were talking about in an earlier > message, or the gcov runtime fix (that would presumably not be > lto-specific)? > >> and we also noticed that forks can make your >> basic blocks appear never executed even though they are executed every run: >> the fork is accounted as 3 independnet runs of the program. First run >> is until fork, the second 2 runs are both variant. >> >> I have patch to track this. Moreover vforks seems to produce repeated >> merging of results. > > Aha, does this explain the gimp issue as well? > >> >> These two factors seems to help to riddle enought the firefox profiles >> so it took us weeks to understand what happens. >>> + if (changed) >>> + { >>> + /* Edge forwarding in particular can cause hot blocks previously >>> + reached by both hot and cold blocks to become dominated only >>> + by cold blocks. This will cause the verification >>> below to fail, >>> + and lead to now cold code in the hot section. This is not easy >>> + to detect and fix during edge forwarding, and in some cases >>> + is only visible after newly unreachable blocks are deleted, >>> + which will be done in fixup_partitions. */ >>> + fixup_partitions (); >> >> Is it really necessary to run this from internal loop of the cfgcleanup? > > The reason I added it here is that just below there is a call to > verify_flow_info, and that will fail with the new verification. > >> It seems >> you will play back and forth game where edge forwarding will remove your fallthru >> and you will be re-adding it? > > fixup_partitions will not add new fall through edges. (It may invoke > force_nonfallthru to do the opposite.) So there shouldn't be any > ping-ponging effect. > >> >> I would wait for cfgcleanup to finish its job (I don't really think it needs to be >> iterative) and then do the fixup possibly cleaning up when the blocks was repoisitoined >> (I suppose it is only case where the code above introduces new cfgcleanup oppurtunities). > > As noted above, I can't do this due to the call to verify_flow_info > for each iteration. One option is to move both down outside the loop. > >> >>> + /* Walk the preds/succs and check if there is at least one already >>> + marked hot. Keep track of the most frequent pred/succ so that we >>> + can mark it hot if we don't find one. */ >>> + FOR_EACH_EDGE (e, ei, edges) >>> + { >>> + basic_block reach_bb = walk_up ? e->src : e->dest; >>> + >>> + if (e->flags & EDGE_DFS_BACK) >>> + continue; >>> + >>> + if (BB_PARTITION (reach_bb) != BB_COLD_PARTITION) >>> + { >>> + found = true; >>> + break; >>> + } >>> + if (e->probability > highest_probability) >>> + highest_probability = e->probability; >> >> When doing predecestor walk if you have two predecestors, one executing 100000 >> times and getting to the block with probability 1%, you want to chose it over >> block executing once and getting to you with probability 100%. >> >> You probably want to look for most likely predecestor here. You need to look >> for highest e->count and if they are all 0 for highest EDGE_FREQUENCY and for >> maximal probability only if all fails? > > Yes, thanks, let me do that. New patch that addresses this is included below. Thanks, Teresa > >> >>> + } >>> + >>> + /* If bb is reached by (or reaches, in the case of !WALK_UP) another hot >>> + block (or unpartitioned, e.g. the entry block) then it is ok. If not, >>> + then the most frequent pred (or succ) needs to be adjusted. In the >>> + case where multiple preds/succs have the same probability (e.g. a >>> + 50-50 branch), then both will be adjusted. */ >>> + if (found) >>> + continue; >>> + >>> + FOR_EACH_EDGE (e, ei, edges) >>> + { >>> + if (e->flags & EDGE_DFS_BACK) >>> + continue; >>> + if (e->probability < highest_probability) >>> + continue; >> >> Again for predecestor walk you need to wind down the bit crazy logic described above. >>> Index: predict.c >>> =================================================================== >>> --- predict.c (revision 202021) >>> +++ predict.c (working copy) >>> @@ -241,6 +241,22 @@ probably_never_executed_bb_p (struct function *fun >>> return false; >>> } >>> >>> + >>> +/* Return true in case edge E is probably never executed. */ >>> + >>> +bool >>> +probably_never_executed_edge_p (struct function *fun, edge e) >>> +{ >>> + gcc_checking_assert (fun); >>> + if (profile_info && flag_branch_probabilities) >>> + return ((e->count + profile_info->runs / 2) / profile_info->runs) == 0; >>> + if ((!profile_info || !flag_branch_probabilities) >>> + && (cgraph_get_node (fun->decl)->frequency >>> + == NODE_FREQUENCY_UNLIKELY_EXECUTED)) >>> + return true; >>> + return false; >> Instead of duplicating the conditional, break out the tests into >> probably_never_executed_count_p, like we have for maybe_hot_count_p. > > ok > >> >> It would be nice to extend this to work w/o profile: probably_never_executed_edge_p >> can return true for EH edges, setjmp edges and we can then walk bodies ignoring >> EH/setjmp flagging blocks that are probably never executed enabling the partitioning >> to do a job w/o profile. > > agreed. although I would prefer to leave that for a follow-on patch so > it can be tuned a bit. > >> >> Otherwise the patch look OK to me. Thanks for working on this. Do we have agreement on C++ way of mixing >> declarations and code? > > According to http://gcc.gnu.org/wiki/CppConventions: > > "In new code variables which are used in a small scope should be > defined at the point of first use, rather than at the top of the > function. Variables which are used throughout the function may be > defined at the top of the function, as in C." > > I think I am following that, but let me know if you see something that > needs to be fixed. > > Thanks, > Teresa > >> 2013-08-29 Teresa Johnson <tejohnson@google.com> Steven Bosscher <steven@gcc.gnu.org> * cfgrtl.c (fixup_new_cold_bb): New routine. (commit_edge_insertions): Invoke fixup_partitions. (find_partition_fixes): New routine. (fixup_partitions): Ditto. (verify_hot_cold_block_grouping): Update comments. (rtl_verify_edges): Invoke find_partition_fixes. (rtl_verify_bb_pointers): Update comments. (rtl_verify_bb_layout): Ditto. * basic-block.h (probably_never_executed_edge_p): Declare. (fixup_partitions): Ditto. * cfgcleanup.c (try_optimize_cfg): Invoke fixup_partitions. * bb-reorder.c (sanitize_hot_paths): New function. (find_rarely_executed_basic_blocks_and_crossing_edges): Invoke sanitize_hot_paths. * predict.c (probably_never_executed_edge_p): New routine. * cfg.c (check_bb_profile): Add partition insanity warnings. Index: cfgrtl.c =================================================================== --- cfgrtl.c (revision 202021) +++ cfgrtl.c (working copy) @@ -1358,6 +1358,43 @@ fixup_partition_crossing (edge e) } } +/* Called when block BB has been reassigned to the cold partition, + because it is now dominated by another cold block, + to ensure that the region crossing attributes are updated. */ + +static void +fixup_new_cold_bb (basic_block bb) +{ + edge e; + edge_iterator ei; + + /* This is called when a hot bb is found to now be dominated + by a cold bb and therefore needs to become cold. Therefore, + its preds will no longer be region crossing. Any non-dominating + preds that were previously hot would also have become cold + in the caller for the same region. Any preds that were previously + region-crossing will be adjusted in fixup_partition_crossing. */ + FOR_EACH_EDGE (e, ei, bb->preds) + { + fixup_partition_crossing (e); + } + + /* Possibly need to make bb's successor edges region crossing, + or remove stale region crossing. */ + FOR_EACH_EDGE (e, ei, bb->succs) + { + /* We can't have fall-through edges across partition boundaries. + Note that force_nonfallthru will do any necessary partition + boundary fixup by calling fixup_partition_crossing itself. */ + if ((e->flags & EDGE_FALLTHRU) + && BB_PARTITION (bb) != BB_PARTITION (e->dest) + && e->dest != EXIT_BLOCK_PTR) + force_nonfallthru (e); + else + fixup_partition_crossing (e); + } +} + /* Attempt to change code to redirect edge E to TARGET. Don't do that on expense of adding new instructions or reordering basic blocks. @@ -1996,6 +2033,14 @@ commit_edge_insertions (void) { basic_block bb; + /* Optimization passes that invoke this routine can cause hot blocks + previously reached by both hot and cold blocks to become dominated only + by cold blocks. This will cause the verification below to fail, + and lead to now cold code in the hot section. In some cases this + may only be visible after newly unreachable blocks are deleted, + which will be done by fixup_partitions. */ + fixup_partitions (); + #ifdef ENABLE_CHECKING verify_flow_info (); #endif @@ -2190,6 +2235,101 @@ get_last_bb_insn (basic_block bb) return end; } +/* Sanity check partition hotness to ensure that basic blocks in + the cold partition don't dominate basic blocks in the hot partition. + If FLAG_ONLY is true, report violations as errors. Otherwise + re-mark the dominated blocks as cold, since this is run after + cfg optimizations that may make hot blocks previously reached + by both hot and cold blocks now only reachable along cold paths. */ + +static vec<basic_block> +find_partition_fixes (bool flag_only) +{ + basic_block bb; + vec<basic_block> bbs_in_cold_partition = vNULL; + vec<basic_block> bbs_to_fix = vNULL; + + /* Callers check this. */ + gcc_checking_assert (crtl->has_bb_partition); + + FOR_EACH_BB (bb) + if ((BB_PARTITION (bb) == BB_COLD_PARTITION)) + bbs_in_cold_partition.safe_push (bb); + + if (bbs_in_cold_partition.is_empty ()) + return vNULL; + + bool dom_calculated_here = !dom_info_available_p (CDI_DOMINATORS); + + if (dom_calculated_here) + calculate_dominance_info (CDI_DOMINATORS); + + while (! bbs_in_cold_partition.is_empty ()) + { + bb = bbs_in_cold_partition.pop (); + /* Any blocks dominated by a block in the cold section + must also be cold. */ + basic_block son; + for (son = first_dom_son (CDI_DOMINATORS, bb); + son; + son = next_dom_son (CDI_DOMINATORS, son)) + { + /* If son is not yet cold, then mark it cold here and + enqueue it for further processing. */ + if ((BB_PARTITION (son) != BB_COLD_PARTITION)) + { + if (flag_only) + error ("non-cold basic block %d dominated " + "by a block in the cold partition (%d)", son->index, bb->index); + else + BB_SET_PARTITION (son, BB_COLD_PARTITION); + bbs_to_fix.safe_push (son); + bbs_in_cold_partition.safe_push (son); + } + } + } + + if (dom_calculated_here) + free_dominance_info (CDI_DOMINATORS); + + return bbs_to_fix; +} + +/* Perform cleanup on the hot/cold bb partitioning after optimization + passes that modify the cfg. */ + +void +fixup_partitions (void) +{ + basic_block bb; + + if (!crtl->has_bb_partition) + return; + + /* Delete any blocks that became unreachable and weren't + already cleaned up, for example during edge forwarding + and convert_jumps_to_returns. This will expose more + opportunities for fixing the partition boundaries here. + Also, the calculation of the dominance graph during verification + will assert if there are unreachable nodes. */ + delete_unreachable_blocks (); + + /* If there are partitions, do a sanity check on them: A basic block in + a cold partition cannot dominate a basic block in a hot partition. + Fixup any that now violate this requirement, as a result of edge + forwarding and unreachable block deletion. */ + vec<basic_block> bbs_to_fix = find_partition_fixes (false); + + /* Do the partition fixup after all necessary blocks have been converted to + cold, so that we only update the region crossings the minimum number of + places, which can require forcing edges to be non fallthru. */ + while (! bbs_to_fix.is_empty ()) + { + bb = bbs_to_fix.pop (); + fixup_new_cold_bb (bb); + } +} + /* Verify, in the basic block chain, that there is at most one switch between hot/cold partitions. This condition will not be true until after reorder_basic_blocks is called. */ @@ -2236,7 +2376,8 @@ verify_hot_cold_block_grouping (void) /* Perform several checks on the edges out of each block, such as the consistency of the branch probabilities, the correctness of hot/cold partition crossing edges, and the number of expected - successor edges. */ + successor edges. Also verify that the dominance relationship + between hot/cold blocks is sane. */ static int rtl_verify_edges (void) @@ -2399,6 +2540,14 @@ rtl_verify_edges (void) } } + /* If there are partitions, do a sanity check on them: A basic block in + a cold partition cannot dominate a basic block in a hot partition. */ + if (crtl->has_bb_partition && !err) + { + vec<basic_block> bbs_to_fix = find_partition_fixes (true); + err = !bbs_to_fix.is_empty (); + } + /* Clean up. */ return err; } @@ -2532,7 +2681,7 @@ rtl_verify_bb_pointers (void) and NOTE_INSN_BASIC_BLOCK - verify that no fall_thru edge crosses hot/cold partition boundaries - verify that there are no pending RTL branch predictions - - verify that there is a single hot/cold partition boundary after bbro + - verify that hot blocks are not dominated by cold blocks In future it can be extended check a lot of other stuff as well (reachability of basic blocks, life information, etc. etc.). */ @@ -2778,7 +2927,8 @@ rtl_verify_bb_layout (void) - check that all insns are in the basic blocks (except the switch handling code, barriers and notes) - check that all returns are followed by barriers - - check that all fallthru edge points to the adjacent blocks. */ + - check that all fallthru edge points to the adjacent blocks + - verify that there is a single hot/cold partition boundary after bbro */ static int rtl_verify_flow_info (void) Index: basic-block.h =================================================================== --- basic-block.h (revision 202021) +++ basic-block.h (working copy) @@ -726,6 +726,7 @@ extern void compute_available (sbitmap *, sbitmap extern bool maybe_hot_bb_p (struct function *, const_basic_block); extern bool maybe_hot_edge_p (edge); extern bool probably_never_executed_bb_p (struct function *, const_basic_block); +extern bool probably_never_executed_edge_p (struct function *, edge); extern bool optimize_bb_for_size_p (const_basic_block); extern bool optimize_bb_for_speed_p (const_basic_block); extern bool optimize_edge_for_size_p (edge); @@ -797,6 +798,7 @@ extern bool contains_no_active_insn_p (const_basic extern bool forwarder_block_p (const_basic_block); extern bool can_fallthru (basic_block, basic_block); extern void emit_barrier_after_bb (basic_block bb); +extern void fixup_partitions (void); /* In cfgbuild.c. */ extern void find_many_sub_basic_blocks (sbitmap); Index: cfgcleanup.c =================================================================== --- cfgcleanup.c (revision 202021) +++ cfgcleanup.c (working copy) @@ -2807,10 +2807,21 @@ try_optimize_cfg (int mode) df_analyze (); } + if (changed) + { + /* Edge forwarding in particular can cause hot blocks previously + reached by both hot and cold blocks to become dominated only + by cold blocks. This will cause the verification below to fail, + and lead to now cold code in the hot section. This is not easy + to detect and fix during edge forwarding, and in some cases + is only visible after newly unreachable blocks are deleted, + which will be done in fixup_partitions. */ + fixup_partitions (); + #ifdef ENABLE_CHECKING - if (changed) - verify_flow_info (); + verify_flow_info (); #endif + } changed_overall |= changed; first_pass = false; Index: bb-reorder.c =================================================================== --- bb-reorder.c (revision 202021) +++ bb-reorder.c (working copy) @@ -1444,27 +1444,157 @@ fix_up_crossing_landing_pad (eh_landing_pad old_lp ei_next (&ei); } + +/* Ensure that all hot bbs are included in a hot path through the + procedure. This is done by calling this function twice, once + with WALK_UP true (to look for paths from the entry to hot bbs) and + once with WALK_UP false (to look for paths from hot bbs to the exit). + Returns the updated value of COLD_BB_COUNT and adds newly-hot bbs + to BBS_IN_HOT_PARTITION. */ + +static unsigned int +sanitize_hot_paths (bool walk_up, unsigned int cold_bb_count, + vec<basic_block> *bbs_in_hot_partition) +{ + /* Callers check this. */ + gcc_checking_assert (cold_bb_count); + + /* Keep examining hot bbs while we still have some left to check + and there are remaining cold bbs. */ + vec<basic_block> hot_bbs_to_check = bbs_in_hot_partition->copy (); + while (! hot_bbs_to_check.is_empty () + && cold_bb_count) + { + basic_block bb = hot_bbs_to_check.pop (); + vec<edge, va_gc> *edges = walk_up ? bb->preds : bb->succs; + edge e; + edge_iterator ei; + int highest_probability = 0; + int highest_freq = 0; + gcov_type highest_count = 0; + bool found = false; + + /* Walk the preds/succs and check if there is at least one already + marked hot. Keep track of the most frequent pred/succ so that we + can mark it hot if we don't find one. */ + FOR_EACH_EDGE (e, ei, edges) + { + basic_block reach_bb = walk_up ? e->src : e->dest; + + if (e->flags & EDGE_DFS_BACK) + continue; + + if (BB_PARTITION (reach_bb) != BB_COLD_PARTITION) + { + found = true; + break; + } + /* The following loop will look for the hottest edge via + the edge count, if it is non-zero, then fallback to the edge + frequency and finally the edge probability. */ + if (e->count > highest_count) + highest_count = e->count; + int edge_freq = EDGE_FREQUENCY (e); + if (edge_freq > highest_freq) + highest_freq = edge_freq; + if (e->probability > highest_probability) + highest_probability = e->probability; + } + + /* If bb is reached by (or reaches, in the case of !WALK_UP) another hot + block (or unpartitioned, e.g. the entry block) then it is ok. If not, + then the most frequent pred (or succ) needs to be adjusted. In the + case where multiple preds/succs have the same frequency (e.g. a + 50-50 branch), then both will be adjusted. */ + if (found) + continue; + + FOR_EACH_EDGE (e, ei, edges) + { + if (e->flags & EDGE_DFS_BACK) + continue; + /* Select the hottest edge using the edge count, if it is non-zero, + then fallback to the edge frequency and finally the edge + probability. */ + if (highest_count) + { + if (e->count < highest_count) + continue; + } + else if (highest_freq) + { + if (EDGE_FREQUENCY (e) < highest_freq) + continue; + } + else if (e->probability < highest_probability) + continue; + + basic_block reach_bb = walk_up ? e->src : e->dest; + + /* We have a hot bb with an immediate dominator that is cold. + The dominator needs to be re-marked hot. */ + BB_SET_PARTITION (reach_bb, BB_HOT_PARTITION); + cold_bb_count--; + + /* Now we need to examine newly-hot reach_bb to see if it is also + dominated by a cold bb. */ + bbs_in_hot_partition->safe_push (reach_bb); + hot_bbs_to_check.safe_push (reach_bb); + } + } + + return cold_bb_count; +} + + /* Find the basic blocks that are rarely executed and need to be moved to a separate section of the .o file (to cut down on paging and improve cache locality). Return a vector of all edges that cross. */ -static vec<edge> +static vec<edge> find_rarely_executed_basic_blocks_and_crossing_edges (void) { vec<edge> crossing_edges = vNULL; basic_block bb; edge e; edge_iterator ei; + unsigned int cold_bb_count = 0; + vec<basic_block> bbs_in_hot_partition = vNULL; /* Mark which partition (hot/cold) each basic block belongs in. */ FOR_EACH_BB (bb) { if (probably_never_executed_bb_p (cfun, bb)) - BB_SET_PARTITION (bb, BB_COLD_PARTITION); + { + BB_SET_PARTITION (bb, BB_COLD_PARTITION); + cold_bb_count++; + } else - BB_SET_PARTITION (bb, BB_HOT_PARTITION); + { + BB_SET_PARTITION (bb, BB_HOT_PARTITION); + bbs_in_hot_partition.safe_push (bb); + } } + /* Ensure that hot bbs are included along a hot path from the entry to exit. + Several different possibilities may include cold bbs along all paths + to/from a hot bb. One is that there are edge weight insanities + due to optimization phases that do not properly update basic block profile + counts. The second is that the entry of the function may not be hot, because + it is entered fewer times than the number of profile training runs, but there + is a loop inside the function that causes blocks within the function to be + above the threshold for hotness. This is fixed by walking up from hot bbs + to the entry block, and then down from hot bbs to the exit, performing + partitioning fixups as necessary. */ + if (cold_bb_count) + { + mark_dfs_back_edges (); + cold_bb_count = sanitize_hot_paths (true, cold_bb_count, + &bbs_in_hot_partition); + if (cold_bb_count) + sanitize_hot_paths (false, cold_bb_count, &bbs_in_hot_partition); + } + /* The format of .gcc_except_table does not allow landing pads to be in a different partition as the throw. Fix this by either moving or duplicating the landing pads. */ Index: predict.c =================================================================== --- predict.c (revision 202021) +++ predict.c (working copy) @@ -241,6 +241,22 @@ probably_never_executed_bb_p (struct function *fun return false; } + +/* Return true in case edge E is probably never executed. */ + +bool +probably_never_executed_edge_p (struct function *fun, edge e) +{ + gcc_checking_assert (fun); + if (profile_info && flag_branch_probabilities) + return ((e->count + profile_info->runs / 2) / profile_info->runs) == 0; + if ((!profile_info || !flag_branch_probabilities) + && (cgraph_get_node (fun->decl)->frequency + == NODE_FREQUENCY_UNLIKELY_EXECUTED)) + return true; + return false; +} + /* Return true if NODE should be optimized for size. */ bool Index: cfg.c =================================================================== --- cfg.c (revision 202021) +++ cfg.c (working copy) @@ -446,6 +446,21 @@ check_bb_profile (basic_block bb, FILE * file, int (flags & TDF_COMMENT) ? ";; " : "", s_indent, (int) lsum, (int) bb->count); } + if (BB_PARTITION (bb) == BB_COLD_PARTITION) + { + /* Warn about inconsistencies in the partitioning that are + currently caused by profile insanities created via optimization. */ + if (!probably_never_executed_bb_p (fun, bb)) + fprintf (file, "%s%sBlock in cold partition with hot count\n", + (flags & TDF_COMMENT) ? ";; " : "", s_indent); + FOR_EACH_EDGE (e, ei, bb->preds) + { + if (!probably_never_executed_edge_p (fun, e)) + fprintf (file, + "%s%sBlock in cold partition with incoming hot edge\n", + (flags & TDF_COMMENT) ? ";; " : "", s_indent); + } + } } ^L void -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-30 7:17 ` Teresa Johnson @ 2013-08-30 9:16 ` Jan Hubicka 2013-08-30 15:13 ` Teresa Johnson 0 siblings, 1 reply; 62+ messages in thread From: Jan Hubicka @ 2013-08-30 9:16 UTC (permalink / raw) To: Teresa Johnson Cc: Jan Hubicka, Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, marxin.liska, Sriraman Tallam, Rong Xu > Great! Is this the LTO merging you were talking about in an earlier > message, or the gcov runtime fix (that would presumably not be > lto-specific)? It is the LTO path - we need to merge profiles there anyway for his code unification work. > > I have patch to track this. Moreover vforks seems to produce repeated > > merging of results. > > Aha, does this explain the gimp issue as well? Not really - we still need to debug why we hit cold section so many times with partitioning. I sitll think easier approach will be to lock the cold section and then start probably with testsuite (i.e. write script to compile the small testcases with FDO + partitioning and see what crash by hitting cold section). > > > > Is it really necessary to run this from internal loop of the cfgcleanup? > > The reason I added it here is that just below there is a call to > verify_flow_info, and that will fail with the new verification. Hmm, OK, I suppose we run the cleanup after partitioning just once or twice, right? We can track this incrementally - I am not sure if we put it from the internal iteration loop we would get anything substantial either. Removing unreachable blocks twice is however ugly. > +/* Ensure that all hot bbs are included in a hot path through the > + procedure. This is done by calling this function twice, once > + with WALK_UP true (to look for paths from the entry to hot bbs) and > + once with WALK_UP false (to look for paths from hot bbs to the exit). > + Returns the updated value of COLD_BB_COUNT and adds newly-hot bbs > + to BBS_IN_HOT_PARTITION. */ > + > +static unsigned int > +sanitize_hot_paths (bool walk_up, unsigned int cold_bb_count, > + vec<basic_block> *bbs_in_hot_partition) > +{ > + /* Callers check this. */ > + gcc_checking_assert (cold_bb_count); > + > + /* Keep examining hot bbs while we still have some left to check > + and there are remaining cold bbs. */ > + vec<basic_block> hot_bbs_to_check = bbs_in_hot_partition->copy (); > + while (! hot_bbs_to_check.is_empty () > + && cold_bb_count) > + { > + basic_block bb = hot_bbs_to_check.pop (); > + vec<edge, va_gc> *edges = walk_up ? bb->preds : bb->succs; > + edge e; > + edge_iterator ei; > + int highest_probability = 0; > + int highest_freq = 0; > + gcov_type highest_count = 0; > + bool found = false; > + > + /* Walk the preds/succs and check if there is at least one already > + marked hot. Keep track of the most frequent pred/succ so that we > + can mark it hot if we don't find one. */ > + FOR_EACH_EDGE (e, ei, edges) > + { > + basic_block reach_bb = walk_up ? e->src : e->dest; > + > + if (e->flags & EDGE_DFS_BACK) > + continue; > + > + if (BB_PARTITION (reach_bb) != BB_COLD_PARTITION) > + { > + found = true; > + break; > + } > + /* The following loop will look for the hottest edge via > + the edge count, if it is non-zero, then fallback to the edge > + frequency and finally the edge probability. */ > + if (e->count > highest_count) > + highest_count = e->count; > + int edge_freq = EDGE_FREQUENCY (e); > + if (edge_freq > highest_freq) > + highest_freq = edge_freq; > + if (e->probability > highest_probability) > + highest_probability = e->probability; > + } > + > + /* If bb is reached by (or reaches, in the case of !WALK_UP) another hot > + block (or unpartitioned, e.g. the entry block) then it is ok. If not, > + then the most frequent pred (or succ) needs to be adjusted. In the > + case where multiple preds/succs have the same frequency (e.g. a > + 50-50 branch), then both will be adjusted. */ > + if (found) > + continue; > + > + FOR_EACH_EDGE (e, ei, edges) > + { > + if (e->flags & EDGE_DFS_BACK) > + continue; > + /* Select the hottest edge using the edge count, if it is non-zero, > + then fallback to the edge frequency and finally the edge > + probability. */ > + if (highest_count) > + { > + if (e->count < highest_count) > + continue; > + } > + else if (highest_freq) The frequency condition needs to be done only when you walk predecestors - when you walk down the edge probabilities are just fine. The patch seems OK to me now. I will make our FDO tester to use partitioning so we get this benchmarked a bit. Honza ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-30 9:16 ` Jan Hubicka @ 2013-08-30 15:13 ` Teresa Johnson 2013-08-30 15:28 ` Jan Hubicka 2013-08-30 21:56 ` Rong Xu 0 siblings, 2 replies; 62+ messages in thread From: Teresa Johnson @ 2013-08-30 15:13 UTC (permalink / raw) To: Jan Hubicka, Rong Xu Cc: Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, marxin.liska, Sriraman Tallam [-- Attachment #1: Type: text/plain, Size: 5789 bytes --] Can someone review and ok the attached patch for trunk? It has been bootstrapped and tested on x86-64-unknown-linux-gnu, and tested by enabling -freorder-blocks-and-partition enabled for a profiledbootstrap as well. (Honza, see more responses inlined below. Rong, please see note below as well). Thanks, Teresa On Fri, Aug 30, 2013 at 2:14 AM, Jan Hubicka <hubicka@ucw.cz> wrote: >> Great! Is this the LTO merging you were talking about in an earlier >> message, or the gcov runtime fix (that would presumably not be >> lto-specific)? > > It is the LTO path - we need to merge profiles there anyway for his code unification > work. Rong - can you send a summary of the approach you are working on? Is it LIPO-specific? > >> > I have patch to track this. Moreover vforks seems to produce repeated >> > merging of results. >> >> Aha, does this explain the gimp issue as well? > > Not really - we still need to debug why we hit cold section so many times with > partitioning. I sitll think easier approach will be to lock the cold section and > then start probably with testsuite (i.e. write script to compile the small testcases > with FDO + partitioning and see what crash by hitting cold section). Ok, that is on my todo list. > >> > >> > Is it really necessary to run this from internal loop of the cfgcleanup? >> >> The reason I added it here is that just below there is a call to >> verify_flow_info, and that will fail with the new verification. > > Hmm, OK, I suppose we run the cleanup after partitioning just once or twice, right? > We can track this incrementally - I am not sure if we put it from the internal iteration > loop we would get anything substantial either. > Removing unreachable blocks twice is however ugly. When I was debugging the issue that led to this change I seemed to see 1-2 iterations typically. Although I haven't measured it scientifically. It would be good to revisit that and see if we can pull both parts out of the loop, but as a separate patch. > >> +/* Ensure that all hot bbs are included in a hot path through the >> + procedure. This is done by calling this function twice, once >> + with WALK_UP true (to look for paths from the entry to hot bbs) and >> + once with WALK_UP false (to look for paths from hot bbs to the exit). >> + Returns the updated value of COLD_BB_COUNT and adds newly-hot bbs >> + to BBS_IN_HOT_PARTITION. */ >> + >> +static unsigned int >> +sanitize_hot_paths (bool walk_up, unsigned int cold_bb_count, >> + vec<basic_block> *bbs_in_hot_partition) >> +{ >> + /* Callers check this. */ >> + gcc_checking_assert (cold_bb_count); >> + >> + /* Keep examining hot bbs while we still have some left to check >> + and there are remaining cold bbs. */ >> + vec<basic_block> hot_bbs_to_check = bbs_in_hot_partition->copy (); >> + while (! hot_bbs_to_check.is_empty () >> + && cold_bb_count) >> + { >> + basic_block bb = hot_bbs_to_check.pop (); >> + vec<edge, va_gc> *edges = walk_up ? bb->preds : bb->succs; >> + edge e; >> + edge_iterator ei; >> + int highest_probability = 0; >> + int highest_freq = 0; >> + gcov_type highest_count = 0; >> + bool found = false; >> + >> + /* Walk the preds/succs and check if there is at least one already >> + marked hot. Keep track of the most frequent pred/succ so that we >> + can mark it hot if we don't find one. */ >> + FOR_EACH_EDGE (e, ei, edges) >> + { >> + basic_block reach_bb = walk_up ? e->src : e->dest; >> + >> + if (e->flags & EDGE_DFS_BACK) >> + continue; >> + >> + if (BB_PARTITION (reach_bb) != BB_COLD_PARTITION) >> + { >> + found = true; >> + break; >> + } >> + /* The following loop will look for the hottest edge via >> + the edge count, if it is non-zero, then fallback to the edge >> + frequency and finally the edge probability. */ >> + if (e->count > highest_count) >> + highest_count = e->count; >> + int edge_freq = EDGE_FREQUENCY (e); >> + if (edge_freq > highest_freq) >> + highest_freq = edge_freq; >> + if (e->probability > highest_probability) >> + highest_probability = e->probability; >> + } >> + >> + /* If bb is reached by (or reaches, in the case of !WALK_UP) another hot >> + block (or unpartitioned, e.g. the entry block) then it is ok. If not, >> + then the most frequent pred (or succ) needs to be adjusted. In the >> + case where multiple preds/succs have the same frequency (e.g. a >> + 50-50 branch), then both will be adjusted. */ >> + if (found) >> + continue; >> + >> + FOR_EACH_EDGE (e, ei, edges) >> + { >> + if (e->flags & EDGE_DFS_BACK) >> + continue; >> + /* Select the hottest edge using the edge count, if it is non-zero, >> + then fallback to the edge frequency and finally the edge >> + probability. */ >> + if (highest_count) >> + { >> + if (e->count < highest_count) >> + continue; >> + } >> + else if (highest_freq) > > The frequency condition needs to be done only when you walk predecestors - when > you walk down the edge probabilities are just fine. True. For simplicity I think it should be fine to leave as-is so there isn't more special casing as the current approach works in both directions. > > The patch seems OK to me now. I will make our FDO tester to use partitioning so we get > this benchmarked a bit. Ok thanks. > > Honza -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 [-- Attachment #2: patch.diff --] [-- Type: application/octet-stream, Size: 19380 bytes --] Patch 3 of 3 split out from the patch I sent in May that fixes problems with -freorder-blocks-and-partition, with changes/fixes discussed in that thread. See http://gcc.gnu.org/ml/gcc-patches/2013-05/threads.html#00388 for more context. This patch sanitizes the partitioning to address issues such as edge weight insanities that sometimes occur due to upstream optimizations, and ensures that hot blocks are not dominated by cold blocks. This needs to be resanitized after certain cfg optimizations that may cause hot blocks previously reached via both hot and cold paths to only be reached by cold paths. The verification code in sanitize_dominator_hotness was contributed by Steven Bosscher. 2013-08-29 Teresa Johnson <tejohnson@google.com> Steven Bosscher <steven@gcc.gnu.org> * cfgrtl.c (fixup_new_cold_bb): New routine. (commit_edge_insertions): Invoke fixup_partitions. (find_partition_fixes): New routine. (fixup_partitions): Ditto. (verify_hot_cold_block_grouping): Update comments. (rtl_verify_edges): Invoke find_partition_fixes. (rtl_verify_bb_pointers): Update comments. (rtl_verify_bb_layout): Ditto. * basic-block.h (probably_never_executed_edge_p): Declare. (fixup_partitions): Ditto. * cfgcleanup.c (try_optimize_cfg): Invoke fixup_partitions. * bb-reorder.c (sanitize_hot_paths): New function. (find_rarely_executed_basic_blocks_and_crossing_edges): Invoke sanitize_hot_paths. * predict.c (probably_never_executed_edge_p): New routine. * cfg.c (check_bb_profile): Add partition insanity warnings. Index: cfgrtl.c =================================================================== --- cfgrtl.c (revision 202021) +++ cfgrtl.c (working copy) @@ -1358,6 +1358,43 @@ fixup_partition_crossing (edge e) } } +/* Called when block BB has been reassigned to the cold partition, + because it is now dominated by another cold block, + to ensure that the region crossing attributes are updated. */ + +static void +fixup_new_cold_bb (basic_block bb) +{ + edge e; + edge_iterator ei; + + /* This is called when a hot bb is found to now be dominated + by a cold bb and therefore needs to become cold. Therefore, + its preds will no longer be region crossing. Any non-dominating + preds that were previously hot would also have become cold + in the caller for the same region. Any preds that were previously + region-crossing will be adjusted in fixup_partition_crossing. */ + FOR_EACH_EDGE (e, ei, bb->preds) + { + fixup_partition_crossing (e); + } + + /* Possibly need to make bb's successor edges region crossing, + or remove stale region crossing. */ + FOR_EACH_EDGE (e, ei, bb->succs) + { + /* We can't have fall-through edges across partition boundaries. + Note that force_nonfallthru will do any necessary partition + boundary fixup by calling fixup_partition_crossing itself. */ + if ((e->flags & EDGE_FALLTHRU) + && BB_PARTITION (bb) != BB_PARTITION (e->dest) + && e->dest != EXIT_BLOCK_PTR) + force_nonfallthru (e); + else + fixup_partition_crossing (e); + } +} + /* Attempt to change code to redirect edge E to TARGET. Don't do that on expense of adding new instructions or reordering basic blocks. @@ -1996,6 +2033,14 @@ commit_edge_insertions (void) { basic_block bb; + /* Optimization passes that invoke this routine can cause hot blocks + previously reached by both hot and cold blocks to become dominated only + by cold blocks. This will cause the verification below to fail, + and lead to now cold code in the hot section. In some cases this + may only be visible after newly unreachable blocks are deleted, + which will be done by fixup_partitions. */ + fixup_partitions (); + #ifdef ENABLE_CHECKING verify_flow_info (); #endif @@ -2190,6 +2235,101 @@ get_last_bb_insn (basic_block bb) return end; } +/* Sanity check partition hotness to ensure that basic blocks in + the cold partition don't dominate basic blocks in the hot partition. + If FLAG_ONLY is true, report violations as errors. Otherwise + re-mark the dominated blocks as cold, since this is run after + cfg optimizations that may make hot blocks previously reached + by both hot and cold blocks now only reachable along cold paths. */ + +static vec<basic_block> +find_partition_fixes (bool flag_only) +{ + basic_block bb; + vec<basic_block> bbs_in_cold_partition = vNULL; + vec<basic_block> bbs_to_fix = vNULL; + + /* Callers check this. */ + gcc_checking_assert (crtl->has_bb_partition); + + FOR_EACH_BB (bb) + if ((BB_PARTITION (bb) == BB_COLD_PARTITION)) + bbs_in_cold_partition.safe_push (bb); + + if (bbs_in_cold_partition.is_empty ()) + return vNULL; + + bool dom_calculated_here = !dom_info_available_p (CDI_DOMINATORS); + + if (dom_calculated_here) + calculate_dominance_info (CDI_DOMINATORS); + + while (! bbs_in_cold_partition.is_empty ()) + { + bb = bbs_in_cold_partition.pop (); + /* Any blocks dominated by a block in the cold section + must also be cold. */ + basic_block son; + for (son = first_dom_son (CDI_DOMINATORS, bb); + son; + son = next_dom_son (CDI_DOMINATORS, son)) + { + /* If son is not yet cold, then mark it cold here and + enqueue it for further processing. */ + if ((BB_PARTITION (son) != BB_COLD_PARTITION)) + { + if (flag_only) + error ("non-cold basic block %d dominated " + "by a block in the cold partition (%d)", son->index, bb->index); + else + BB_SET_PARTITION (son, BB_COLD_PARTITION); + bbs_to_fix.safe_push (son); + bbs_in_cold_partition.safe_push (son); + } + } + } + + if (dom_calculated_here) + free_dominance_info (CDI_DOMINATORS); + + return bbs_to_fix; +} + +/* Perform cleanup on the hot/cold bb partitioning after optimization + passes that modify the cfg. */ + +void +fixup_partitions (void) +{ + basic_block bb; + + if (!crtl->has_bb_partition) + return; + + /* Delete any blocks that became unreachable and weren't + already cleaned up, for example during edge forwarding + and convert_jumps_to_returns. This will expose more + opportunities for fixing the partition boundaries here. + Also, the calculation of the dominance graph during verification + will assert if there are unreachable nodes. */ + delete_unreachable_blocks (); + + /* If there are partitions, do a sanity check on them: A basic block in + a cold partition cannot dominate a basic block in a hot partition. + Fixup any that now violate this requirement, as a result of edge + forwarding and unreachable block deletion. */ + vec<basic_block> bbs_to_fix = find_partition_fixes (false); + + /* Do the partition fixup after all necessary blocks have been converted to + cold, so that we only update the region crossings the minimum number of + places, which can require forcing edges to be non fallthru. */ + while (! bbs_to_fix.is_empty ()) + { + bb = bbs_to_fix.pop (); + fixup_new_cold_bb (bb); + } +} + /* Verify, in the basic block chain, that there is at most one switch between hot/cold partitions. This condition will not be true until after reorder_basic_blocks is called. */ @@ -2236,7 +2376,8 @@ verify_hot_cold_block_grouping (void) /* Perform several checks on the edges out of each block, such as the consistency of the branch probabilities, the correctness of hot/cold partition crossing edges, and the number of expected - successor edges. */ + successor edges. Also verify that the dominance relationship + between hot/cold blocks is sane. */ static int rtl_verify_edges (void) @@ -2399,6 +2540,14 @@ rtl_verify_edges (void) } } + /* If there are partitions, do a sanity check on them: A basic block in + a cold partition cannot dominate a basic block in a hot partition. */ + if (crtl->has_bb_partition && !err) + { + vec<basic_block> bbs_to_fix = find_partition_fixes (true); + err = !bbs_to_fix.is_empty (); + } + /* Clean up. */ return err; } @@ -2532,7 +2681,7 @@ rtl_verify_bb_pointers (void) and NOTE_INSN_BASIC_BLOCK - verify that no fall_thru edge crosses hot/cold partition boundaries - verify that there are no pending RTL branch predictions - - verify that there is a single hot/cold partition boundary after bbro + - verify that hot blocks are not dominated by cold blocks In future it can be extended check a lot of other stuff as well (reachability of basic blocks, life information, etc. etc.). */ @@ -2778,7 +2927,8 @@ rtl_verify_bb_layout (void) - check that all insns are in the basic blocks (except the switch handling code, barriers and notes) - check that all returns are followed by barriers - - check that all fallthru edge points to the adjacent blocks. */ + - check that all fallthru edge points to the adjacent blocks + - verify that there is a single hot/cold partition boundary after bbro */ static int rtl_verify_flow_info (void) Index: basic-block.h =================================================================== --- basic-block.h (revision 202021) +++ basic-block.h (working copy) @@ -726,6 +726,7 @@ extern void compute_available (sbitmap *, sbitmap extern bool maybe_hot_bb_p (struct function *, const_basic_block); extern bool maybe_hot_edge_p (edge); extern bool probably_never_executed_bb_p (struct function *, const_basic_block); +extern bool probably_never_executed_edge_p (struct function *, edge); extern bool optimize_bb_for_size_p (const_basic_block); extern bool optimize_bb_for_speed_p (const_basic_block); extern bool optimize_edge_for_size_p (edge); @@ -797,6 +798,7 @@ extern bool contains_no_active_insn_p (const_basic extern bool forwarder_block_p (const_basic_block); extern bool can_fallthru (basic_block, basic_block); extern void emit_barrier_after_bb (basic_block bb); +extern void fixup_partitions (void); /* In cfgbuild.c. */ extern void find_many_sub_basic_blocks (sbitmap); Index: cfgcleanup.c =================================================================== --- cfgcleanup.c (revision 202021) +++ cfgcleanup.c (working copy) @@ -2807,10 +2807,21 @@ try_optimize_cfg (int mode) df_analyze (); } + if (changed) + { + /* Edge forwarding in particular can cause hot blocks previously + reached by both hot and cold blocks to become dominated only + by cold blocks. This will cause the verification below to fail, + and lead to now cold code in the hot section. This is not easy + to detect and fix during edge forwarding, and in some cases + is only visible after newly unreachable blocks are deleted, + which will be done in fixup_partitions. */ + fixup_partitions (); + #ifdef ENABLE_CHECKING - if (changed) - verify_flow_info (); + verify_flow_info (); #endif + } changed_overall |= changed; first_pass = false; Index: bb-reorder.c =================================================================== --- bb-reorder.c (revision 202021) +++ bb-reorder.c (working copy) @@ -1444,27 +1444,157 @@ fix_up_crossing_landing_pad (eh_landing_pad old_lp ei_next (&ei); } + +/* Ensure that all hot bbs are included in a hot path through the + procedure. This is done by calling this function twice, once + with WALK_UP true (to look for paths from the entry to hot bbs) and + once with WALK_UP false (to look for paths from hot bbs to the exit). + Returns the updated value of COLD_BB_COUNT and adds newly-hot bbs + to BBS_IN_HOT_PARTITION. */ + +static unsigned int +sanitize_hot_paths (bool walk_up, unsigned int cold_bb_count, + vec<basic_block> *bbs_in_hot_partition) +{ + /* Callers check this. */ + gcc_checking_assert (cold_bb_count); + + /* Keep examining hot bbs while we still have some left to check + and there are remaining cold bbs. */ + vec<basic_block> hot_bbs_to_check = bbs_in_hot_partition->copy (); + while (! hot_bbs_to_check.is_empty () + && cold_bb_count) + { + basic_block bb = hot_bbs_to_check.pop (); + vec<edge, va_gc> *edges = walk_up ? bb->preds : bb->succs; + edge e; + edge_iterator ei; + int highest_probability = 0; + int highest_freq = 0; + gcov_type highest_count = 0; + bool found = false; + + /* Walk the preds/succs and check if there is at least one already + marked hot. Keep track of the most frequent pred/succ so that we + can mark it hot if we don't find one. */ + FOR_EACH_EDGE (e, ei, edges) + { + basic_block reach_bb = walk_up ? e->src : e->dest; + + if (e->flags & EDGE_DFS_BACK) + continue; + + if (BB_PARTITION (reach_bb) != BB_COLD_PARTITION) + { + found = true; + break; + } + /* The following loop will look for the hottest edge via + the edge count, if it is non-zero, then fallback to the edge + frequency and finally the edge probability. */ + if (e->count > highest_count) + highest_count = e->count; + int edge_freq = EDGE_FREQUENCY (e); + if (edge_freq > highest_freq) + highest_freq = edge_freq; + if (e->probability > highest_probability) + highest_probability = e->probability; + } + + /* If bb is reached by (or reaches, in the case of !WALK_UP) another hot + block (or unpartitioned, e.g. the entry block) then it is ok. If not, + then the most frequent pred (or succ) needs to be adjusted. In the + case where multiple preds/succs have the same frequency (e.g. a + 50-50 branch), then both will be adjusted. */ + if (found) + continue; + + FOR_EACH_EDGE (e, ei, edges) + { + if (e->flags & EDGE_DFS_BACK) + continue; + /* Select the hottest edge using the edge count, if it is non-zero, + then fallback to the edge frequency and finally the edge + probability. */ + if (highest_count) + { + if (e->count < highest_count) + continue; + } + else if (highest_freq) + { + if (EDGE_FREQUENCY (e) < highest_freq) + continue; + } + else if (e->probability < highest_probability) + continue; + + basic_block reach_bb = walk_up ? e->src : e->dest; + + /* We have a hot bb with an immediate dominator that is cold. + The dominator needs to be re-marked hot. */ + BB_SET_PARTITION (reach_bb, BB_HOT_PARTITION); + cold_bb_count--; + + /* Now we need to examine newly-hot reach_bb to see if it is also + dominated by a cold bb. */ + bbs_in_hot_partition->safe_push (reach_bb); + hot_bbs_to_check.safe_push (reach_bb); + } + } + + return cold_bb_count; +} + + /* Find the basic blocks that are rarely executed and need to be moved to a separate section of the .o file (to cut down on paging and improve cache locality). Return a vector of all edges that cross. */ -static vec<edge> +static vec<edge> find_rarely_executed_basic_blocks_and_crossing_edges (void) { vec<edge> crossing_edges = vNULL; basic_block bb; edge e; edge_iterator ei; + unsigned int cold_bb_count = 0; + vec<basic_block> bbs_in_hot_partition = vNULL; /* Mark which partition (hot/cold) each basic block belongs in. */ FOR_EACH_BB (bb) { if (probably_never_executed_bb_p (cfun, bb)) - BB_SET_PARTITION (bb, BB_COLD_PARTITION); + { + BB_SET_PARTITION (bb, BB_COLD_PARTITION); + cold_bb_count++; + } else - BB_SET_PARTITION (bb, BB_HOT_PARTITION); + { + BB_SET_PARTITION (bb, BB_HOT_PARTITION); + bbs_in_hot_partition.safe_push (bb); + } } + /* Ensure that hot bbs are included along a hot path from the entry to exit. + Several different possibilities may include cold bbs along all paths + to/from a hot bb. One is that there are edge weight insanities + due to optimization phases that do not properly update basic block profile + counts. The second is that the entry of the function may not be hot, because + it is entered fewer times than the number of profile training runs, but there + is a loop inside the function that causes blocks within the function to be + above the threshold for hotness. This is fixed by walking up from hot bbs + to the entry block, and then down from hot bbs to the exit, performing + partitioning fixups as necessary. */ + if (cold_bb_count) + { + mark_dfs_back_edges (); + cold_bb_count = sanitize_hot_paths (true, cold_bb_count, + &bbs_in_hot_partition); + if (cold_bb_count) + sanitize_hot_paths (false, cold_bb_count, &bbs_in_hot_partition); + } + /* The format of .gcc_except_table does not allow landing pads to be in a different partition as the throw. Fix this by either moving or duplicating the landing pads. */ Index: predict.c =================================================================== --- predict.c (revision 202021) +++ predict.c (working copy) @@ -241,6 +241,22 @@ probably_never_executed_bb_p (struct function *fun return false; } + +/* Return true in case edge E is probably never executed. */ + +bool +probably_never_executed_edge_p (struct function *fun, edge e) +{ + gcc_checking_assert (fun); + if (profile_info && flag_branch_probabilities) + return ((e->count + profile_info->runs / 2) / profile_info->runs) == 0; + if ((!profile_info || !flag_branch_probabilities) + && (cgraph_get_node (fun->decl)->frequency + == NODE_FREQUENCY_UNLIKELY_EXECUTED)) + return true; + return false; +} + /* Return true if NODE should be optimized for size. */ bool Index: cfg.c =================================================================== --- cfg.c (revision 202021) +++ cfg.c (working copy) @@ -446,6 +446,21 @@ check_bb_profile (basic_block bb, FILE * file, int (flags & TDF_COMMENT) ? ";; " : "", s_indent, (int) lsum, (int) bb->count); } + if (BB_PARTITION (bb) == BB_COLD_PARTITION) + { + /* Warn about inconsistencies in the partitioning that are + currently caused by profile insanities created via optimization. */ + if (!probably_never_executed_bb_p (fun, bb)) + fprintf (file, "%s%sBlock in cold partition with hot count\n", + (flags & TDF_COMMENT) ? ";; " : "", s_indent); + FOR_EACH_EDGE (e, ei, bb->preds) + { + if (!probably_never_executed_edge_p (fun, e)) + fprintf (file, + "%s%sBlock in cold partition with incoming hot edge\n", + (flags & TDF_COMMENT) ? ";; " : "", s_indent); + } + } } \f void ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-30 15:13 ` Teresa Johnson @ 2013-08-30 15:28 ` Jan Hubicka 2013-08-30 15:54 ` Teresa Johnson 2013-08-30 21:56 ` Rong Xu 1 sibling, 1 reply; 62+ messages in thread From: Jan Hubicka @ 2013-08-30 15:28 UTC (permalink / raw) To: Teresa Johnson Cc: Jan Hubicka, Rong Xu, Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, marxin.liska, Sriraman Tallam > > > > The frequency condition needs to be done only when you walk predecestors - when > > you walk down the edge probabilities are just fine. > > True. For simplicity I think it should be fine to leave as-is so there > isn't more special casing as the current approach works in both > directions. Yep, you are right. Frequencies are safe both directions. I think this change belongs to profile feedback category, so the patch is OK. Honza ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-30 15:28 ` Jan Hubicka @ 2013-08-30 15:54 ` Teresa Johnson 0 siblings, 0 replies; 62+ messages in thread From: Teresa Johnson @ 2013-08-30 15:54 UTC (permalink / raw) To: Jan Hubicka Cc: Rong Xu, Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, marxin.liska, Sriraman Tallam On Fri, Aug 30, 2013 at 8:26 AM, Jan Hubicka <hubicka@ucw.cz> wrote: >> > >> > The frequency condition needs to be done only when you walk predecestors - when >> > you walk down the edge probabilities are just fine. >> >> True. For simplicity I think it should be fine to leave as-is so there >> isn't more special casing as the current approach works in both >> directions. > > Yep, you are right. Frequencies are safe both directions. > > I think this change belongs to profile feedback category, so the patch is OK. ok, thanks. Teresa > > Honza -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-30 15:13 ` Teresa Johnson 2013-08-30 15:28 ` Jan Hubicka @ 2013-08-30 21:56 ` Rong Xu 1 sibling, 0 replies; 62+ messages in thread From: Rong Xu @ 2013-08-30 21:56 UTC (permalink / raw) To: Teresa Johnson Cc: Jan Hubicka, Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, marxin.liska, Sriraman Tallam On Fri, Aug 30, 2013 at 7:50 AM, Teresa Johnson <tejohnson@google.com> wrote: > Can someone review and ok the attached patch for trunk? It has been > bootstrapped and tested on x86-64-unknown-linux-gnu, and tested by > enabling -freorder-blocks-and-partition enabled for a > profiledbootstrap as well. > > (Honza, see more responses inlined below. Rong, please see note below as well). > > Thanks, > Teresa > > On Fri, Aug 30, 2013 at 2:14 AM, Jan Hubicka <hubicka@ucw.cz> wrote: >>> Great! Is this the LTO merging you were talking about in an earlier >>> message, or the gcov runtime fix (that would presumably not be >>> lto-specific)? >> >> It is the LTO path - we need to merge profiles there anyway for his code unification >> work. > > Rong - can you send a summary of the approach you are working on? Is > it LIPO-specific? I'm also working to improve COMDAT handling in FDO/LIPO. It's applicable to regular FDO. Our motivation case is different from the case talked here: COMDAT function F is defined in module A and is picked by linker as the out-of-line copy in profile-gen phrase. It gets all the counters. In profile-use compilation F may not be emitted in A (like all the callsites are ipa-inlined in A), We choose instance of F in module B as the out-of-line copy, and it's not optimized. We are seeing more of this kind of cases in LIPO due to multiple COMDAT copies brought by the auxiliary modules. Since COMDAT function may be inlined after instrumentation, multiple copies of the counters make be co-exists We want to differentiate inlined-copy counters or out-of-line-copy counters. (In LIPO, we actually encourage the inline of COMDAT in IPA-inline to get more context sensitive counters.) Our current solution is to have another instrumentation only for COMDAT functions. * For each comdat_key, we create a global var pointing to the gcov_fn_info of the out-of-line copy. * This global var is initialized by the instrumentation code placed in the function entry, to the value of gcov_fn_info of current module. This is post IPA-inline instrumentation. So at most one instrumentation (the one picked by linker) is executed. * Expand gcov_fn_info to points to the global var. In the run time, we can differentiate if the counter out-of-line copy or inlined copy. We set a tag in gcda file accordingly. (in the case of both out-of-line copy and inlined copy, we treat it as out-of-line copy). In the profile-use phrase, we use this info to resolve the decls of COMDAT functions, and also make sure we only emit the out-of-line copy of the COMDAT function (even if it's not referenced in the module). This has been done and we are testing it now. I talked to Teresa about the her case some time back. It seems that merging the counters is an easy change with the above patch because we have the address of the out-of-line gcov_fn_info. We can do a simple in-memory merge if the checksum matches. The concern is that this may reduce the context sensitive information. -Rong > >> >>> > I have patch to track this. Moreover vforks seems to produce repeated >>> > merging of results. >>> >>> Aha, does this explain the gimp issue as well? >> >> Not really - we still need to debug why we hit cold section so many times with >> partitioning. I sitll think easier approach will be to lock the cold section and >> then start probably with testsuite (i.e. write script to compile the small testcases >> with FDO + partitioning and see what crash by hitting cold section). > > Ok, that is on my todo list. > >> >>> > >>> > Is it really necessary to run this from internal loop of the cfgcleanup? >>> >>> The reason I added it here is that just below there is a call to >>> verify_flow_info, and that will fail with the new verification. >> >> Hmm, OK, I suppose we run the cleanup after partitioning just once or twice, right? >> We can track this incrementally - I am not sure if we put it from the internal iteration >> loop we would get anything substantial either. >> Removing unreachable blocks twice is however ugly. > > When I was debugging the issue that led to this change I seemed to see > 1-2 iterations typically. Although I haven't measured it > scientifically. It would be good to revisit that and see if we can > pull both parts out of the loop, but as a separate patch. > >> >>> +/* Ensure that all hot bbs are included in a hot path through the >>> + procedure. This is done by calling this function twice, once >>> + with WALK_UP true (to look for paths from the entry to hot bbs) and >>> + once with WALK_UP false (to look for paths from hot bbs to the exit). >>> + Returns the updated value of COLD_BB_COUNT and adds newly-hot bbs >>> + to BBS_IN_HOT_PARTITION. */ >>> + >>> +static unsigned int >>> +sanitize_hot_paths (bool walk_up, unsigned int cold_bb_count, >>> + vec<basic_block> *bbs_in_hot_partition) >>> +{ >>> + /* Callers check this. */ >>> + gcc_checking_assert (cold_bb_count); >>> + >>> + /* Keep examining hot bbs while we still have some left to check >>> + and there are remaining cold bbs. */ >>> + vec<basic_block> hot_bbs_to_check = bbs_in_hot_partition->copy (); >>> + while (! hot_bbs_to_check.is_empty () >>> + && cold_bb_count) >>> + { >>> + basic_block bb = hot_bbs_to_check.pop (); >>> + vec<edge, va_gc> *edges = walk_up ? bb->preds : bb->succs; >>> + edge e; >>> + edge_iterator ei; >>> + int highest_probability = 0; >>> + int highest_freq = 0; >>> + gcov_type highest_count = 0; >>> + bool found = false; >>> + >>> + /* Walk the preds/succs and check if there is at least one already >>> + marked hot. Keep track of the most frequent pred/succ so that we >>> + can mark it hot if we don't find one. */ >>> + FOR_EACH_EDGE (e, ei, edges) >>> + { >>> + basic_block reach_bb = walk_up ? e->src : e->dest; >>> + >>> + if (e->flags & EDGE_DFS_BACK) >>> + continue; >>> + >>> + if (BB_PARTITION (reach_bb) != BB_COLD_PARTITION) >>> + { >>> + found = true; >>> + break; >>> + } >>> + /* The following loop will look for the hottest edge via >>> + the edge count, if it is non-zero, then fallback to the edge >>> + frequency and finally the edge probability. */ >>> + if (e->count > highest_count) >>> + highest_count = e->count; >>> + int edge_freq = EDGE_FREQUENCY (e); >>> + if (edge_freq > highest_freq) >>> + highest_freq = edge_freq; >>> + if (e->probability > highest_probability) >>> + highest_probability = e->probability; >>> + } >>> + >>> + /* If bb is reached by (or reaches, in the case of !WALK_UP) another hot >>> + block (or unpartitioned, e.g. the entry block) then it is ok. If not, >>> + then the most frequent pred (or succ) needs to be adjusted. In the >>> + case where multiple preds/succs have the same frequency (e.g. a >>> + 50-50 branch), then both will be adjusted. */ >>> + if (found) >>> + continue; >>> + >>> + FOR_EACH_EDGE (e, ei, edges) >>> + { >>> + if (e->flags & EDGE_DFS_BACK) >>> + continue; >>> + /* Select the hottest edge using the edge count, if it is non-zero, >>> + then fallback to the edge frequency and finally the edge >>> + probability. */ >>> + if (highest_count) >>> + { >>> + if (e->count < highest_count) >>> + continue; >>> + } >>> + else if (highest_freq) >> >> The frequency condition needs to be done only when you walk predecestors - when >> you walk down the edge probabilities are just fine. > > True. For simplicity I think it should be fine to leave as-is so there > isn't more special casing as the current approach works in both > directions. > >> >> The patch seems OK to me now. I will make our FDO tester to use partitioning so we get >> this benchmarked a bit. > > Ok thanks. > >> >> Honza > > > > -- > Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-27 18:12 ` Teresa Johnson 2013-08-28 16:59 ` Jan Hubicka @ 2013-08-31 16:20 ` Jan Hubicka 2013-08-31 23:40 ` Jan Hubicka 1 sibling, 1 reply; 62+ messages in thread From: Jan Hubicka @ 2013-08-31 16:20 UTC (permalink / raw) To: Teresa Johnson Cc: Jan Hubicka, Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, marxin.liska, Sriraman Tallam Hi, With Martin we made script for testing the profiling failures. First do ld --verbose >~/script then apply --- /home/jh/script2 2013-08-31 17:59:11.000000000 +0200 +++ /home/jh/script 2013-08-31 17:39:40.000000000 +0200 @@ -1,12 +1,3 @@ -GNU ld (GNU Binutils for Debian) 2.20.1-system.20100303 - Supported emulations: - elf_x86_64 - elf_i386 - i386linux - elf_l1om -using internal linker script: -================================================== -/* Script for -z combreloc: combine and sort reloc sections */ OUTPUT_FORMAT("elf64-x86-64", "elf64-x86-64", "elf64-x86-64") OUTPUT_ARCH(i386:x86-64) @@ -55,6 +46,7 @@ KEEP (*(.init)) } =0x90909090 .plt : { *(.plt) *(.iplt) } + .text.unlikely (NOLOAD) : { *(.text.unlikely .text.*_unlikely .text.unlikely.*) } .text : { *(.text.unlikely .text.*_unlikely) @@ -218,4 +210,3 @@ } -================================================== then create t.c as: __attribute__ ((noinline)) t() { printf ("test\n"); } main(int argc) { if (argc>1) t(); return 0; } and dotests as: for name in $* do rm a.out *.gcda 2>/dev/null ./xgcc -B ./ -Ofast -fprofile-generate $name --static 2>/dev/null if [ -f a.out ] then ./a.out >/dev/null 2>/dev/null || continue ./xgcc -B ./ -Ofast -fprofile-use -freorder-blocks-and-partition -Wl,-T,/home/jh/script --static $name 2>/dev/null ./a.out t >/dev/null 2>/dev/null || echo FAIL $name else echo skip $name fi done Then run: jh@gcc10:~/trunk/build/gcc$ sh dotests t.c FAIL t.c You should get FAIL if things are fine, because t.c depends behaviour on number of command line parameters. If that fail you can run i.e. jh@gcc10:~/trunk/build/gcc$ sh dotests ~/trunk/gcc/testsuite/gcc.c-torture/execute/*.c skip /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20000402-1.c FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20000422-1.c FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20000910-2.c skip /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20010329-1.c Those with fail get cold section executed. When you run tehm again through dotests you can do gdb a.out and see what function gets split incorrectly. Honza ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-31 16:20 ` Jan Hubicka @ 2013-08-31 23:40 ` Jan Hubicka 2013-09-24 8:07 ` Teresa Johnson 0 siblings, 1 reply; 62+ messages in thread From: Jan Hubicka @ 2013-08-31 23:40 UTC (permalink / raw) To: Jan Hubicka Cc: Teresa Johnson, Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, marxin.liska, Sriraman Tallam Hi, I run my script on execute testsuite and looked into few testcases. The problem I found was roundoff errors - i.e. when expanding switch we set 50% chage that out of bound value is above or bellow. Both gets rounded to 0, because switch is executed once and the value is bellow. Partly this can be fixed by making probably_never_executed to consider frequencies when counts are too coarse: Index: predict.c =================================================================== --- predict.c (revision 202133) +++ predict.c (working copy) @@ -232,8 +232,22 @@ bool probably_never_executed_bb_p (struct function *fun, const_basic_block bb) { gcc_checking_assert (fun); - if (profile_info && flag_branch_probabilities) - return ((bb->count + profile_info->runs / 2) / profile_info->runs) == 0; + if (profile_status_for_function (fun) == PROFILE_READ) + { + if ((bb->count * 4 + profile_info->runs / 2) / profile_info->runs > 0) + return false; + if (!bb->frequency) + return true; + if (!ENTRY_BLOCK_PTR->frequency) + return false; + if (ENTRY_BLOCK_PTR->count && ENTRY_BLOCK_PTR->count < REG_BR_PROB_BASE) + { + return (RDIV (bb->frequency * ENTRY_BLOCK_PTR->count, + ENTRY_BLOCK_PTR->frequency) + < REG_BR_PROB_BASE / 4); + } + return true; + } if ((!profile_info || !flag_branch_probabilities) && (cgraph_get_node (fun->decl)->frequency == NODE_FREQUENCY_UNLIKELY_EXECUTED)) In other cases it was mostly loop unrolling in combination with jump threading. So I modified my script to separately report when failure happens for test trained once and test trained hundred times. FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20000422-1.c FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20000910-2.c FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20020413-1.c FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20030903-1.c FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20031204-1.c FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20031204-1.c FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20060420-1.c FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20060905-1.c FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20120427-1.c FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20120427-2.c FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20120808-1.c FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20121108-1.c FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20121108-1.c FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/920501-6.c FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/920501-6.c FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/920726-1.c FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/981001-1.c FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/981001-1.c FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/990628-1.c FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/991216-2.c FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/991216-2.c FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/cmpdi-1.c FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/cmpdi-1.c FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/float-floor.c FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/float-floor.c FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr33870-1.c FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr33870-1.c FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr33870.c FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr33870.c FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr36093.c FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr37573.c FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr43784.c FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr43784.c FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/switch-1.c FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/switch-1.c FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/va-arg-22.c FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/va-arg-22.c FAIL1 is failure after one run, FIAL is failure after 100 train runs. We should take look at FAILs and see if there are bugs to fix. For FAIL1 I think it is kind of design problem: while implementing counts&frequencies the idea was that small counts do not matter, so integer arithmetic is all right. I wonder if with current C++ wonderland we can't simply switch count to a better representation. Either sreal or fixedpoint with capping (the integer overflow issues are tiring, too). Honza ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-31 23:40 ` Jan Hubicka @ 2013-09-24 8:07 ` Teresa Johnson 2013-09-24 13:44 ` Jan Hubicka 2013-09-24 18:28 ` Jan Hubicka 0 siblings, 2 replies; 62+ messages in thread From: Teresa Johnson @ 2013-09-24 8:07 UTC (permalink / raw) To: Jan Hubicka; +Cc: gcc-patches, marxin.liska Hi Honza, I am finally getting back to working on this after a few weeks of working on some other priorities. On Sat, Aug 31, 2013 at 2:46 PM, Jan Hubicka <hubicka@ucw.cz> wrote: > Hi, > I run my script on execute testsuite and looked into few testcases. The problem I found > was roundoff errors - i.e. when expanding switch we set 50% chage that out of bound > value is above or bellow. Both gets rounded to 0, because switch is executed once > and the value is bellow. > > Partly this can be fixed by making probably_never_executed to consider frequencies when > counts are too coarse: > > Index: predict.c > =================================================================== > --- predict.c (revision 202133) > +++ predict.c (working copy) > @@ -232,8 +232,22 @@ bool > probably_never_executed_bb_p (struct function *fun, const_basic_block bb) > { > gcc_checking_assert (fun); > - if (profile_info && flag_branch_probabilities) > - return ((bb->count + profile_info->runs / 2) / profile_info->runs) == 0; > + if (profile_status_for_function (fun) == PROFILE_READ) > + { > + if ((bb->count * 4 + profile_info->runs / 2) / profile_info->runs > 0) > + return false; > + if (!bb->frequency) > + return true; > + if (!ENTRY_BLOCK_PTR->frequency) > + return false; > + if (ENTRY_BLOCK_PTR->count && ENTRY_BLOCK_PTR->count < REG_BR_PROB_BASE) > + { > + return (RDIV (bb->frequency * ENTRY_BLOCK_PTR->count, > + ENTRY_BLOCK_PTR->frequency) > + < REG_BR_PROB_BASE / 4); > + } > + return true; > + } > if ((!profile_info || !flag_branch_probabilities) > && (cgraph_get_node (fun->decl)->frequency > == NODE_FREQUENCY_UNLIKELY_EXECUTED)) Did you mean to commit the above change? I see that it went in as part of r202258 but doesn't show up in the ChangeLog entry for that revision. > > In other cases it was mostly loop unrolling in combination with jump threading. So > I modified my script to separately report when failure happens for test trained > once and test trained hundred times. Thanks for the linker script. I reproduced your results. I looked at a couple cases. The first was one that failed after 1 training run only (20000910-2.c). It was due to jump threading, which you noted was a problem. For this one I think we can handle it in the partitioning, since there is an FDO insanity that we could probably treat more conservatively when splitting. I looked at one that failed after 100 as well (20031204-1.c). In this case, it was due to expansion which was creating multiple branches/bbs from a logical OR and guessing incorrectly on how to assign the counts: if (octets == 4 && (*cp == ':' || *cp == '\0')) { The (*cp == ':' || *cp == '\0') part looked like the following going into RTL expansion: [20031204-1.c : 31:33] _29 = _28 == 58; [20031204-1.c : 31:33] _30 = _28 == 0; [20031204-1.c : 31:33] _31 = _29 | _30; [20031204-1.c : 31:18] if (_31 != 0) goto <bb 16>; else goto <bb 19>; where the result of the OR was always true, so bb 16 had a count of 100 and bb 19 a count of 0. When it was expanded, the expanded version of the above turned into 2 bbs with a branch in between. Both comparisons were done in the first bb, but the first bb checked whether the result of the *cp == '\0' compare was true, and if not branched to the check for whether the *cp == ':' compare was true. It gave the branch to the second check against ':' a count of 0, so that bb got a count of 0 and was split out, and put the count of 100 on the fall through assuming the compare with '\0' always evaluated to true. In reality, this OR condition was always true because *cp was ':', not '\0'. Therefore, the count of 0 on the second block with the check for ':' was incorrect, we ended up trying to execute it, and failed. Presumably we had the correct profile data for both blocks, but the accuracy was reduced when the OR was represented as a logical computation with a single branch. We could change the expansion code to do something different, e.g. treat as a 50-50 branch. But we would still end up with integer truncation issues when there was a single training run. But that could be dealt with conservatively in the bbpart code as I suggested for the jump threading issue above. I.e. a cold block with incoming non-cold edges conservatively not marked cold for splitting. > > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20000422-1.c > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20000910-2.c > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20020413-1.c > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20030903-1.c > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20031204-1.c > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20031204-1.c > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20060420-1.c > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20060905-1.c > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20120427-1.c > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20120427-2.c > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20120808-1.c > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20121108-1.c > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20121108-1.c > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/920501-6.c > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/920501-6.c > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/920726-1.c > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/981001-1.c > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/981001-1.c > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/990628-1.c > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/991216-2.c > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/991216-2.c > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/cmpdi-1.c > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/cmpdi-1.c > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/float-floor.c > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/float-floor.c > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr33870-1.c > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr33870-1.c > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr33870.c > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr33870.c > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr36093.c > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr37573.c > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr43784.c > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr43784.c > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/switch-1.c > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/switch-1.c > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/va-arg-22.c > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/va-arg-22.c > > FAIL1 is failure after one run, FIAL is failure after 100 train runs. > We should take look at FAILs and see if there are bugs to fix. For FAIL1 > I think it is kind of design problem: while implementing counts&frequencies > the idea was that small counts do not matter, so integer arithmetic is all > right. > > I wonder if with current C++ wonderland we can't simply switch count > to a better representation. Either sreal or fixedpoint with capping > (the integer overflow issues are tiring, too). It also seems like we should be able to detect the profile insanities caused by integer truncation and handle them conservatively. That being said, I see some sreal uses already in the profile.c code, so presumably we could use this for the counts as well if it turns out to be necessary? BTW, Rong also implemented his runtime patch to do the COMDAT profile merging. However, that ended up having some issues, that were solvable but would have caused us to lose all context sensitivity from COMDATS inlined during the profile-gen build. I am going to go back to solving this in the profile-use phase as we discussed in the separate thread on the COMDAT inlining patch I had been working on. Thanks, Teresa > > Honza -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-09-24 8:07 ` Teresa Johnson @ 2013-09-24 13:44 ` Jan Hubicka 2013-09-24 19:06 ` Teresa Johnson 2013-09-26 20:55 ` Rong Xu 2013-09-24 18:28 ` Jan Hubicka 1 sibling, 2 replies; 62+ messages in thread From: Jan Hubicka @ 2013-09-24 13:44 UTC (permalink / raw) To: Teresa Johnson; +Cc: Jan Hubicka, gcc-patches, marxin.liska > Hi Honza, > > I am finally getting back to working on this after a few weeks of > working on some other priorities. I am also trying to return to this, so good timming ;) Martin has got smaller C++ programs (Inkscape) to not touch cold segment during the startup with FDO (w/o partitioning). Firefox still does, I think the problem are lost samples due to different linker decisions even with LTO. (i.e. linker pick an object from .a libraryat profile-generate time that i never passes later.). I plan to look into that today. > > Did you mean to commit the above change? I see that it went in as part > of r202258 but doesn't show up in the ChangeLog entry for that > revision. Yes, I meant to check it in, but did not mean to do so w/o Changelog. I wil fix that. > > > > > In other cases it was mostly loop unrolling in combination with jump threading. So > > I modified my script to separately report when failure happens for test trained > > once and test trained hundred times. > > Thanks for the linker script. I reproduced your results. I looked at a > couple cases. The first was one that failed after 1 training run only > (20000910-2.c). It was due to jump threading, which you noted was a > problem. For this one I think we can handle it in the partitioning, > since there is an FDO insanity that we could probably treat more > conservatively when splitting. We should fix the roundoff issues - when I was introducing the frequency/probability/count system I made an assumptions that parts of programs with very low counts do not matter, since they are not part of hot spot (and I cared only about the hot spot). Now we care about identifying unlikely executed spots and we need to fix this. > > I looked at one that failed after 100 as well (20031204-1.c). In this > case, it was due to expansion which was creating multiple branches/bbs > from a logical OR and guessing incorrectly on how to assign the > counts: > > if (octets == 4 && (*cp == ':' || *cp == '\0')) { > > The (*cp == ':' || *cp == '\0') part looked like the following going > into RTL expansion: > > [20031204-1.c : 31:33] _29 = _28 == 58; > [20031204-1.c : 31:33] _30 = _28 == 0; > [20031204-1.c : 31:33] _31 = _29 | _30; > [20031204-1.c : 31:18] if (_31 != 0) > goto <bb 16>; > else > goto <bb 19>; > > where the result of the OR was always true, so bb 16 had a count of > 100 and bb 19 a count of 0. When it was expanded, the expanded version > of the above turned into 2 bbs with a branch in between. Both > comparisons were done in the first bb, but the first bb checked > whether the result of the *cp == '\0' compare was true, and if not > branched to the check for whether the *cp == ':' compare was true. It > gave the branch to the second check against ':' a count of 0, so that > bb got a count of 0 and was split out, and put the count of 100 on the > fall through assuming the compare with '\0' always evaluated to true. > In reality, this OR condition was always true because *cp was ':', not > '\0'. Therefore, the count of 0 on the second block with the check for > ':' was incorrect, we ended up trying to execute it, and failed. > > Presumably we had the correct profile data for both blocks, but the > accuracy was reduced when the OR was represented as a logical > computation with a single branch. We could change the expansion code > to do something different, e.g. treat as a 50-50 branch. But we would > still end up with integer truncation issues when there was a single > training run. But that could be dealt with conservatively in the > bbpart code as I suggested for the jump threading issue above. I.e. a > cold block with incoming non-cold edges conservatively not marked cold > for splitting. > > > > > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20000422-1.c > > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20000910-2.c > > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20020413-1.c > > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20030903-1.c > > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20031204-1.c > > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20031204-1.c > > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20060420-1.c > > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20060905-1.c > > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20120427-1.c > > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20120427-2.c > > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20120808-1.c > > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20121108-1.c > > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20121108-1.c > > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/920501-6.c > > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/920501-6.c > > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/920726-1.c > > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/981001-1.c > > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/981001-1.c > > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/990628-1.c > > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/991216-2.c > > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/991216-2.c > > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/cmpdi-1.c > > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/cmpdi-1.c > > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/float-floor.c > > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/float-floor.c > > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr33870-1.c > > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr33870-1.c > > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr33870.c > > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr33870.c > > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr36093.c > > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr37573.c > > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr43784.c > > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr43784.c > > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/switch-1.c > > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/switch-1.c > > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/va-arg-22.c > > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/va-arg-22.c > > > > FAIL1 is failure after one run, FIAL is failure after 100 train runs. > > We should take look at FAILs and see if there are bugs to fix. For FAIL1 > > I think it is kind of design problem: while implementing counts&frequencies > > the idea was that small counts do not matter, so integer arithmetic is all > > right. > > > > I wonder if with current C++ wonderland we can't simply switch count > > to a better representation. Either sreal or fixedpoint with capping > > (the integer overflow issues are tiring, too). > > It also seems like we should be able to detect the profile insanities > caused by integer truncation and handle them conservatively. That > being said, I see some sreal uses already in the profile.c code, so > presumably we could use this for the counts as well if it turns out to > be necessary? Yes, I was thinking about this, too. We would need to do some evaulation of compile time implications but switching counts from gcov_type to profile_counter_t that is typedef to sreal seems sane idea. We could switch CFG code first. there should not be many hot spots where counts are involved. We can offline the common calculation we already moved to macros that We will need to invent also REG representation for them. Now we have INT_LIST for that we may have SREAL list and introduce SREAL as valid RTX argument. This can be done incrementally. > > BTW, Rong also implemented his runtime patch to do the COMDAT profile > merging. However, that ended up having some issues, that were solvable > but would have caused us to lose all context sensitivity from COMDATS > inlined during the profile-gen build. I am going to go back to solving > this in the profile-use phase as we discussed in the separate thread > on the COMDAT inlining patch I had been working on. Yes, lets move ahead with this, too. I think i should dig out the change that made frequencies to be guessed again. As for COMDAT merging, i would like to see the patch. I am experimenting now with a patch to also privatize COMDATs during -fprofile-generate to avoid problems with lost profiles mentioned above. As for context sensitivity, one approach would be to have two sets of counters for every comdat - one merged globally and one counting local instances. We can then privatize always and at profile read in stage just clone every comdat and have two instances - one for offline copy and one for inlining. This is not different from how I always wanted to handle GNU extern inlines (that also do have this issue - when you do not inline it, the unit do not see any profile of it). We can just tie the two functions together so "inline" version stay prior inlining and then have linker to redirect to inline version instead of offline version in such cases. It already knows how to skip aliases and this is not terribly different from that. Honza > > Thanks, > Teresa > > > > > Honza > > > > -- > Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-09-24 13:44 ` Jan Hubicka @ 2013-09-24 19:06 ` Teresa Johnson 2013-09-26 20:55 ` Rong Xu 1 sibling, 0 replies; 62+ messages in thread From: Teresa Johnson @ 2013-09-24 19:06 UTC (permalink / raw) To: Jan Hubicka, Rong Xu; +Cc: gcc-patches, marxin.liska Rong - can you answer the questions below on the comdat patch? On Tue, Sep 24, 2013 at 5:31 AM, Jan Hubicka <hubicka@ucw.cz> wrote: >> Hi Honza, >> >> I am finally getting back to working on this after a few weeks of >> working on some other priorities. > > I am also trying to return to this, so good timming ;) > Martin has got smaller C++ programs (Inkscape) to not touch cold segment > during the startup with FDO (w/o partitioning). Firefox still does, I think > the problem are lost samples due to different linker decisions even with LTO. > (i.e. linker pick an object from .a libraryat profile-generate time that i never > passes later.). > > I plan to look into that today. >> >> Did you mean to commit the above change? I see that it went in as part >> of r202258 but doesn't show up in the ChangeLog entry for that >> revision. > > Yes, I meant to check it in, but did not mean to do so w/o Changelog. I wil > fix that. Should the same fix be applied to probably_never_executed_edge_p? >> >> > >> > In other cases it was mostly loop unrolling in combination with jump threading. So >> > I modified my script to separately report when failure happens for test trained >> > once and test trained hundred times. >> >> Thanks for the linker script. I reproduced your results. I looked at a >> couple cases. The first was one that failed after 1 training run only >> (20000910-2.c). It was due to jump threading, which you noted was a >> problem. For this one I think we can handle it in the partitioning, >> since there is an FDO insanity that we could probably treat more >> conservatively when splitting. > > We should fix the roundoff issues - when I was introducing the > frequency/probability/count system I made an assumptions that parts of programs > with very low counts do not matter, since they are not part of hot spot (and I > cared only about the hot spot). Now we care about identifying unlikely > executed spots and we need to fix this. >> >> I looked at one that failed after 100 as well (20031204-1.c). In this >> case, it was due to expansion which was creating multiple branches/bbs >> from a logical OR and guessing incorrectly on how to assign the >> counts: >> >> if (octets == 4 && (*cp == ':' || *cp == '\0')) { >> >> The (*cp == ':' || *cp == '\0') part looked like the following going >> into RTL expansion: >> >> [20031204-1.c : 31:33] _29 = _28 == 58; >> [20031204-1.c : 31:33] _30 = _28 == 0; >> [20031204-1.c : 31:33] _31 = _29 | _30; >> [20031204-1.c : 31:18] if (_31 != 0) >> goto <bb 16>; >> else >> goto <bb 19>; >> >> where the result of the OR was always true, so bb 16 had a count of >> 100 and bb 19 a count of 0. When it was expanded, the expanded version >> of the above turned into 2 bbs with a branch in between. Both >> comparisons were done in the first bb, but the first bb checked >> whether the result of the *cp == '\0' compare was true, and if not >> branched to the check for whether the *cp == ':' compare was true. It >> gave the branch to the second check against ':' a count of 0, so that >> bb got a count of 0 and was split out, and put the count of 100 on the >> fall through assuming the compare with '\0' always evaluated to true. >> In reality, this OR condition was always true because *cp was ':', not >> '\0'. Therefore, the count of 0 on the second block with the check for >> ':' was incorrect, we ended up trying to execute it, and failed. >> >> Presumably we had the correct profile data for both blocks, but the >> accuracy was reduced when the OR was represented as a logical >> computation with a single branch. We could change the expansion code >> to do something different, e.g. treat as a 50-50 branch. But we would >> still end up with integer truncation issues when there was a single >> training run. But that could be dealt with conservatively in the >> bbpart code as I suggested for the jump threading issue above. I.e. a >> cold block with incoming non-cold edges conservatively not marked cold >> for splitting. >> >> > >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20000422-1.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20000910-2.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20020413-1.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20030903-1.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20031204-1.c >> > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20031204-1.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20060420-1.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20060905-1.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20120427-1.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20120427-2.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20120808-1.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20121108-1.c >> > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20121108-1.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/920501-6.c >> > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/920501-6.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/920726-1.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/981001-1.c >> > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/981001-1.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/990628-1.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/991216-2.c >> > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/991216-2.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/cmpdi-1.c >> > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/cmpdi-1.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/float-floor.c >> > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/float-floor.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr33870-1.c >> > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr33870-1.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr33870.c >> > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr33870.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr36093.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr37573.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr43784.c >> > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr43784.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/switch-1.c >> > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/switch-1.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/va-arg-22.c >> > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/va-arg-22.c >> > >> > FAIL1 is failure after one run, FIAL is failure after 100 train runs. >> > We should take look at FAILs and see if there are bugs to fix. For FAIL1 >> > I think it is kind of design problem: while implementing counts&frequencies >> > the idea was that small counts do not matter, so integer arithmetic is all >> > right. >> > >> > I wonder if with current C++ wonderland we can't simply switch count >> > to a better representation. Either sreal or fixedpoint with capping >> > (the integer overflow issues are tiring, too). >> >> It also seems like we should be able to detect the profile insanities >> caused by integer truncation and handle them conservatively. That >> being said, I see some sreal uses already in the profile.c code, so >> presumably we could use this for the counts as well if it turns out to >> be necessary? > > Yes, I was thinking about this, too. We would need to do some evaulation of > compile time implications but switching counts from gcov_type to > profile_counter_t that is typedef to sreal seems sane idea. > > We could switch CFG code first. there should not be many hot spots where > counts are involved. We can offline the common calculation we already moved to > macros that > > We will need to invent also REG representation for them. Now we have INT_LIST > for that we may have SREAL list and introduce SREAL as valid RTX argument. > This can be done incrementally. >> >> BTW, Rong also implemented his runtime patch to do the COMDAT profile >> merging. However, that ended up having some issues, that were solvable >> but would have caused us to lose all context sensitivity from COMDATS >> inlined during the profile-gen build. I am going to go back to solving >> this in the profile-use phase as we discussed in the separate thread >> on the COMDAT inlining patch I had been working on. > > Yes, lets move ahead with this, too. I think i should dig out the change > that made frequencies to be guessed again. I think I have that and was building my patch on top of it. Rong: > > As for COMDAT merging, i would like to see the patch. I am experimenting > now with a patch to also privatize COMDATs during -fprofile-generate to > avoid problems with lost profiles mentioned above. > > As for context sensitivity, one approach would be to have two sets of > counters for every comdat - one merged globally and one counting local > instances. We can then privatize always and at profile read in stage > just clone every comdat and have two instances - one for offline copy > and one for inlining. > > This is not different from how I always wanted to handle GNU extern inlines > (that also do have this issue - when you do not inline it, the unit do not see > any profile of it). > > We can just tie the two functions together so "inline" version stay prior > inlining and then have linker to redirect to inline version instead of offline > version in such cases. It already knows how to skip aliases and this is not > terribly different from that. > > Honza >> >> Thanks, >> Teresa >> >> > >> > Honza >> >> >> >> -- >> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-09-24 13:44 ` Jan Hubicka 2013-09-24 19:06 ` Teresa Johnson @ 2013-09-26 20:55 ` Rong Xu 2013-09-26 22:23 ` Jan Hubicka 1 sibling, 1 reply; 62+ messages in thread From: Rong Xu @ 2013-09-26 20:55 UTC (permalink / raw) To: Jan Hubicka; +Cc: Teresa Johnson, gcc-patches, marxin.liska On Tue, Sep 24, 2013 at 5:31 AM, Jan Hubicka <hubicka@ucw.cz> wrote: >> Hi Honza, >> >> I am finally getting back to working on this after a few weeks of >> working on some other priorities. > > I am also trying to return to this, so good timming ;) > Martin has got smaller C++ programs (Inkscape) to not touch cold segment > during the startup with FDO (w/o partitioning). Firefox still does, I think > the problem are lost samples due to different linker decisions even with LTO. > (i.e. linker pick an object from .a libraryat profile-generate time that i never > passes later.). > > I plan to look into that today. >> >> Did you mean to commit the above change? I see that it went in as part >> of r202258 but doesn't show up in the ChangeLog entry for that >> revision. > > Yes, I meant to check it in, but did not mean to do so w/o Changelog. I wil > fix that. >> >> > >> > In other cases it was mostly loop unrolling in combination with jump threading. So >> > I modified my script to separately report when failure happens for test trained >> > once and test trained hundred times. >> >> Thanks for the linker script. I reproduced your results. I looked at a >> couple cases. The first was one that failed after 1 training run only >> (20000910-2.c). It was due to jump threading, which you noted was a >> problem. For this one I think we can handle it in the partitioning, >> since there is an FDO insanity that we could probably treat more >> conservatively when splitting. > > We should fix the roundoff issues - when I was introducing the > frequency/probability/count system I made an assumptions that parts of programs > with very low counts do not matter, since they are not part of hot spot (and I > cared only about the hot spot). Now we care about identifying unlikely > executed spots and we need to fix this. >> >> I looked at one that failed after 100 as well (20031204-1.c). In this >> case, it was due to expansion which was creating multiple branches/bbs >> from a logical OR and guessing incorrectly on how to assign the >> counts: >> >> if (octets == 4 && (*cp == ':' || *cp == '\0')) { >> >> The (*cp == ':' || *cp == '\0') part looked like the following going >> into RTL expansion: >> >> [20031204-1.c : 31:33] _29 = _28 == 58; >> [20031204-1.c : 31:33] _30 = _28 == 0; >> [20031204-1.c : 31:33] _31 = _29 | _30; >> [20031204-1.c : 31:18] if (_31 != 0) >> goto <bb 16>; >> else >> goto <bb 19>; >> >> where the result of the OR was always true, so bb 16 had a count of >> 100 and bb 19 a count of 0. When it was expanded, the expanded version >> of the above turned into 2 bbs with a branch in between. Both >> comparisons were done in the first bb, but the first bb checked >> whether the result of the *cp == '\0' compare was true, and if not >> branched to the check for whether the *cp == ':' compare was true. It >> gave the branch to the second check against ':' a count of 0, so that >> bb got a count of 0 and was split out, and put the count of 100 on the >> fall through assuming the compare with '\0' always evaluated to true. >> In reality, this OR condition was always true because *cp was ':', not >> '\0'. Therefore, the count of 0 on the second block with the check for >> ':' was incorrect, we ended up trying to execute it, and failed. >> >> Presumably we had the correct profile data for both blocks, but the >> accuracy was reduced when the OR was represented as a logical >> computation with a single branch. We could change the expansion code >> to do something different, e.g. treat as a 50-50 branch. But we would >> still end up with integer truncation issues when there was a single >> training run. But that could be dealt with conservatively in the >> bbpart code as I suggested for the jump threading issue above. I.e. a >> cold block with incoming non-cold edges conservatively not marked cold >> for splitting. >> >> > >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20000422-1.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20000910-2.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20020413-1.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20030903-1.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20031204-1.c >> > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20031204-1.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20060420-1.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20060905-1.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20120427-1.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20120427-2.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20120808-1.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20121108-1.c >> > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/20121108-1.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/920501-6.c >> > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/920501-6.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/920726-1.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/981001-1.c >> > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/981001-1.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/990628-1.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/991216-2.c >> > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/991216-2.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/cmpdi-1.c >> > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/cmpdi-1.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/float-floor.c >> > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/float-floor.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr33870-1.c >> > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr33870-1.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr33870.c >> > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr33870.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr36093.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr37573.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr43784.c >> > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/pr43784.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/switch-1.c >> > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/switch-1.c >> > FAIL1 /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/va-arg-22.c >> > FAIL /home/jh/trunk/gcc/testsuite/gcc.c-torture/execute/va-arg-22.c >> > >> > FAIL1 is failure after one run, FIAL is failure after 100 train runs. >> > We should take look at FAILs and see if there are bugs to fix. For FAIL1 >> > I think it is kind of design problem: while implementing counts&frequencies >> > the idea was that small counts do not matter, so integer arithmetic is all >> > right. >> > >> > I wonder if with current C++ wonderland we can't simply switch count >> > to a better representation. Either sreal or fixedpoint with capping >> > (the integer overflow issues are tiring, too). >> >> It also seems like we should be able to detect the profile insanities >> caused by integer truncation and handle them conservatively. That >> being said, I see some sreal uses already in the profile.c code, so >> presumably we could use this for the counts as well if it turns out to >> be necessary? > > Yes, I was thinking about this, too. We would need to do some evaulation of > compile time implications but switching counts from gcov_type to > profile_counter_t that is typedef to sreal seems sane idea. > > We could switch CFG code first. there should not be many hot spots where > counts are involved. We can offline the common calculation we already moved to > macros that > > We will need to invent also REG representation for them. Now we have INT_LIST > for that we may have SREAL list and introduce SREAL as valid RTX argument. > This can be done incrementally. >> >> BTW, Rong also implemented his runtime patch to do the COMDAT profile >> merging. However, that ended up having some issues, that were solvable >> but would have caused us to lose all context sensitivity from COMDATS >> inlined during the profile-gen build. I am going to go back to solving >> this in the profile-use phase as we discussed in the separate thread >> on the COMDAT inlining patch I had been working on. > > Yes, lets move ahead with this, too. I think i should dig out the change > that made frequencies to be guessed again. > > As for COMDAT merging, i would like to see the patch. I am experimenting > now with a patch to also privatize COMDATs during -fprofile-generate to > avoid problems with lost profiles mentioned above. > Do you mean you privatize every COMDAT function in the profile-generate? We discussed this idea internally and we thought it would not work for large applications (like in google) due to size. > As for context sensitivity, one approach would be to have two sets of > counters for every comdat - one merged globally and one counting local > instances. We can then privatize always and at profile read in stage > just clone every comdat and have two instances - one for offline copy > and one for inlining. > In my implementation, I also allow multiple sets of COMDAT profile co-existing in one compilation. Due to the auxiliary modules in LIPO, I actually have more than two. But I'm wondering how do you determine which profile to use for each call-site -- the inline decision may not be the same for profile-generate and profile-use compilation. > This is not different from how I always wanted to handle GNU extern inlines > (that also do have this issue - when you do not inline it, the unit do not see > any profile of it). > > We can just tie the two functions together so "inline" version stay prior > inlining and then have linker to redirect to inline version instead of offline > version in such cases. It already knows how to skip aliases and this is not > terribly different from that. > > Honza >> >> Thanks, >> Teresa >> >> > >> > Honza >> >> >> >> -- >> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-09-26 20:55 ` Rong Xu @ 2013-09-26 22:23 ` Jan Hubicka 2013-09-26 22:54 ` Rong Xu 0 siblings, 1 reply; 62+ messages in thread From: Jan Hubicka @ 2013-09-26 22:23 UTC (permalink / raw) To: Rong Xu; +Cc: Jan Hubicka, Teresa Johnson, gcc-patches, marxin.liska > > As for COMDAT merging, i would like to see the patch. I am experimenting > > now with a patch to also privatize COMDATs during -fprofile-generate to > > avoid problems with lost profiles mentioned above. > > > > Do you mean you privatize every COMDAT function in the profile-generate? > We discussed this idea internally and we thought it would not work for > large applications (like in google) due to size. Yes, Martin and I plan to test this on firefox. In a way you already have all the COMDAT functions unshared in the object files, so the resulting binary should not be completely off the limits. But I do not have any quantitative data, yet, since we hit bug in constant folding and devirtualization I fixed in meantime but we did not re-run the tests yet. > > > As for context sensitivity, one approach would be to have two sets of > > counters for every comdat - one merged globally and one counting local > > instances. We can then privatize always and at profile read in stage > > just clone every comdat and have two instances - one for offline copy > > and one for inlining. > > > > In my implementation, I also allow multiple sets of COMDAT profile > co-existing in one compilation. > Due to the auxiliary modules in LIPO, I actually have more than two. How does auxiliary modules work? > > But I'm wondering how do you determine which profile to use for each > call-site -- the inline decision may not > be the same for profile-generate and profile-use compilation. My suggestion was to simply use the module local profile for all inline sites within the given module and the global profile for the offline copy of the function (that one will, in the case it survives linking, be shared across all the modules anyway). I think this may work in the cases where i.e. use of hash templates in one module is very different (in average size) from other module. I did not really put much effort into it - I currently worry primarily about the cases where profile is lost completely since it gets attached to a function not surviving final linking (or because we inline something we did not inlined at profile time). As for context sensitivity, we may try to consider developing more consistent solution for this. COMDAT functions are definitely not only that may exhibit context sensitive behaviour. One approach would be to always have multiple counters for each function and hash based on cbacktraces collected by indirect call profiling instrumentation. In a way this is same path profiling, but that would definitely add quite some overhead + we will need to think of resonable way to represent this within compiler. How do you decide what functions you want to have multiple profiles for? Honza ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-09-26 22:23 ` Jan Hubicka @ 2013-09-26 22:54 ` Rong Xu 0 siblings, 0 replies; 62+ messages in thread From: Rong Xu @ 2013-09-26 22:54 UTC (permalink / raw) To: Jan Hubicka; +Cc: Teresa Johnson, gcc-patches, marxin.liska On Thu, Sep 26, 2013 at 2:54 PM, Jan Hubicka <hubicka@ucw.cz> wrote: >> > As for COMDAT merging, i would like to see the patch. I am experimenting >> > now with a patch to also privatize COMDATs during -fprofile-generate to >> > avoid problems with lost profiles mentioned above. >> > >> >> Do you mean you privatize every COMDAT function in the profile-generate? >> We discussed this idea internally and we thought it would not work for >> large applications (like in google) due to size. > > Yes, Martin and I plan to test this on firefox. In a way you already have all > the COMDAT functions unshared in the object files, so the resulting binary > should not be completely off the limits. But I do not have any quantitative > data, yet, since we hit bug in constant folding and devirtualization I fixed in > meantime but we did not re-run the tests yet. LInker removes a great numbers of duplicated copies, esp for those template functions. We don't have a quantitative numbers either. But I'll collect some soon. > >> >> > As for context sensitivity, one approach would be to have two sets of >> > counters for every comdat - one merged globally and one counting local >> > instances. We can then privatize always and at profile read in stage >> > just clone every comdat and have two instances - one for offline copy >> > and one for inlining. >> > >> >> In my implementation, I also allow multiple sets of COMDAT profile >> co-existing in one compilation. >> Due to the auxiliary modules in LIPO, I actually have more than two. > > How does auxiliary modules work? It pulls in multiple profiles from other compilation. So there might be multiple inlined profiles. >> >> But I'm wondering how do you determine which profile to use for each >> call-site -- the inline decision may not >> be the same for profile-generate and profile-use compilation. > > My suggestion was to simply use the module local profile for all inline sites > within the given module and the global profile for the offline copy of the > function (that one will, in the case it survives linking, be shared across > all the modules anyway). For simple example like: callsite1 --> comcat_function_foo callsite2 --> comdat_function_foo callsite1 is inlined in profile-generate, it has its own inlined profile counter. callsite2 is not inlined and the profile goes to the offline copies. let's callsite 1 is cold (0 counter) and callsite 2 is hot. Using local profile (the cold one) for callsite2 will not be correct. > > I think this may work in the cases where i.e. use of hash templates in one > module is very different (in average size) from other module. > I did not really put much effort into it - I currently worry primarily about > the cases where profile is lost completely since it gets attached to a function > not surviving final linking (or because we inline something we did not inlined > at profile time). > > As for context sensitivity, we may try to consider developing more consistent > solution for this. COMDAT functions are definitely not only that may exhibit > context sensitive behaviour. > One approach would be to always have multiple counters for each function and > hash based on cbacktraces collected by indirect call profiling instrumentation. > In a way this is same path profiling, but that would definitely add quite some > overhead + we will need to think of resonable way to represent this within > compiler. > > How do you decide what functions you want to have multiple profiles for? I do the instrumentation after ipa-inline for comdat function. I know if a callsite is inlined or not. In profile-use phrase, I also need to provide to the context (which module this is from) to pick the right profile. > > Honza ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-09-24 8:07 ` Teresa Johnson 2013-09-24 13:44 ` Jan Hubicka @ 2013-09-24 18:28 ` Jan Hubicka 2013-09-24 18:51 ` Teresa Johnson 1 sibling, 1 reply; 62+ messages in thread From: Jan Hubicka @ 2013-09-24 18:28 UTC (permalink / raw) To: Teresa Johnson; +Cc: Jan Hubicka, gcc-patches, marxin.liska > > I looked at one that failed after 100 as well (20031204-1.c). In this > case, it was due to expansion which was creating multiple branches/bbs > from a logical OR and guessing incorrectly on how to assign the > counts: > > if (octets == 4 && (*cp == ':' || *cp == '\0')) { > > The (*cp == ':' || *cp == '\0') part looked like the following going > into RTL expansion: > > [20031204-1.c : 31:33] _29 = _28 == 58; > [20031204-1.c : 31:33] _30 = _28 == 0; > [20031204-1.c : 31:33] _31 = _29 | _30; > [20031204-1.c : 31:18] if (_31 != 0) > goto <bb 16>; > else > goto <bb 19>; > > where the result of the OR was always true, so bb 16 had a count of > 100 and bb 19 a count of 0. When it was expanded, the expanded version > of the above turned into 2 bbs with a branch in between. Both > comparisons were done in the first bb, but the first bb checked > whether the result of the *cp == '\0' compare was true, and if not > branched to the check for whether the *cp == ':' compare was true. It > gave the branch to the second check against ':' a count of 0, so that > bb got a count of 0 and was split out, and put the count of 100 on the > fall through assuming the compare with '\0' always evaluated to true. > In reality, this OR condition was always true because *cp was ':', not > '\0'. Therefore, the count of 0 on the second block with the check for > ':' was incorrect, we ended up trying to execute it, and failed. I see, we produce: ;; if (_26 != 0) (insn 94 93 95 (set (reg:CCZ 17 flags) (compare:CCZ (reg:QI 107 [ D.2184 ]) (const_int 0 [0]))) a.c:31 -1 (nil)) (insn 95 94 96 (set (reg:QI 122 [ D.2186 ]) (eq:QI (reg:CCZ 17 flags) (const_int 0 [0]))) a.c:31 -1 (nil)) (insn 96 95 97 (set (reg:CCZ 17 flags) (compare:CCZ (reg:QI 122 [ D.2186 ]) (const_int 0 [0]))) a.c:31 -1 (nil)) (jump_insn 97 96 98 (set (pc) (if_then_else (ne (reg:CCZ 17 flags) (const_int 0 [0])) (label_ref 100) (pc))) a.c:31 -1 (expr_list:REG_BR_PROB (const_int 6100 [0x17d4]) (nil))) (insn 98 97 99 (set (reg:CCZ 17 flags) (compare:CCZ (reg:QI 108 [ D.2186 ]) (const_int 0 [0]))) a.c:31 -1 (nil)) (jump_insn 99 98 100 (set (pc) (if_then_else (eq (reg:CCZ 17 flags) (const_int 0 [0])) (label_ref 0) (pc))) a.c:31 -1 (expr_list:REG_BR_PROB (const_int 3900 [0xf3c]) (nil))) (code_label 100 99 0 14 "" [0 uses]) That is because we TER together "_26 = _25 | _24" and "if (_26 != 0)" First I think the logic of do_jump should really be moved to trees. It is not doing things that can not be adequately represented by gimple. I am not that certain we want to move it before profiling though. > > Presumably we had the correct profile data for both blocks, but the > accuracy was reduced when the OR was represented as a logical > computation with a single branch. We could change the expansion code > to do something different, e.g. treat as a 50-50 branch. But we would > still end up with integer truncation issues when there was a single > training run. But that could be dealt with conservatively in the Yep, but it is still better than what we have now - if the test above was in hot part of program (i.e. not executed once), we will end up optimizing the second conditional for size. So I think it is do_jump bug to not distribute probabilities across the two conditoinals introduced. > bbpart code as I suggested for the jump threading issue above. I.e. a > cold block with incoming non-cold edges conservatively not marked cold > for splitting. Yep, we can probably do that, but we ought to fix the individual cases above at least for resonable number of runs. Will you look into logic of do_jump or shall I try to dive in? Honza ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-09-24 18:28 ` Jan Hubicka @ 2013-09-24 18:51 ` Teresa Johnson 2013-09-25 23:10 ` Teresa Johnson ` (2 more replies) 0 siblings, 3 replies; 62+ messages in thread From: Teresa Johnson @ 2013-09-24 18:51 UTC (permalink / raw) To: Jan Hubicka; +Cc: gcc-patches, marxin.liska On Tue, Sep 24, 2013 at 10:57 AM, Jan Hubicka <hubicka@ucw.cz> wrote: >> >> I looked at one that failed after 100 as well (20031204-1.c). In this >> case, it was due to expansion which was creating multiple branches/bbs >> from a logical OR and guessing incorrectly on how to assign the >> counts: >> >> if (octets == 4 && (*cp == ':' || *cp == '\0')) { >> >> The (*cp == ':' || *cp == '\0') part looked like the following going >> into RTL expansion: >> >> [20031204-1.c : 31:33] _29 = _28 == 58; >> [20031204-1.c : 31:33] _30 = _28 == 0; >> [20031204-1.c : 31:33] _31 = _29 | _30; >> [20031204-1.c : 31:18] if (_31 != 0) >> goto <bb 16>; >> else >> goto <bb 19>; >> >> where the result of the OR was always true, so bb 16 had a count of >> 100 and bb 19 a count of 0. When it was expanded, the expanded version >> of the above turned into 2 bbs with a branch in between. Both >> comparisons were done in the first bb, but the first bb checked >> whether the result of the *cp == '\0' compare was true, and if not >> branched to the check for whether the *cp == ':' compare was true. It >> gave the branch to the second check against ':' a count of 0, so that >> bb got a count of 0 and was split out, and put the count of 100 on the >> fall through assuming the compare with '\0' always evaluated to true. >> In reality, this OR condition was always true because *cp was ':', not >> '\0'. Therefore, the count of 0 on the second block with the check for >> ':' was incorrect, we ended up trying to execute it, and failed. > > I see, we produce: > ;; if (_26 != 0) > > (insn 94 93 95 (set (reg:CCZ 17 flags) > (compare:CCZ (reg:QI 107 [ D.2184 ]) > (const_int 0 [0]))) a.c:31 -1 > (nil)) > > (insn 95 94 96 (set (reg:QI 122 [ D.2186 ]) > (eq:QI (reg:CCZ 17 flags) > (const_int 0 [0]))) a.c:31 -1 > (nil)) > > (insn 96 95 97 (set (reg:CCZ 17 flags) > (compare:CCZ (reg:QI 122 [ D.2186 ]) > (const_int 0 [0]))) a.c:31 -1 > (nil)) > > (jump_insn 97 96 98 (set (pc) > (if_then_else (ne (reg:CCZ 17 flags) > (const_int 0 [0])) > (label_ref 100) > (pc))) a.c:31 -1 > (expr_list:REG_BR_PROB (const_int 6100 [0x17d4]) > (nil))) > > (insn 98 97 99 (set (reg:CCZ 17 flags) > (compare:CCZ (reg:QI 108 [ D.2186 ]) > (const_int 0 [0]))) a.c:31 -1 > (nil)) > > (jump_insn 99 98 100 (set (pc) > (if_then_else (eq (reg:CCZ 17 flags) > (const_int 0 [0])) > (label_ref 0) > (pc))) a.c:31 -1 > (expr_list:REG_BR_PROB (const_int 3900 [0xf3c]) > (nil))) > > (code_label 100 99 0 14 "" [0 uses]) > > That is because we TER together "_26 = _25 | _24" and "if (_26 != 0)" > > First I think the logic of do_jump should really be moved to trees. It is not > doing things that can not be adequately represented by gimple. > > I am not that certain we want to move it before profiling though. >> >> Presumably we had the correct profile data for both blocks, but the >> accuracy was reduced when the OR was represented as a logical >> computation with a single branch. We could change the expansion code >> to do something different, e.g. treat as a 50-50 branch. But we would >> still end up with integer truncation issues when there was a single >> training run. But that could be dealt with conservatively in the > > Yep, but it is still better than what we have now - if the test above was > in hot part of program (i.e. not executed once), we will end up optimizing > the second conditional for size. > > So I think it is do_jump bug to not distribute probabilities across the two > conditoinals introduced. >> bbpart code as I suggested for the jump threading issue above. I.e. a >> cold block with incoming non-cold edges conservatively not marked cold >> for splitting. > > Yep, we can probably do that, but we ought to fix the individual cases > above at least for resonable number of runs. I made this change and it removed a few of the failures. I looked at another case that still failed with 1 train run but passed with 100. It turned out to be another truncation issue exposed by RTL expansion, where we created some control flow for a memset builtin which was in a block with an execution count of 1. Some of the blocks got frequencies less than half the original block, so the count was rounded down or truncated to 0. I noticed that in this case (as well as the jump threading case I fixed by looking for non-zero incoming edges in partitioning) that the bb frequency was non-zero. Why not just have probably_never_executed_bb_p return simply return false bb->frequency is non-zero (right now it does the opposite - returns true when bb->frequency is 0)? Making this change removed a bunch of other failures. With this change as well, there are only 3 cases that still fail with 1 train run that pass with 100. Need to look at those. > > Will you look into logic of do_jump or shall I try to dive in? I can take a look, but probably won't have a chance until late this week. If you don't get to it before then I will see if I can figure out why it is applying the branch probabilities this way. Teresa > > Honza -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-09-24 18:51 ` Teresa Johnson @ 2013-09-25 23:10 ` Teresa Johnson 2013-09-26 8:44 ` Teresa Johnson 2013-09-26 22:26 ` Jan Hubicka 2013-10-01 17:36 ` Teresa Johnson 2 siblings, 1 reply; 62+ messages in thread From: Teresa Johnson @ 2013-09-25 23:10 UTC (permalink / raw) To: Jan Hubicka; +Cc: gcc-patches, marxin.liska On Tue, Sep 24, 2013 at 11:25 AM, Teresa Johnson <tejohnson@google.com> wrote: > On Tue, Sep 24, 2013 at 10:57 AM, Jan Hubicka <hubicka@ucw.cz> wrote: >>> >>> I looked at one that failed after 100 as well (20031204-1.c). In this >>> case, it was due to expansion which was creating multiple branches/bbs >>> from a logical OR and guessing incorrectly on how to assign the >>> counts: >>> >>> if (octets == 4 && (*cp == ':' || *cp == '\0')) { >>> >>> The (*cp == ':' || *cp == '\0') part looked like the following going >>> into RTL expansion: >>> >>> [20031204-1.c : 31:33] _29 = _28 == 58; >>> [20031204-1.c : 31:33] _30 = _28 == 0; >>> [20031204-1.c : 31:33] _31 = _29 | _30; >>> [20031204-1.c : 31:18] if (_31 != 0) >>> goto <bb 16>; >>> else >>> goto <bb 19>; >>> >>> where the result of the OR was always true, so bb 16 had a count of >>> 100 and bb 19 a count of 0. When it was expanded, the expanded version >>> of the above turned into 2 bbs with a branch in between. Both >>> comparisons were done in the first bb, but the first bb checked >>> whether the result of the *cp == '\0' compare was true, and if not >>> branched to the check for whether the *cp == ':' compare was true. It >>> gave the branch to the second check against ':' a count of 0, so that >>> bb got a count of 0 and was split out, and put the count of 100 on the >>> fall through assuming the compare with '\0' always evaluated to true. >>> In reality, this OR condition was always true because *cp was ':', not >>> '\0'. Therefore, the count of 0 on the second block with the check for >>> ':' was incorrect, we ended up trying to execute it, and failed. >> >> I see, we produce: >> ;; if (_26 != 0) >> >> (insn 94 93 95 (set (reg:CCZ 17 flags) >> (compare:CCZ (reg:QI 107 [ D.2184 ]) >> (const_int 0 [0]))) a.c:31 -1 >> (nil)) >> >> (insn 95 94 96 (set (reg:QI 122 [ D.2186 ]) >> (eq:QI (reg:CCZ 17 flags) >> (const_int 0 [0]))) a.c:31 -1 >> (nil)) >> >> (insn 96 95 97 (set (reg:CCZ 17 flags) >> (compare:CCZ (reg:QI 122 [ D.2186 ]) >> (const_int 0 [0]))) a.c:31 -1 >> (nil)) >> >> (jump_insn 97 96 98 (set (pc) >> (if_then_else (ne (reg:CCZ 17 flags) >> (const_int 0 [0])) >> (label_ref 100) >> (pc))) a.c:31 -1 >> (expr_list:REG_BR_PROB (const_int 6100 [0x17d4]) >> (nil))) >> >> (insn 98 97 99 (set (reg:CCZ 17 flags) >> (compare:CCZ (reg:QI 108 [ D.2186 ]) >> (const_int 0 [0]))) a.c:31 -1 >> (nil)) >> >> (jump_insn 99 98 100 (set (pc) >> (if_then_else (eq (reg:CCZ 17 flags) >> (const_int 0 [0])) >> (label_ref 0) >> (pc))) a.c:31 -1 >> (expr_list:REG_BR_PROB (const_int 3900 [0xf3c]) >> (nil))) >> >> (code_label 100 99 0 14 "" [0 uses]) >> >> That is because we TER together "_26 = _25 | _24" and "if (_26 != 0)" >> >> First I think the logic of do_jump should really be moved to trees. It is not >> doing things that can not be adequately represented by gimple. >> >> I am not that certain we want to move it before profiling though. >>> >>> Presumably we had the correct profile data for both blocks, but the >>> accuracy was reduced when the OR was represented as a logical >>> computation with a single branch. We could change the expansion code >>> to do something different, e.g. treat as a 50-50 branch. But we would >>> still end up with integer truncation issues when there was a single >>> training run. But that could be dealt with conservatively in the >> >> Yep, but it is still better than what we have now - if the test above was >> in hot part of program (i.e. not executed once), we will end up optimizing >> the second conditional for size. >> >> So I think it is do_jump bug to not distribute probabilities across the two >> conditoinals introduced. >>> bbpart code as I suggested for the jump threading issue above. I.e. a >>> cold block with incoming non-cold edges conservatively not marked cold >>> for splitting. >> >> Yep, we can probably do that, but we ought to fix the individual cases >> above at least for resonable number of runs. > > I made this change and it removed a few of the failures. > > I looked at another case that still failed with 1 train run but passed > with 100. It turned out to be another truncation issue exposed by RTL > expansion, where we created some control flow for a memset builtin > which was in a block with an execution count of 1. Some of the blocks > got frequencies less than half the original block, so the count was > rounded down or truncated to 0. I noticed that in this case (as well > as the jump threading case I fixed by looking for non-zero incoming > edges in partitioning) that the bb frequency was non-zero. > > Why not just have probably_never_executed_bb_p return simply return > false bb->frequency is non-zero (right now it does the opposite - > returns true when bb->frequency is 0)? Making this change removed a > bunch of other failures. With this change as well, there are only 3 > cases that still fail with 1 train run that pass with 100. Need to > look at those. FYI, These turned out to be more jump threading issues. I am currently working on getting jump threading profile updates to work properly, I think I'm pretty close. Haven't had a chance to look at do_jump yet. Teresa > >> >> Will you look into logic of do_jump or shall I try to dive in? > > I can take a look, but probably won't have a chance until late this > week. If you don't get to it before then I will see if I can figure > out why it is applying the branch probabilities this way. > > Teresa > >> >> Honza > > > > -- > Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-09-25 23:10 ` Teresa Johnson @ 2013-09-26 8:44 ` Teresa Johnson 0 siblings, 0 replies; 62+ messages in thread From: Teresa Johnson @ 2013-09-26 8:44 UTC (permalink / raw) To: Jan Hubicka; +Cc: gcc-patches, marxin.liska On Wed, Sep 25, 2013 at 2:33 PM, Teresa Johnson <tejohnson@google.com> wrote: > On Tue, Sep 24, 2013 at 11:25 AM, Teresa Johnson <tejohnson@google.com> wrote: >> On Tue, Sep 24, 2013 at 10:57 AM, Jan Hubicka <hubicka@ucw.cz> wrote: >>>> >>>> I looked at one that failed after 100 as well (20031204-1.c). In this >>>> case, it was due to expansion which was creating multiple branches/bbs >>>> from a logical OR and guessing incorrectly on how to assign the >>>> counts: >>>> >>>> if (octets == 4 && (*cp == ':' || *cp == '\0')) { >>>> >>>> The (*cp == ':' || *cp == '\0') part looked like the following going >>>> into RTL expansion: >>>> >>>> [20031204-1.c : 31:33] _29 = _28 == 58; >>>> [20031204-1.c : 31:33] _30 = _28 == 0; >>>> [20031204-1.c : 31:33] _31 = _29 | _30; >>>> [20031204-1.c : 31:18] if (_31 != 0) >>>> goto <bb 16>; >>>> else >>>> goto <bb 19>; >>>> >>>> where the result of the OR was always true, so bb 16 had a count of >>>> 100 and bb 19 a count of 0. When it was expanded, the expanded version >>>> of the above turned into 2 bbs with a branch in between. Both >>>> comparisons were done in the first bb, but the first bb checked >>>> whether the result of the *cp == '\0' compare was true, and if not >>>> branched to the check for whether the *cp == ':' compare was true. It >>>> gave the branch to the second check against ':' a count of 0, so that >>>> bb got a count of 0 and was split out, and put the count of 100 on the >>>> fall through assuming the compare with '\0' always evaluated to true. >>>> In reality, this OR condition was always true because *cp was ':', not >>>> '\0'. Therefore, the count of 0 on the second block with the check for >>>> ':' was incorrect, we ended up trying to execute it, and failed. >>> >>> I see, we produce: >>> ;; if (_26 != 0) >>> >>> (insn 94 93 95 (set (reg:CCZ 17 flags) >>> (compare:CCZ (reg:QI 107 [ D.2184 ]) >>> (const_int 0 [0]))) a.c:31 -1 >>> (nil)) >>> >>> (insn 95 94 96 (set (reg:QI 122 [ D.2186 ]) >>> (eq:QI (reg:CCZ 17 flags) >>> (const_int 0 [0]))) a.c:31 -1 >>> (nil)) >>> >>> (insn 96 95 97 (set (reg:CCZ 17 flags) >>> (compare:CCZ (reg:QI 122 [ D.2186 ]) >>> (const_int 0 [0]))) a.c:31 -1 >>> (nil)) >>> >>> (jump_insn 97 96 98 (set (pc) >>> (if_then_else (ne (reg:CCZ 17 flags) >>> (const_int 0 [0])) >>> (label_ref 100) >>> (pc))) a.c:31 -1 >>> (expr_list:REG_BR_PROB (const_int 6100 [0x17d4]) >>> (nil))) >>> >>> (insn 98 97 99 (set (reg:CCZ 17 flags) >>> (compare:CCZ (reg:QI 108 [ D.2186 ]) >>> (const_int 0 [0]))) a.c:31 -1 >>> (nil)) >>> >>> (jump_insn 99 98 100 (set (pc) >>> (if_then_else (eq (reg:CCZ 17 flags) >>> (const_int 0 [0])) >>> (label_ref 0) >>> (pc))) a.c:31 -1 >>> (expr_list:REG_BR_PROB (const_int 3900 [0xf3c]) >>> (nil))) >>> >>> (code_label 100 99 0 14 "" [0 uses]) >>> >>> That is because we TER together "_26 = _25 | _24" and "if (_26 != 0)" >>> >>> First I think the logic of do_jump should really be moved to trees. It is not >>> doing things that can not be adequately represented by gimple. >>> >>> I am not that certain we want to move it before profiling though. >>>> >>>> Presumably we had the correct profile data for both blocks, but the >>>> accuracy was reduced when the OR was represented as a logical >>>> computation with a single branch. We could change the expansion code >>>> to do something different, e.g. treat as a 50-50 branch. But we would >>>> still end up with integer truncation issues when there was a single >>>> training run. But that could be dealt with conservatively in the >>> >>> Yep, but it is still better than what we have now - if the test above was >>> in hot part of program (i.e. not executed once), we will end up optimizing >>> the second conditional for size. >>> >>> So I think it is do_jump bug to not distribute probabilities across the two >>> conditoinals introduced. >>>> bbpart code as I suggested for the jump threading issue above. I.e. a >>>> cold block with incoming non-cold edges conservatively not marked cold >>>> for splitting. >>> >>> Yep, we can probably do that, but we ought to fix the individual cases >>> above at least for resonable number of runs. >> >> I made this change and it removed a few of the failures. >> >> I looked at another case that still failed with 1 train run but passed >> with 100. It turned out to be another truncation issue exposed by RTL >> expansion, where we created some control flow for a memset builtin >> which was in a block with an execution count of 1. Some of the blocks >> got frequencies less than half the original block, so the count was >> rounded down or truncated to 0. I noticed that in this case (as well >> as the jump threading case I fixed by looking for non-zero incoming >> edges in partitioning) that the bb frequency was non-zero. >> >> Why not just have probably_never_executed_bb_p return simply return >> false bb->frequency is non-zero (right now it does the opposite - >> returns true when bb->frequency is 0)? Making this change removed a >> bunch of other failures. With this change as well, there are only 3 >> cases that still fail with 1 train run that pass with 100. Need to >> look at those. > > FYI, These turned out to be more jump threading issues. I am currently > working on getting jump threading profile updates to work properly, I > think I'm pretty close. Haven't had a chance to look at do_jump yet. Correction, it was not the 3 tests I mentioned above, but a different set of tests being affected by this jump threading issue. I have a patch I am regression testing. But there are other profile insanities being caused upstream of jump threading that I haven't tracked down. So I will also test and send the patch to handle some of these in the splitting/cold cold detection code. It also contains the change to probably_never_executed_bb_p that I mention above to return false when bb->frequency is non-zero. Teresa > > Teresa > >> >>> >>> Will you look into logic of do_jump or shall I try to dive in? >> >> I can take a look, but probably won't have a chance until late this >> week. If you don't get to it before then I will see if I can figure >> out why it is applying the branch probabilities this way. >> >> Teresa >> >>> >>> Honza >> >> >> >> -- >> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 > > > > -- > Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-09-24 18:51 ` Teresa Johnson 2013-09-25 23:10 ` Teresa Johnson @ 2013-09-26 22:26 ` Jan Hubicka 2013-09-27 14:50 ` Teresa Johnson 2013-10-01 17:36 ` Teresa Johnson 2 siblings, 1 reply; 62+ messages in thread From: Jan Hubicka @ 2013-09-26 22:26 UTC (permalink / raw) To: Teresa Johnson; +Cc: Jan Hubicka, gcc-patches, marxin.liska > > Why not just have probably_never_executed_bb_p return simply return > false bb->frequency is non-zero (right now it does the opposite - We want to have frequencies guessed for functions that was not trained in the profiling run (that was patch I posted earlier that I think did not go in, yet). Currently I return true when frequency indicate that BB is executed at least in 1/4th of all executions. With the cases discussed I see we may need to reduce this threshold. In general I do not like much hard tests for 0 because meaning of 0 depends on REG_BR_FREQ_BASE that is supposed to be changeable and we may want to make frequencies sreal, too. I suppose we may introduce --param for this. You are also right that I should update probably_never_executed_edge_p (I intended so, but obviously the code ended up in mainline accidentally). I however saw at least one case of jump threading where this trick did not help: the jump threading update confused itself by scaling via counts rather than frequencies and ended up with dropping everything to 0. This makes it more tempting to try to go with sreals for those.... Honza > returns true when bb->frequency is 0)? Making this change removed a > bunch of other failures. With this change as well, there are only 3 > cases that still fail with 1 train run that pass with 100. Need to > look at those. > > > > > Will you look into logic of do_jump or shall I try to dive in? > > I can take a look, but probably won't have a chance until late this > week. If you don't get to it before then I will see if I can figure > out why it is applying the branch probabilities this way. > > Teresa > > > > > Honza > > > > -- > Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-09-26 22:26 ` Jan Hubicka @ 2013-09-27 14:50 ` Teresa Johnson 2013-09-29 17:34 ` Teresa Johnson 0 siblings, 1 reply; 62+ messages in thread From: Teresa Johnson @ 2013-09-27 14:50 UTC (permalink / raw) To: Jan Hubicka; +Cc: gcc-patches, marxin.liska On Thu, Sep 26, 2013 at 3:02 PM, Jan Hubicka <hubicka@ucw.cz> wrote: >> >> Why not just have probably_never_executed_bb_p return simply return >> false bb->frequency is non-zero (right now it does the opposite - > > We want to have frequencies guessed for functions that was not trained > in the profiling run (that was patch I posted earlier that I think did not > go in, yet). Right, but for splitting and bb layout purposes, for these statically guessed unprofiled functions we in fact don't want to do any splitting or treat the bbs as never executed (which shouldn't be a change from the status quo since all the bbs in these functions are currently 0 weight, it's only when we inline in the case of comdats that they appear colder than the surrounding code, but in fact we don't want this). The only other caller to probably_never_executed_bb_p is compute_function_frequency, but in the case of statically guessed functions they will have profile_status != PROFILE_READ and won't invoke probably_never_executed_bb_p. But re-reading our most recent exchange on the comdat profile issue, it sounds like you were suggesting guessing profiles for all 0-weight functions early, then dropping them from PROFILE_READ to PROFILE_GUESSED only once we determine in ipa-inline that there is a potentially non-zero call path to them. In that case with the change I describe above to probably_never_executed_bb_p, the 0-weight functions with 0 calls to them will incorrectly be marked as NODE_FREQUENCY_NORMAL, which would be bad as they would not be size optimized or moved into the cold section. So it seems like we want different handling of these guessed frequencies in compute_function_frequency and bb-reorder.c. Actually I think we can handle this by checking if the function entry block has a 0 count. If so, then we just look at the bb counts and not the frequencies for determining bb hotness as the frequencies would presumably have been statically-guessed. This will ensure that the cgraph node continues to be marked unlikely and size-optimized. If the function entry block has a non-zero count, then we look at both the bb count and the bb frequency - if they are both zero then the bb is probably never executed, but if either is non-zero then we should treat the block as possibly executed (which will come into play for splitting and bb layout). Teresa > > Currently I return true when frequency indicate that BB is executed at least in > 1/4th of all executions. With the cases discussed I see we may need to reduce > this threshold. In general I do not like much hard tests for 0 because meaning > of 0 depends on REG_BR_FREQ_BASE that is supposed to be changeable and we may > want to make frequencies sreal, too. > > I suppose we may introduce --param for this. You are also right that I should > update probably_never_executed_edge_p (I intended so, but obviously the code > ended up in mainline accidentally). > > I however saw at least one case of jump threading where this trick did not > help: the jump threading update confused itself by scaling via counts rather > than frequencies and ended up with dropping everything to 0. This makes it > more tempting to try to go with sreals for those.... > > Honza > >> returns true when bb->frequency is 0)? Making this change removed a >> bunch of other failures. With this change as well, there are only 3 >> cases that still fail with 1 train run that pass with 100. Need to >> look at those. >> >> > >> > Will you look into logic of do_jump or shall I try to dive in? >> >> I can take a look, but probably won't have a chance until late this >> week. If you don't get to it before then I will see if I can figure >> out why it is applying the branch probabilities this way. >> >> Teresa >> >> > >> > Honza >> >> >> >> -- >> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-09-27 14:50 ` Teresa Johnson @ 2013-09-29 17:34 ` Teresa Johnson 2013-10-02 16:19 ` Jan Hubicka 0 siblings, 1 reply; 62+ messages in thread From: Teresa Johnson @ 2013-09-29 17:34 UTC (permalink / raw) To: Jan Hubicka; +Cc: gcc-patches, marxin.liska On Fri, Sep 27, 2013 at 7:15 AM, Teresa Johnson <tejohnson@google.com> wrote: > On Thu, Sep 26, 2013 at 3:02 PM, Jan Hubicka <hubicka@ucw.cz> wrote: >>> >>> Why not just have probably_never_executed_bb_p return simply return >>> false bb->frequency is non-zero (right now it does the opposite - >> >> We want to have frequencies guessed for functions that was not trained >> in the profiling run (that was patch I posted earlier that I think did not >> go in, yet). > > Right, but for splitting and bb layout purposes, for these statically > guessed unprofiled functions we in fact don't want to do any splitting > or treat the bbs as never executed (which shouldn't be a change from > the status quo since all the bbs in these functions are currently 0 > weight, it's only when we inline in the case of comdats that they > appear colder than the surrounding code, but in fact we don't want > this). > > The only other caller to probably_never_executed_bb_p is > compute_function_frequency, but in the case of statically guessed > functions they will have profile_status != PROFILE_READ and won't > invoke probably_never_executed_bb_p. But re-reading our most recent > exchange on the comdat profile issue, it sounds like you were > suggesting guessing profiles for all 0-weight functions early, then > dropping them from PROFILE_READ to PROFILE_GUESSED only once we > determine in ipa-inline that there is a potentially non-zero call path > to them. In that case with the change I describe above to > probably_never_executed_bb_p, the 0-weight functions with 0 calls to > them will incorrectly be marked as NODE_FREQUENCY_NORMAL, which would > be bad as they would not be size optimized or moved into the cold > section. > > So it seems like we want different handling of these guessed > frequencies in compute_function_frequency and bb-reorder.c. Actually I > think we can handle this by checking if the function entry block has a > 0 count. If so, then we just look at the bb counts and not the > frequencies for determining bb hotness as the frequencies would > presumably have been statically-guessed. This will ensure that the > cgraph node continues to be marked unlikely and size-optimized. If the > function entry block has a non-zero count, then we look at both the bb > count and the bb frequency - if they are both zero then the bb is > probably never executed, but if either is non-zero then we should > treat the block as possibly executed (which will come into play for > splitting and bb layout). Here is a patch to handle the profile insanities conservatively during splitting. It also simplifies the probably_never_executed* code to treat missing counts within a profiled function differently (conservatively, based on frequency) from the case where the whole function has a guessed profile. That way, once a patch to guess profiles for non-executed functions is added, they will continue to have their nodes marked as unlikely. I also pulled the guts of the probably_never_executed_bb_p code out to a helper that is then invoked by both the bb and edge versions of this function, so they stay in sync. This gets rid of a number of the failures with splitting + the linker script to make the unlikely section non-executable. I have a patch to fix some jump threading insanities that I will send separately. Bootstrapped and regression tested on x86_64. Also tested with an lto profiledbootstrap. Ok for trunk? Thanks, Teresa 2013-09-29 Teresa Johnson <tejohnson@google.com> * bb-reorder.c (find_rarely_executed_basic_blocks_and_crossing_edges): Treat profile insanities conservatively. * predict.c (probably_never_executed): New function. Treat profile insanities conservatively. (probably_never_executed_bb_p): Invoke probably_never_executed. (probably_never_executed_edge_p): Invoke probably_never_executed. Index: bb-reorder.c =================================================================== --- bb-reorder.c (revision 202947) +++ bb-reorder.c (working copy) @@ -1564,8 +1564,25 @@ find_rarely_executed_basic_blocks_and_crossing_edg /* Mark which partition (hot/cold) each basic block belongs in. */ FOR_EACH_BB (bb) { + bool cold_bb = false; if (probably_never_executed_bb_p (cfun, bb)) { + /* Handle profile insanities created by upstream optimizations + by also checking the incoming edge weights. If there is a non-cold + incoming edge, conservatively prevent this block from being split + into the cold section. */ + cold_bb = true; + FOR_EACH_EDGE (e, ei, bb->preds) + { + if (!probably_never_executed_edge_p (cfun, e)) + { + cold_bb = false; + break; + } + } + } + if (cold_bb) + { BB_SET_PARTITION (bb, BB_COLD_PARTITION); cold_bb_count++; } Index: predict.c =================================================================== --- predict.c (revision 202947) +++ predict.c (working copy) @@ -226,26 +226,26 @@ maybe_hot_edge_p (edge e) } -/* Return true in case BB is probably never executed. */ -bool -probably_never_executed_bb_p (struct function *fun, const_basic_block bb) +/* Return true if profile COUNT and FREQUENCY, or function FUN static + node frequency reflects never being executed. */ + +static bool +probably_never_executed (struct function *fun, + gcov_type count, int frequency) { gcc_checking_assert (fun); if (profile_status_for_function (fun) == PROFILE_READ) { - if ((bb->count * 4 + profile_info->runs / 2) / profile_info->runs > 0) + if ((count * 4 + profile_info->runs / 2) / profile_info->runs > 0) return false; - if (!bb->frequency) - return true; - if (!ENTRY_BLOCK_PTR->frequency) - return false; - if (ENTRY_BLOCK_PTR->count && ENTRY_BLOCK_PTR->count < REG_BR_PROB_BASE) - { - return (RDIV (bb->frequency * ENTRY_BLOCK_PTR->count, - ENTRY_BLOCK_PTR->frequency) - < REG_BR_PROB_BASE / 4); - } + // If this is a profiled function (entry bb non-zero count), then base + // the coldness decision on the frequency. This will handle cases where + // counts are not updated properly during optimizations or expansion. + if (ENTRY_BLOCK_PTR->count) + return frequency == 0; + // Unprofiled function, frequencies statically assigned. All bbs are + // treated as cold. return true; } if ((!profile_info || !flag_branch_probabilities) @@ -256,19 +256,21 @@ maybe_hot_edge_p (edge e) } +/* Return true in case BB is probably never executed. */ + +bool +probably_never_executed_bb_p (struct function *fun, const_basic_block bb) +{ + return probably_never_executed (fun, bb->count, bb->frequency); +} + + /* Return true in case edge E is probably never executed. */ bool probably_never_executed_edge_p (struct function *fun, edge e) { - gcc_checking_assert (fun); - if (profile_info && flag_branch_probabilities) - return ((e->count + profile_info->runs / 2) / profile_info->runs) == 0; - if ((!profile_info || !flag_branch_probabilities) - && (cgraph_get_node (fun->decl)->frequency - == NODE_FREQUENCY_UNLIKELY_EXECUTED)) - return true; - return false; + return probably_never_executed (fun, e->count, EDGE_FREQUENCY (e)); } /* Return true if NODE should be optimized for size. */ > > Teresa > >> >> Currently I return true when frequency indicate that BB is executed at least in >> 1/4th of all executions. With the cases discussed I see we may need to reduce >> this threshold. In general I do not like much hard tests for 0 because meaning >> of 0 depends on REG_BR_FREQ_BASE that is supposed to be changeable and we may >> want to make frequencies sreal, too. >> >> I suppose we may introduce --param for this. You are also right that I should >> update probably_never_executed_edge_p (I intended so, but obviously the code >> ended up in mainline accidentally). >> >> I however saw at least one case of jump threading where this trick did not >> help: the jump threading update confused itself by scaling via counts rather >> than frequencies and ended up with dropping everything to 0. This makes it >> more tempting to try to go with sreals for those.... >> >> Honza >> >>> returns true when bb->frequency is 0)? Making this change removed a >>> bunch of other failures. With this change as well, there are only 3 >>> cases that still fail with 1 train run that pass with 100. Need to >>> look at those. >>> >>> > >>> > Will you look into logic of do_jump or shall I try to dive in? >>> >>> I can take a look, but probably won't have a chance until late this >>> week. If you don't get to it before then I will see if I can figure >>> out why it is applying the branch probabilities this way. >>> >>> Teresa >>> >>> > >>> > Honza >>> >>> >>> >>> -- >>> Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 > > > > -- > Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-09-29 17:34 ` Teresa Johnson @ 2013-10-02 16:19 ` Jan Hubicka 2013-10-02 17:55 ` Teresa Johnson 2013-10-03 13:42 ` Teresa Johnson 0 siblings, 2 replies; 62+ messages in thread From: Jan Hubicka @ 2013-10-02 16:19 UTC (permalink / raw) To: Teresa Johnson; +Cc: Jan Hubicka, gcc-patches, marxin.liska > 2013-09-29 Teresa Johnson <tejohnson@google.com> > > * bb-reorder.c (find_rarely_executed_basic_blocks_and_crossing_edges): > Treat profile insanities conservatively. > * predict.c (probably_never_executed): New function. Treat profile > insanities conservatively. > (probably_never_executed_bb_p): Invoke probably_never_executed. > (probably_never_executed_edge_p): Invoke probably_never_executed. > > Index: bb-reorder.c > =================================================================== > --- bb-reorder.c (revision 202947) > +++ bb-reorder.c (working copy) > @@ -1564,8 +1564,25 @@ find_rarely_executed_basic_blocks_and_crossing_edg > /* Mark which partition (hot/cold) each basic block belongs in. */ > FOR_EACH_BB (bb) > { > + bool cold_bb = false; whitespace here > if (probably_never_executed_bb_p (cfun, bb)) > { > + /* Handle profile insanities created by upstream optimizations > + by also checking the incoming edge weights. If there is a non-cold > + incoming edge, conservatively prevent this block from being split > + into the cold section. */ > + cold_bb = true; > + FOR_EACH_EDGE (e, ei, bb->preds) > + { > + if (!probably_never_executed_edge_p (cfun, e)) > + { > + cold_bb = false; > + break; > + } > + } You can probably elimnate the extra braces. So we won't propagate deeper in the CFG, right? This change is OK. > + } > + if (cold_bb) > + { > BB_SET_PARTITION (bb, BB_COLD_PARTITION); > cold_bb_count++; > } > Index: predict.c > =================================================================== > --- predict.c (revision 202947) > +++ predict.c (working copy) > @@ -226,26 +226,26 @@ maybe_hot_edge_p (edge e) > } > > > -/* Return true in case BB is probably never executed. */ > > -bool > -probably_never_executed_bb_p (struct function *fun, const_basic_block bb) > +/* Return true if profile COUNT and FREQUENCY, or function FUN static > + node frequency reflects never being executed. */ > + > +static bool > +probably_never_executed (struct function *fun, > + gcov_type count, int frequency) > { > gcc_checking_assert (fun); > if (profile_status_for_function (fun) == PROFILE_READ) > { > - if ((bb->count * 4 + profile_info->runs / 2) / profile_info->runs > 0) > + if ((count * 4 + profile_info->runs / 2) / profile_info->runs > 0) > return false; > - if (!bb->frequency) > - return true; > - if (!ENTRY_BLOCK_PTR->frequency) > - return false; > - if (ENTRY_BLOCK_PTR->count && ENTRY_BLOCK_PTR->count < REG_BR_PROB_BASE) > - { > - return (RDIV (bb->frequency * ENTRY_BLOCK_PTR->count, > - ENTRY_BLOCK_PTR->frequency) > - < REG_BR_PROB_BASE / 4); > - } > + // If this is a profiled function (entry bb non-zero count), then base > + // the coldness decision on the frequency. This will handle cases where > + // counts are not updated properly during optimizations or expansion. > + if (ENTRY_BLOCK_PTR->count) > + return frequency == 0; > + // Unprofiled function, frequencies statically assigned. All bbs are > + // treated as cold. I would avoid combining C and C++ comments in the function. Did you get some data on how many basic blocks we now consider hot? The previous implemntation consdered block as never executed when frequencies indicates that it is executed in at most 1/4th of invocations of program. You essentially chnage to 1/10000. The first seems bit too high given the way we distribute probabilities in dojump and firends, second looks too low. The change introducing probably_never_executed with the current logic is OK. We may want to fine tune the ratio. Honza > return true; > } > if ((!profile_info || !flag_branch_probabilities) > @@ -256,19 +256,21 @@ maybe_hot_edge_p (edge e) > } > > > +/* Return true in case BB is probably never executed. */ > + > +bool > +probably_never_executed_bb_p (struct function *fun, const_basic_block bb) > +{ > + return probably_never_executed (fun, bb->count, bb->frequency); > +} > + > + > /* Return true in case edge E is probably never executed. */ > > bool > probably_never_executed_edge_p (struct function *fun, edge e) > { > - gcc_checking_assert (fun); > - if (profile_info && flag_branch_probabilities) > - return ((e->count + profile_info->runs / 2) / profile_info->runs) == 0; > - if ((!profile_info || !flag_branch_probabilities) > - && (cgraph_get_node (fun->decl)->frequency > - == NODE_FREQUENCY_UNLIKELY_EXECUTED)) > - return true; > - return false; > + return probably_never_executed (fun, e->count, EDGE_FREQUENCY (e)); > } > > /* Return true if NODE should be optimized for size. */ ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-10-02 16:19 ` Jan Hubicka @ 2013-10-02 17:55 ` Teresa Johnson 2013-10-02 18:10 ` Jan Hubicka 2013-10-03 13:42 ` Teresa Johnson 1 sibling, 1 reply; 62+ messages in thread From: Teresa Johnson @ 2013-10-02 17:55 UTC (permalink / raw) To: Jan Hubicka; +Cc: gcc-patches, marxin.liska On Wed, Oct 2, 2013 at 9:19 AM, Jan Hubicka <hubicka@ucw.cz> wrote: >> 2013-09-29 Teresa Johnson <tejohnson@google.com> >> >> * bb-reorder.c (find_rarely_executed_basic_blocks_and_crossing_edges): >> Treat profile insanities conservatively. >> * predict.c (probably_never_executed): New function. Treat profile >> insanities conservatively. >> (probably_never_executed_bb_p): Invoke probably_never_executed. >> (probably_never_executed_edge_p): Invoke probably_never_executed. >> >> Index: bb-reorder.c >> =================================================================== >> --- bb-reorder.c (revision 202947) >> +++ bb-reorder.c (working copy) >> @@ -1564,8 +1564,25 @@ find_rarely_executed_basic_blocks_and_crossing_edg >> /* Mark which partition (hot/cold) each basic block belongs in. */ >> FOR_EACH_BB (bb) >> { >> + bool cold_bb = false; > > whitespace here meaning add a line of whitespace? Ok, done. > >> if (probably_never_executed_bb_p (cfun, bb)) >> { >> + /* Handle profile insanities created by upstream optimizations >> + by also checking the incoming edge weights. If there is a non-cold >> + incoming edge, conservatively prevent this block from being split >> + into the cold section. */ >> + cold_bb = true; >> + FOR_EACH_EDGE (e, ei, bb->preds) >> + { >> + if (!probably_never_executed_edge_p (cfun, e)) >> + { >> + cold_bb = false; >> + break; >> + } >> + } > > You can probably elimnate the extra braces. > So we won't propagate deeper in the CFG, right? Done. > > This change is OK. > >> + } >> + if (cold_bb) >> + { >> BB_SET_PARTITION (bb, BB_COLD_PARTITION); >> cold_bb_count++; >> } >> Index: predict.c >> =================================================================== >> --- predict.c (revision 202947) >> +++ predict.c (working copy) >> @@ -226,26 +226,26 @@ maybe_hot_edge_p (edge e) >> } >> >> >> -/* Return true in case BB is probably never executed. */ >> >> -bool >> -probably_never_executed_bb_p (struct function *fun, const_basic_block bb) >> +/* Return true if profile COUNT and FREQUENCY, or function FUN static >> + node frequency reflects never being executed. */ >> + >> +static bool >> +probably_never_executed (struct function *fun, >> + gcov_type count, int frequency) >> { >> gcc_checking_assert (fun); >> if (profile_status_for_function (fun) == PROFILE_READ) >> { >> - if ((bb->count * 4 + profile_info->runs / 2) / profile_info->runs > 0) >> + if ((count * 4 + profile_info->runs / 2) / profile_info->runs > 0) >> return false; >> - if (!bb->frequency) >> - return true; >> - if (!ENTRY_BLOCK_PTR->frequency) >> - return false; >> - if (ENTRY_BLOCK_PTR->count && ENTRY_BLOCK_PTR->count < REG_BR_PROB_BASE) >> - { >> - return (RDIV (bb->frequency * ENTRY_BLOCK_PTR->count, >> - ENTRY_BLOCK_PTR->frequency) >> - < REG_BR_PROB_BASE / 4); >> - } >> + // If this is a profiled function (entry bb non-zero count), then base >> + // the coldness decision on the frequency. This will handle cases where >> + // counts are not updated properly during optimizations or expansion. >> + if (ENTRY_BLOCK_PTR->count) >> + return frequency == 0; >> + // Unprofiled function, frequencies statically assigned. All bbs are >> + // treated as cold. > > I would avoid combining C and C++ comments in the function. Fixed. > Did you get some data on how many basic blocks we now consider hot? No, I can do that. > > The previous implemntation consdered block as never executed when frequencies > indicates that it is executed in at most 1/4th of invocations of program. > You essentially chnage to 1/10000. The first seems bit too high given the > way we distribute probabilities in dojump and firends, second looks too low. But why do we want to consider blocks as "probably never executed" when the frequency suggests they are sometimes executed? AFAICT, there are 2 main callers of this routine: 1) function splitting in bb-layout 2) function cgraph node weight Where #2 will affect optimization of the function for size and also function layout by the linker. I would argue that for function splitting, we really want to know when it is probably *never* executed - i.e. completely cold, since the cost of jumping back and forth to the cold section is likely to be high. I am not sure for #2 what the right ratio is. For function layout, we may also want to place only really cold *never* executed functions into the cold section, but I am less sure about optimization for size. Perhaps we really need two different interfaces to test for different levels of coldness: probably_never_executed() -> returns true when there is profile information for the function and the bb has 0 count and 0 frequency. -> invoked from bb-reorder.cc to drive function splitting -> may want to consider invoking this as an additional check before putting function into unlikely text section in the future. possibly_never_executed() -> essentially the existing logic in probably_never_executed_bb_p -> invoked when marking the cgraph node > > The change introducing probably_never_executed with the current logic is OK. Ok, I will commit the two approved parts for now (the change to bb-reorder.c and the addition of probably_never_executed that uses the existing logic from probably_never_executed_bb_p. Thanks, Teresa > We may want to fine tune the ratio. > > Honza >> return true; >> } >> if ((!profile_info || !flag_branch_probabilities) >> @@ -256,19 +256,21 @@ maybe_hot_edge_p (edge e) >> } >> >> >> +/* Return true in case BB is probably never executed. */ >> + >> +bool >> +probably_never_executed_bb_p (struct function *fun, const_basic_block bb) >> +{ >> + return probably_never_executed (fun, bb->count, bb->frequency); >> +} >> + >> + >> /* Return true in case edge E is probably never executed. */ >> >> bool >> probably_never_executed_edge_p (struct function *fun, edge e) >> { >> - gcc_checking_assert (fun); >> - if (profile_info && flag_branch_probabilities) >> - return ((e->count + profile_info->runs / 2) / profile_info->runs) == 0; >> - if ((!profile_info || !flag_branch_probabilities) >> - && (cgraph_get_node (fun->decl)->frequency >> - == NODE_FREQUENCY_UNLIKELY_EXECUTED)) >> - return true; >> - return false; >> + return probably_never_executed (fun, e->count, EDGE_FREQUENCY (e)); >> } >> >> /* Return true if NODE should be optimized for size. */ -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-10-02 17:55 ` Teresa Johnson @ 2013-10-02 18:10 ` Jan Hubicka 0 siblings, 0 replies; 62+ messages in thread From: Jan Hubicka @ 2013-10-02 18:10 UTC (permalink / raw) To: Teresa Johnson; +Cc: Jan Hubicka, gcc-patches, marxin.liska > But why do we want to consider blocks as "probably never executed" > when the frequency suggests they are sometimes executed? Well, probably never executed is mean to reffer to one run. If you have something like code handling fatal errors, you probably still want to have it in cold secion even if user may have trained the program on a testsuite that triggers them once or twice per thousdand of runs. We may just make the predicate more strict, but lets do that incrementally so we know how much things change. I am somewhat concerned that we are not that effective on breaking out cold code so -fprofile-use does not lead to as significant code size reductions as the theory would suggest, so perhaps I am just overfly conservative about this. Getting the splitting to work reliably is definitely going to be a win. > Perhaps we really need two different interfaces to test for different > levels of coldness: > > probably_never_executed() > -> returns true when there is profile information for the function > and the bb has 0 count and 0 frequency. > -> invoked from bb-reorder.cc to drive function splitting > -> may want to consider invoking this as an additional check before > putting function into unlikely text section in the future. > > possibly_never_executed() > -> essentially the existing logic in probably_never_executed_bb_p > -> invoked when marking the cgraph node Perhaps... Advantage of hot/normal/cold split is that it is easy to understand, but if necessary (i.e, it becomes impossible to tune well) we may add more stages... Honza ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-10-02 16:19 ` Jan Hubicka 2013-10-02 17:55 ` Teresa Johnson @ 2013-10-03 13:42 ` Teresa Johnson 2013-10-03 23:37 ` Teresa Johnson 1 sibling, 1 reply; 62+ messages in thread From: Teresa Johnson @ 2013-10-03 13:42 UTC (permalink / raw) To: Jan Hubicka; +Cc: gcc-patches, marxin.liska On Wed, Oct 2, 2013 at 9:19 AM, Jan Hubicka <hubicka@ucw.cz> wrote: >> 2013-09-29 Teresa Johnson <tejohnson@google.com> >> >> * bb-reorder.c (find_rarely_executed_basic_blocks_and_crossing_edges): >> Treat profile insanities conservatively. >> * predict.c (probably_never_executed): New function. Treat profile >> insanities conservatively. >> (probably_never_executed_bb_p): Invoke probably_never_executed. >> (probably_never_executed_edge_p): Invoke probably_never_executed. >> >> Index: bb-reorder.c >> =================================================================== >> --- bb-reorder.c (revision 202947) >> +++ bb-reorder.c (working copy) >> @@ -1564,8 +1564,25 @@ find_rarely_executed_basic_blocks_and_crossing_edg >> /* Mark which partition (hot/cold) each basic block belongs in. */ >> FOR_EACH_BB (bb) >> { >> + bool cold_bb = false; > > whitespace here > >> if (probably_never_executed_bb_p (cfun, bb)) >> { >> + /* Handle profile insanities created by upstream optimizations >> + by also checking the incoming edge weights. If there is a non-cold >> + incoming edge, conservatively prevent this block from being split >> + into the cold section. */ >> + cold_bb = true; >> + FOR_EACH_EDGE (e, ei, bb->preds) >> + { >> + if (!probably_never_executed_edge_p (cfun, e)) >> + { >> + cold_bb = false; >> + break; >> + } >> + } > > You can probably elimnate the extra braces. > So we won't propagate deeper in the CFG, right? > > This change is OK. > >> + } >> + if (cold_bb) >> + { >> BB_SET_PARTITION (bb, BB_COLD_PARTITION); >> cold_bb_count++; >> } >> Index: predict.c >> =================================================================== >> --- predict.c (revision 202947) >> +++ predict.c (working copy) >> @@ -226,26 +226,26 @@ maybe_hot_edge_p (edge e) >> } >> >> >> -/* Return true in case BB is probably never executed. */ >> >> -bool >> -probably_never_executed_bb_p (struct function *fun, const_basic_block bb) >> +/* Return true if profile COUNT and FREQUENCY, or function FUN static >> + node frequency reflects never being executed. */ >> + >> +static bool >> +probably_never_executed (struct function *fun, >> + gcov_type count, int frequency) >> { >> gcc_checking_assert (fun); >> if (profile_status_for_function (fun) == PROFILE_READ) >> { >> - if ((bb->count * 4 + profile_info->runs / 2) / profile_info->runs > 0) >> + if ((count * 4 + profile_info->runs / 2) / profile_info->runs > 0) >> return false; >> - if (!bb->frequency) >> - return true; >> - if (!ENTRY_BLOCK_PTR->frequency) >> - return false; >> - if (ENTRY_BLOCK_PTR->count && ENTRY_BLOCK_PTR->count < REG_BR_PROB_BASE) >> - { >> - return (RDIV (bb->frequency * ENTRY_BLOCK_PTR->count, >> - ENTRY_BLOCK_PTR->frequency) >> - < REG_BR_PROB_BASE / 4); >> - } >> + // If this is a profiled function (entry bb non-zero count), then base >> + // the coldness decision on the frequency. This will handle cases where >> + // counts are not updated properly during optimizations or expansion. >> + if (ENTRY_BLOCK_PTR->count) >> + return frequency == 0; >> + // Unprofiled function, frequencies statically assigned. All bbs are >> + // treated as cold. > > I would avoid combining C and C++ comments in the function. > Did you get some data on how many basic blocks we now consider hot? > > The previous implemntation consdered block as never executed when frequencies > indicates that it is executed in at most 1/4th of invocations of program. > You essentially chnage to 1/10000. The first seems bit too high given the > way we distribute probabilities in dojump and firends, second looks too low. Actually, I don't think the current code is detecting when the frequencies indicate it was executed 1/4 time. The current code takes a ratio of the entry block count, and compares it to REG_BR_PROB_BASE/4, which seems like the wrong comparison for profile counts. Shouldn't this be something like: gcov_type computed_count = RDIV (frequency * ENTRY_BLOCK_PTR->count, ENTRY_BLOCK_PTR->frequency) if ((computed_count * 4 + profile_info->runs / 2) / profile_info->runs > 0) return false; return true; i.e. do the same check we do for bb->count above. And the check guarding this is looking for ENTRY_BLOCK_PTR->count < REG_BR_PROB_BASE, which also doesn't seem right, although we need to ensure that we don't overflow when multiplying by frequency. Teresa > > The change introducing probably_never_executed with the current logic is OK. > We may want to fine tune the ratio. > > Honza >> return true; >> } >> if ((!profile_info || !flag_branch_probabilities) >> @@ -256,19 +256,21 @@ maybe_hot_edge_p (edge e) >> } >> >> >> +/* Return true in case BB is probably never executed. */ >> + >> +bool >> +probably_never_executed_bb_p (struct function *fun, const_basic_block bb) >> +{ >> + return probably_never_executed (fun, bb->count, bb->frequency); >> +} >> + >> + >> /* Return true in case edge E is probably never executed. */ >> >> bool >> probably_never_executed_edge_p (struct function *fun, edge e) >> { >> - gcc_checking_assert (fun); >> - if (profile_info && flag_branch_probabilities) >> - return ((e->count + profile_info->runs / 2) / profile_info->runs) == 0; >> - if ((!profile_info || !flag_branch_probabilities) >> - && (cgraph_get_node (fun->decl)->frequency >> - == NODE_FREQUENCY_UNLIKELY_EXECUTED)) >> - return true; >> - return false; >> + return probably_never_executed (fun, e->count, EDGE_FREQUENCY (e)); >> } >> >> /* Return true if NODE should be optimized for size. */ -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-10-03 13:42 ` Teresa Johnson @ 2013-10-03 23:37 ` Teresa Johnson 0 siblings, 0 replies; 62+ messages in thread From: Teresa Johnson @ 2013-10-03 23:37 UTC (permalink / raw) To: Jan Hubicka; +Cc: gcc-patches, marxin.liska On Thu, Oct 3, 2013 at 6:41 AM, Teresa Johnson <tejohnson@google.com> wrote: > On Wed, Oct 2, 2013 at 9:19 AM, Jan Hubicka <hubicka@ucw.cz> wrote: >>> 2013-09-29 Teresa Johnson <tejohnson@google.com> >>> >>> * bb-reorder.c (find_rarely_executed_basic_blocks_and_crossing_edges): >>> Treat profile insanities conservatively. >>> * predict.c (probably_never_executed): New function. Treat profile >>> insanities conservatively. >>> (probably_never_executed_bb_p): Invoke probably_never_executed. >>> (probably_never_executed_edge_p): Invoke probably_never_executed. >>> >>> Index: bb-reorder.c >>> =================================================================== >>> --- bb-reorder.c (revision 202947) >>> +++ bb-reorder.c (working copy) >>> @@ -1564,8 +1564,25 @@ find_rarely_executed_basic_blocks_and_crossing_edg >>> /* Mark which partition (hot/cold) each basic block belongs in. */ >>> FOR_EACH_BB (bb) >>> { >>> + bool cold_bb = false; >> >> whitespace here >> >>> if (probably_never_executed_bb_p (cfun, bb)) >>> { >>> + /* Handle profile insanities created by upstream optimizations >>> + by also checking the incoming edge weights. If there is a non-cold >>> + incoming edge, conservatively prevent this block from being split >>> + into the cold section. */ >>> + cold_bb = true; >>> + FOR_EACH_EDGE (e, ei, bb->preds) >>> + { >>> + if (!probably_never_executed_edge_p (cfun, e)) >>> + { >>> + cold_bb = false; >>> + break; >>> + } >>> + } >> >> You can probably elimnate the extra braces. >> So we won't propagate deeper in the CFG, right? >> >> This change is OK. >> >>> + } >>> + if (cold_bb) >>> + { >>> BB_SET_PARTITION (bb, BB_COLD_PARTITION); >>> cold_bb_count++; >>> } >>> Index: predict.c >>> =================================================================== >>> --- predict.c (revision 202947) >>> +++ predict.c (working copy) >>> @@ -226,26 +226,26 @@ maybe_hot_edge_p (edge e) >>> } >>> >>> >>> -/* Return true in case BB is probably never executed. */ >>> >>> -bool >>> -probably_never_executed_bb_p (struct function *fun, const_basic_block bb) >>> +/* Return true if profile COUNT and FREQUENCY, or function FUN static >>> + node frequency reflects never being executed. */ >>> + >>> +static bool >>> +probably_never_executed (struct function *fun, >>> + gcov_type count, int frequency) >>> { >>> gcc_checking_assert (fun); >>> if (profile_status_for_function (fun) == PROFILE_READ) >>> { >>> - if ((bb->count * 4 + profile_info->runs / 2) / profile_info->runs > 0) >>> + if ((count * 4 + profile_info->runs / 2) / profile_info->runs > 0) >>> return false; >>> - if (!bb->frequency) >>> - return true; >>> - if (!ENTRY_BLOCK_PTR->frequency) >>> - return false; >>> - if (ENTRY_BLOCK_PTR->count && ENTRY_BLOCK_PTR->count < REG_BR_PROB_BASE) >>> - { >>> - return (RDIV (bb->frequency * ENTRY_BLOCK_PTR->count, >>> - ENTRY_BLOCK_PTR->frequency) >>> - < REG_BR_PROB_BASE / 4); >>> - } >>> + // If this is a profiled function (entry bb non-zero count), then base >>> + // the coldness decision on the frequency. This will handle cases where >>> + // counts are not updated properly during optimizations or expansion. >>> + if (ENTRY_BLOCK_PTR->count) >>> + return frequency == 0; >>> + // Unprofiled function, frequencies statically assigned. All bbs are >>> + // treated as cold. >> >> I would avoid combining C and C++ comments in the function. >> Did you get some data on how many basic blocks we now consider hot? >> >> The previous implemntation consdered block as never executed when frequencies >> indicates that it is executed in at most 1/4th of invocations of program. >> You essentially chnage to 1/10000. The first seems bit too high given the >> way we distribute probabilities in dojump and firends, second looks too low. > > Actually, I don't think the current code is detecting when the > frequencies indicate it was executed 1/4 time. The current code takes > a ratio of the entry block count, and compares it to > REG_BR_PROB_BASE/4, which seems like the wrong comparison for profile > counts. Shouldn't this be something like: > > gcov_type computed_count = RDIV (frequency * ENTRY_BLOCK_PTR->count, > ENTRY_BLOCK_PTR->frequency) > if ((computed_count * 4 + profile_info->runs / 2) / profile_info->runs > 0) > return false; > return true; > > i.e. do the same check we do for bb->count above. And the check > guarding this is looking for ENTRY_BLOCK_PTR->count < > REG_BR_PROB_BASE, which also doesn't seem right, although we need to > ensure that we don't overflow when multiplying by frequency. I have a variant of this change that works well, and has the proper overflow checking. This gets rid of 13 failures when there is 1 train run. Changing the required ratio to 1/100 training runs instead of 1/4 reduces the single train run failures by another 5, since smaller frequencies are handled even when the count has been truncated to 0. I found a couple more failures with 1 train run were due to inlining profile update issues, which I have a fix for. At that point, the failures are the same between the 1 train run and 100 train run cases. After that, there are 2 remaining failures (both with 1 train and 100 train runs), that go away when I disable loop unrolling, that I haven't looked at yet. I'll send a patch with the above changes though hopefully tonight. What do you think of changing the required execution count to profile run ratio to 1/100? Teresa > > Teresa > >> >> The change introducing probably_never_executed with the current logic is OK. >> We may want to fine tune the ratio. >> >> Honza >>> return true; >>> } >>> if ((!profile_info || !flag_branch_probabilities) >>> @@ -256,19 +256,21 @@ maybe_hot_edge_p (edge e) >>> } >>> >>> >>> +/* Return true in case BB is probably never executed. */ >>> + >>> +bool >>> +probably_never_executed_bb_p (struct function *fun, const_basic_block bb) >>> +{ >>> + return probably_never_executed (fun, bb->count, bb->frequency); >>> +} >>> + >>> + >>> /* Return true in case edge E is probably never executed. */ >>> >>> bool >>> probably_never_executed_edge_p (struct function *fun, edge e) >>> { >>> - gcc_checking_assert (fun); >>> - if (profile_info && flag_branch_probabilities) >>> - return ((e->count + profile_info->runs / 2) / profile_info->runs) == 0; >>> - if ((!profile_info || !flag_branch_probabilities) >>> - && (cgraph_get_node (fun->decl)->frequency >>> - == NODE_FREQUENCY_UNLIKELY_EXECUTED)) >>> - return true; >>> - return false; >>> + return probably_never_executed (fun, e->count, EDGE_FREQUENCY (e)); >>> } >>> >>> /* Return true if NODE should be optimized for size. */ > > > > -- > Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-09-24 18:51 ` Teresa Johnson 2013-09-25 23:10 ` Teresa Johnson 2013-09-26 22:26 ` Jan Hubicka @ 2013-10-01 17:36 ` Teresa Johnson 2 siblings, 0 replies; 62+ messages in thread From: Teresa Johnson @ 2013-10-01 17:36 UTC (permalink / raw) To: Jan Hubicka; +Cc: gcc-patches, marxin.liska On Tue, Sep 24, 2013 at 11:25 AM, Teresa Johnson <tejohnson@google.com> wrote: > On Tue, Sep 24, 2013 at 10:57 AM, Jan Hubicka <hubicka@ucw.cz> wrote: >>> >>> I looked at one that failed after 100 as well (20031204-1.c). In this >>> case, it was due to expansion which was creating multiple branches/bbs >>> from a logical OR and guessing incorrectly on how to assign the >>> counts: >>> >>> if (octets == 4 && (*cp == ':' || *cp == '\0')) { >>> >>> The (*cp == ':' || *cp == '\0') part looked like the following going >>> into RTL expansion: >>> >>> [20031204-1.c : 31:33] _29 = _28 == 58; >>> [20031204-1.c : 31:33] _30 = _28 == 0; >>> [20031204-1.c : 31:33] _31 = _29 | _30; >>> [20031204-1.c : 31:18] if (_31 != 0) >>> goto <bb 16>; >>> else >>> goto <bb 19>; >>> >>> where the result of the OR was always true, so bb 16 had a count of >>> 100 and bb 19 a count of 0. When it was expanded, the expanded version >>> of the above turned into 2 bbs with a branch in between. Both >>> comparisons were done in the first bb, but the first bb checked >>> whether the result of the *cp == '\0' compare was true, and if not >>> branched to the check for whether the *cp == ':' compare was true. It >>> gave the branch to the second check against ':' a count of 0, so that >>> bb got a count of 0 and was split out, and put the count of 100 on the >>> fall through assuming the compare with '\0' always evaluated to true. >>> In reality, this OR condition was always true because *cp was ':', not >>> '\0'. Therefore, the count of 0 on the second block with the check for >>> ':' was incorrect, we ended up trying to execute it, and failed. >> >> I see, we produce: >> ;; if (_26 != 0) >> >> (insn 94 93 95 (set (reg:CCZ 17 flags) >> (compare:CCZ (reg:QI 107 [ D.2184 ]) >> (const_int 0 [0]))) a.c:31 -1 >> (nil)) >> >> (insn 95 94 96 (set (reg:QI 122 [ D.2186 ]) >> (eq:QI (reg:CCZ 17 flags) >> (const_int 0 [0]))) a.c:31 -1 >> (nil)) >> >> (insn 96 95 97 (set (reg:CCZ 17 flags) >> (compare:CCZ (reg:QI 122 [ D.2186 ]) >> (const_int 0 [0]))) a.c:31 -1 >> (nil)) >> >> (jump_insn 97 96 98 (set (pc) >> (if_then_else (ne (reg:CCZ 17 flags) >> (const_int 0 [0])) >> (label_ref 100) >> (pc))) a.c:31 -1 >> (expr_list:REG_BR_PROB (const_int 6100 [0x17d4]) >> (nil))) >> >> (insn 98 97 99 (set (reg:CCZ 17 flags) >> (compare:CCZ (reg:QI 108 [ D.2186 ]) >> (const_int 0 [0]))) a.c:31 -1 >> (nil)) >> >> (jump_insn 99 98 100 (set (pc) >> (if_then_else (eq (reg:CCZ 17 flags) >> (const_int 0 [0])) >> (label_ref 0) >> (pc))) a.c:31 -1 >> (expr_list:REG_BR_PROB (const_int 3900 [0xf3c]) >> (nil))) >> >> (code_label 100 99 0 14 "" [0 uses]) >> >> That is because we TER together "_26 = _25 | _24" and "if (_26 != 0)" >> >> First I think the logic of do_jump should really be moved to trees. It is not >> doing things that can not be adequately represented by gimple. >> >> I am not that certain we want to move it before profiling though. >>> >>> Presumably we had the correct profile data for both blocks, but the >>> accuracy was reduced when the OR was represented as a logical >>> computation with a single branch. We could change the expansion code >>> to do something different, e.g. treat as a 50-50 branch. But we would >>> still end up with integer truncation issues when there was a single >>> training run. But that could be dealt with conservatively in the >> >> Yep, but it is still better than what we have now - if the test above was >> in hot part of program (i.e. not executed once), we will end up optimizing >> the second conditional for size. >> >> So I think it is do_jump bug to not distribute probabilities across the two >> conditoinals introduced. >>> bbpart code as I suggested for the jump threading issue above. I.e. a >>> cold block with incoming non-cold edges conservatively not marked cold >>> for splitting. >> >> Yep, we can probably do that, but we ought to fix the individual cases >> above at least for resonable number of runs. > > I made this change and it removed a few of the failures. > > I looked at another case that still failed with 1 train run but passed > with 100. It turned out to be another truncation issue exposed by RTL > expansion, where we created some control flow for a memset builtin > which was in a block with an execution count of 1. Some of the blocks > got frequencies less than half the original block, so the count was > rounded down or truncated to 0. I noticed that in this case (as well > as the jump threading case I fixed by looking for non-zero incoming > edges in partitioning) that the bb frequency was non-zero. > > Why not just have probably_never_executed_bb_p return simply return > false bb->frequency is non-zero (right now it does the opposite - > returns true when bb->frequency is 0)? Making this change removed a > bunch of other failures. With this change as well, there are only 3 > cases that still fail with 1 train run that pass with 100. Need to > look at those. > >> >> Will you look into logic of do_jump or shall I try to dive in? > > I can take a look, but probably won't have a chance until late this > week. If you don't get to it before then I will see if I can figure > out why it is applying the branch probabilities this way. Turned out not to be too tricky to fix the do_jump issue affecting 20031204-1.c. The patch below fixes the issue for this test case (along with the patch I posted a couple days ago that handles profile insanities conservatively in the case where there is 1 training run and we truncate counts): Index: dojump.c =================================================================== --- dojump.c (revision 202947) +++ dojump.c (working copy) @@ -325,15 +325,20 @@ break; case TRUTH_ORIF_EXPR: + /* Spread the probability evenly between the two conditions. So + the first condition has half the total probability of being true, + and therefore has half the probability of being false + (i.e. falls through to the second condition). If we reach the + second condition, it will be true with the original probability. */ if (if_true_label == NULL_RTX) { drop_through_label = gen_label_rtx (); - do_jump (op0, NULL_RTX, drop_through_label, prob); + do_jump (op0, NULL_RTX, drop_through_label, prob / 2); do_jump (op1, if_false_label, NULL_RTX, prob); } else { - do_jump (op0, NULL_RTX, if_true_label, prob); + do_jump (op0, NULL_RTX, if_true_label, prob / 2); do_jump (op1, if_false_label, if_true_label, prob); } break; I am regression testing this now and will post the patch for review in a separate thread. Honza, any comments on the patch I posted a couple days ago that treats the profile insanities conservatively? Thanks, Teresa > > Teresa > >> >> Honza > > > > -- > Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-17 21:02 ` Jan Hubicka 2013-08-19 13:51 ` Teresa Johnson @ 2013-08-19 15:34 ` Teresa Johnson 2013-08-21 15:31 ` Jan Hubicka 1 sibling, 1 reply; 62+ messages in thread From: Teresa Johnson @ 2013-08-19 15:34 UTC (permalink / raw) To: Jan Hubicka Cc: Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, marxin.liska, Sriraman Tallam On Sat, Aug 17, 2013 at 1:44 PM, Jan Hubicka <hubicka@ucw.cz> wrote: >> >> patch for updating counts based on estimated frequencies to address >> inlined comdats with 0 profile counts: >> >> 013-08-16 Teresa Johnson <tejohnson@google.com> >> >> * tree-inline.c (copy_bb): Compute count based on frequency. >> (copy_edges_for_bb): Ditto. >> (copy_cfg_body): Ditto. >> (copy_body): Pass down frequency. >> (expand_call_inline): Ditto. >> (tree_function_versioning): Ditto. >> * predict.c (init_and_estimate_bb_frequencies): New function. >> (rebuild_frequencies): Invoke init_and_estimate_bb_frequencies. >> * predict.h (init_and_estimate_bb_frequencies): Declare. >> * profile.c (branch_prob): Invoke init_and_estimate_bb_frequencies. >> * ipa-inline-transform.c (update_noncloned_frequencies): Scale edge >> counts. >> (clone_inlined_nodes): Compute edge count scale if needed. > > I do not see why inliner needs to care about scaling more than it does right > now. So you have init_and_estimate_bb_frequencies that force profile guessing > on a given function body. In addition to that I thing you need something like > freqs_to_counts that will compute counts based on freqs with given scale > (actually you can do that as part of propagation before frequencies are scalled > to the usual 0...FREQ_MAX scale and precision is lost). > > Because offline COMDAT functoin will be porduced for every COMDAT used, I think > it is bad to porduce any COMDAT (or any reachable function via calls with non-0 > count) that has empty profile (either because it got lost by COMDAT merging > or because of reading mismatch). The approach this patch takes is to simply treat those functions the same as we would if we didn't feed back profile data in the first place, by using the frequencies. This is sufficient except when one is inlined, which is why I have the special handling in the inliner itself. > > So I guess you can just check functions with 0 count and non-0 count callers > and initialize their guessed profile. > Some capping will probably be needed to not propagate insanely large numbers.. But at profile read time we don't have access to the inter-module calls. Presumably having guessed profiles for these routines should help the O2 profile-use case as well (i.e. better optimized out-of-line copy), so I wouldn't want to limit it to IPO or LIPO compiles where we can identify inter-module call counts at some point in the compilation. > > Since new direct calls can be discovered later, inline may want to do that > again each time it inlines non-0 count call of COMDAT with 0 count... How about an approach like this: - Invoke init_and_estimate_bb_frequencies as I am doing to guess the profiles at profile read time for functions with 0 counts. - At inline time, invoke some kind of freqs_to_counts routine for any 0-count routine that is reached by non-zero call edges. It would take the sum of all incoming call edge counts and synthesize counts for the bbs using the guessed profile frequencies applied earlier by init_and_estimate_bb_frequencies. Then the inliner can do its normal bb count scaling. Does that seem like a reasonable approach? There is one other fix in this patch: - The clone_inlined_nodes/update_noncloned_frequencies changes below are handling a different case: 0-count call edge in this module, with non-zero callee node count due to calls from other modules. It will allow update_noncloned_frequencies to scale down the edge counts in callee's cloned call tree. This was a fix I made for the callgraph-based linker plugin function reordering, and not splitting (since it is using both the node and edge weights to make ordering decisions). Here's a description of the issue when I was debugging it: ---- In this case, because the callee we are inlining does not have any other callers and is not external, we call update_noncloned_frequencies from clone_inlined_nodes instead of creating a clone. This routine does not attempt to scale the outgoing edge weight counters on the callee, since the assumption must be that there are no other callers so all the weight is attributed to the current edge that we are inlining. In this case this is clearly not correct, because the caller's count is 0. I'm assuming that this is happening because the callee we are inlining is a comdat, so its non-zero weights must have come from a different module. It seems like update_noncloned_frequencies should go ahead and scale the counts too. ---- For the above case, I think the right place to fix this is probably during clone_inlined_nodes/update_noncloned_frequencies, as scaling is handled by cgraph_clone_node in the case where we need cloning (also called from clone_inlined_nodes). Teresa > > Honza >> >> Index: tree-inline.c >> =================================================================== >> --- tree-inline.c (revision 201644) >> +++ tree-inline.c (working copy) >> @@ -1502,7 +1502,7 @@ remap_gimple_stmt (gimple stmt, copy_body_data *id >> >> static basic_block >> copy_bb (copy_body_data *id, basic_block bb, int frequency_scale, >> - gcov_type count_scale) >> + gcov_type count_scale, gcov_type freq_to_count_scale) >> { >> gimple_stmt_iterator gsi, copy_gsi, seq_gsi; >> basic_block copy_basic_block; >> @@ -1519,7 +1519,13 @@ copy_bb (copy_body_data *id, basic_block bb, int f >> basic_block_info automatically. */ >> copy_basic_block = create_basic_block (NULL, (void *) 0, >> (basic_block) prev->aux); >> - copy_basic_block->count = apply_scale (bb->count, count_scale); >> + copy_basic_block->count >> + = (count_scale >> + ? apply_scale (bb->count, count_scale) >> + /* When the callee bb counts were all zero (e.g. this was a COMDAT >> + that didn't get profile counts) then we compute the new bb counts >> + via the statically-estimated frequencies. */ >> + : RDIV ((gcov_type)bb->frequency * freq_to_count_scale, BB_FREQ_MAX)); >> >> /* We are going to rebuild frequencies from scratch. These values >> have just small importance to drive canonicalize_loop_headers. */ >> @@ -1888,7 +1894,8 @@ update_ssa_across_abnormal_edges (basic_block bb, >> debug stmts are left after a statement that must end the basic block. */ >> >> static bool >> -copy_edges_for_bb (basic_block bb, gcov_type count_scale, basic_block ret_bb, >> +copy_edges_for_bb (basic_block bb, gcov_type count_scale, >> + basic_block ret_bb, >> bool can_make_abnormal_goto) >> { >> basic_block new_bb = (basic_block) bb->aux; >> @@ -1912,7 +1919,14 @@ static bool >> && old_edge->dest->aux != EXIT_BLOCK_PTR) >> flags |= EDGE_FALLTHRU; >> new_edge = make_edge (new_bb, (basic_block) old_edge->dest->aux, flags); >> - new_edge->count = apply_scale (old_edge->count, count_scale); >> + basic_block new_src_bb = (basic_block) old_edge->src->aux; >> + new_edge->count >> + = (count_scale >> + ? apply_scale (old_edge->count, count_scale) >> + // The bb counts have already been scaled with >> freq_to_count_scale >> + // when that is non-zero, so just scale that new bb count by >> + // the edge probability. >> + : apply_probability (new_src_bb->count, old_edge->probability)); >> new_edge->probability = old_edge->probability; >> } >> >> @@ -2282,7 +2296,8 @@ redirect_all_calls (copy_body_data * id, basic_blo >> another function. Walks FN via CFG, returns new fndecl. */ >> >> static tree >> -copy_cfg_body (copy_body_data * id, gcov_type count, int frequency_scale, >> +copy_cfg_body (copy_body_data * id, gcov_type count, >> + int frequency, int frequency_scale, >> basic_block entry_block_map, basic_block exit_block_map, >> bitmap blocks_to_copy, basic_block new_entry) >> { >> @@ -2293,15 +2308,20 @@ static tree >> basic_block bb; >> tree new_fndecl = NULL; >> bool need_debug_cleanup = false; >> - gcov_type count_scale; >> + gcov_type count_scale = 0; >> + gcov_type freq_to_count_scale = 0; >> int last; >> int incoming_frequency = 0; >> gcov_type incoming_count = 0; >> >> - if (ENTRY_BLOCK_PTR_FOR_FUNCTION (src_cfun)->count) >> - count_scale >> - = GCOV_COMPUTE_SCALE (count, >> - ENTRY_BLOCK_PTR_FOR_FUNCTION (src_cfun)->count); >> + basic_block entry_bb = ENTRY_BLOCK_PTR_FOR_FUNCTION (src_cfun); >> + if (entry_bb->count) >> + count_scale = GCOV_COMPUTE_SCALE (count, entry_bb->count); >> + /* When the callee bb counts were all zero (e.g. this was a COMDAT >> + that didn't get profile counts) then we compute the new bb counts >> + via the statically-estimated frequencies. */ >> + else if (entry_bb->frequency) >> + freq_to_count_scale = RDIV (count * frequency, entry_bb->frequency); >> else >> count_scale = REG_BR_PROB_BASE; >> >> @@ -2323,7 +2343,13 @@ static tree >> incoming_frequency += EDGE_FREQUENCY (e); >> incoming_count += e->count; >> } >> - incoming_count = apply_scale (incoming_count, count_scale); >> + incoming_count >> + = (count_scale >> + ? apply_scale (incoming_count, count_scale) >> + /* When the callee bb counts were all zero (e.g. this was a COMDAT >> + that didn't get profile counts) then we compute the >> new bb counts >> + via the statically-estimated frequencies. */ >> + : RDIV (incoming_frequency * freq_to_count_scale, BB_FREQ_MAX)); >> incoming_frequency >> = apply_scale ((gcov_type)incoming_frequency, frequency_scale); >> ENTRY_BLOCK_PTR->count = incoming_count; >> @@ -2350,7 +2376,8 @@ static tree >> FOR_EACH_BB_FN (bb, cfun_to_copy) >> if (!blocks_to_copy || bitmap_bit_p (blocks_to_copy, bb->index)) >> { >> - basic_block new_bb = copy_bb (id, bb, frequency_scale, count_scale); >> + basic_block new_bb = copy_bb (id, bb, frequency_scale, count_scale, >> + freq_to_count_scale); >> bb->aux = new_bb; >> new_bb->aux = bb; >> new_bb->loop_father = entry_block_map->loop_father; >> @@ -2364,7 +2391,8 @@ static tree >> FOR_ALL_BB_FN (bb, cfun_to_copy) >> if (!blocks_to_copy >> || (bb->index > 0 && bitmap_bit_p (blocks_to_copy, bb->index))) >> - need_debug_cleanup |= copy_edges_for_bb (bb, count_scale, exit_block_map, >> + need_debug_cleanup |= copy_edges_for_bb (bb, count_scale, >> + exit_block_map, >> can_make_abormal_goto); >> >> if (new_entry) >> @@ -2562,7 +2590,8 @@ copy_tree_body (copy_body_data *id) >> another function. */ >> >> static tree >> -copy_body (copy_body_data *id, gcov_type count, int frequency_scale, >> +copy_body (copy_body_data *id, gcov_type count, int frequency, >> + int frequency_scale, >> basic_block entry_block_map, basic_block exit_block_map, >> bitmap blocks_to_copy, basic_block new_entry) >> { >> @@ -2571,7 +2600,8 @@ static tree >> >> /* If this body has a CFG, walk CFG and copy. */ >> gcc_assert (ENTRY_BLOCK_PTR_FOR_FUNCTION (DECL_STRUCT_FUNCTION (fndecl))); >> - body = copy_cfg_body (id, count, frequency_scale, entry_block_map, >> exit_block_map, >> + body = copy_cfg_body (id, count, frequency, frequency_scale, >> + entry_block_map, exit_block_map, >> blocks_to_copy, new_entry); >> copy_debug_stmts (id); >> >> @@ -4172,7 +4202,7 @@ expand_call_inline (basic_block bb, gimple stmt, c >> function in any way before this point, as this CALL_EXPR may be >> a self-referential call; if we're calling ourselves, we need to >> duplicate our body before altering anything. */ >> - copy_body (id, bb->count, >> + copy_body (id, bb->count, bb->frequency, >> GCOV_COMPUTE_SCALE (cg_edge->frequency, CGRAPH_FREQ_BASE), >> bb, return_block, NULL, NULL); >> >> @@ -5299,8 +5329,9 @@ tree_function_versioning (tree old_decl, tree new_ >> } >> >> /* Copy the Function's body. */ >> - copy_body (&id, old_entry_block->count, REG_BR_PROB_BASE, >> - ENTRY_BLOCK_PTR, EXIT_BLOCK_PTR, blocks_to_copy, new_entry); >> + copy_body (&id, old_entry_block->count, old_entry_block->frequency, >> + REG_BR_PROB_BASE, ENTRY_BLOCK_PTR, EXIT_BLOCK_PTR, >> + blocks_to_copy, new_entry); >> >> /* Renumber the lexical scoping (non-code) blocks consecutively. */ >> number_blocks (new_decl); >> Index: predict.c >> =================================================================== >> --- predict.c (revision 201644) >> +++ predict.c (working copy) >> @@ -2976,6 +2976,24 @@ make_pass_strip_predict_hints (gcc::context *ctxt) >> return new pass_strip_predict_hints (ctxt); >> } >> >> +/* Initialize loop edges and compute estimated bb frequencies when there >> + is no profile data available. */ >> + >> +void >> +init_and_estimate_bb_frequencies (void) >> +{ >> + if (profile_status == PROFILE_READ && counts_to_freqs ()) >> + return; >> + >> + loop_optimizer_init (0); >> + add_noreturn_fake_exit_edges (); >> + mark_irreducible_loops (); >> + connect_infinite_loops_to_exit (); >> + estimate_bb_frequencies (); >> + remove_fake_exit_edges (); >> + loop_optimizer_finalize (); >> +} >> + >> /* Rebuild function frequencies. Passes are in general expected to >> maintain profile by hand, however in some cases this is not possible: >> for example when inlining several functions with loops freuqencies might run >> @@ -2986,15 +3004,7 @@ rebuild_frequencies (void) >> { >> timevar_push (TV_REBUILD_FREQUENCIES); >> if (profile_status == PROFILE_GUESSED) >> - { >> - loop_optimizer_init (0); >> - add_noreturn_fake_exit_edges (); >> - mark_irreducible_loops (); >> - connect_infinite_loops_to_exit (); >> - estimate_bb_frequencies (); >> - remove_fake_exit_edges (); >> - loop_optimizer_finalize (); >> - } >> + init_and_estimate_bb_frequencies (); >> else if (profile_status == PROFILE_READ) >> counts_to_freqs (); >> else >> Index: predict.h >> =================================================================== >> --- predict.h (revision 201644) >> +++ predict.h (working copy) >> @@ -38,6 +38,7 @@ enum prediction >> extern void predict_insn_def (rtx, enum br_predictor, enum prediction); >> extern int counts_to_freqs (void); >> extern void estimate_bb_frequencies (void); >> +extern void init_and_estimate_bb_frequencies (void); >> extern const char *predictor_name (enum br_predictor); >> extern tree build_predict_expr (enum br_predictor, enum prediction); >> extern void tree_estimate_probability (void); >> Index: profile.c >> =================================================================== >> --- profile.c (revision 201644) >> +++ profile.c (working copy) >> @@ -1305,6 +1305,12 @@ branch_prob (void) >> >> values.release (); >> free_edge_list (el); >> + >> + /* Call after setting profile_status to PROFILE_READ, will then >> + invoke counts_to_freqs and if the sum of the counts is zero, will >> + estimate the frequencies. */ >> + init_and_estimate_bb_frequencies (); >> + >> coverage_end_function (lineno_checksum, cfg_checksum); >> } >> ^L >> Index: ipa-inline-transform.c >> =================================================================== >> --- ipa-inline-transform.c (revision 201644) >> +++ ipa-inline-transform.c (working copy) >> @@ -51,7 +51,7 @@ int nfunctions_inlined; >> >> static void >> update_noncloned_frequencies (struct cgraph_node *node, >> - int freq_scale) >> + gcov_type count_scale, int freq_scale) >> { >> struct cgraph_edge *e; >> >> @@ -60,14 +60,16 @@ update_noncloned_frequencies (struct cgraph_node * >> freq_scale = 1; >> for (e = node->callees; e; e = e->next_callee) >> { >> + e->count = apply_scale (e->count, count_scale); >> e->frequency = e->frequency * (gcov_type) freq_scale / CGRAPH_FREQ_BASE; >> if (e->frequency > CGRAPH_FREQ_MAX) >> e->frequency = CGRAPH_FREQ_MAX; >> if (!e->inline_failed) >> - update_noncloned_frequencies (e->callee, freq_scale); >> + update_noncloned_frequencies (e->callee, count_scale, freq_scale); >> } >> for (e = node->indirect_calls; e; e = e->next_callee) >> { >> + e->count = apply_scale (e->count, count_scale); >> e->frequency = e->frequency * (gcov_type) freq_scale / CGRAPH_FREQ_BASE; >> if (e->frequency > CGRAPH_FREQ_MAX) >> e->frequency = CGRAPH_FREQ_MAX; >> @@ -169,7 +171,13 @@ clone_inlined_nodes (struct cgraph_edge *e, bool d >> } >> duplicate = false; >> e->callee->symbol.externally_visible = false; >> - update_noncloned_frequencies (e->callee, e->frequency); >> + // In the case of a COMDAT, the callee's count may be from other >> + // modules, and we need to scale it for the current module's calls >> + // (e.g. e->count may be 0 despite e->callee->count > 0). >> + gcov_type count_scale = REG_BR_PROB_BASE; >> + if (e->callee->count > e->count) >> + count_scale = GCOV_COMPUTE_SCALE (e->count, e->callee->count); >> + update_noncloned_frequencies (e->callee, count_scale, e->frequency); >> } >> else >> { -- Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition 2013-08-19 15:34 ` Teresa Johnson @ 2013-08-21 15:31 ` Jan Hubicka 0 siblings, 0 replies; 62+ messages in thread From: Jan Hubicka @ 2013-08-21 15:31 UTC (permalink / raw) To: Teresa Johnson Cc: Jan Hubicka, Bernhard Reutner-Fischer, gcc-patches, Steven Bosscher, Jeff Law, marxin.liska, Sriraman Tallam > > > > Because offline COMDAT functoin will be porduced for every COMDAT used, I think > > it is bad to porduce any COMDAT (or any reachable function via calls with non-0 > > count) that has empty profile (either because it got lost by COMDAT merging > > or because of reading mismatch). > > The approach this patch takes is to simply treat those functions the > same as we would if we didn't feed back profile data in the first > place, by using the frequencies. This is sufficient except when one is > inlined, which is why I have the special handling in the inliner > itself. Yes, my orignal plan was to have per-function profile_status that specify if profile is read, guessed or absent and do function local decision sanely with each setting. Here we read the function, we set profile to READ (with all counts being 0). We should drop it to GUESSED when we see that there are non-0 count edges calling the function in question and probably we should see if it is obviously hot (i.e. reachable by a hot call) and promote its function profile to HOT then to get code placement less bad... > > > > Since new direct calls can be discovered later, inline may want to do that > > again each time it inlines non-0 count call of COMDAT with 0 count... > > How about an approach like this: > - Invoke init_and_estimate_bb_frequencies as I am doing to guess the > profiles at profile read time for functions with 0 counts. I see, here we are out of sync. We always used to go with estimated frequencies for functions with 0 counts, but it seems that this code broke when prediction was moved before profiling. (we also should keep edge probabilities from predict.c in that case) The esitmated profile is already there before reading the profile in, so we only do not want to overwrite it. Does the following work for you? Index: tree-profile.c =================================================================== --- tree-profile.c (revision 201838) +++ tree-profile.c (working copy) @@ -604,6 +604,34 @@ pop_cfun (); } + /* See if 0 count function has non-0 count callers. In this case we + lost some profile. Drop its function profile to PROFILE_GUESSED. */ + FOR_EACH_DEFINED_FUNCTION (node) + { + struct cgraph_edge *e; + bool called = false; + if (node->count) + continue; + for (e = node->callers; e; e = e->next_caller) + { + if (e->count) + called = true; + if (cgraph_maybe_hot_edge_p (e)) + break; + } + if (e || called + && profile_status_for_function + (DECL_STRUCT_FUNCTION (node->symbol.decl)) == PROFILE_READ) + { + if (dump_file) + fprintf (stderr, "Dropping 0 profile for %s/%i.%s based on calls.\n", + cgraph_node_name (node), node->symbol.order, + e ? "function is hot" : "function is normal"); + profile_status_for_function (DECL_STRUCT_FUNCTION (node->symbol.decl)) + = (flag_guess_branch_prob ? PROFILE_GUESSED : PROFILE_ABSENT); + node->frequency = e ? NODE_FREQUENCY_HOT : NODE_FREQUENCY_NORMAL; + } + } del_node_map(); return 0; Index: predict.c =================================================================== --- predict.c (revision 201838) +++ predict.c (working copy) @@ -2715,6 +2715,9 @@ gcov_type count_max, true_count_max = 0; basic_block bb; + if (!ENTRY_BLOCK_PTR->count) + return 0; + FOR_BB_BETWEEN (bb, ENTRY_BLOCK_PTR, NULL, next_bb) true_count_max = MAX (bb->count, true_count_max); > - At inline time, invoke some kind of freqs_to_counts routine for any > 0-count routine that is reached by non-zero call edges. It would take We should not need that since frequencies ought to be there. > the sum of all incoming call edge counts and synthesize counts for the > bbs using the guessed profile frequencies applied earlier by > init_and_estimate_bb_frequencies. Then the inliner can do its normal > bb count scaling. Yes, i guess we should go this way. Still we will need to watch overly large values. Recrusive inlining can probably easily produce quite a nonsense here. We wil also need to solve problem that in this case cgraph edges will have 0 profile. We probably want to play the game there and just do the scaling for edge count, since IPA passes probably do not want to care about partial profiles. > > Does that seem like a reasonable approach? > > There is one other fix in this patch: > - The clone_inlined_nodes/update_noncloned_frequencies changes below > are handling a different case: 0-count call edge in this module, with > non-zero callee node count due to calls from other modules. It will > allow update_noncloned_frequencies to scale down the edge counts in > callee's cloned call tree. This was a fix I made for the > callgraph-based linker plugin function reordering, and not splitting > (since it is using both the node and edge weights to make ordering > decisions). Here's a description of the issue when I was debugging it: Yes, it seems resonable. I did not really care about comdats at a time I was writting this function.. > >> Index: ipa-inline-transform.c > >> =================================================================== > >> --- ipa-inline-transform.c (revision 201644) > >> +++ ipa-inline-transform.c (working copy) > >> @@ -51,7 +51,7 @@ int nfunctions_inlined; > >> > >> static void > >> update_noncloned_frequencies (struct cgraph_node *node, > >> - int freq_scale) > >> + gcov_type count_scale, int freq_scale) > >> { > >> struct cgraph_edge *e; > >> > >> @@ -60,14 +60,16 @@ update_noncloned_frequencies (struct cgraph_node * > >> freq_scale = 1; > >> for (e = node->callees; e; e = e->next_callee) > >> { > >> + e->count = apply_scale (e->count, count_scale); > >> e->frequency = e->frequency * (gcov_type) freq_scale / CGRAPH_FREQ_BASE; > >> if (e->frequency > CGRAPH_FREQ_MAX) > >> e->frequency = CGRAPH_FREQ_MAX; > >> if (!e->inline_failed) > >> - update_noncloned_frequencies (e->callee, freq_scale); > >> + update_noncloned_frequencies (e->callee, count_scale, freq_scale); > >> } > >> for (e = node->indirect_calls; e; e = e->next_callee) > >> { > >> + e->count = apply_scale (e->count, count_scale); > >> e->frequency = e->frequency * (gcov_type) freq_scale / CGRAPH_FREQ_BASE; > >> if (e->frequency > CGRAPH_FREQ_MAX) > >> e->frequency = CGRAPH_FREQ_MAX; > >> @@ -169,7 +171,13 @@ clone_inlined_nodes (struct cgraph_edge *e, bool d > >> } > >> duplicate = false; > >> e->callee->symbol.externally_visible = false; > >> - update_noncloned_frequencies (e->callee, e->frequency); > >> + // In the case of a COMDAT, the callee's count may be from other > >> + // modules, and we need to scale it for the current module's calls > >> + // (e.g. e->count may be 0 despite e->callee->count > 0). > >> + gcov_type count_scale = REG_BR_PROB_BASE; > >> + if (e->callee->count > e->count) > >> + count_scale = GCOV_COMPUTE_SCALE (e->count, e->callee->count); > >> + update_noncloned_frequencies (e->callee, count_scale, e->frequency); Please fix the comment to the usual style and go ahead with this change. Thanks, Honza > >> } > >> else > >> { > > > > -- > Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413 ^ permalink raw reply [flat|nested] 62+ messages in thread
end of thread, other threads:[~2013-10-03 23:37 UTC | newest] Thread overview: 62+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2013-08-01 16:32 [PATCH] Sanitize block partitioning under -freorder-blocks-and-partition Teresa Johnson 2013-08-02 11:22 ` Bernhard Reutner-Fischer 2013-08-02 14:51 ` Teresa Johnson 2013-08-02 15:05 ` Jan Hubicka 2013-08-02 23:05 ` Steven Bosscher 2013-08-03 4:53 ` Teresa Johnson 2013-08-03 4:48 ` Teresa Johnson 2013-08-05 13:36 ` Teresa Johnson 2013-08-05 14:11 ` Jan Hubicka 2013-08-05 14:57 ` Teresa Johnson 2013-08-06 3:01 ` Teresa Johnson [not found] ` <20130808222332.GA31755@kam.mff.cuni.cz> 2013-08-08 23:04 ` Teresa Johnson 2013-08-09 9:58 ` Jan Hubicka 2013-08-09 14:38 ` Teresa Johnson 2013-08-09 15:28 ` Jan Hubicka 2013-08-09 15:54 ` Martin Liška 2013-08-09 21:03 ` Teresa Johnson 2013-08-09 21:02 ` Teresa Johnson 2013-08-09 22:43 ` Jan Hubicka 2013-08-11 12:21 ` Jan Hubicka 2013-08-11 13:25 ` Teresa Johnson 2013-08-11 15:55 ` Martin Liška 2013-08-11 17:55 ` Jan Hubicka 2013-08-11 21:05 ` Jan Hubicka 2013-08-17 15:54 ` Teresa Johnson 2013-08-17 21:02 ` Jan Hubicka 2013-08-19 13:51 ` Teresa Johnson 2013-08-19 15:16 ` Jan Hubicka 2013-08-19 17:48 ` Teresa Johnson 2013-08-19 19:56 ` Martin Liška 2013-08-27 18:12 ` Teresa Johnson 2013-08-28 16:59 ` Jan Hubicka 2013-08-28 18:35 ` Teresa Johnson 2013-08-30 7:17 ` Teresa Johnson 2013-08-30 9:16 ` Jan Hubicka 2013-08-30 15:13 ` Teresa Johnson 2013-08-30 15:28 ` Jan Hubicka 2013-08-30 15:54 ` Teresa Johnson 2013-08-30 21:56 ` Rong Xu 2013-08-31 16:20 ` Jan Hubicka 2013-08-31 23:40 ` Jan Hubicka 2013-09-24 8:07 ` Teresa Johnson 2013-09-24 13:44 ` Jan Hubicka 2013-09-24 19:06 ` Teresa Johnson 2013-09-26 20:55 ` Rong Xu 2013-09-26 22:23 ` Jan Hubicka 2013-09-26 22:54 ` Rong Xu 2013-09-24 18:28 ` Jan Hubicka 2013-09-24 18:51 ` Teresa Johnson 2013-09-25 23:10 ` Teresa Johnson 2013-09-26 8:44 ` Teresa Johnson 2013-09-26 22:26 ` Jan Hubicka 2013-09-27 14:50 ` Teresa Johnson 2013-09-29 17:34 ` Teresa Johnson 2013-10-02 16:19 ` Jan Hubicka 2013-10-02 17:55 ` Teresa Johnson 2013-10-02 18:10 ` Jan Hubicka 2013-10-03 13:42 ` Teresa Johnson 2013-10-03 23:37 ` Teresa Johnson 2013-10-01 17:36 ` Teresa Johnson 2013-08-19 15:34 ` Teresa Johnson 2013-08-21 15:31 ` Jan Hubicka
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).