* [PATCH] AutoFDO patch for trunk
@ 2014-05-07 21:24 Dehao Chen
2014-05-07 21:47 ` Xinliang David Li
` (2 more replies)
0 siblings, 3 replies; 25+ messages in thread
From: Dehao Chen @ 2014-05-07 21:24 UTC (permalink / raw)
To: GCC Patches; +Cc: Jan Hubicka, David Li
[-- Attachment #1: Type: text/plain, Size: 1635 bytes --]
Hi,
I'm planning to port the AutoFDO patch upstream. Attached is the
prepared patch. You can also find the patch in
http://codereview.appspot.com/99010043
I've tested the patch with SPECCPU2006. For the CINT2006 benchmarks,
the speedup comparison between O2, FDO and AutoFDO is as follows:
Reference: o2
(1): auto_fdo
(2): fdo
Benchmark Base:Reference (1) (2)
-----------------------------------------------------------------
spec/2006/int/C++/471.omnetpp 23.18 +3.11% +5.09%
spec/2006/int/C++/473.astar 21.15 +6.79% +9.80%
spec/2006/int/C++/483.xalancbmk 36.68 +11.56% +14.47%
spec/2006/int/C/400.perlbench 34.57 +6.59% +18.56%
spec/2006/int/C/401.bzip2 23.17 +0.95% +2.49%
spec/2006/int/C/403.gcc 32.33 +8.27% +9.76%
spec/2006/int/C/429.mcf 42.13 +4.72% +5.23%
spec/2006/int/C/445.gobmk 26.53 -1.39% +0.05%
spec/2006/int/C/456.hmmer 23.72 +7.12% +7.87%
spec/2006/int/C/458.sjeng 26.17 +4.65% +6.04%
spec/2006/int/C/462.libquantum 57.23 +4.04% +1.42%
spec/2006/int/C/464.h264ref 46.3 +1.07% +8.97%
geometric mean +4.73% +7.36%
The majority of the performance difference between AutoFDO and FDO
comes from the lack of instruction level discriminator support. Cary
Coutant is planning to port that patch upstream too.
Please let me know if you have any question about this patch, and
thanks in advance for reviewing such a huge patch.
Dehao
[-- Attachment #2: diff.txt --]
[-- Type: text/plain, Size: 76989 bytes --]
Index: gcc/cfghooks.c
===================================================================
--- gcc/cfghooks.c (revision 210180)
+++ gcc/cfghooks.c (working copy)
@@ -833,6 +833,9 @@ make_forwarder_block (basic_block bb, bool (*redir
fallthru = split_block_after_labels (bb);
dummy = fallthru->src;
+ dummy->count = 0;
+ dummy->frequency = 0;
+ fallthru->count = 0;
bb = fallthru->dest;
/* Redirect back edges we want to keep. */
@@ -842,20 +845,13 @@ make_forwarder_block (basic_block bb, bool (*redir
if (redirect_edge_p (e))
{
+ dummy->frequency += EDGE_FREQUENCY (e);
+ dummy->count += e->count;
+ fallthru->count += e->count;
ei_next (&ei);
continue;
}
- dummy->frequency -= EDGE_FREQUENCY (e);
- dummy->count -= e->count;
- if (dummy->frequency < 0)
- dummy->frequency = 0;
- if (dummy->count < 0)
- dummy->count = 0;
- fallthru->count -= e->count;
- if (fallthru->count < 0)
- fallthru->count = 0;
-
e_src = e->src;
jump = redirect_edge_and_branch_force (e, bb);
if (jump != NULL)
Index: gcc/loop-unroll.c
===================================================================
--- gcc/loop-unroll.c (revision 210180)
+++ gcc/loop-unroll.c (working copy)
@@ -998,6 +998,13 @@ decide_unroll_runtime_iterations (struct loop *loo
return;
}
+ /* In AutoFDO, the profile is not accurate. If the calculated trip count
+ is larger than the header count, then the profile is not accurate
+ enough to make correct unroll decisions. */
+ if (flag_auto_profile
+ && expected_loop_iterations (loop) > loop->header->count)
+ return;
+
/* Success; now force nunroll to be power of 2, as we are unable to
cope with overflows in computation of number of iterations. */
for (i = 1; 2 * i <= nunroll; i *= 2)
Index: gcc/passes.def
===================================================================
--- gcc/passes.def (revision 210180)
+++ gcc/passes.def (working copy)
@@ -91,6 +91,7 @@ along with GCC; see the file COPYING3. If not see
NEXT_PASS (pass_rebuild_cgraph_edges);
NEXT_PASS (pass_inline_parameters);
POP_INSERT_PASSES ()
+ NEXT_PASS (pass_ipa_auto_profile);
NEXT_PASS (pass_ipa_free_inline_summary);
NEXT_PASS (pass_ipa_tree_profile);
PUSH_INSERT_PASSES_WITHIN (pass_ipa_tree_profile)
Index: gcc/dwarf2out.c
===================================================================
--- gcc/dwarf2out.c (revision 210180)
+++ gcc/dwarf2out.c (working copy)
@@ -2481,6 +2481,40 @@ const struct gcc_debug_hooks dwarf2_debug_hooks =
1, /* start_end_main_source_file */
TYPE_SYMTAB_IS_DIE /* tree_type_symtab_field */
};
+
+const struct gcc_debug_hooks auto_profile_debug_hooks =
+{
+ debug_nothing_charstar,
+ debug_nothing_charstar,
+ debug_nothing_void,
+ debug_nothing_int_charstar,
+ debug_nothing_int_charstar,
+ debug_nothing_int_charstar,
+ debug_nothing_int,
+ debug_nothing_int_int, /* begin_block */
+ debug_nothing_int_int, /* end_block */
+ dwarf2out_ignore_block, /* ignore_block */
+ debug_nothing_int_charstar_int_bool, /* source_line */
+ debug_nothing_int_charstar, /* begin_prologue */
+ debug_nothing_int_charstar, /* end_prologue */
+ debug_nothing_int_charstar, /* begin_epilogue */
+ debug_nothing_int_charstar, /* end_epilogue */
+ debug_nothing_tree, /* begin_function */
+ debug_nothing_int, /* end_function */
+ debug_nothing_tree, /* function_decl */
+ debug_nothing_tree, /* global_decl */
+ debug_nothing_tree_int, /* type_decl */
+ debug_nothing_tree_tree_tree_bool, /* imported_module_or_decl */
+ debug_nothing_tree, /* deferred_inline_function */
+ debug_nothing_tree, /* outlining_inline_function */
+ debug_nothing_rtx, /* label */
+ debug_nothing_int, /* handle_pch */
+ debug_nothing_rtx, /* var_location */
+ debug_nothing_void, /* switch_text_section */
+ debug_nothing_tree_tree, /* set_name */
+ 0, /* start_end_main_source_file */
+ TYPE_SYMTAB_IS_ADDRESS /* tree_type_symtab_field */
+};
\f
/* NOTE: In the comments in this file, many references are made to
"Debugging Information Entries". This term is abbreviated as `DIE'
@@ -16799,10 +16833,9 @@ add_src_coords_attributes (dw_die_ref die, tree de
static void
add_linkage_name (dw_die_ref die, tree decl)
{
- if (debug_info_level > DINFO_LEVEL_TERSE
+ if (debug_info_level > DINFO_LEVEL_NONE
&& (TREE_CODE (decl) == FUNCTION_DECL || TREE_CODE (decl) == VAR_DECL)
&& TREE_PUBLIC (decl)
- && !DECL_ABSTRACT (decl)
&& !(TREE_CODE (decl) == VAR_DECL && DECL_REGISTER (decl))
&& die->die_tag != DW_TAG_member)
{
Index: gcc/timevar.def
===================================================================
--- gcc/timevar.def (revision 210180)
+++ gcc/timevar.def (working copy)
@@ -87,6 +87,7 @@ DEFTIMEVAR (TV_WHOPR_PARTITIONING , "whopr part
DEFTIMEVAR (TV_WHOPR_LTRANS , "whopr ltrans")
DEFTIMEVAR (TV_IPA_REFERENCE , "ipa reference")
DEFTIMEVAR (TV_IPA_PROFILE , "ipa profile")
+DEFTIMEVAR (TV_IPA_AUTOFDO , "auto profile")
DEFTIMEVAR (TV_IPA_PURE_CONST , "ipa pure const")
DEFTIMEVAR (TV_IPA_PTA , "ipa points-to")
DEFTIMEVAR (TV_IPA_SRA , "ipa SRA")
Index: gcc/gcov-io.h
===================================================================
--- gcc/gcov-io.h (revision 210180)
+++ gcc/gcov-io.h (working copy)
@@ -237,6 +237,9 @@ typedef unsigned HOST_WIDEST_INT gcov_type_unsigne
#define GCOV_TAG_PROGRAM_SUMMARY ((gcov_unsigned_t)0xa3000000)
#define GCOV_TAG_SUMMARY_LENGTH(NUM) \
(1 + GCOV_COUNTERS_SUMMABLE * (10 + 3 * 2) + (NUM) * 5)
+#define GCOV_TAG_AFDO_FILE_NAMES ((gcov_unsigned_t)0xaa000000)
+#define GCOV_TAG_AFDO_FUNCTION ((gcov_unsigned_t)0xac000000)
+#define GCOV_TAG_AFDO_WORKING_SET ((gcov_unsigned_t)0xaf000000)
/* Counters that are collected. */
Index: gcc/cfg-flags.def
===================================================================
--- gcc/cfg-flags.def (revision 210180)
+++ gcc/cfg-flags.def (working copy)
@@ -93,6 +93,9 @@ DEF_BASIC_BLOCK_FLAG(VISITED, 13)
demand, and is available after calling compute_transaction_bits(). */
DEF_BASIC_BLOCK_FLAG(IN_TRANSACTION, 14)
+/* Set on blocks that has been annotated during AutoFDO profile
+ attribution. */
+DEF_BASIC_BLOCK_FLAG(ANNOTATED, 15)
#endif
#ifdef DEF_EDGE_FLAG
@@ -177,6 +180,9 @@ DEF_EDGE_FLAG(TM_UNINSTRUMENTED, 15)
/* Abort (over) edge out of a GIMPLE_TRANSACTION statement. */
DEF_EDGE_FLAG(TM_ABORT, 16)
+/* Annotated during AutoFDO profile attribution. */
+DEF_EDGE_FLAG(ANNOTATED, 17)
+
#endif
/*
Index: gcc/ira-int.h
===================================================================
--- gcc/ira-int.h (revision 210180)
+++ gcc/ira-int.h (working copy)
@@ -41,9 +41,10 @@ along with GCC; see the file COPYING3. If not see
analogous to REG_FREQ_FROM_BB. When optimizing for size, or
profile driven feedback is available and the function is never
executed, frequency is always equivalent. Otherwise rescale the
- edge frequency. */
+ edge frequency. For AutoFDO, even if a function is not sampled,
+ it can still be executed, thus frequency rescale is still used. */
#define REG_FREQ_FROM_EDGE_FREQ(freq) \
- (optimize_size || (flag_branch_probabilities \
+ (optimize_size || (flag_branch_probabilities && !flag_auto_profile \
&& !ENTRY_BLOCK_PTR_FOR_FN (cfun)->count) \
? REG_FREQ_MAX : (freq * REG_FREQ_MAX / BB_FREQ_MAX) \
? (freq * REG_FREQ_MAX / BB_FREQ_MAX) : 1)
Index: gcc/toplev.c
===================================================================
--- gcc/toplev.c (revision 210180)
+++ gcc/toplev.c (working copy)
@@ -78,6 +78,7 @@ along with GCC; see the file COPYING3. If not see
#include "diagnostic-color.h"
#include "context.h"
#include "pass_manager.h"
+#include "auto-profile.h"
#include "optabs.h"
#if defined(DBX_DEBUGGING_INFO) || defined(XCOFF_DEBUGGING_INFO)
@@ -660,6 +661,10 @@ compile_file (void)
targetm.asm_out.output_ident (ident_str);
}
+ /* Auto profile finalization. */
+ if (flag_auto_profile)
+ end_auto_profile ();
+
/* Invoke registered plugin callbacks. */
invoke_plugin_callbacks (PLUGIN_FINISH_UNIT, NULL);
@@ -1418,6 +1423,11 @@ process_options (void)
error ("target system does not support the \"%s\" debug format",
debug_type_names[write_symbols]);
+#ifdef DWARF2_DEBUGGING_INFO
+ if (flag_auto_profile && debug_info_level == DINFO_LEVEL_NONE)
+ debug_hooks = &auto_profile_debug_hooks;
+#endif
+
/* We know which debug output will be used so we can set flag_var_tracking
and flag_var_tracking_uninit if the user has not specified them. */
if (debug_info_level < DINFO_LEVEL_NORMAL
Index: gcc/tree-profile.c
===================================================================
--- gcc/tree-profile.c (revision 210180)
+++ gcc/tree-profile.c (working copy)
@@ -696,7 +696,7 @@ bool
pass_ipa_tree_profile::gate (function *)
{
/* When profile instrumentation, use or test coverage shall be performed. */
- return (!in_lto_p
+ return (!in_lto_p && !flag_auto_profile
&& (flag_branch_probabilities || flag_test_coverage
|| profile_arc_flag));
}
Index: gcc/tree-inline.c
===================================================================
--- gcc/tree-inline.c (revision 210180)
+++ gcc/tree-inline.c (working copy)
@@ -1977,9 +1977,10 @@ copy_edges_for_bb (basic_block bb, gcov_type count
edge new_edge;
flags = old_edge->flags;
+ flags &= (~EDGE_ANNOTATED);
/* Return edges do get a FALLTHRU flag when the get inlined. */
- if (old_edge->dest->index == EXIT_BLOCK && !old_edge->flags
+ if (old_edge->dest->index == EXIT_BLOCK && !flags
&& old_edge->dest->aux != EXIT_BLOCK_PTR_FOR_FN (cfun))
flags |= EDGE_FALLTHRU;
new_edge = make_edge (new_bb, (basic_block) old_edge->dest->aux, flags);
@@ -2191,6 +2192,9 @@ initialize_cfun (tree new_fndecl, tree callee_fnde
else
count_scale = REG_BR_PROB_BASE;
+ if (count_scale > REG_BR_PROB_BASE)
+ count_scale = REG_BR_PROB_BASE;
+
/* Register specific tree functions. */
gimple_register_cfg_hooks ();
@@ -2452,6 +2456,9 @@ copy_cfg_body (copy_body_data * id, gcov_type coun
else
count_scale = REG_BR_PROB_BASE;
+ if (count_scale > REG_BR_PROB_BASE)
+ count_scale = REG_BR_PROB_BASE;
+
/* Register specific tree functions. */
gimple_register_cfg_hooks ();
Index: gcc/predict.c
===================================================================
--- gcc/predict.c (revision 210180)
+++ gcc/predict.c (working copy)
@@ -161,7 +161,7 @@ set_hot_bb_threshold (gcov_type min)
/* Return TRUE if frequency FREQ is considered to be hot. */
-static inline bool
+bool
maybe_hot_count_p (struct function *fun, gcov_type count)
{
if (fun && profile_status_for_fn (fun) != PROFILE_READ)
Index: gcc/debug.h
===================================================================
--- gcc/debug.h (revision 210180)
+++ gcc/debug.h (working copy)
@@ -169,6 +169,7 @@ extern const struct gcc_debug_hooks dbx_debug_hook
extern const struct gcc_debug_hooks sdb_debug_hooks;
extern const struct gcc_debug_hooks xcoff_debug_hooks;
extern const struct gcc_debug_hooks dwarf2_debug_hooks;
+extern const struct gcc_debug_hooks auto_profile_debug_hooks;
extern const struct gcc_debug_hooks vmsdbg_debug_hooks;
/* Dwarf2 frame information. */
Index: gcc/basic-block.h
===================================================================
--- gcc/basic-block.h (revision 210180)
+++ gcc/basic-block.h (working copy)
@@ -718,6 +718,7 @@ extern struct edge_list *pre_edge_rev_lcm (int, sb
extern void compute_available (sbitmap *, sbitmap *, sbitmap *, sbitmap *);
/* In predict.c */
+extern bool maybe_hot_count_p (struct function *, gcov_type);
extern bool maybe_hot_bb_p (struct function *, const_basic_block);
extern bool maybe_hot_edge_p (edge);
extern bool probably_never_executed_bb_p (struct function *, const_basic_block);
Index: gcc/bb-reorder.c
===================================================================
--- gcc/bb-reorder.c (revision 210180)
+++ gcc/bb-reorder.c (working copy)
@@ -2663,7 +2663,7 @@ pass_partition_blocks::gate (function *fun)
user defined section attributes. Don't call it if either case
arises. */
return (flag_reorder_blocks_and_partition
- && optimize
+ && optimize &&!flag_auto_profile
/* See gate_handle_reorder_blocks. We should not partition if
we are going to omit the reordering. */
&& optimize_function_for_speed_p (fun)
Index: gcc/coverage.c
===================================================================
--- gcc/coverage.c (revision 210180)
+++ gcc/coverage.c (working copy)
@@ -54,6 +54,7 @@ along with GCC; see the file COPYING3. If not see
#include "intl.h"
#include "filenames.h"
#include "target.h"
+#include "auto-profile.h"
#include "gcov-io.h"
#include "gcov-io.c"
@@ -1175,7 +1176,9 @@ coverage_init (const char *filename)
bbg_file_stamp = local_tick;
- if (flag_branch_probabilities)
+ if (flag_auto_profile)
+ init_auto_profile ();
+ else if (flag_branch_probabilities)
read_counts_file ();
/* Name of bbg file. */
Index: gcc/profile.c
===================================================================
--- gcc/profile.c (revision 210180)
+++ gcc/profile.c (working copy)
@@ -106,6 +106,12 @@ static int total_num_times_called;
static int total_hist_br_prob[20];
static int total_num_branches;
+void add_working_set (gcov_working_set_t *set) {
+ int i = 0;
+ for (; i < NUM_GCOV_WORKING_SETS; i++)
+ gcov_working_sets[i] = set[i];
+}
+
/* Forward declarations. */
static void find_spanning_tree (struct edge_list *);
Index: gcc/profile.h
===================================================================
--- gcc/profile.h (revision 210180)
+++ gcc/profile.h (working copy)
@@ -47,6 +47,7 @@ extern void init_node_map (bool);
extern void del_node_map (void);
extern void get_working_sets (void);
+extern void add_working_set (gcov_working_set_t *);
/* In predict.c. */
extern gcov_type get_hot_bb_threshold (void);
Index: gcc/regs.h
===================================================================
--- gcc/regs.h (revision 210180)
+++ gcc/regs.h (working copy)
@@ -137,6 +137,7 @@ extern size_t reg_info_p_size;
frequency. */
#define REG_FREQ_FROM_BB(bb) (optimize_size \
|| (flag_branch_probabilities \
+ && !flag_auto_profile \
&& !ENTRY_BLOCK_PTR_FOR_FN (cfun)->count) \
? REG_FREQ_MAX \
: ((bb)->frequency * REG_FREQ_MAX / BB_FREQ_MAX)\
Index: gcc/common.opt
===================================================================
--- gcc/common.opt (revision 210180)
+++ gcc/common.opt (working copy)
@@ -878,6 +878,16 @@ fauto-inc-dec
Common Report Var(flag_auto_inc_dec) Init(1)
Generate auto-inc/dec instructions
+fauto-profile
+Common Report Var(flag_auto_profile) Optimization
+Use sample profile information for call graph node weights. The default
+profile file is fbdata.afdo in 'pwd'.
+
+fauto-profile=
+Common Joined RejectNegative Var(auto_profile_file)
+Use sample profile information for call graph node weights. The profile
+file is specified in the argument.
+
; -fcheck-bounds causes gcc to generate array bounds checks.
; For C, C++ and ObjC: defaults off.
; For Java: defaults to on.
Index: gcc/Makefile.in
===================================================================
--- gcc/Makefile.in (revision 210180)
+++ gcc/Makefile.in (working copy)
@@ -1163,6 +1163,7 @@ OBJS = \
alias.o \
alloc-pool.o \
auto-inc-dec.o \
+ auto-profile.o \
bb-reorder.o \
bitmap.o \
bt-load.o \
Index: gcc/opts.c
===================================================================
--- gcc/opts.c (revision 210180)
+++ gcc/opts.c (working copy)
@@ -627,6 +627,11 @@ default_options_optimization (struct gcc_options *
default_param_value (PARAM_MIN_CROSSJUMP_INSNS),
opts->x_param_values, opts_set->x_param_values);
+ if (flag_auto_profile)
+ maybe_set_param_value
+ (PARAM_EARLY_INLINER_MAX_ITERATIONS, 10,
+ opts->x_param_values, opts_set->x_param_values);
+
/* Allow default optimizations to be specified on a per-machine basis. */
maybe_default_options (opts, opts_set,
targetm_common.option_optimization_table,
@@ -1720,6 +1725,56 @@ common_handle_option (struct gcc_options *opts,
opts->x_flag_devirtualize_speculatively = false;
break;
+ case OPT_fauto_profile_:
+ opts->x_auto_profile_file = xstrdup (arg);
+ opts->x_flag_auto_profile = true;
+ value = true;
+ /* No break here - do -fauto-profile processing. */
+ case OPT_fauto_profile:
+ if (!opts_set->x_flag_branch_probabilities)
+ opts->x_flag_branch_probabilities = value;
+ if (!opts_set->x_flag_profile_values)
+ opts->x_flag_profile_values = value;
+ if (!opts_set->x_flag_unroll_loops)
+ opts->x_flag_unroll_loops = value;
+ if (!opts_set->x_flag_peel_loops)
+ opts->x_flag_peel_loops = value;
+ if (!opts_set->x_flag_tracer)
+ opts->x_flag_tracer = value;
+ if (!opts_set->x_flag_value_profile_transformations)
+ opts->x_flag_value_profile_transformations = value;
+ if (!opts_set->x_flag_inline_functions)
+ opts->x_flag_inline_functions = value;
+ if (!opts_set->x_flag_ipa_cp)
+ opts->x_flag_ipa_cp = value;
+ if (!opts_set->x_flag_ipa_cp_clone
+ && value && opts->x_flag_ipa_cp)
+ opts->x_flag_ipa_cp_clone = value;
+ if (!opts_set->x_flag_predictive_commoning)
+ opts->x_flag_predictive_commoning = value;
+ if (!opts_set->x_flag_unswitch_loops)
+ opts->x_flag_unswitch_loops = value;
+ if (!opts_set->x_flag_gcse_after_reload)
+ opts->x_flag_gcse_after_reload = value;
+ if (!opts_set->x_flag_tree_loop_vectorize
+ && !opts_set->x_flag_tree_vectorize)
+ opts->x_flag_tree_loop_vectorize = value;
+ if (!opts_set->x_flag_tree_slp_vectorize
+ && !opts_set->x_flag_tree_vectorize)
+ opts->x_flag_tree_slp_vectorize = value;
+ if (!opts_set->x_flag_vect_cost_model)
+ opts->x_flag_vect_cost_model = VECT_COST_MODEL_DYNAMIC;
+ if (!opts_set->x_flag_tree_loop_distribute_patterns)
+ opts->x_flag_tree_loop_distribute_patterns = value;
+ /* Indirect call profiling should do all useful transformations
+ speculative devirutalization does. */
+ if (!opts_set->x_flag_devirtualize_speculatively
+ && opts->x_flag_value_profile_transformations)
+ opts->x_flag_devirtualize_speculatively = false;
+ if (!opts_set->x_flag_profile_correction)
+ opts->x_flag_profile_correction = value;
+ break;
+
case OPT_fprofile_generate_:
opts->x_profile_data_prefix = xstrdup (arg);
value = true;
Index: gcc/tree-cfg.c
===================================================================
--- gcc/tree-cfg.c (revision 210180)
+++ gcc/tree-cfg.c (working copy)
@@ -1877,6 +1877,14 @@ gimple_merge_blocks (basic_block a, basic_block b)
}
}
+ /* When merging two BBs, if their counts are different, the larger
+ count is selected as the new bb count. */
+ if (flag_auto_profile && a->count != b->count)
+ {
+ a->count = MAX (a->count, b->count);
+ a->frequency = MAX (a->frequency, b->frequency);
+ }
+
/* Merge the sequences. */
last = gsi_last_bb (a);
gsi_insert_seq_after (&last, bb_seq (b), GSI_NEW_STMT);
Index: gcc/cgraphclones.c
===================================================================
--- gcc/cgraphclones.c (revision 210180)
+++ gcc/cgraphclones.c (working copy)
@@ -435,6 +435,11 @@ cgraph_clone_node (struct cgraph_node *n, tree dec
}
else
count_scale = 0;
+ /* In AutoFDO, if edge count is larger than callee's entry block
+ count, we will not update the original callee because it may
+ mistakenly mark some hot function as cold. */
+ if (flag_auto_profile && count >= n->count)
+ update_original = false;
if (update_original)
{
n->count -= count;
Index: gcc/value-prof.c
===================================================================
--- gcc/value-prof.c (revision 210180)
+++ gcc/value-prof.c (working copy)
@@ -133,11 +133,10 @@ static bool gimple_divmod_fixed_value_transform (g
static bool gimple_mod_pow2_value_transform (gimple_stmt_iterator *);
static bool gimple_mod_subtract_transform (gimple_stmt_iterator *);
static bool gimple_stringops_transform (gimple_stmt_iterator *);
-static bool gimple_ic_transform (gimple_stmt_iterator *);
/* Allocate histogram value. */
-static histogram_value
+histogram_value
gimple_alloc_histogram_value (struct function *fun ATTRIBUTE_UNUSED,
enum hist_type type, gimple stmt, tree value)
{
@@ -1286,6 +1285,9 @@ del_node_map (void)
struct cgraph_node*
find_func_by_profile_id (int profile_id)
{
+ if (flag_auto_profile)
+ return cgraph_node_for_asm (
+ get_identifier ((const char *)(size_t)profile_id));
void **val = pointer_map_contains (cgraph_node_map,
(void *)(size_t)profile_id);
if (val)
@@ -1300,7 +1302,7 @@ find_func_by_profile_id (int profile_id)
may ICE. Here we only do very minimal sanity check just to make compiler happy.
Returns true if TARGET is considered ok for call CALL_STMT. */
-static bool
+bool
check_ic_target (gimple call_stmt, struct cgraph_node *target)
{
location_t locus;
@@ -1463,7 +1465,7 @@ gimple_ic (gimple icall_stmt, struct cgraph_node *
this call to:
*/
-static bool
+bool
gimple_ic_transform (gimple_stmt_iterator *gsi)
{
gimple stmt = gsi_stmt (*gsi);
@@ -1492,8 +1494,8 @@ gimple_ic_transform (gimple_stmt_iterator *gsi)
/* The order of CHECK_COUNTER calls is important -
since check_counter can correct the third parameter
and we want to make count <= all <= bb_all. */
- if ( check_counter (stmt, "ic", &all, &bb_all, bb_all)
- || check_counter (stmt, "ic", &count, &all, all))
+ if (!flag_auto_profile && (check_counter (stmt, "ic", &all, &bb_all, bb_all)
+ || check_counter (stmt, "ic", &count, &all, all)))
{
gimple_remove_histogram_value (cfun, stmt, histogram);
return false;
Index: gcc/value-prof.h
===================================================================
--- gcc/value-prof.h (revision 210180)
+++ gcc/value-prof.h (working copy)
@@ -75,6 +75,8 @@ typedef vec<histogram_value> histogram_values;
extern void gimple_find_values_to_profile (histogram_values *);
extern bool gimple_value_profile_transformations (void);
+histogram_value gimple_alloc_histogram_value (struct function *, enum hist_type,
+ gimple stmt, tree);
histogram_value gimple_histogram_value (struct function *, gimple);
histogram_value gimple_histogram_value_of_type (struct function *, gimple,
enum hist_type);
@@ -89,6 +91,8 @@ void verify_histograms (void);
void free_histograms (void);
void stringop_block_profile (gimple, unsigned int *, HOST_WIDE_INT *);
gimple gimple_ic (gimple, struct cgraph_node *, int, gcov_type, gcov_type);
+bool check_ic_target (gimple, struct cgraph_node *);
+bool gimple_ic_transform (gimple_stmt_iterator *);
/* In tree-profile.c. */
Index: gcc/tree-pass.h
===================================================================
--- gcc/tree-pass.h (revision 210180)
+++ gcc/tree-pass.h (working copy)
@@ -452,6 +452,7 @@ extern simple_ipa_opt_pass *make_pass_ipa_lower_em
extern simple_ipa_opt_pass
*make_pass_ipa_function_and_variable_visibility (gcc::context *ctxt);
extern simple_ipa_opt_pass *make_pass_ipa_tree_profile (gcc::context *ctxt);
+extern simple_ipa_opt_pass *make_pass_ipa_auto_profile (gcc::context *ctxt);
extern simple_ipa_opt_pass *make_pass_early_local_passes (gcc::context *ctxt);
Index: gcc/cfgrtl.c
===================================================================
--- gcc/cfgrtl.c (revision 210180)
+++ gcc/cfgrtl.c (working copy)
@@ -2479,7 +2479,8 @@ rtl_verify_edges (void)
| EDGE_IRREDUCIBLE_LOOP
| EDGE_LOOP_EXIT
| EDGE_CROSSING
- | EDGE_PRESERVE)) == 0)
+ | EDGE_PRESERVE
+ | EDGE_ANNOTATED)) == 0)
n_branch++;
if (e->flags & EDGE_ABNORMAL_CALL)
Index: gcc/auto-profile.c
===================================================================
--- gcc/auto-profile.c (revision 0)
+++ gcc/auto-profile.c (revision 0)
@@ -0,0 +1,1584 @@
+/* Calculate branch probabilities, and basic block execution counts.
+ Copyright (C) 2012. Free Software Foundation, Inc.
+ Contributed by Dehao Chen (dehao@google.com)
+
+This file is part of GCC.
+
+GCC is free software; you can redistribute it and/or modify it under
+the terms of the GNU General Public License as published by the Free
+Software Foundation; either version 3, or (at your option) any later
+version.
+
+GCC is distributed in the hope that it will be useful, but WITHOUT ANY
+WARRANTY; without even the implied warranty of MERCHANTABILITY or
+FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
+for more details.
+
+You should have received a copy of the GNU General Public License
+along with GCC; see the file COPYING3. If not see
+<http://www.gnu.org/licenses/>. */
+
+/* Read and annotate call graph profile from the auto profile data
+ file. */
+
+#include <string.h>
+#include <map>
+#include <vector>
+#include <set>
+
+#include "config.h"
+#include "system.h"
+#include "coretypes.h"
+#include "tree.h"
+#include "tree-pass.h"
+#include "flags.h"
+#include "basic-block.h"
+#include "diagnostic-core.h"
+#include "gcov-io.h"
+#include "input.h"
+#include "profile.h"
+#include "langhooks.h"
+#include "opts.h"
+#include "tree-pass.h"
+#include "cfgloop.h"
+#include "tree-ssa-alias.h"
+#include "tree-cfg.h"
+#include "tree-cfgcleanup.h"
+#include "tree-ssa-operands.h"
+#include "tree-into-ssa.h"
+#include "internal-fn.h"
+#include "is-a.h"
+#include "gimple-expr.h"
+#include "gimple.h"
+#include "gimple-iterator.h"
+#include "gimple-ssa.h"
+#include "cgraph.h"
+#include "value-prof.h"
+#include "coverage.h"
+#include "params.h"
+#include "ipa-inline.h"
+#include "tree-inline.h"
+#include "auto-profile.h"
+
+/* The following routines implements AutoFDO optimization.
+
+ This optimization uses sampling profiles to annotate basic block counts
+ and uses heuristics to estimate branch probabilities.
+
+ There are three phases in AutoFDO:
+
+ Phase 1: Read profile from the profile data file.
+ The following info is read from the profile datafile:
+ * string_table: a map between function name and its index.
+ * autofdo_source_profile: a map from function_instance name to
+ function_instance. This is represented as a forest of
+ function_instances.
+ * WorkingSet: a histogram of how many instructions are covered for a
+ given percentage of total cycles.
+
+ Phase 2: Early inline.
+ Early inline uses autofdo_source_profile to find if a callsite is:
+ * inlined in the profiled binary.
+ * callee body is hot in the profiling run.
+ If both condition satisfies, early inline will inline the callsite
+ regardless of the code growth.
+
+ Phase 3: Annotate control flow graph.
+ AutoFDO uses a separate pass to:
+ * Annotate basic block count
+ * Estimate branch probability
+
+ After the above 3 phases, all profile is readily annotated on the GCC IR.
+ AutoFDO tries to reuse all FDO infrastructure as much as possible to make
+ use of the profile. E.g. it uses existing mechanism to calculate the basic
+ block/edge frequency, as well as the cgraph node/edge count.
+*/
+
+#define DEFAULT_AUTO_PROFILE_FILE "fbdata.afdo"
+
+namespace autofdo {
+
+/* Represent a source location: (function_decl, lineno). */
+typedef std::pair<tree, unsigned> decl_lineno;
+
+/* Represent an inline stack. vector[0] is the leaf node. */
+typedef std::vector<decl_lineno> inline_stack;
+
+/* String array that stores function names. */
+typedef std::vector<const char *> string_vector;
+
+/* Map from function name's index in string_table to target's
+ execution count. */
+typedef std::map<unsigned, gcov_type> icall_target_map;
+
+/* Set of gimple stmts. Used to track if the stmt has already been promoted
+ to direct call. */
+typedef std::set<gimple> stmt_set;
+
+/* Represent count info of an inline stack. */
+struct count_info
+{
+ /* Sampled count of the inline stack. */
+ gcov_type count;
+
+ /* Map from indirect call target to its sample count. */
+ icall_target_map targets;
+
+ /* Whether this inline stack is already used in annotation.
+
+ Each inline stack should only be used to annotate IR once.
+ This will be enforced when instruction-level discriminator
+ is supported. */
+ bool annotated;
+};
+
+/* operator< for "const char *". */
+struct string_compare
+{
+ bool operator() (const char *a, const char *b) const
+ { return strcmp (a, b) < 0; }
+};
+
+/* Store a string array, indexed by string position in the array. */
+class string_table {
+public:
+ static string_table *create ();
+
+ /* For a given string, returns its index. */
+ int get_index (const char *name) const;
+
+ /* For a given decl, returns the index of the decl name. */
+ int get_index_by_decl (tree decl) const;
+
+ /* For a given index, returns the string. */
+ const char *get_name (int index) const;
+
+private:
+ string_table () {}
+ bool read ();
+
+ typedef std::map<const char *, unsigned, string_compare> string_index_map;
+ string_vector vector_;
+ string_index_map map_;
+};
+
+/* Profile of a function instance:
+ 1. total_count of the function.
+ 2. head_count of the function (only valid when function is a top-level
+ function_instance, i.e. it is the original copy instead of the
+ inlined copy).
+ 3. map from source location (decl_lineno) of the inlined callsite to
+ profile (count_info).
+ 4. map from callsite to callee function_instance. */
+class function_instance {
+public:
+ typedef std::vector<function_instance *> function_instance_stack;
+
+ /* Read the profile and return a function_instance with head count as
+ HEAD_COUNT. Recursively read callsites to create nested function_instances
+ too. STACK is used to track the recursive creation process. */
+ static function_instance *read_function_instance (
+ function_instance_stack *stack, gcov_type head_count);
+
+ /* Recursively deallocate all callsites (nested function_instances). */
+ ~function_instance ();
+
+ /* Accessors. */
+ int name () const { return name_; }
+ gcov_type total_count () const { return total_count_; }
+ gcov_type head_count () const { return head_count_; }
+
+ /* Recursively traverse STACK starting from LEVEL to find the corresponding
+ function_instance. */
+ function_instance *get_function_instance (const inline_stack &stack,
+ unsigned level);
+
+ /* Store the profile info for LOC in INFO. Return TRUE if profile info
+ is found. */
+ bool get_count_info (location_t loc, count_info *info) const;
+
+ /* Read the inlined indirect call target profile for STMT and store it in
+ MAP, return the total count for all inlined indirect calls. */
+ gcov_type find_icall_target_map (gimple stmt, icall_target_map *map) const;
+
+ /* Sum of counts that is used during annotation. */
+ gcov_type total_annotated_count () const;
+
+ /* Mark LOC as annotated. */
+ void mark_annotated (location_t loc);
+
+private:
+ /* Callsite, represented as (decl_lineno, callee_function_name_index). */
+ typedef std::pair<unsigned, unsigned> callsite;
+
+ /* Map from callsite to callee function_instance. */
+ typedef std::map<callsite, function_instance *> callsite_map;
+
+ function_instance (unsigned name, gcov_type head_count)
+ : name_(name), total_count_(0), head_count_(head_count) {}
+
+ /* Traverse callsites of the current function_instance to find one at the
+ location of LINENO and callee name represented in DECL. */
+ function_instance *get_function_instance_by_decl (
+ unsigned lineno, tree decl) const;
+
+ /* Map from source location (decl_lineno) to profile (count_info). */
+ typedef std::map<unsigned, count_info> position_count_map;
+
+ /* function_instance name index in the string_table. */
+ unsigned name_;
+
+ /* Total sample count. */
+ gcov_type total_count_;
+
+ /* Entry BB's sample count. */
+ gcov_type head_count_;
+
+ /* Map from callsite location to callee function_instance. */
+ callsite_map callsites;
+
+ /* Map from source location to count_info. */
+ position_count_map pos_counts;
+};
+
+/* Profile for all functions. */
+class autofdo_source_profile {
+public:
+ static autofdo_source_profile *create ()
+ {
+ autofdo_source_profile *map = new autofdo_source_profile ();
+ if (map->read ())
+ return map;
+ delete map;
+ return NULL;
+ }
+
+ ~autofdo_source_profile ();
+
+ /* For a given DECL, returns the top-level function_instance. */
+ function_instance *get_function_instance_by_decl (tree decl) const;
+
+ /* Find count_info for a given gimple STMT. If found, store the count_info
+ in INFO and return true; otherwise return false. */
+ bool get_count_info (gimple stmt, count_info *info) const;
+
+ /* Find total count of the callee of EDGE. */
+ gcov_type get_callsite_total_count (struct cgraph_edge *edge) const;
+
+ /* Update value profile INFO for STMT from the inlined indirect callsite.
+ Return true if INFO is updated. */
+ bool update_inlined_ind_target (gimple stmt, count_info *info);
+
+ /* Mark LOC as annotated. */
+ void mark_annotated (location_t loc);
+
+private:
+ /* Map from function_instance name index (in string_table) to
+ function_instance. */
+ typedef std::map<unsigned, function_instance *>
+ name_function_instance_map;
+
+ autofdo_source_profile () {}
+
+ /* Read AutoFDO profile and returns TRUE on success. */
+ bool read ();
+
+ /* Return the function_instance in the profile that correspond to the
+ inline STACK. */
+ function_instance *get_function_instance_by_inline_stack (
+ const inline_stack &stack) const;
+
+ name_function_instance_map map_;
+};
+
+/* Store the strings read from the profile data file. */
+static string_table *afdo_string_table;
+
+/* Store the AutoFDO source profile. */
+static autofdo_source_profile *afdo_source_profile;
+
+/* gcov_ctr_summary structure to store the profile_info. */
+static struct gcov_ctr_summary *afdo_profile_info;
+
+/* Helper functions. */
+
+/* Return the original name of NAME: strip the suffix that starts
+ with '.' */
+
+static const char *get_original_name (const char *name)
+{
+ char *ret = xstrdup (name);
+ char *find = strchr (ret, '.');
+ if (find != NULL)
+ *find = 0;
+ return ret;
+}
+
+/* Return the combined location, which is a 32bit integer in which
+ higher 16 bits stores the line offset of LOC to the start lineno
+ of DECL, The lower 16 bits stores the discrimnator. */
+
+static unsigned
+get_combined_location (location_t loc, tree decl)
+{
+ return ((LOCATION_LINE (loc) - DECL_SOURCE_LINE (decl)) << 16);
+}
+
+/* Return the function decl of a given lexical BLOCK. */
+
+static tree
+get_function_decl_from_block (tree block)
+{
+ tree decl;
+
+ if (LOCATION_LOCUS (BLOCK_SOURCE_LOCATION (block) == UNKNOWN_LOCATION))
+ return NULL_TREE;
+
+ for (decl = BLOCK_ABSTRACT_ORIGIN (block);
+ decl && (TREE_CODE (decl) == BLOCK);
+ decl = BLOCK_ABSTRACT_ORIGIN (decl))
+ if (TREE_CODE (decl) == FUNCTION_DECL)
+ break;
+ return decl;
+}
+
+/* Store inline stack for STMT in STACK. */
+
+static void
+get_inline_stack (location_t locus, inline_stack *stack)
+{
+ if (LOCATION_LOCUS (locus) == UNKNOWN_LOCATION)
+ return;
+
+ tree block = LOCATION_BLOCK (locus);
+ if (block && TREE_CODE (block) == BLOCK)
+ {
+ int level = 0;
+ for (block = BLOCK_SUPERCONTEXT (block);
+ block && (TREE_CODE (block) == BLOCK);
+ block = BLOCK_SUPERCONTEXT (block))
+ {
+ location_t tmp_locus = BLOCK_SOURCE_LOCATION (block);
+ if (LOCATION_LOCUS (tmp_locus) == UNKNOWN_LOCATION)
+ continue;
+
+ tree decl = get_function_decl_from_block (block);
+ stack->push_back (std::make_pair (
+ decl, get_combined_location (locus, decl)));
+ locus = tmp_locus;
+ level++;
+ }
+ }
+ stack->push_back (std::make_pair (
+ current_function_decl,
+ get_combined_location (locus, current_function_decl)));
+}
+
+/* Return STMT's combined location, which is a 32bit integer in which
+ higher 16 bits stores the line offset of LOC to the start lineno
+ of DECL, The lower 16 bits stores the discrimnator. */
+
+static unsigned
+get_relative_location_for_stmt (gimple stmt)
+{
+ location_t locus = gimple_location (stmt);
+ if (LOCATION_LOCUS (locus) == UNKNOWN_LOCATION)
+ return UNKNOWN_LOCATION;
+
+ for (tree block = gimple_block (stmt);
+ block && (TREE_CODE (block) == BLOCK);
+ block = BLOCK_SUPERCONTEXT (block))
+ if (LOCATION_LOCUS (BLOCK_SOURCE_LOCATION (block)) != UNKNOWN_LOCATION)
+ return get_combined_location (
+ locus, get_function_decl_from_block (block));
+ return get_combined_location (locus, current_function_decl);
+}
+
+/* Return true if BB contains indirect call. */
+
+static bool
+has_indirect_call (basic_block bb)
+{
+ gimple_stmt_iterator gsi;
+
+ for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi))
+ {
+ gimple stmt = gsi_stmt (gsi);
+ if (gimple_code (stmt) == GIMPLE_CALL
+ && gimple_call_fn (stmt)
+ && TREE_CODE (gimple_call_fn (stmt)) != FUNCTION_DECL)
+ return true;
+ }
+ return false;
+}
+
+/* Member functions for string_table. */
+
+string_table *
+string_table::create ()
+{
+ string_table *map = new string_table();
+ if (map->read ())
+ return map;
+ delete map;
+ return NULL;
+}
+
+int
+string_table::get_index (const char *name) const
+{
+ if (name == NULL)
+ return -1;
+ string_index_map::const_iterator iter = map_.find (name);
+ if (iter == map_.end())
+ return -1;
+ else
+ return iter->second;
+}
+
+int
+string_table::get_index_by_decl (tree decl) const
+{
+ const char *name = get_original_name (
+ IDENTIFIER_POINTER (DECL_ASSEMBLER_NAME (decl)));
+ int ret = get_index (name);
+ if (ret != -1)
+ return ret;
+ ret = get_index (lang_hooks.dwarf_name (decl, 0));
+ if (ret != -1)
+ return ret;
+ if (DECL_ABSTRACT_ORIGIN (decl))
+ return get_index_by_decl (DECL_ABSTRACT_ORIGIN (decl));
+ else
+ return -1;
+}
+
+const char *
+string_table::get_name (int index) const
+{
+ gcc_assert (index > 0 && index < (int) vector_.size());
+ return vector_[index];
+}
+
+bool
+string_table::read ()
+{
+ if (gcov_read_unsigned () != GCOV_TAG_AFDO_FILE_NAMES)
+ return false;
+ /* Skip the length of the section. */
+ gcov_read_unsigned ();
+ /* Read in the file name table. */
+ unsigned string_num = gcov_read_unsigned ();
+ for (unsigned i = 0; i < string_num; i++)
+ {
+ vector_.push_back (get_original_name (gcov_read_string ()));
+ map_[vector_.back()] = i;
+ }
+ return true;
+}
+
+
+/* Member functions for function_instance. */
+
+function_instance::~function_instance ()
+{
+ for (callsite_map::iterator iter = callsites.begin();
+ iter != callsites.end(); ++iter)
+ delete iter->second;
+}
+
+/* Traverse callsites of the current function_instance to find one at the
+ location of LINENO and callee name represented in DECL. */
+
+function_instance *
+function_instance::get_function_instance_by_decl (
+ unsigned lineno, tree decl) const
+{
+ int func_name_idx = afdo_string_table->get_index_by_decl (decl);
+ if (func_name_idx != -1)
+ {
+ callsite_map::const_iterator ret = callsites.find (
+ std::make_pair (lineno, func_name_idx));
+ if (ret != callsites.end ())
+ return ret->second;
+ }
+ func_name_idx = afdo_string_table->get_index (
+ lang_hooks.dwarf_name (decl, 0));
+ if (func_name_idx != -1)
+ {
+ callsite_map::const_iterator ret = callsites.find (
+ std::make_pair (lineno, func_name_idx));
+ if (ret != callsites.end ())
+ return ret->second;
+ }
+ if (DECL_ABSTRACT_ORIGIN (decl))
+ return get_function_instance_by_decl (lineno, DECL_ABSTRACT_ORIGIN (decl));
+ else
+ return NULL;
+}
+
+/* Recursively traverse STACK starting from LEVEL to find the corresponding
+ function_instance. */
+
+function_instance *
+function_instance::get_function_instance (
+ const inline_stack &stack, unsigned level)
+{
+ if (level == 0)
+ return this;
+ function_instance *s =
+ get_function_instance_by_decl (stack[level].second, stack[level - 1].first);
+ if (s)
+ return s->get_function_instance (stack, level - 1);
+ else
+ return NULL;
+}
+
+/* Store the profile info for LOC in INFO. Return TRUE if profile info
+ is found. */
+
+bool
+function_instance::get_count_info (location_t loc, count_info *info) const
+{
+ position_count_map::const_iterator iter = pos_counts.find (loc);
+ if (iter == pos_counts.end ())
+ return false;
+ *info = iter->second;
+ return true;
+}
+
+/* Mark LOC as annotated. */
+
+void
+function_instance::mark_annotated (location_t loc)
+{
+ position_count_map::iterator iter = pos_counts.find (loc);
+ if (iter == pos_counts.end ())
+ return;
+ iter->second.annotated = true;
+}
+
+/* Read the inlinied indirect call target profile for STMT and store it in
+ MAP, return the total count for all inlined indirect calls. */
+
+gcov_type
+function_instance::find_icall_target_map (
+ gimple stmt, icall_target_map *map) const
+{
+ gcov_type ret = 0;
+ unsigned stmt_offset = get_relative_location_for_stmt (stmt);
+
+ for (callsite_map::const_iterator iter = callsites.begin();
+ iter != callsites.end(); ++iter)
+ {
+ unsigned callee = iter->second->name();
+ /* Check if callsite location match the stmt. */
+ if (iter->first.first != stmt_offset)
+ continue;
+ struct cgraph_node *node = find_func_by_profile_id (
+ (unsigned long long) afdo_string_table->get_name (callee));
+ if (node == NULL)
+ continue;
+ if (!check_ic_target (stmt, node))
+ continue;
+ (*map)[callee] = iter->second->total_count ();
+ ret += iter->second->total_count ();
+ }
+ return ret;
+}
+
+/* Read the profile and create a function_instance with head count as
+ HEAD_COUNT. Recursively read callsites to create nested function_instances
+ too. STACK is used to track the recursive creation process. */
+
+function_instance *
+function_instance::read_function_instance (
+ function_instance_stack *stack, gcov_type head_count)
+{
+ unsigned name = gcov_read_unsigned ();
+ unsigned num_pos_counts = gcov_read_unsigned ();
+ unsigned num_callsites = gcov_read_unsigned ();
+ function_instance *s = new function_instance (name, head_count);
+ stack->push_back(s);
+
+ for (unsigned i = 0; i < num_pos_counts; i++)
+ {
+ unsigned offset = gcov_read_unsigned () & 0xffff0000;
+ unsigned num_targets = gcov_read_unsigned ();
+ gcov_type count = gcov_read_counter ();
+ s->pos_counts[offset].count = count;
+ for (unsigned j = 0; j < stack->size(); j++)
+ (*stack)[j]->total_count_ += count;
+ for (unsigned j = 0; j < num_targets; j++)
+ {
+ /* Only indirect call target histogram is supported now. */
+ gcov_read_unsigned ();
+ gcov_type target_idx = gcov_read_counter ();
+ s->pos_counts[offset].targets[target_idx] =
+ gcov_read_counter ();
+ }
+ }
+ for (unsigned i = 0; i < num_callsites; i++) {
+ unsigned offset = gcov_read_unsigned ();
+ function_instance *callee_function_instance =
+ read_function_instance (stack, 0);
+ s->callsites[std::make_pair (offset, callee_function_instance->name ())] =
+ callee_function_instance;
+ }
+ stack->pop_back();
+ return s;
+}
+
+/* Sum of counts that is used during annotation. */
+
+gcov_type
+function_instance::total_annotated_count () const
+{
+ gcov_type ret = 0;
+ for (callsite_map::const_iterator iter = callsites.begin();
+ iter != callsites.end(); ++iter)
+ ret += iter->second->total_annotated_count ();
+ for (position_count_map::const_iterator iter = pos_counts.begin();
+ iter != pos_counts.end(); ++iter)
+ if (iter->second.annotated)
+ ret += iter->second.count;
+ return ret;
+}
+
+
+/* Member functions for autofdo_source_profile. */
+
+autofdo_source_profile::~autofdo_source_profile ()
+{
+ for (name_function_instance_map::const_iterator iter = map_.begin ();
+ iter != map_.end (); ++iter)
+ delete iter->second;
+}
+
+/* For a given DECL, returns the top-level function_instance. */
+
+function_instance *
+autofdo_source_profile::get_function_instance_by_decl (tree decl) const
+{
+ int index = afdo_string_table->get_index_by_decl (decl);
+ if (index == -1)
+ return NULL;
+ name_function_instance_map::const_iterator ret = map_.find (index);
+ return ret == map_.end() ? NULL : ret->second;
+}
+
+/* Find count_info for a given gimple STMT. If found, store the count_info
+ in INFO and return true; otherwise return false. */
+
+bool
+autofdo_source_profile::get_count_info (gimple stmt, count_info *info) const
+{
+ if (LOCATION_LOCUS (gimple_location (stmt)) == cfun->function_end_locus)
+ return false;
+
+ inline_stack stack;
+ get_inline_stack (gimple_location (stmt), &stack);
+ if (stack.size () == 0)
+ return false;
+ function_instance *s = get_function_instance_by_inline_stack (stack);
+ if (s == NULL)
+ return false;
+ return s->get_count_info (stack[0].second, info);
+}
+
+/* Mark LOC as annotated. */
+
+void
+autofdo_source_profile::mark_annotated (location_t loc) {
+ inline_stack stack;
+ get_inline_stack (loc, &stack);
+ if (stack.size () == 0)
+ return;
+ function_instance *s = get_function_instance_by_inline_stack (stack);
+ if (s == NULL)
+ return;
+ s->mark_annotated (stack[0].second);
+}
+
+/* Update value profile INFO for STMT from the inlined indirect callsite.
+ Return true if INFO is updated. */
+
+bool
+autofdo_source_profile::update_inlined_ind_target (
+ gimple stmt, count_info *info)
+{
+ if (LOCATION_LOCUS (gimple_location (stmt)) == cfun->function_end_locus)
+ return false;
+
+ count_info old_info;
+ get_count_info (stmt, &old_info);
+ gcov_type total = 0;
+ for (icall_target_map::const_iterator iter = old_info.targets.begin();
+ iter != old_info.targets.end(); ++iter)
+ total += iter->second;
+
+ /* Program behavior changed, original promoted (and inlined) target is not
+ hot any more. Will avoid promote the original target.
+
+ To check if original promoted target is still hot, we check the total
+ count of the unpromoted targets (stored in old_info). If it is no less
+ than half of the callsite count (stored in INFO), the original promoted
+ target is considered not hot any more. */
+ if (total >= info->count * 0.5)
+ return false;
+
+ inline_stack stack;
+ get_inline_stack (gimple_location (stmt), &stack);
+ if (stack.size () == 0)
+ return false;
+ function_instance *s = get_function_instance_by_inline_stack (stack);
+ if (s == NULL)
+ return false;
+ icall_target_map map;
+ if (s->find_icall_target_map (stmt, &map) == 0)
+ return false;
+ for (icall_target_map::const_iterator iter = map.begin();
+ iter != map.end(); ++iter)
+ info->targets[iter->first] = iter->second;
+ return true;
+}
+
+/* Find total count of the callee of EDGE. */
+
+gcov_type
+autofdo_source_profile::get_callsite_total_count (
+ struct cgraph_edge *edge) const
+{
+ inline_stack stack;
+ stack.push_back (std::make_pair(edge->callee->decl, 0));
+ get_inline_stack (gimple_location (edge->call_stmt), &stack);
+
+ function_instance *s = get_function_instance_by_inline_stack (stack);
+ if (s == NULL ||
+ afdo_string_table->get_index (IDENTIFIER_POINTER (DECL_ASSEMBLER_NAME (edge->callee->decl))) != s->name ())
+ return 0;
+ else
+ return s->total_count ();
+}
+
+/* Read AutoFDO profile and returns TRUE on success. */
+
+bool
+autofdo_source_profile::read ()
+{
+ if (gcov_read_unsigned () != GCOV_TAG_AFDO_FUNCTION)
+ {
+ inform (0, "Not expected TAG.");
+ return false;
+ }
+
+ /* Skip the length of the section. */
+ gcov_read_unsigned ();
+
+ /* Read in the function/callsite profile, and store it in local
+ data structure. */
+ unsigned function_num = gcov_read_unsigned ();
+ for (unsigned i = 0; i < function_num; i++)
+ {
+ function_instance::function_instance_stack stack;
+ function_instance *s = function_instance::read_function_instance (
+ &stack, gcov_read_counter ());
+ afdo_profile_info->sum_all += s->total_count ();
+ map_[s->name ()] = s;
+ }
+ return true;
+}
+
+/* Return the function_instance in the profile that correspond to the
+ inline STACK. */
+
+function_instance *
+autofdo_source_profile::get_function_instance_by_inline_stack (
+ const inline_stack &stack) const
+{
+ name_function_instance_map::const_iterator iter = map_.find (
+ afdo_string_table->get_index_by_decl (
+ stack[stack.size() - 1].first));
+ return iter == map_.end()
+ ? NULL
+ : iter->second->get_function_instance (stack, stack.size() - 1);
+}
+
+/* Module profile is only used by LIPO. Here we simply ignore it. */
+
+static void
+fake_read_autofdo_module_profile ()
+{
+ /* Read in the module info. */
+ gcov_read_unsigned ();
+
+ /* Skip the length of the section. */
+ gcov_read_unsigned ();
+
+ /* Read in the file name table. */
+ unsigned total_module_num = gcov_read_unsigned ();
+ gcc_assert (total_module_num == 0);
+}
+
+static void
+read_profile (void)
+{
+ if (gcov_open (auto_profile_file, 1) == 0)
+ error ("Cannot open profile file %s.", auto_profile_file);
+
+ if (gcov_read_unsigned () != GCOV_DATA_MAGIC)
+ error ("AutoFDO profile magic number does not mathch.");
+
+ /* Skip the version number. */
+ gcov_read_unsigned ();
+
+ /* Skip the empty integer. */
+ gcov_read_unsigned ();
+
+ /* string_table. */
+ afdo_string_table = string_table::create ();
+ if (afdo_string_table == NULL)
+ error ("Cannot read string table from %s.", auto_profile_file);
+
+ /* autofdo_source_profile. */
+ afdo_source_profile = autofdo_source_profile::create ();
+ if (afdo_source_profile == NULL)
+ error ("Cannot read function profile from %s.", auto_profile_file);
+
+ /* autofdo_module_profile. */
+ fake_read_autofdo_module_profile ();
+
+ /* Read in the working set. */
+ if (gcov_read_unsigned () != GCOV_TAG_AFDO_WORKING_SET)
+ error ("Cannot read working set from %s.", auto_profile_file);
+
+ /* Skip the length of the section. */
+ gcov_read_unsigned ();
+ gcov_working_set_t set[128];
+ for (unsigned i = 0; i < 128; i++)
+ {
+ set[i].num_counters = gcov_read_unsigned ();
+ set[i].min_counter = gcov_read_counter ();
+ }
+ add_working_set (set);
+}
+
+/* From AutoFDO profiles, find values inside STMT for that we want to measure
+ histograms for indirect-call optimization. */
+
+static void
+afdo_indirect_call (gimple_stmt_iterator *gsi, const icall_target_map &map,
+ bool transform)
+{
+ gimple stmt = gsi_stmt (*gsi);
+ tree callee;
+
+ if (map.size() == 0 || gimple_code (stmt) != GIMPLE_CALL
+ || gimple_call_fndecl (stmt) != NULL_TREE)
+ return;
+
+ callee = gimple_call_fn (stmt);
+
+ histogram_value hist = gimple_alloc_histogram_value (
+ cfun, HIST_TYPE_INDIR_CALL, stmt, callee);
+ hist->n_counters = 3;
+ hist->hvalue.counters = XNEWVEC (gcov_type, hist->n_counters);
+ gimple_add_histogram_value (cfun, stmt, hist);
+
+ gcov_type total = 0;
+ icall_target_map::const_iterator max_iter = map.end();
+
+ for (icall_target_map::const_iterator iter = map.begin();
+ iter != map.end(); ++iter)
+ {
+ total += iter->second;
+ if (max_iter == map.end() || max_iter->second < iter->second)
+ max_iter = iter;
+ }
+
+ hist->hvalue.counters[0] = (unsigned long long)
+ afdo_string_table->get_name (max_iter->first);
+ hist->hvalue.counters[1] = max_iter->second;
+ hist->hvalue.counters[2] = total;
+
+ if (!transform)
+ return;
+
+ if (gimple_ic_transform (gsi))
+ {
+ struct cgraph_edge *indirect_edge =
+ cgraph_edge (cgraph_get_node (current_function_decl), stmt);
+ struct cgraph_node *direct_call =
+ find_func_by_profile_id ((int)hist->hvalue.counters[0]);
+ if (DECL_STRUCT_FUNCTION (direct_call->decl) == NULL)
+ return;
+ struct cgraph_edge *new_edge =
+ cgraph_turn_edge_to_speculative (indirect_edge, direct_call, 0, 0);
+ cgraph_redirect_edge_call_stmt_to_callee (new_edge);
+ gimple_remove_histogram_value (cfun, stmt, hist);
+ inline_call (new_edge, true, NULL, NULL, false);
+ return;
+ }
+ return;
+}
+
+/* From AutoFDO profiles, find values inside STMT for that we want to measure
+ histograms and adds them to list VALUES. */
+
+static void
+afdo_vpt (gimple_stmt_iterator *gsi, const icall_target_map &map,
+ bool transform)
+{
+ afdo_indirect_call (gsi, map, transform);
+}
+
+/* For a given BB, return its execution count. Add the location of annotated
+ stmt to ANNOTATED. Attach value profile if a stmt is not in PROMOTED,
+ because we only want to promot an indirect call once. */
+
+static gcov_type
+afdo_get_bb_count (basic_block bb, const stmt_set &promoted)
+{
+ gimple_stmt_iterator gsi;
+ edge e;
+ edge_iterator ei;
+ gcov_type max_count = 0;
+ bool has_annotated = false;
+
+ for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi))
+ {
+ count_info info;
+ gimple stmt = gsi_stmt (gsi);
+ if (stmt->code == GIMPLE_DEBUG)
+ continue;
+ if (afdo_source_profile->get_count_info (stmt, &info))
+ {
+ if (info.count > max_count)
+ max_count = info.count;
+ has_annotated = true;
+ if (info.targets.size() > 0 && promoted.find (stmt) == promoted.end ())
+ afdo_vpt (&gsi, info.targets, false);
+ }
+ }
+
+ if (!has_annotated)
+ return 0;
+
+ for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi))
+ afdo_source_profile->mark_annotated (gimple_location (gsi_stmt (gsi)));
+ for (gsi = gsi_start_phis (bb); !gsi_end_p (gsi); gsi_next (&gsi))
+ {
+ gimple phi = gsi_stmt (gsi);
+ size_t i;
+ for (i = 0; i < gimple_phi_num_args (phi); i++)
+ afdo_source_profile->mark_annotated (gimple_phi_arg_location (phi, i));
+ }
+ FOR_EACH_EDGE (e, ei, bb->succs)
+ afdo_source_profile->mark_annotated (e->goto_locus);
+
+ bb->flags |= BB_ANNOTATED;
+ return max_count;
+}
+
+/* BB1 and BB2 are in an equivalent class iff:
+ 1. BB1 dominates BB2.
+ 2. BB2 post-dominates BB1.
+ 3. BB1 and BB2 are in the same loop nest.
+ This function finds the equivalent class for each basic block, and
+ stores a pointer to the first BB in its equivalent class. Meanwhile,
+ set bb counts for the same equivalent class to be idenical. */
+
+static void
+afdo_find_equiv_class (void)
+{
+ basic_block bb;
+
+ FOR_ALL_BB_FN (bb, cfun)
+ bb->aux = NULL;
+
+ FOR_ALL_BB_FN (bb, cfun)
+ {
+ vec<basic_block> dom_bbs;
+ basic_block bb1;
+ int i;
+
+ if (bb->aux != NULL)
+ continue;
+ bb->aux = bb;
+ dom_bbs = get_dominated_by (CDI_DOMINATORS, bb);
+ FOR_EACH_VEC_ELT (dom_bbs, i, bb1)
+ if (bb1->aux == NULL
+ && dominated_by_p (CDI_POST_DOMINATORS, bb, bb1)
+ && bb1->loop_father == bb->loop_father)
+ {
+ bb1->aux = bb;
+ if (bb1->count > bb->count && (bb1->flags & BB_ANNOTATED) != 0)
+ {
+ bb->count = MAX (bb->count, bb1->count);
+ bb->flags |= BB_ANNOTATED;
+ }
+ }
+ dom_bbs = get_dominated_by (CDI_POST_DOMINATORS, bb);
+ FOR_EACH_VEC_ELT (dom_bbs, i, bb1)
+ if (bb1->aux == NULL
+ && dominated_by_p (CDI_DOMINATORS, bb, bb1)
+ && bb1->loop_father == bb->loop_father)
+ {
+ bb1->aux = bb;
+ if (bb1->count > bb->count && (bb1->flags & BB_ANNOTATED) != 0)
+ {
+ bb->count = MAX (bb->count, bb1->count);
+ bb->flags |= BB_ANNOTATED;
+ }
+ }
+ }
+}
+
+/* If a basic block's count is known, and only one of its in/out edges' count
+ is unknown, its count can be calculated.
+ Meanwhile, if all of the in/out edges' counts are known, then the basic
+ block's unknown count can also be calculated.
+ IS_SUCC is true if out edges of a basic blocks are examined.
+ Return TRUE if any basic block/edge count is changed. */
+
+static bool
+afdo_propagate_edge (bool is_succ)
+{
+ basic_block bb;
+ bool changed = false;
+
+ FOR_EACH_BB_FN (bb, cfun)
+ {
+ edge e, unknown_edge = NULL;
+ edge_iterator ei;
+ int num_unknown_edge = 0;
+ gcov_type total_known_count = 0;
+
+ FOR_EACH_EDGE (e, ei, is_succ ? bb->succs : bb->preds)
+ if ((e->flags & EDGE_ANNOTATED) == 0)
+ num_unknown_edge ++, unknown_edge = e;
+ else
+ total_known_count += e->count;
+
+ if (num_unknown_edge == 0)
+ {
+ if (total_known_count > bb->count)
+ {
+ bb->count = total_known_count;
+ changed = true;
+ }
+ if ((bb->flags & BB_ANNOTATED) == 0)
+ {
+ bb->flags |= BB_ANNOTATED;
+ changed = true;
+ }
+ }
+ else if (num_unknown_edge == 1
+ && (bb->flags & BB_ANNOTATED) != 0)
+ {
+ if (bb->count >= total_known_count)
+ unknown_edge->count = bb->count - total_known_count;
+ else
+ unknown_edge->count = 0;
+ unknown_edge->flags |= EDGE_ANNOTATED;
+ changed = true;
+ }
+ }
+ return changed;
+}
+
+/* Special propagation for circuit expressions. Because GCC translates
+ control flow into data flow for circuit expressions. E.g.
+ BB1:
+ if (a && b)
+ BB2
+ else
+ BB3
+
+ will be translated into:
+
+ BB1:
+ if (a)
+ goto BB.t1
+ else
+ goto BB.t3
+ BB.t1:
+ if (b)
+ goto BB.t2
+ else
+ goto BB.t3
+ BB.t2:
+ goto BB.t3
+ BB.t3:
+ tmp = PHI (0 (BB1), 0 (BB.t1), 1 (BB.t2)
+ if (tmp)
+ goto BB2
+ else
+ goto BB3
+
+ In this case, we need to propagate through PHI to determine the edge
+ count of BB1->BB.t1, BB.t1->BB.t2. */
+
+static void
+afdo_propagate_circuit (void)
+{
+ basic_block bb;
+ FOR_ALL_BB_FN (bb, cfun)
+ {
+ gimple phi_stmt;
+ tree cmp_rhs, cmp_lhs;
+ gimple cmp_stmt = last_stmt (bb);
+ edge e;
+ edge_iterator ei;
+
+ if (!cmp_stmt || gimple_code (cmp_stmt) != GIMPLE_COND)
+ continue;
+ cmp_rhs = gimple_cond_rhs (cmp_stmt);
+ cmp_lhs = gimple_cond_lhs (cmp_stmt);
+ if (!TREE_CONSTANT (cmp_rhs)
+ || !(integer_zerop (cmp_rhs) || integer_onep (cmp_rhs)))
+ continue;
+ if (TREE_CODE (cmp_lhs) != SSA_NAME)
+ continue;
+ if ((bb->flags & BB_ANNOTATED) == 0)
+ continue;
+ phi_stmt = SSA_NAME_DEF_STMT (cmp_lhs);
+ while (phi_stmt && gimple_code (phi_stmt) == GIMPLE_ASSIGN
+ && gimple_assign_single_p (phi_stmt)
+ && TREE_CODE (gimple_assign_rhs1 (phi_stmt)) == SSA_NAME)
+ phi_stmt = SSA_NAME_DEF_STMT (gimple_assign_rhs1 (phi_stmt));
+ if (!phi_stmt || gimple_code (phi_stmt) != GIMPLE_PHI)
+ continue;
+ FOR_EACH_EDGE (e, ei, bb->succs)
+ {
+ unsigned i, total = 0;
+ edge only_one;
+ bool check_value_one = (((integer_onep (cmp_rhs))
+ ^ (gimple_cond_code (cmp_stmt) == EQ_EXPR))
+ ^ ((e->flags & EDGE_TRUE_VALUE) != 0));
+ if ((e->flags & EDGE_ANNOTATED) == 0)
+ continue;
+ for (i = 0; i < gimple_phi_num_args (phi_stmt); i++)
+ {
+ tree val = gimple_phi_arg_def (phi_stmt, i);
+ edge ep = gimple_phi_arg_edge (phi_stmt, i);
+
+ if (!TREE_CONSTANT (val) || !(integer_zerop (val)
+ || integer_onep (val)))
+ continue;
+ if (check_value_one ^ integer_onep (val))
+ continue;
+ total++;
+ only_one = ep;
+ if (e->probability == 0 && (ep->flags & EDGE_ANNOTATED) == 0)
+ {
+ ep->probability = 0;
+ ep->count = 0;
+ ep->flags |= EDGE_ANNOTATED;
+ }
+ }
+ if (total == 1 && (only_one->flags & EDGE_ANNOTATED) == 0)
+ {
+ only_one->probability = e->probability;
+ only_one->count = e->count;
+ only_one->flags |= EDGE_ANNOTATED;
+ }
+ }
+ }
+}
+
+/* Propagate the basic block count and edge count on the control flow
+ graph. We do the propagation iteratively until stablize. */
+
+static void
+afdo_propagate (void)
+{
+ basic_block bb;
+ bool changed = true;
+ int i = 0;
+
+ FOR_ALL_BB_FN (bb, cfun)
+ {
+ bb->count = ((basic_block) bb->aux)->count;
+ if ((((basic_block) bb->aux)->flags & BB_ANNOTATED) != 0)
+ bb->flags |= BB_ANNOTATED;
+ }
+
+ while (changed && i++ < 10)
+ {
+ changed = false;
+
+ if (afdo_propagate_edge (true))
+ changed = true;
+ if (afdo_propagate_edge (false))
+ changed = true;
+ afdo_propagate_circuit ();
+ }
+}
+
+/* Propagate counts on control flow graph and calculate branch
+ probabilities. */
+
+static void
+afdo_calculate_branch_prob (void)
+{
+ basic_block bb;
+ bool has_sample = false;
+
+ FOR_EACH_BB_FN (bb, cfun)
+ if (bb->count > 0)
+ has_sample = true;
+
+ if (!has_sample)
+ return;
+
+ calculate_dominance_info (CDI_POST_DOMINATORS);
+ calculate_dominance_info (CDI_DOMINATORS);
+ loop_optimizer_init (0);
+
+ afdo_find_equiv_class ();
+ afdo_propagate ();
+
+ FOR_EACH_BB_FN (bb, cfun)
+ {
+ edge e;
+ edge_iterator ei;
+ int num_unknown_succ = 0;
+ gcov_type total_count = 0;
+
+ FOR_EACH_EDGE (e, ei, bb->succs)
+ {
+ if ((e->flags & EDGE_ANNOTATED) == 0)
+ num_unknown_succ ++;
+ else
+ total_count += e->count;
+ }
+ if (num_unknown_succ == 0 && total_count > 0)
+ {
+ FOR_EACH_EDGE (e, ei, bb->succs)
+ e->probability =
+ (double) e->count * REG_BR_PROB_BASE / total_count;
+ }
+ }
+ FOR_ALL_BB_FN (bb, cfun)
+ {
+ edge e;
+ edge_iterator ei;
+
+ FOR_EACH_EDGE (e, ei, bb->succs)
+ e->count =
+ (double) bb->count * e->probability / REG_BR_PROB_BASE;
+ bb->aux = NULL;
+ }
+
+ loop_optimizer_finalize ();
+ free_dominance_info (CDI_DOMINATORS);
+ free_dominance_info (CDI_POST_DOMINATORS);
+}
+
+/* Perform value profile transformation using AutoFDO profile. Add the
+ promoted stmts to PROMOTED_STMTS. Return TRUE if there is any
+ indirect call promoted. */
+
+static bool
+afdo_vpt_for_early_inline (stmt_set *promoted_stmts)
+{
+ basic_block bb;
+ if (afdo_source_profile->get_function_instance_by_decl (
+ current_function_decl) == NULL)
+ return false;
+
+ compute_inline_parameters (cgraph_get_node (current_function_decl), true);
+
+ bool has_vpt = false;
+ FOR_EACH_BB_FN (bb, cfun)
+ {
+ if (!has_indirect_call (bb))
+ continue;
+ gimple_stmt_iterator gsi;
+
+ gcov_type bb_count = 0;
+ for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi))
+ {
+ count_info info;
+ gimple stmt = gsi_stmt (gsi);
+ if (afdo_source_profile->get_count_info (stmt, &info))
+ bb_count = MAX (bb_count, info.count);
+ }
+
+ for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi))
+ {
+ gimple stmt = gsi_stmt (gsi);
+ /* IC_promotion and early_inline_2 is done in multiple iterations.
+ No need to promoted the stmt if its in promoted_stmts (means
+ it is already been promoted in the previous iterations). */
+ if (gimple_code (stmt) != GIMPLE_CALL
+ || TREE_CODE (gimple_call_fn (stmt)) == FUNCTION_DECL
+ || promoted_stmts->find (stmt) != promoted_stmts->end ())
+ continue;
+
+ count_info info;
+ afdo_source_profile->get_count_info (stmt, &info);
+ info.count = bb_count;
+ if (afdo_source_profile->update_inlined_ind_target (stmt, &info))
+ {
+ /* Promote the indirect call and update the promoted_stmts. */
+ promoted_stmts->insert (stmt);
+ afdo_vpt (&gsi, info.targets, true);
+ has_vpt = true;
+ }
+ }
+ }
+ if (has_vpt)
+ {
+ optimize_inline_calls (current_function_decl);
+ return true;
+ }
+ else
+ return false;
+}
+
+/* Annotate auto profile to the control flow graph. Do not annotate value
+ profile for stmts in PROMOTED_STMTS. */
+
+static void
+afdo_annotate_cfg (const stmt_set &promoted_stmts)
+{
+ basic_block bb;
+ const function_instance *s =
+ afdo_source_profile->get_function_instance_by_decl (
+ current_function_decl);
+
+ if (s == NULL)
+ return;
+ cgraph_get_node (current_function_decl)->count = s->head_count ();
+ ENTRY_BLOCK_PTR_FOR_FN (cfun)->count = s->head_count ();
+ gcov_type max_count = ENTRY_BLOCK_PTR_FOR_FN (cfun)->count;
+
+ FOR_EACH_BB_FN (bb, cfun)
+ {
+ edge e;
+ edge_iterator ei;
+
+ bb->count = 0;
+ bb->flags &= (~BB_ANNOTATED);
+ FOR_EACH_EDGE (e, ei, bb->succs)
+ {
+ e->count = 0;
+ e->flags &= (~EDGE_ANNOTATED);
+ }
+
+ bb->count = afdo_get_bb_count (bb, promoted_stmts);
+ if (bb->count > max_count)
+ max_count = bb->count;
+ }
+ if (ENTRY_BLOCK_PTR_FOR_FN (cfun)->count >
+ ENTRY_BLOCK_PTR_FOR_FN (cfun)->next_bb->count)
+ {
+ ENTRY_BLOCK_PTR_FOR_FN (cfun)->next_bb->count =
+ ENTRY_BLOCK_PTR_FOR_FN (cfun)->count;
+ ENTRY_BLOCK_PTR_FOR_FN (cfun)->next_bb->flags |= BB_ANNOTATED;
+ }
+ if (ENTRY_BLOCK_PTR_FOR_FN (cfun)->count >
+ EXIT_BLOCK_PTR_FOR_FN (cfun)->prev_bb->count)
+ {
+ EXIT_BLOCK_PTR_FOR_FN (cfun)->prev_bb->count =
+ ENTRY_BLOCK_PTR_FOR_FN (cfun)->count;
+ EXIT_BLOCK_PTR_FOR_FN (cfun)->prev_bb->flags |= BB_ANNOTATED;
+ }
+ afdo_source_profile->mark_annotated (
+ DECL_SOURCE_LOCATION (current_function_decl));
+ afdo_source_profile->mark_annotated (cfun->function_start_locus);
+ afdo_source_profile->mark_annotated (cfun->function_end_locus);
+ if (max_count > 0)
+ {
+ afdo_calculate_branch_prob ();
+ counts_to_freqs ();
+ profile_status_for_fn (cfun) = PROFILE_READ;
+ }
+ if (flag_value_profile_transformations)
+ gimple_value_profile_transformations ();
+}
+
+/* Wrapper function to invoke early inliner. */
+
+static void early_inline ()
+{
+ compute_inline_parameters (cgraph_get_node (current_function_decl), true);
+ unsigned todo = early_inliner (cfun);
+ if (todo & TODO_update_ssa_any)
+ update_ssa (TODO_update_ssa);
+}
+
+/* Use AutoFDO profile to annoate the control flow graph.
+ Return the todo flag. */
+
+static unsigned int
+auto_profile (void)
+{
+ struct cgraph_node *node;
+
+ if (cgraph_state == CGRAPH_STATE_FINISHED)
+ return 0;
+
+ init_node_map (true);
+ profile_info = autofdo::afdo_profile_info;
+
+ FOR_EACH_FUNCTION (node)
+ {
+ if (!gimple_has_body_p (node->decl))
+ continue;
+
+ /* Don't profile functions produced for builtin stuff. */
+ if (DECL_SOURCE_LOCATION (node->decl) == BUILTINS_LOCATION)
+ continue;
+
+ push_cfun (DECL_STRUCT_FUNCTION (node->decl));
+
+ /* First do indirect call promotion and early inline to make the
+ IR match the profiled binary before actual annotation.
+
+ This is needed because an indirect call might have been promoted
+ and inlined in the profiled binary. If we do not promote and
+ inline these indirect calls before annotation, the profile for
+ these promoted functions will be lost.
+
+ e.g. foo() --indirect_call--> bar()
+ In profiled binary, the callsite is promoted and inlined, making
+ the profile look like:
+
+ foo: {
+ loc_foo_1: count_1
+ bar@loc_foo_2: {
+ loc_bar_1: count_2
+ loc_bar_2: count_3
+ }
+ }
+
+ Before AutoFDO pass, loc_foo_2 is not promoted thus not inlined.
+ If we perform annotation on it, the profile inside bar@loc_foo2
+ will be wasted.
+
+ To avoid this, we promote loc_foo_2 and inline the promoted bar
+ function before annotation, so the profile inside bar@loc_foo2
+ will be useful. */
+ autofdo::stmt_set promoted_stmts;
+ for (int i = 0; i < PARAM_VALUE (PARAM_EARLY_INLINER_MAX_ITERATIONS); i++)
+ {
+ if (!flag_value_profile_transformations
+ || !autofdo::afdo_vpt_for_early_inline (&promoted_stmts))
+ break;
+ early_inline ();
+ }
+
+ early_inline ();
+ autofdo::afdo_annotate_cfg (promoted_stmts);
+ compute_function_frequency ();
+ update_ssa (TODO_update_ssa);
+
+ /* Local pure-const may imply need to fixup the cfg. */
+ if (execute_fixup_cfg () & TODO_cleanup_cfg)
+ cleanup_tree_cfg ();
+
+ free_dominance_info (CDI_DOMINATORS);
+ free_dominance_info (CDI_POST_DOMINATORS);
+ rebuild_cgraph_edges ();
+ pop_cfun ();
+ }
+
+ return TODO_rebuild_cgraph_edges;
+}
+} /* namespace autofdo. */
+
+/* Read the profile from the profile data file. */
+
+void
+init_auto_profile (void)
+{
+ if (auto_profile_file == NULL)
+ auto_profile_file = DEFAULT_AUTO_PROFILE_FILE;
+
+ autofdo::afdo_profile_info = (struct gcov_ctr_summary *)
+ xcalloc (1, sizeof (struct gcov_ctr_summary));
+ autofdo::afdo_profile_info->runs = 1;
+ autofdo::afdo_profile_info->sum_max = 0;
+ autofdo::afdo_profile_info->sum_all = 0;
+
+ /* Read the profile from the profile file. */
+ autofdo::read_profile ();
+}
+
+/* Free the resources. */
+
+void
+end_auto_profile (void)
+{
+ delete autofdo::afdo_source_profile;
+ delete autofdo::afdo_string_table;
+ profile_info = NULL;
+}
+
+/* Returns TRUE if EDGE is hot enough to be inlined early. */
+
+bool
+afdo_callsite_hot_enough_for_early_inline (struct cgraph_edge *edge)
+{
+ gcov_type count =
+ autofdo::afdo_source_profile->get_callsite_total_count (edge);
+ if (count > 0)
+ {
+ bool is_hot;
+ const struct gcov_ctr_summary *saved_profile_info = profile_info;
+ /* At earling inline stage, profile_info is not set yet. We need to
+ temporarily set it to afdo_profile_info to calculate hotness. */
+ profile_info = autofdo::afdo_profile_info;
+ is_hot = maybe_hot_count_p (NULL, count);
+ profile_info = saved_profile_info;
+ return is_hot;
+ }
+ else
+ return false;
+}
+
+namespace {
+
+const pass_data pass_data_ipa_auto_profile =
+{
+ SIMPLE_IPA_PASS,
+ "afdo", /* name */
+ OPTGROUP_NONE, /* optinfo_flags */
+ true, /* has_execute */
+ TV_IPA_AUTOFDO, /* tv_id */
+ 0, /* properties_required */
+ 0, /* properties_provided */
+ 0, /* properties_destroyed */
+ 0, /* todo_flags_start */
+ 0, /* todo_flags_finish */
+};
+
+class pass_ipa_auto_profile : public simple_ipa_opt_pass
+{
+public:
+ pass_ipa_auto_profile(gcc::context *ctxt)
+ : simple_ipa_opt_pass(pass_data_ipa_auto_profile, ctxt)
+ {}
+
+ /* opt_pass methods: */
+ virtual bool gate (function *)
+ {
+ return flag_auto_profile;
+ }
+ virtual unsigned int execute (function *)
+ {
+ return autofdo::auto_profile ();
+ }
+}; // class pass_ipa_auto_profile
+
+} // anon namespace
+
+simple_ipa_opt_pass *
+make_pass_ipa_auto_profile (gcc::context *ctxt)
+{
+ return new pass_ipa_auto_profile (ctxt);
+}
+
Index: gcc/auto-profile.h
===================================================================
--- gcc/auto-profile.h (revision 0)
+++ gcc/auto-profile.h (revision 0)
@@ -0,0 +1,31 @@
+/* auto-profile.h - Defines data exported from auto-profile.c
+ Copyright (C) 2012. Free Software Foundation, Inc.
+ Contributed by Dehao Chen (dehao@google.com)
+
+This file is part of GCC.
+
+GCC is free software; you can redistribute it and/or modify it under
+the terms of the GNU General Public License as published by the Free
+Software Foundation; either version 3, or (at your option) any later
+version.
+
+GCC is distributed in the hope that it will be useful, but WITHOUT ANY
+WARRANTY; without even the implied warranty of MERCHANTABILITY or
+FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
+for more details.
+
+You should have received a copy of the GNU General Public License
+along with GCC; see the file COPYING3. If not see
+<http://www.gnu.org/licenses/>. */
+
+#ifndef AUTO_PROFILE_H
+#define AUTO_PROFILE_H
+
+/* Read, process, finalize AutoFDO data structures. */
+extern void init_auto_profile (void);
+extern void end_auto_profile (void);
+
+/* Returns TRUE if EDGE is hot enough to be inlined early. */
+extern bool afdo_callsite_hot_enough_for_early_inline (struct cgraph_edge *);
+
+#endif /* AUTO_PROFILE_H */
Index: gcc/tree-ssa-live.c
===================================================================
--- gcc/tree-ssa-live.c (revision 210180)
+++ gcc/tree-ssa-live.c (working copy)
@@ -607,7 +607,7 @@ remove_unused_scope_block_p (tree scope)
;
/* When not generating debug info we can eliminate info on unused
variables. */
- else if (debug_info_level == DINFO_LEVEL_NONE)
+ else if (!flag_auto_profile && debug_info_level == DINFO_LEVEL_NONE)
{
/* Even for -g0 don't prune outer scopes from artificial
functions, otherwise diagnostics using tree_nonartificial_location
Index: gcc/ipa-inline.c
===================================================================
--- gcc/ipa-inline.c (revision 210180)
+++ gcc/ipa-inline.c (working copy)
@@ -121,6 +121,7 @@ along with GCC; see the file COPYING3. If not see
#include "ipa-inline.h"
#include "ipa-utils.h"
#include "sreal.h"
+#include "auto-profile.h"
#include "cilk.h"
/* Statistics we collect about inlining algorithm. */
@@ -450,6 +451,8 @@ want_early_inline_function_p (struct cgraph_edge *
if (DECL_DISREGARD_INLINE_LIMITS (callee->decl))
;
+ else if (flag_auto_profile && afdo_callsite_hot_enough_for_early_inline (e))
+ ;
else if (!DECL_DECLARED_INLINE_P (callee->decl)
&& !flag_inline_small_functions)
{
@@ -2368,41 +2371,8 @@ early_inline_small_functions (struct cgraph_node *
return inlined;
}
-/* Do inlining of small functions. Doing so early helps profiling and other
- passes to be somewhat more effective and avoids some code duplication in
- later real inlining pass for testcases with very many function calls. */
-
-namespace {
-
-const pass_data pass_data_early_inline =
+unsigned int early_inliner (function *fun)
{
- GIMPLE_PASS, /* type */
- "einline", /* name */
- OPTGROUP_INLINE, /* optinfo_flags */
- true, /* has_execute */
- TV_EARLY_INLINING, /* tv_id */
- PROP_ssa, /* properties_required */
- 0, /* properties_provided */
- 0, /* properties_destroyed */
- 0, /* todo_flags_start */
- 0, /* todo_flags_finish */
-};
-
-class pass_early_inline : public gimple_opt_pass
-{
-public:
- pass_early_inline (gcc::context *ctxt)
- : gimple_opt_pass (pass_data_early_inline, ctxt)
- {}
-
- /* opt_pass methods: */
- virtual unsigned int execute (function *);
-
-}; // class pass_early_inline
-
-unsigned int
-pass_early_inline::execute (function *fun)
-{
struct cgraph_node *node = cgraph_get_node (current_function_decl);
struct cgraph_edge *edge;
unsigned int todo = 0;
@@ -2502,6 +2472,44 @@ early_inline_small_functions (struct cgraph_node *
return todo;
}
+/* Do inlining of small functions. Doing so early helps profiling and other
+ passes to be somewhat more effective and avoids some code duplication in
+ later real inlining pass for testcases with very many function calls. */
+
+namespace {
+
+const pass_data pass_data_early_inline =
+{
+ GIMPLE_PASS, /* type */
+ "einline", /* name */
+ OPTGROUP_INLINE, /* optinfo_flags */
+ true, /* has_execute */
+ TV_EARLY_INLINING, /* tv_id */
+ PROP_ssa, /* properties_required */
+ 0, /* properties_provided */
+ 0, /* properties_destroyed */
+ 0, /* todo_flags_start */
+ 0, /* todo_flags_finish */
+};
+
+class pass_early_inline : public gimple_opt_pass
+{
+public:
+ pass_early_inline (gcc::context *ctxt)
+ : gimple_opt_pass (pass_data_early_inline, ctxt)
+ {}
+
+ /* opt_pass methods: */
+ virtual unsigned int execute (function *);
+
+}; // class pass_early_inline
+
+unsigned int
+pass_early_inline::execute (function *fun)
+{
+ return early_inliner (fun);
+}
+
} // anon namespace
gimple_opt_pass *
Index: gcc/ipa-inline.h
===================================================================
--- gcc/ipa-inline.h (revision 210180)
+++ gcc/ipa-inline.h (working copy)
@@ -345,3 +345,5 @@ reset_edge_growth_cache (struct cgraph_edge *edge)
edge_growth_cache[edge->uid] = zero;
}
}
+
+unsigned int early_inliner (function *fun);
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH] AutoFDO patch for trunk 2014-05-07 21:24 [PATCH] AutoFDO patch for trunk Dehao Chen @ 2014-05-07 21:47 ` Xinliang David Li 2014-05-15 21:04 ` Jan Hubicka 2014-05-16 0:46 ` Jan Hubicka 2 siblings, 0 replies; 25+ messages in thread From: Xinliang David Li @ 2014-05-07 21:47 UTC (permalink / raw) To: Dehao Chen; +Cc: GCC Patches, Jan Hubicka Have you announced the autofdo profile tool to gcc list? David On Wed, May 7, 2014 at 2:24 PM, Dehao Chen <dehao@google.com> wrote: > Hi, > > I'm planning to port the AutoFDO patch upstream. Attached is the > prepared patch. You can also find the patch in > http://codereview.appspot.com/99010043 > > I've tested the patch with SPECCPU2006. For the CINT2006 benchmarks, > the speedup comparison between O2, FDO and AutoFDO is as follows: > > Reference: o2 > (1): auto_fdo > (2): fdo > > Benchmark Base:Reference (1) (2) > ----------------------------------------------------------------- > spec/2006/int/C++/471.omnetpp 23.18 +3.11% +5.09% > spec/2006/int/C++/473.astar 21.15 +6.79% +9.80% > spec/2006/int/C++/483.xalancbmk 36.68 +11.56% +14.47% > spec/2006/int/C/400.perlbench 34.57 +6.59% +18.56% > spec/2006/int/C/401.bzip2 23.17 +0.95% +2.49% > spec/2006/int/C/403.gcc 32.33 +8.27% +9.76% > spec/2006/int/C/429.mcf 42.13 +4.72% +5.23% > spec/2006/int/C/445.gobmk 26.53 -1.39% +0.05% > spec/2006/int/C/456.hmmer 23.72 +7.12% +7.87% > spec/2006/int/C/458.sjeng 26.17 +4.65% +6.04% > spec/2006/int/C/462.libquantum 57.23 +4.04% +1.42% > spec/2006/int/C/464.h264ref 46.3 +1.07% +8.97% > > geometric mean +4.73% +7.36% > > The majority of the performance difference between AutoFDO and FDO > comes from the lack of instruction level discriminator support. Cary > Coutant is planning to port that patch upstream too. > > Please let me know if you have any question about this patch, and > thanks in advance for reviewing such a huge patch. > > Dehao ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH] AutoFDO patch for trunk 2014-05-07 21:24 [PATCH] AutoFDO patch for trunk Dehao Chen 2014-05-07 21:47 ` Xinliang David Li @ 2014-05-15 21:04 ` Jan Hubicka 2014-10-09 0:24 ` Dehao Chen 2014-05-16 0:46 ` Jan Hubicka 2 siblings, 1 reply; 25+ messages in thread From: Jan Hubicka @ 2014-05-15 21:04 UTC (permalink / raw) To: Dehao Chen; +Cc: GCC Patches, Jan Hubicka, David Li > Hi, > > I'm planning to port the AutoFDO patch upstream. Attached is the > prepared patch. You can also find the patch in > http://codereview.appspot.com/99010043 > > I've tested the patch with SPECCPU2006. For the CINT2006 benchmarks, > the speedup comparison between O2, FDO and AutoFDO is as follows: > > Reference: o2 > (1): auto_fdo > (2): fdo > > Benchmark Base:Reference (1) (2) > ----------------------------------------------------------------- > spec/2006/int/C++/471.omnetpp 23.18 +3.11% +5.09% > spec/2006/int/C++/473.astar 21.15 +6.79% +9.80% > spec/2006/int/C++/483.xalancbmk 36.68 +11.56% +14.47% > spec/2006/int/C/400.perlbench 34.57 +6.59% +18.56% > spec/2006/int/C/401.bzip2 23.17 +0.95% +2.49% > spec/2006/int/C/403.gcc 32.33 +8.27% +9.76% > spec/2006/int/C/429.mcf 42.13 +4.72% +5.23% > spec/2006/int/C/445.gobmk 26.53 -1.39% +0.05% > spec/2006/int/C/456.hmmer 23.72 +7.12% +7.87% > spec/2006/int/C/458.sjeng 26.17 +4.65% +6.04% > spec/2006/int/C/462.libquantum 57.23 +4.04% +1.42% > spec/2006/int/C/464.h264ref 46.3 +1.07% +8.97% > > geometric mean +4.73% +7.36% The results are very nice indeed. So basically it adds another 50% to static estimation, that is not bad. The patch is long and missing a changelog, I will add my comments and I think it would be great to break it up where possible - for some infrastructure preparation patches and the actual reading infrastructure. > Index: gcc/cfghooks.c > =================================================================== > --- gcc/cfghooks.c (revision 210180) > +++ gcc/cfghooks.c (working copy) > @@ -833,6 +833,9 @@ make_forwarder_block (basic_block bb, bool (*redir > > fallthru = split_block_after_labels (bb); > dummy = fallthru->src; > + dummy->count = 0; > + dummy->frequency = 0; > + fallthru->count = 0; > bb = fallthru->dest; > > /* Redirect back edges we want to keep. */ > @@ -842,20 +845,13 @@ make_forwarder_block (basic_block bb, bool (*redir > > if (redirect_edge_p (e)) > { > + dummy->frequency += EDGE_FREQUENCY (e); > + dummy->count += e->count; > + fallthru->count += e->count; > ei_next (&ei); > continue; > } > > - dummy->frequency -= EDGE_FREQUENCY (e); > - dummy->count -= e->count; > - if (dummy->frequency < 0) > - dummy->frequency = 0; > - if (dummy->count < 0) > - dummy->count = 0; > - fallthru->count -= e->count; > - if (fallthru->count < 0) > - fallthru->count = 0; > - Can you elaborate why these changes are needed? They look unrelated, so it is a bug in forwarder block profile updating? I do not see why source of the edge needs to have profile cleaned. > Index: gcc/loop-unroll.c > =================================================================== > --- gcc/loop-unroll.c (revision 210180) > +++ gcc/loop-unroll.c (working copy) > @@ -998,6 +998,13 @@ decide_unroll_runtime_iterations (struct loop *loo > return; > } > > + /* In AutoFDO, the profile is not accurate. If the calculated trip count > + is larger than the header count, then the profile is not accurate > + enough to make correct unroll decisions. */ > + if (flag_auto_profile > + && expected_loop_iterations (loop) > loop->header->count) > + return; This is another independent change. It does make sense, but I would preffer to have it in a separate function (expected_loop_iterations_reliable_p) rather than inline in the code. There are other loop optimizations that are driven by this, and they should consistently use this predicate. In fact current -fprofile-use enabled -funroll-loops flag that enables all unrolling independently of reliablility of the profile. I suppose we need to imporove this scheme and have something liek unroll-loop-with-reliable-profile path that would default for profile feedback optimization. I also do not think your check makes a lot of sense. You want to compare loop header count and loop preheader or so, or this is not reliable enough for auto-FDO profiles? In any case, lets handle this separately of the main autoFDO patches (and not forget on it). > Index: gcc/dwarf2out.c > =================================================================== > --- gcc/dwarf2out.c (revision 210180) > +++ gcc/dwarf2out.c (working copy) > @@ -2481,6 +2481,40 @@ const struct gcc_debug_hooks dwarf2_debug_hooks = > 1, /* start_end_main_source_file */ > TYPE_SYMTAB_IS_DIE /* tree_type_symtab_field */ > }; > + > +const struct gcc_debug_hooks auto_profile_debug_hooks = > +{ > + debug_nothing_charstar, > + debug_nothing_charstar, > + debug_nothing_void, > + debug_nothing_int_charstar, > + debug_nothing_int_charstar, > + debug_nothing_int_charstar, > + debug_nothing_int, > + debug_nothing_int_int, /* begin_block */ > + debug_nothing_int_int, /* end_block */ > + dwarf2out_ignore_block, /* ignore_block */ > + debug_nothing_int_charstar_int_bool, /* source_line */ > + debug_nothing_int_charstar, /* begin_prologue */ > + debug_nothing_int_charstar, /* end_prologue */ > + debug_nothing_int_charstar, /* begin_epilogue */ > + debug_nothing_int_charstar, /* end_epilogue */ > + debug_nothing_tree, /* begin_function */ > + debug_nothing_int, /* end_function */ > + debug_nothing_tree, /* function_decl */ > + debug_nothing_tree, /* global_decl */ > + debug_nothing_tree_int, /* type_decl */ > + debug_nothing_tree_tree_tree_bool, /* imported_module_or_decl */ > + debug_nothing_tree, /* deferred_inline_function */ > + debug_nothing_tree, /* outlining_inline_function */ > + debug_nothing_rtx, /* label */ > + debug_nothing_int, /* handle_pch */ > + debug_nothing_rtx, /* var_location */ > + debug_nothing_void, /* switch_text_section */ > + debug_nothing_tree_tree, /* set_name */ > + 0, /* start_end_main_source_file */ > + TYPE_SYMTAB_IS_ADDRESS /* tree_type_symtab_field */ > +}; Note that I can not approve changes to debug machine (and do not feel confident in it), so lets do this as independent change, too. You need those to reliably output line info with autofdo enabled or why this is needed at all? > \f > /* NOTE: In the comments in this file, many references are made to > "Debugging Information Entries". This term is abbreviated as `DIE' > @@ -16799,10 +16833,9 @@ add_src_coords_attributes (dw_die_ref die, tree de > static void > add_linkage_name (dw_die_ref die, tree decl) > { > - if (debug_info_level > DINFO_LEVEL_TERSE > + if (debug_info_level > DINFO_LEVEL_NONE > && (TREE_CODE (decl) == FUNCTION_DECL || TREE_CODE (decl) == VAR_DECL) > && TREE_PUBLIC (decl) > - && !DECL_ABSTRACT (decl) > && !(TREE_CODE (decl) == VAR_DECL && DECL_REGISTER (decl)) > && die->die_tag != DW_TAG_member) > { > /* Counters that are collected. */ > Index: gcc/cfg-flags.def > =================================================================== > --- gcc/cfg-flags.def (revision 210180) > +++ gcc/cfg-flags.def (working copy) > @@ -93,6 +93,9 @@ DEF_BASIC_BLOCK_FLAG(VISITED, 13) > demand, and is available after calling compute_transaction_bits(). */ > DEF_BASIC_BLOCK_FLAG(IN_TRANSACTION, 14) > > +/* Set on blocks that has been annotated during AutoFDO profile > + attribution. */ > +DEF_BASIC_BLOCK_FLAG(ANNOTATED, 15) ANNOTATED is uninformative name, perhaps AUTOFDO_ANNOTATED? What is the annotation used for? > Index: gcc/ira-int.h > =================================================================== > --- gcc/ira-int.h (revision 210180) > +++ gcc/ira-int.h (working copy) > @@ -41,9 +41,10 @@ along with GCC; see the file COPYING3. If not see > analogous to REG_FREQ_FROM_BB. When optimizing for size, or > profile driven feedback is available and the function is never > executed, frequency is always equivalent. Otherwise rescale the > - edge frequency. */ > + edge frequency. For AutoFDO, even if a function is not sampled, > + it can still be executed, thus frequency rescale is still used. */ > #define REG_FREQ_FROM_EDGE_FREQ(freq) \ > - (optimize_size || (flag_branch_probabilities \ > + (optimize_size || (flag_branch_probabilities && !flag_auto_profile \ > && !ENTRY_BLOCK_PTR_FOR_FN (cfun)->count) \ > ? REG_FREQ_MAX : (freq * REG_FREQ_MAX / BB_FREQ_MAX) \ > ? (freq * REG_FREQ_MAX / BB_FREQ_MAX) : 1) This also needs more consistent solution. I think this macro is out of date and hsould test optimize_function_for_size_p predicate and the predicate should do the right thing for auto-FDO. This change is pre-approved for mainline. > Index: gcc/tree-profile.c > =================================================================== > --- gcc/tree-profile.c (revision 210180) > +++ gcc/tree-profile.c (working copy) > @@ -696,7 +696,7 @@ bool > pass_ipa_tree_profile::gate (function *) > { > /* When profile instrumentation, use or test coverage shall be performed. */ > - return (!in_lto_p > + return (!in_lto_p && !flag_auto_profile > && (flag_branch_probabilities || flag_test_coverage > || profile_arc_flag)); Update comment here. > } > Index: gcc/tree-inline.c > =================================================================== > --- gcc/tree-inline.c (revision 210180) > +++ gcc/tree-inline.c (working copy) > @@ -1977,9 +1977,10 @@ copy_edges_for_bb (basic_block bb, gcov_type count > edge new_edge; > > flags = old_edge->flags; > + flags &= (~EDGE_ANNOTATED); Probably comment on why you want to clear annotations. > @@ -2191,6 +2192,9 @@ initialize_cfun (tree new_fndecl, tree callee_fnde > else > count_scale = REG_BR_PROB_BASE; > > + if (count_scale > REG_BR_PROB_BASE) > + count_scale = REG_BR_PROB_BASE; > + > /* Register specific tree functions. */ > gimple_register_cfg_hooks (); > > @@ -2452,6 +2456,9 @@ copy_cfg_body (copy_body_data * id, gcov_type coun > else > count_scale = REG_BR_PROB_BASE; > > + if (count_scale > REG_BR_PROB_BASE) > + count_scale = REG_BR_PROB_BASE; > + > /* Register specific tree functions. */ > gimple_register_cfg_hooks (); > These changes also looks independent and should go in separately (with an explanation) > Index: gcc/bb-reorder.c > =================================================================== > --- gcc/bb-reorder.c (revision 210180) > +++ gcc/bb-reorder.c (working copy) > @@ -2663,7 +2663,7 @@ pass_partition_blocks::gate (function *fun) > user defined section attributes. Don't call it if either case > arises. */ > return (flag_reorder_blocks_and_partition > - && optimize > + && optimize &&!flag_auto_profile Formatting error. Why we want !flag_auto_profile? Can't we just arrange flag_reorder_blocks_and_partition to default to false? In fact I would like to see flag_reorder_blocks_and_partition to offload obviously cold code (in partiuclar the EH cleanups) even without profile. That should work well with autofdo. > Index: gcc/coverage.c > =================================================================== > --- gcc/coverage.c (revision 210180) > +++ gcc/coverage.c (working copy) > @@ -54,6 +54,7 @@ along with GCC; see the file COPYING3. If not see > #include "intl.h" > #include "filenames.h" > #include "target.h" > +#include "auto-profile.h" > > #include "gcov-io.h" > #include "gcov-io.c" > @@ -1175,7 +1176,9 @@ coverage_init (const char *filename) > > bbg_file_stamp = local_tick; > > - if (flag_branch_probabilities) > + if (flag_auto_profile) > + init_auto_profile (); Perhaps initialization can happen from toplev.c instead? > Index: gcc/profile.c > =================================================================== > --- gcc/profile.c (revision 210180) > +++ gcc/profile.c (working copy) > @@ -106,6 +106,12 @@ static int total_num_times_called; > static int total_hist_br_prob[20]; > static int total_num_branches; > > +void add_working_set (gcov_working_set_t *set) { > + int i = 0; > + for (; i < NUM_GCOV_WORKING_SETS; i++) > + gcov_working_sets[i] = set[i]; > +} Comment here. > Index: gcc/regs.h > =================================================================== > --- gcc/regs.h (revision 210180) > +++ gcc/regs.h (working copy) > @@ -137,6 +137,7 @@ extern size_t reg_info_p_size; > frequency. */ > #define REG_FREQ_FROM_BB(bb) (optimize_size \ > || (flag_branch_probabilities \ > + && !flag_auto_profile \ > && !ENTRY_BLOCK_PTR_FOR_FN (cfun)->count) \ > ? REG_FREQ_MAX \ > : ((bb)->frequency * REG_FREQ_MAX / BB_FREQ_MAX)\ Again, I think optimize_function_for_size is good predicate here. > Index: gcc/common.opt > =================================================================== > --- gcc/common.opt (revision 210180) > +++ gcc/common.opt (working copy) > @@ -878,6 +878,16 @@ fauto-inc-dec > Common Report Var(flag_auto_inc_dec) Init(1) > Generate auto-inc/dec instructions > > +fauto-profile > +Common Report Var(flag_auto_profile) Optimization > +Use sample profile information for call graph node weights. The default > +profile file is fbdata.afdo in 'pwd'. > + > +fauto-profile= > +Common Joined RejectNegative Var(auto_profile_file) > +Use sample profile information for call graph node weights. The profile > +file is specified in the argument. I believe auto-fdo will need a separate section in the manual describing the whole machinery. > Index: gcc/opts.c > =================================================================== > --- gcc/opts.c (revision 210180) > +++ gcc/opts.c (working copy) > @@ -627,6 +627,11 @@ default_options_optimization (struct gcc_options * > default_param_value (PARAM_MIN_CROSSJUMP_INSNS), > opts->x_param_values, opts_set->x_param_values); > > + if (flag_auto_profile) > + maybe_set_param_value > + (PARAM_EARLY_INLINER_MAX_ITERATIONS, 10, > + opts->x_param_values, opts_set->x_param_values); > + > /* Allow default optimizations to be specified on a per-machine basis. */ > maybe_default_options (opts, opts_set, > targetm_common.option_optimization_table, > @@ -1720,6 +1725,56 @@ common_handle_option (struct gcc_options *opts, > opts->x_flag_devirtualize_speculatively = false; > break; > > + case OPT_fauto_profile_: > + opts->x_auto_profile_file = xstrdup (arg); > + opts->x_flag_auto_profile = true; > + value = true; > + /* No break here - do -fauto-profile processing. */ > + case OPT_fauto_profile: > + if (!opts_set->x_flag_branch_probabilities) > + opts->x_flag_branch_probabilities = value; > + if (!opts_set->x_flag_profile_values) > + opts->x_flag_profile_values = value; > + if (!opts_set->x_flag_unroll_loops) > + opts->x_flag_unroll_loops = value; > + if (!opts_set->x_flag_peel_loops) > + opts->x_flag_peel_loops = value; > + if (!opts_set->x_flag_tracer) > + opts->x_flag_tracer = value; > + if (!opts_set->x_flag_value_profile_transformations) > + opts->x_flag_value_profile_transformations = value; > + if (!opts_set->x_flag_inline_functions) > + opts->x_flag_inline_functions = value; > + if (!opts_set->x_flag_ipa_cp) > + opts->x_flag_ipa_cp = value; > + if (!opts_set->x_flag_ipa_cp_clone > + && value && opts->x_flag_ipa_cp) > + opts->x_flag_ipa_cp_clone = value; > + if (!opts_set->x_flag_predictive_commoning) > + opts->x_flag_predictive_commoning = value; > + if (!opts_set->x_flag_unswitch_loops) > + opts->x_flag_unswitch_loops = value; > + if (!opts_set->x_flag_gcse_after_reload) > + opts->x_flag_gcse_after_reload = value; > + if (!opts_set->x_flag_tree_loop_vectorize > + && !opts_set->x_flag_tree_vectorize) > + opts->x_flag_tree_loop_vectorize = value; > + if (!opts_set->x_flag_tree_slp_vectorize > + && !opts_set->x_flag_tree_vectorize) > + opts->x_flag_tree_slp_vectorize = value; > + if (!opts_set->x_flag_vect_cost_model) > + opts->x_flag_vect_cost_model = VECT_COST_MODEL_DYNAMIC; > + if (!opts_set->x_flag_tree_loop_distribute_patterns) > + opts->x_flag_tree_loop_distribute_patterns = value; > + /* Indirect call profiling should do all useful transformations > + speculative devirutalization does. */ > + if (!opts_set->x_flag_devirtualize_speculatively > + && opts->x_flag_value_profile_transformations) > + opts->x_flag_devirtualize_speculatively = false; > + if (!opts_set->x_flag_profile_correction) > + opts->x_flag_profile_correction = value; > + break; Instead of cut&paste from profile-use I would go for function setting the common flags. > Index: gcc/tree-cfg.c > =================================================================== > --- gcc/tree-cfg.c (revision 210180) > +++ gcc/tree-cfg.c (working copy) > @@ -1877,6 +1877,14 @@ gimple_merge_blocks (basic_block a, basic_block b) > } > } > > + /* When merging two BBs, if their counts are different, the larger > + count is selected as the new bb count. */ > + if (flag_auto_profile && a->count != b->count) > + { > + a->count = MAX (a->count, b->count); > + a->frequency = MAX (a->frequency, b->frequency); > + } Separate change and OK for mainline. (perhaps with comment update that this if to handle better inconsistent profiles) > + > /* Merge the sequences. */ > last = gsi_last_bb (a); > gsi_insert_seq_after (&last, bb_seq (b), GSI_NEW_STMT); > Index: gcc/cgraphclones.c > =================================================================== > --- gcc/cgraphclones.c (revision 210180) > +++ gcc/cgraphclones.c (working copy) > @@ -435,6 +435,11 @@ cgraph_clone_node (struct cgraph_node *n, tree dec > } > else > count_scale = 0; > + /* In AutoFDO, if edge count is larger than callee's entry block > + count, we will not update the original callee because it may > + mistakenly mark some hot function as cold. */ > + if (flag_auto_profile && count >= n->count) > + update_original = false; This looks like a hack. Lets go with it later - I see what you are shooting for here and non-autoFDO has kind of same problem, too. Counts are not at all that reliable after some inlining. I wonder how this hack would fare for -fprofile-use > @@ -1286,6 +1285,9 @@ del_node_map (void) > struct cgraph_node* > find_func_by_profile_id (int profile_id) > { > + if (flag_auto_profile) > + return cgraph_node_for_asm ( > + get_identifier ((const char *)(size_t)profile_id)); I think with LTO we will need to make this machinery safe for symbol mangling. Any plans on this? > @@ -1492,8 +1494,8 @@ gimple_ic_transform (gimple_stmt_iterator *gsi) > /* The order of CHECK_COUNTER calls is important - > since check_counter can correct the third parameter > and we want to make count <= all <= bb_all. */ > - if ( check_counter (stmt, "ic", &all, &bb_all, bb_all) > - || check_counter (stmt, "ic", &count, &all, all)) > + if (!flag_auto_profile && (check_counter (stmt, "ic", &all, &bb_all, bb_all) > + || check_counter (stmt, "ic", &count, &all, all))) Why you skip check for auto profile here, but not for other transforms? Similar check is now done at ipa-profile later. I suppose you may want to make autofdo code to make the counters just look right when they are supposed to be right. > Index: gcc/auto-profile.c > =================================================================== > --- gcc/auto-profile.c (revision 0) > +++ gcc/auto-profile.c (revision 0) > @@ -0,0 +1,1584 @@ > +/* Calculate branch probabilities, and basic block execution counts. > + Copyright (C) 2012. Free Software Foundation, Inc. > + Contributed by Dehao Chen (dehao@google.com) I will look into this file incrementally. You want to update copyright year. > Index: gcc/ipa-inline.c > =================================================================== > --- gcc/ipa-inline.c (revision 210180) > +++ gcc/ipa-inline.c (working copy) > @@ -121,6 +121,7 @@ along with GCC; see the file COPYING3. If not see > #include "ipa-inline.h" > #include "ipa-utils.h" > #include "sreal.h" > +#include "auto-profile.h" > #include "cilk.h" > > /* Statistics we collect about inlining algorithm. */ > @@ -450,6 +451,8 @@ want_early_inline_function_p (struct cgraph_edge * > > if (DECL_DISREGARD_INLINE_LIMITS (callee->decl)) > ; > + else if (flag_auto_profile && afdo_callsite_hot_enough_for_early_inline (e)) > + ; You want to explain what happens here (I remember it from presentation :) > Index: gcc/ipa-inline.h > =================================================================== > --- gcc/ipa-inline.h (revision 210180) > +++ gcc/ipa-inline.h (working copy) > @@ -345,3 +345,5 @@ reset_edge_growth_cache (struct cgraph_edge *edge) > edge_growth_cache[edge->uid] = zero; > } > } > + > +unsigned int early_inliner (function *fun); Can't this be handled by scheduling early inliner as subpass of autofdo or something similar? I would rather make this explicit in PM. Honza ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH] AutoFDO patch for trunk 2014-05-15 21:04 ` Jan Hubicka @ 2014-10-09 0:24 ` Dehao Chen 2014-10-09 22:34 ` Jan Hubicka 2014-10-14 15:08 ` Jan Hubicka 0 siblings, 2 replies; 25+ messages in thread From: Dehao Chen @ 2014-10-09 0:24 UTC (permalink / raw) To: Jan Hubicka; +Cc: GCC Patches, David Li [-- Attachment #1: Type: text/plain, Size: 12812 bytes --] Hi, Honza, Sorry for the delay. I just picked up the original patch, and updated it with your comments. I've addressed most of your comments. Something else to discuss inlined. I had refactored the patch to make it much less intrusive. New patch is attached (ChangeLog will be added in the final version of the patch). The performance of the new patch is the same as the original patch. Thanks, Dehao >> Index: gcc/bb-reorder.c >> =================================================================== >> --- gcc/bb-reorder.c (revision 210180) >> +++ gcc/bb-reorder.c (working copy) >> @@ -2663,7 +2663,7 @@ pass_partition_blocks::gate (function *fun) >> user defined section attributes. Don't call it if either case >> arises. */ >> return (flag_reorder_blocks_and_partition >> - && optimize >> + && optimize &&!flag_auto_profile > > Formatting error. Why we want !flag_auto_profile? Can't we just arrange > flag_reorder_blocks_and_partition to default to false? > > In fact I would like to see flag_reorder_blocks_and_partition to offload obviously > cold code (in partiuclar the EH cleanups) even without profile. That should work > well with autofdo. This will cause bzip2 performance to degrade 6%. I haven't had time to triage the problem. Will investigate this later. >> Index: gcc/coverage.c >> =================================================================== >> --- gcc/coverage.c (revision 210180) >> +++ gcc/coverage.c (working copy) >> @@ -54,6 +54,7 @@ along with GCC; see the file COPYING3. If not see >> #include "intl.h" >> #include "filenames.h" >> #include "target.h" >> +#include "auto-profile.h" >> >> #include "gcov-io.h" >> #include "gcov-io.c" >> @@ -1175,7 +1176,9 @@ coverage_init (const char *filename) >> >> bbg_file_stamp = local_tick; >> >> - if (flag_branch_probabilities) >> + if (flag_auto_profile) >> + init_auto_profile (); > > Perhaps initialization can happen from toplev.c instead? This is actually reading of afdo profile. I've renamed the function to make it consistent. >> Index: gcc/cgraphclones.c >> =================================================================== >> --- gcc/cgraphclones.c (revision 210180) >> +++ gcc/cgraphclones.c (working copy) >> @@ -435,6 +435,11 @@ cgraph_clone_node (struct cgraph_node *n, tree dec >> } >> else >> count_scale = 0; >> + /* In AutoFDO, if edge count is larger than callee's entry block >> + count, we will not update the original callee because it may >> + mistakenly mark some hot function as cold. */ >> + if (flag_auto_profile && count >= n->count) >> + update_original = false; > > This looks like a hack. Lets go with it later - I see what you are shooting for here > and non-autoFDO has kind of same problem, too. Counts are not at all that reliable > after some inlining. > > I wonder how this hack would fare for -fprofile-use It does not affect -fprofile-use performance in my testing. However, for AutoFDO, this happens much more often. >> @@ -1286,6 +1285,9 @@ del_node_map (void) >> struct cgraph_node* >> find_func_by_profile_id (int profile_id) >> { >> + if (flag_auto_profile) >> + return cgraph_node_for_asm ( >> + get_identifier ((const char *)(size_t)profile_id)); > > I think with LTO we will need to make this machinery safe for symbol mangling. Any plans on this? No plan yet. Let's get AutoFDO in first, and spend some separate effort to make AutoFDO+LTO work? >> Index: gcc/ipa-inline.h >> =================================================================== >> --- gcc/ipa-inline.h (revision 210180) >> +++ gcc/ipa-inline.h (working copy) >> @@ -345,3 +345,5 @@ reset_edge_growth_cache (struct cgraph_edge *edge) >> edge_growth_cache[edge->uid] = zero; >> } >> } >> + >> +unsigned int early_inliner (function *fun); > > Can't this be handled by scheduling early inliner as subpass of autofdo or something similar? > I would rather make this explicit in PM. See my comment below in auto-profile.c >> Index: gcc/auto-profile.c >> =================================================================== >> --- gcc/auto-profile.c (revision 0) >> +++ gcc/auto-profile.c (revision 0) >> + After the above 3 phases, all profile is readily annotated on the GCC IR. >> + AutoFDO tries to reuse all FDO infrastructure as much as possible to make >> + use of the profile. E.g. it uses existing mechanism to calculate the basic >> + block/edge frequency, as well as the cgraph node/edge count. >> +*/ > > I suppose the phases can be made explicit to the PM, so early inliner can stay > a pass. Unfortunately, this is more complicated: while(changed) { vpt(); einline(); annotate(); } >> +/* Store a string array, indexed by string position in the array. */ >> +class string_table { >> +public: >> + static string_table *create (); >> + >> + /* For a given string, returns its index. */ >> + int get_index (const char *name) const; >> + >> + /* For a given decl, returns the index of the decl name. */ >> + int get_index_by_decl (tree decl) const; >> + >> + /* For a given index, returns the string. */ >> + const char *get_name (int index) const; > > I wonder how these should work with LTO. I suppose we can tell user to not LTO > for train run, we should teach the table to ignore LTO generated suffixes as > well as random seeds (like coverage code already does). > > What happens when two static functions have same name? Shall we first focus on non-LTO version. And there could be separate effort to make AutoFDO+LTO work? For static functions, there will only be one copy in the profile. the profile will be merged. I agree that this could potentially cause problem. But OTOH, in a good practice, these functions should *not* be hot functions. At least in our practice, we didn't see any hot functions like this. > >> +/* Profile for all functions. */ >> +class autofdo_source_profile { >> +public: >> + static autofdo_source_profile *create () >> + { >> + autofdo_source_profile *map = new autofdo_source_profile (); > > I think we still want empty lines here. Otherwise I trust you on following GCC C++ coding standards :) >> + if (map->read ()) >> + return map; >> + delete map; >> + return NULL; >> + } > >> +/* Helper functions. */ >> + >> +/* Return the original name of NAME: strip the suffix that starts >> + with '.' */ >> + >> +static const char *get_original_name (const char *name) >> +{ >> + char *ret = xstrdup (name); >> + char *find = strchr (ret, '.'); >> + if (find != NULL) >> + *find = 0; >> + return ret; >> +} > > Stripping all suffixes will likely make conflicts on static functions, right? > Example like this will get you a conflict > m() > { > void t(void) > { > } > t(); > } > m2() > { > void t2(void) > { > } > t2(); > } > > that is of course not that important given that nested functions are not top priority > but it would be nice to handle it gratefuly. I agree. My original intention is to make the function name handling as simple as possible. The current impl is based on the asumption that all important (hot) functions deserve a separate name (not just suffix difference). We could definitely refine the function name handling in the future patches. > I would definitely merge logic here with logic in gcov that already knows how to skip > random seeds and other cases that makes filenames to diverge... Could you point me to the gcov logic that handles skipping function name suffix? >> + >> +/* Return the combined location, which is a 32bit integer in which >> + higher 16 bits stores the line offset of LOC to the start lineno >> + of DECL, The lower 16 bits stores the discrimnator. */ >> + >> +static unsigned >> +get_combined_location (location_t loc, tree decl) >> +{ >> + return ((LOCATION_LINE (loc) - DECL_SOURCE_LINE (decl)) << 16); >> +} > > What happens on overflows here? I guess they are possible - 65536 lines is not infinity > these days. If a function is longer than 65535 lines, then some lines may have incorrect count attributed. But in pactice, we did not find any function that is over 10K lines. > The discriminator is added later? Yes, we need to update this when discrminator support is ported upstream. > > Please review the file for the coding standards. I'll do a final pass of the coding standard after all code logics are OK. >> + >> +static void >> +fake_read_autofdo_module_profile () > > Probably could get in only after LIPO? Yes, this is only there to make sure that the profile format is compatible with google branch. But after LIPO is upstream, we can add the actual content here. >> +{ >> + /* Read in the module info. */ >> + gcov_read_unsigned (); >> + >> + /* Skip the length of the section. */ >> + gcov_read_unsigned (); > > You probably want to check that length is 0 or so. We already checked num_module is 0, whch is equivalent with checking length is 0. >> +/* If a basic block's count is known, and only one of its in/out edges' count >> + is unknown, its count can be calculated. >> + Meanwhile, if all of the in/out edges' counts are known, then the basic >> + block's unknown count can also be calculated. >> + IS_SUCC is true if out edges of a basic blocks are examined. >> + Return TRUE if any basic block/edge count is changed. */ > > Interesting profile cleanup. > I wonder wy EDGE_ANNOTATED is exported now (i.e. why it is not maintained only withing > auto-FDO). Do you have more use for it outside auto-FDO? > > I was thinking that having a flag about profile being reliable may be useful idea, but never > really tried to implement it. Good point, removed the ANNOTATED flags. >> + >> +/* Special propagation for circuit expressions. Because GCC translates >> + control flow into data flow for circuit expressions. E.g. >> + BB1: >> + if (a && b) >> + BB2 >> + else >> + BB3 >> + >> + will be translated into: >> + >> + BB1: >> + if (a) >> + goto BB.t1 >> + else >> + goto BB.t3 >> + BB.t1: >> + if (b) >> + goto BB.t2 >> + else >> + goto BB.t3 >> + BB.t2: >> + goto BB.t3 >> + BB.t3: >> + tmp = PHI (0 (BB1), 0 (BB.t1), 1 (BB.t2) >> + if (tmp) >> + goto BB2 >> + else >> + goto BB3 >> + >> + In this case, we need to propagate through PHI to determine the edge >> + count of BB1->BB.t1, BB.t1->BB.t2. */ > > So here you have known probabilities of BB1, BB2 and BB3 and you need to guess all the rest? > Why it need so much of gimple matching? This is basically because some basic blocks have no stmt so that we can not get its count from source level profile. However, we can get edge count through PHI nodes. >> + while (changed && i++ < 10) >> + { >> + changed = false; >> + >> + if (afdo_propagate_edge (true)) >> + changed = true; >> + if (afdo_propagate_edge (false)) >> + changed = true; >> + afdo_propagate_circuit (); >> + } > > I wonder if there is any code to share with same (except for the circuit thing) code in profile.c > and gcov.c? Profile.c also have the minimal flow algorithm for fixing broken profiles, perhaps > it could be useful here? The initial SampleFDO implementation uses MCF algorithm to calculate edge counts (the current mcf.c actually comes from that effort). However, from our performance tuning effort, we found that MCF is an overkill for AutoFDO purpose, and it's extremely slow and hard to debug. Thus we implemented this ad-hoc heuristic for propagation. >> + FOR_ALL_BB_FN (bb, cfun) >> + { >> + edge e; >> + edge_iterator ei; >> + >> + FOR_EACH_EDGE (e, ei, bb->succs) >> + e->count = >> + (double) bb->count * e->probability / REG_BR_PROB_BASE; >> + bb->aux = NULL; >> + } >> + >> + loop_optimizer_finalize (); >> + free_dominance_info (CDI_DOMINATORS); >> + free_dominance_info (CDI_POST_DOMINATORS); >> +} > > I wonder if using the estimated profile to fill in the missing gaps would make sense? > I.e. running the standard branch probabilities->freuqencies propagation code bug this > time taking the determined counts as known values? Not sure if I follow. Could you elaborate? >> + >> +/* Perform value profile transformation using AutoFDO profile. Add the >> + promoted stmts to PROMOTED_STMTS. Return TRUE if there is any >> + indirect call promoted. */ >> + >> +static bool >> +afdo_vpt_for_early_inline (stmt_set *promoted_stmts) > > What exactly this function does? It inlines according to the inline decisions > made at train run time and recorded to profile feedback? As replied above, we need iterative vpt-einline to make sure the IR looks exactly the same as profiling binary before annotation. [-- Attachment #2: diff.txt --] [-- Type: text/plain, Size: 73937 bytes --] Index: gcc/Makefile.in =================================================================== --- gcc/Makefile.in (revision 215826) +++ gcc/Makefile.in (working copy) @@ -1153,6 +1153,7 @@ alias.o \ alloc-pool.o \ auto-inc-dec.o \ + auto-profile.o \ bb-reorder.o \ bitmap.o \ bt-load.o \ Index: gcc/cgraphclones.c =================================================================== --- gcc/cgraphclones.c (revision 215826) +++ gcc/cgraphclones.c (working copy) @@ -453,6 +453,11 @@ } else count_scale = 0; + /* In AutoFDO, if edge count is larger than callee's entry block + count, we will not update the original callee because it may + mistakenly mark some hot function as cold. */ + if (flag_auto_profile && gcov_count >= count) + update_original = false; if (update_original) { count -= gcov_count; Index: gcc/gcov-io.h =================================================================== --- gcc/gcov-io.h (revision 215826) +++ gcc/gcov-io.h (working copy) @@ -244,6 +244,9 @@ #define GCOV_TAG_PROGRAM_SUMMARY ((gcov_unsigned_t)0xa3000000) #define GCOV_TAG_SUMMARY_LENGTH(NUM) \ (1 + GCOV_COUNTERS_SUMMABLE * (10 + 3 * 2) + (NUM) * 5) +#define GCOV_TAG_AFDO_FILE_NAMES ((gcov_unsigned_t)0xaa000000) +#define GCOV_TAG_AFDO_FUNCTION ((gcov_unsigned_t)0xac000000) +#define GCOV_TAG_AFDO_WORKING_SET ((gcov_unsigned_t)0xaf000000) /* Counters that are collected. */ Index: gcc/bb-reorder.c =================================================================== --- gcc/bb-reorder.c (revision 215826) +++ gcc/bb-reorder.c (working copy) @@ -1569,15 +1569,14 @@ /* Mark which partition (hot/cold) each basic block belongs in. */ FOR_EACH_BB_FN (bb, cfun) { - bool cold_bb = false; + bool cold_bb = probably_never_executed_bb_p (cfun, bb); - if (probably_never_executed_bb_p (cfun, bb)) + if (!flag_auto_profile && cold_bb) { /* Handle profile insanities created by upstream optimizations by also checking the incoming edge weights. If there is a non-cold incoming edge, conservatively prevent this block from being split into the cold section. */ - cold_bb = true; FOR_EACH_EDGE (e, ei, bb->preds) if (!probably_never_executed_edge_p (cfun, e)) { @@ -2672,7 +2671,7 @@ user defined section attributes. Don't call it if either case arises. */ return (flag_reorder_blocks_and_partition - && optimize + && optimize && !flag_auto_profile /* See gate_handle_reorder_blocks. We should not partition if we are going to omit the reordering. */ && optimize_function_for_speed_p (fun) Index: gcc/predict.c =================================================================== --- gcc/predict.c (revision 215826) +++ gcc/predict.c (working copy) @@ -161,7 +161,7 @@ /* Return TRUE if frequency FREQ is considered to be hot. */ -static inline bool +bool maybe_hot_count_p (struct function *fun, gcov_type count) { if (fun && profile_status_for_fn (fun) != PROFILE_READ) @@ -2852,7 +2852,7 @@ /* Don't overwrite the estimated frequencies when the profile for the function is missing. We may drop this function PROFILE_GUESSED later in drop_profile (). */ - if (!ENTRY_BLOCK_PTR_FOR_FN (cfun)->count) + if (!flag_auto_profile && !ENTRY_BLOCK_PTR_FOR_FN (cfun)->count) return 0; FOR_BB_BETWEEN (bb, ENTRY_BLOCK_PTR_FOR_FN (cfun), NULL, next_bb) @@ -3223,7 +3223,8 @@ count_max = MAX (bb->count, count_max); if (profile_status_for_fn (cfun) == PROFILE_GUESSED - || (profile_status_for_fn (cfun) == PROFILE_READ && count_max < REG_BR_PROB_BASE/10)) + || (!flag_auto_profile && profile_status_for_fn (cfun) == PROFILE_READ + && count_max < REG_BR_PROB_BASE/10)) { loop_optimizer_init (0); add_noreturn_fake_exit_edges (); Index: gcc/auto-profile.c =================================================================== --- gcc/auto-profile.c (revision 0) +++ gcc/auto-profile.c (revision 0) @@ -0,0 +1,1662 @@ +/* Read and annotate call graph profile from the auto profile data file. + Copyright (C) 2014. Free Software Foundation, Inc. + Contributed by Dehao Chen (dehao@google.com) + +This file is part of GCC. + +GCC is free software; you can redistribute it and/or modify it under +the terms of the GNU General Public License as published by the Free +Software Foundation; either version 3, or (at your option) any later +version. + +GCC is distributed in the hope that it will be useful, but WITHOUT ANY +WARRANTY; without even the implied warranty of MERCHANTABILITY or +FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License +for more details. + +You should have received a copy of the GNU General Public License +along with GCC; see the file COPYING3. If not see +<http://www.gnu.org/licenses/>. */ + +#include <string.h> +#include <map> +#include <vector> +#include <set> + +#include "config.h" +#include "system.h" +#include "coretypes.h" +#include "tree.h" +#include "tree-pass.h" +#include "flags.h" +#include "basic-block.h" +#include "diagnostic-core.h" +#include "gcov-io.h" +#include "input.h" +#include "profile.h" +#include "langhooks.h" +#include "opts.h" +#include "tree-pass.h" +#include "cfgloop.h" +#include "tree-ssa-alias.h" +#include "tree-cfg.h" +#include "tree-cfgcleanup.h" +#include "tree-ssa-operands.h" +#include "tree-into-ssa.h" +#include "internal-fn.h" +#include "is-a.h" +#include "gimple-expr.h" +#include "gimple.h" +#include "gimple-iterator.h" +#include "gimple-ssa.h" +#include "cgraph.h" +#include "value-prof.h" +#include "coverage.h" +#include "params.h" +#include "ipa-inline.h" +#include "tree-inline.h" +#include "auto-profile.h" + +/* The following routines implements AutoFDO optimization. + + This optimization uses sampling profiles to annotate basic block counts + and uses heuristics to estimate branch probabilities. + + There are three phases in AutoFDO: + + Phase 1: Read profile from the profile data file. + The following info is read from the profile datafile: + * string_table: a map between function name and its index. + * autofdo_source_profile: a map from function_instance name to + function_instance. This is represented as a forest of + function_instances. + * WorkingSet: a histogram of how many instructions are covered for a + given percentage of total cycles. + + Phase 2: Early inline + VPT. + Early inline uses autofdo_source_profile to find if a callsite is: + * inlined in the profiled binary. + * callee body is hot in the profiling run. + If both condition satisfies, early inline will inline the callsite + regardless of the code growth. + Phase 2 is an iterative process. During each iteration, we also check + if an indirect callsite is promoted and inlined in the profiling run. + If yes, vpt will happen to force promote it and in the next iteration, + einline will inline the promoted callsite in the next iteration. + + Phase 3: Annotate control flow graph. + AutoFDO uses a separate pass to: + * Annotate basic block count + * Estimate branch probability + + After the above 3 phases, all profile is readily annotated on the GCC IR. + AutoFDO tries to reuse all FDO infrastructure as much as possible to make + use of the profile. E.g. it uses existing mechanism to calculate the basic + block/edge frequency, as well as the cgraph node/edge count. +*/ + +#define DEFAULT_AUTO_PROFILE_FILE "fbdata.afdo" + +namespace autofdo { + +/* Represent a source location: (function_decl, lineno). */ +typedef std::pair<tree, unsigned> decl_lineno; + +/* Represent an inline stack. vector[0] is the leaf node. */ +typedef std::vector<decl_lineno> inline_stack; + +/* String array that stores function names. */ +typedef std::vector<const char *> string_vector; + +/* Map from function name's index in string_table to target's + execution count. */ +typedef std::map<unsigned, gcov_type> icall_target_map; + +/* Set of gimple stmts. Used to track if the stmt has already been promoted + to direct call. */ +typedef std::set<gimple> stmt_set; + +/* Represent count info of an inline stack. */ +struct count_info +{ + /* Sampled count of the inline stack. */ + gcov_type count; + + /* Map from indirect call target to its sample count. */ + icall_target_map targets; + + /* Whether this inline stack is already used in annotation. + + Each inline stack should only be used to annotate IR once. + This will be enforced when instruction-level discriminator + is supported. */ + bool annotated; +}; + +/* operator< for "const char *". */ +struct string_compare +{ + bool operator() (const char *a, const char *b) const + { return strcmp (a, b) < 0; } +}; + +/* Store a string array, indexed by string position in the array. */ +class string_table { +public: + static string_table *create (); + + /* For a given string, returns its index. */ + int get_index (const char *name) const; + + /* For a given decl, returns the index of the decl name. */ + int get_index_by_decl (tree decl) const; + + /* For a given index, returns the string. */ + const char *get_name (int index) const; + +private: + string_table () {} + bool read (); + + typedef std::map<const char *, unsigned, string_compare> string_index_map; + string_vector vector_; + string_index_map map_; +}; + +/* Profile of a function instance: + 1. total_count of the function. + 2. head_count of the function (only valid when function is a top-level + function_instance, i.e. it is the original copy instead of the + inlined copy). + 3. map from source location (decl_lineno) of the inlined callsite to + profile (count_info). + 4. map from callsite to callee function_instance. */ +class function_instance { +public: + typedef std::vector<function_instance *> function_instance_stack; + + /* Read the profile and return a function_instance with head count as + HEAD_COUNT. Recursively read callsites to create nested function_instances + too. STACK is used to track the recursive creation process. */ + static function_instance *read_function_instance ( + function_instance_stack *stack, gcov_type head_count); + + /* Recursively deallocate all callsites (nested function_instances). */ + ~function_instance (); + + /* Accessors. */ + int name () const { return name_; } + gcov_type total_count () const { return total_count_; } + gcov_type head_count () const { return head_count_; } + + /* Recursively traverse STACK starting from LEVEL to find the corresponding + function_instance. */ + function_instance *get_function_instance (const inline_stack &stack, + unsigned level); + + /* Store the profile info for LOC in INFO. Return TRUE if profile info + is found. */ + bool get_count_info (location_t loc, count_info *info) const; + + /* Read the inlined indirect call target profile for STMT and store it in + MAP, return the total count for all inlined indirect calls. */ + gcov_type find_icall_target_map (gimple stmt, icall_target_map *map) const; + + /* Sum of counts that is used during annotation. */ + gcov_type total_annotated_count () const; + + /* Mark LOC as annotated. */ + void mark_annotated (location_t loc); + +private: + /* Callsite, represented as (decl_lineno, callee_function_name_index). */ + typedef std::pair<unsigned, unsigned> callsite; + + /* Map from callsite to callee function_instance. */ + typedef std::map<callsite, function_instance *> callsite_map; + + function_instance (unsigned name, gcov_type head_count) + : name_(name), total_count_(0), head_count_(head_count) {} + + /* Traverse callsites of the current function_instance to find one at the + location of LINENO and callee name represented in DECL. */ + function_instance *get_function_instance_by_decl ( + unsigned lineno, tree decl) const; + + /* Map from source location (decl_lineno) to profile (count_info). */ + typedef std::map<unsigned, count_info> position_count_map; + + /* function_instance name index in the string_table. */ + unsigned name_; + + /* Total sample count. */ + gcov_type total_count_; + + /* Entry BB's sample count. */ + gcov_type head_count_; + + /* Map from callsite location to callee function_instance. */ + callsite_map callsites; + + /* Map from source location to count_info. */ + position_count_map pos_counts; +}; + +/* Profile for all functions. */ +class autofdo_source_profile { +public: + static autofdo_source_profile *create () + { + autofdo_source_profile *map = new autofdo_source_profile (); + + if (map->read ()) + return map; + delete map; + return NULL; + } + + ~autofdo_source_profile (); + + /* For a given DECL, returns the top-level function_instance. */ + function_instance *get_function_instance_by_decl (tree decl) const; + + /* Find count_info for a given gimple STMT. If found, store the count_info + in INFO and return true; otherwise return false. */ + bool get_count_info (gimple stmt, count_info *info) const; + + /* Find total count of the callee of EDGE. */ + gcov_type get_callsite_total_count (struct cgraph_edge *edge) const; + + /* Update value profile INFO for STMT from the inlined indirect callsite. + Return true if INFO is updated. */ + bool update_inlined_ind_target (gimple stmt, count_info *info); + + /* Mark LOC as annotated. */ + void mark_annotated (location_t loc); + +private: + /* Map from function_instance name index (in string_table) to + function_instance. */ + typedef std::map<unsigned, function_instance *> + name_function_instance_map; + + autofdo_source_profile () {} + + /* Read AutoFDO profile and returns TRUE on success. */ + bool read (); + + /* Return the function_instance in the profile that correspond to the + inline STACK. */ + function_instance *get_function_instance_by_inline_stack ( + const inline_stack &stack) const; + + name_function_instance_map map_; +}; + +/* Store the strings read from the profile data file. */ +static string_table *afdo_string_table; + +/* Store the AutoFDO source profile. */ +static autofdo_source_profile *afdo_source_profile; + +/* gcov_ctr_summary structure to store the profile_info. */ +static struct gcov_ctr_summary *afdo_profile_info; + +/* Helper functions. */ + +/* Return the original name of NAME: strip the suffix that starts + with '.' */ + +static const char *get_original_name (const char *name) +{ + char *ret = xstrdup (name); + char *find = strchr (ret, '.'); + if (find != NULL) + *find = 0; + return ret; +} + +/* Return the combined location, which is a 32bit integer in which + higher 16 bits stores the line offset of LOC to the start lineno + of DECL, The lower 16 bits stores the discrimnator. */ + +static unsigned +get_combined_location (location_t loc, tree decl) +{ + return ((LOCATION_LINE (loc) - DECL_SOURCE_LINE (decl)) << 16); +} + +/* Return the function decl of a given lexical BLOCK. */ + +static tree +get_function_decl_from_block (tree block) +{ + tree decl; + + if (LOCATION_LOCUS (BLOCK_SOURCE_LOCATION (block) == UNKNOWN_LOCATION)) + return NULL_TREE; + + for (decl = BLOCK_ABSTRACT_ORIGIN (block); + decl && (TREE_CODE (decl) == BLOCK); + decl = BLOCK_ABSTRACT_ORIGIN (decl)) + if (TREE_CODE (decl) == FUNCTION_DECL) + break; + return decl; +} + +/* Store inline stack for STMT in STACK. */ + +static void +get_inline_stack (location_t locus, inline_stack *stack) +{ + if (LOCATION_LOCUS (locus) == UNKNOWN_LOCATION) + return; + + tree block = LOCATION_BLOCK (locus); + if (block && TREE_CODE (block) == BLOCK) + { + int level = 0; + for (block = BLOCK_SUPERCONTEXT (block); + block && (TREE_CODE (block) == BLOCK); + block = BLOCK_SUPERCONTEXT (block)) + { + location_t tmp_locus = BLOCK_SOURCE_LOCATION (block); + if (LOCATION_LOCUS (tmp_locus) == UNKNOWN_LOCATION) + continue; + + tree decl = get_function_decl_from_block (block); + stack->push_back (std::make_pair ( + decl, get_combined_location (locus, decl))); + locus = tmp_locus; + level++; + } + } + stack->push_back (std::make_pair ( + current_function_decl, + get_combined_location (locus, current_function_decl))); +} + +/* Return STMT's combined location, which is a 32bit integer in which + higher 16 bits stores the line offset of LOC to the start lineno + of DECL, The lower 16 bits stores the discrimnator. */ + +static unsigned +get_relative_location_for_stmt (gimple stmt) +{ + location_t locus = gimple_location (stmt); + if (LOCATION_LOCUS (locus) == UNKNOWN_LOCATION) + return UNKNOWN_LOCATION; + + for (tree block = gimple_block (stmt); + block && (TREE_CODE (block) == BLOCK); + block = BLOCK_SUPERCONTEXT (block)) + if (LOCATION_LOCUS (BLOCK_SOURCE_LOCATION (block)) != UNKNOWN_LOCATION) + return get_combined_location ( + locus, get_function_decl_from_block (block)); + return get_combined_location (locus, current_function_decl); +} + +/* Return true if BB contains indirect call. */ + +static bool +has_indirect_call (basic_block bb) +{ + gimple_stmt_iterator gsi; + + for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi)) + { + gimple stmt = gsi_stmt (gsi); + if (gimple_code (stmt) == GIMPLE_CALL + && (gimple_call_fn (stmt) == NULL + || TREE_CODE (gimple_call_fn (stmt)) != FUNCTION_DECL)) + return true; + } + return false; +} + +/* Member functions for string_table. */ + +string_table * +string_table::create () +{ + string_table *map = new string_table(); + if (map->read ()) + return map; + delete map; + return NULL; +} + +/* Return the index of a given function NAME. */ + +int +string_table::get_index (const char *name) const +{ + if (name == NULL) + return -1; + string_index_map::const_iterator iter = map_.find (name); + if (iter == map_.end()) + return -1; + else + return iter->second; +} + +/* Return the index of a given function DECL. */ + +int +string_table::get_index_by_decl (tree decl) const +{ + const char *name = get_original_name ( + IDENTIFIER_POINTER (DECL_ASSEMBLER_NAME (decl))); + int ret = get_index (name); + if (ret != -1) + return ret; + ret = get_index (lang_hooks.dwarf_name (decl, 0)); + if (ret != -1) + return ret; + if (DECL_ABSTRACT_ORIGIN (decl)) + return get_index_by_decl (DECL_ABSTRACT_ORIGIN (decl)); + else + return -1; +} + +/* Return the function name of a given INDEX. */ + +const char * +string_table::get_name (int index) const +{ + gcc_assert (index > 0 && index < (int) vector_.size()); + return vector_[index]; +} + +/* Read the string table. Return TRUE if reading is successful. */ + +bool +string_table::read () +{ + if (gcov_read_unsigned () != GCOV_TAG_AFDO_FILE_NAMES) + return false; + /* Skip the length of the section. */ + gcov_read_unsigned (); + /* Read in the file name table. */ + unsigned string_num = gcov_read_unsigned (); + for (unsigned i = 0; i < string_num; i++) + { + vector_.push_back (get_original_name (gcov_read_string ())); + map_[vector_.back()] = i; + } + return true; +} + + +/* Member functions for function_instance. */ + +function_instance::~function_instance () +{ + for (callsite_map::iterator iter = callsites.begin(); + iter != callsites.end(); ++iter) + delete iter->second; +} + +/* Traverse callsites of the current function_instance to find one at the + location of LINENO and callee name represented in DECL. */ + +function_instance * +function_instance::get_function_instance_by_decl ( + unsigned lineno, tree decl) const +{ + int func_name_idx = afdo_string_table->get_index_by_decl (decl); + if (func_name_idx != -1) + { + callsite_map::const_iterator ret = callsites.find ( + std::make_pair (lineno, func_name_idx)); + if (ret != callsites.end ()) + return ret->second; + } + func_name_idx = afdo_string_table->get_index ( + lang_hooks.dwarf_name (decl, 0)); + if (func_name_idx != -1) + { + callsite_map::const_iterator ret = callsites.find ( + std::make_pair (lineno, func_name_idx)); + if (ret != callsites.end ()) + return ret->second; + } + if (DECL_ABSTRACT_ORIGIN (decl)) + return get_function_instance_by_decl (lineno, DECL_ABSTRACT_ORIGIN (decl)); + else + return NULL; +} + +/* Recursively traverse STACK starting from LEVEL to find the corresponding + function_instance. */ + +function_instance * +function_instance::get_function_instance ( + const inline_stack &stack, unsigned level) +{ + if (level == 0) + return this; + function_instance *s = + get_function_instance_by_decl (stack[level].second, stack[level - 1].first); + if (s) + return s->get_function_instance (stack, level - 1); + else + return NULL; +} + +/* Store the profile info for LOC in INFO. Return TRUE if profile info + is found. */ + +bool +function_instance::get_count_info (location_t loc, count_info *info) const +{ + position_count_map::const_iterator iter = pos_counts.find (loc); + if (iter == pos_counts.end ()) + return false; + *info = iter->second; + return true; +} + +/* Mark LOC as annotated. */ + +void +function_instance::mark_annotated (location_t loc) +{ + position_count_map::iterator iter = pos_counts.find (loc); + if (iter == pos_counts.end ()) + return; + iter->second.annotated = true; +} + +/* Read the inlinied indirect call target profile for STMT and store it in + MAP, return the total count for all inlined indirect calls. */ + +gcov_type +function_instance::find_icall_target_map ( + gimple stmt, icall_target_map *map) const +{ + gcov_type ret = 0; + unsigned stmt_offset = get_relative_location_for_stmt (stmt); + + for (callsite_map::const_iterator iter = callsites.begin(); + iter != callsites.end(); ++iter) + { + unsigned callee = iter->second->name(); + /* Check if callsite location match the stmt. */ + if (iter->first.first != stmt_offset) + continue; + struct cgraph_node *node = find_func_by_profile_id ( + (unsigned long long) afdo_string_table->get_name (callee)); + if (node == NULL) + continue; + if (!check_ic_target (stmt, node)) + continue; + (*map)[callee] = iter->second->total_count (); + ret += iter->second->total_count (); + } + return ret; +} + +/* Read the profile and create a function_instance with head count as + HEAD_COUNT. Recursively read callsites to create nested function_instances + too. STACK is used to track the recursive creation process. */ + +/* function instance profile format: + + ENTRY_COUNT: 8 bytes + NAME_INDEX: 4 bytes + NUM_POS_COUNTS: 4 bytes + NUM_CALLSITES: 4 byte + POS_COUNT_1: + POS_1_OFFSET: 4 bytes + NUM_TARGETS: 4 bytes + COUNT: 8 bytes + TARGET_1: + VALUE_PROFILE_TYPE: 4 bytes + TARGET_IDX: 8 bytes + COUNT: 8 bytes + TARGET_2 + ... + TARGET_n + POS_COUNT_2 + ... + POS_COUNT_N + CALLSITE_1: + CALLSITE_1_OFFSET: 4 bytes + FUNCTION_INSTANCE_PROFILE (nested) + CALLSITE_2 + ... + CALLSITE_n. */ + +function_instance * +function_instance::read_function_instance ( + function_instance_stack *stack, gcov_type head_count) +{ + unsigned name = gcov_read_unsigned (); + unsigned num_pos_counts = gcov_read_unsigned (); + unsigned num_callsites = gcov_read_unsigned (); + function_instance *s = new function_instance (name, head_count); + stack->push_back(s); + + for (unsigned i = 0; i < num_pos_counts; i++) + { + unsigned offset = gcov_read_unsigned () & 0xffff0000; + unsigned num_targets = gcov_read_unsigned (); + gcov_type count = gcov_read_counter (); + s->pos_counts[offset].count = count; + for (unsigned j = 0; j < stack->size(); j++) + (*stack)[j]->total_count_ += count; + for (unsigned j = 0; j < num_targets; j++) + { + /* Only indirect call target histogram is supported now. */ + gcov_read_unsigned (); + gcov_type target_idx = gcov_read_counter (); + s->pos_counts[offset].targets[target_idx] = + gcov_read_counter (); + } + } + for (unsigned i = 0; i < num_callsites; i++) { + unsigned offset = gcov_read_unsigned (); + function_instance *callee_function_instance = + read_function_instance (stack, 0); + s->callsites[std::make_pair (offset, callee_function_instance->name ())] = + callee_function_instance; + } + stack->pop_back(); + return s; +} + +/* Sum of counts that is used during annotation. */ + +gcov_type +function_instance::total_annotated_count () const +{ + gcov_type ret = 0; + for (callsite_map::const_iterator iter = callsites.begin(); + iter != callsites.end(); ++iter) + ret += iter->second->total_annotated_count (); + for (position_count_map::const_iterator iter = pos_counts.begin(); + iter != pos_counts.end(); ++iter) + if (iter->second.annotated) + ret += iter->second.count; + return ret; +} + + +/* Member functions for autofdo_source_profile. */ + +autofdo_source_profile::~autofdo_source_profile () +{ + for (name_function_instance_map::const_iterator iter = map_.begin (); + iter != map_.end (); ++iter) + delete iter->second; +} + +/* For a given DECL, returns the top-level function_instance. */ + +function_instance * +autofdo_source_profile::get_function_instance_by_decl (tree decl) const +{ + int index = afdo_string_table->get_index_by_decl (decl); + if (index == -1) + return NULL; + name_function_instance_map::const_iterator ret = map_.find (index); + return ret == map_.end() ? NULL : ret->second; +} + +/* Find count_info for a given gimple STMT. If found, store the count_info + in INFO and return true; otherwise return false. */ + +bool +autofdo_source_profile::get_count_info (gimple stmt, count_info *info) const +{ + if (LOCATION_LOCUS (gimple_location (stmt)) == cfun->function_end_locus) + return false; + + inline_stack stack; + get_inline_stack (gimple_location (stmt), &stack); + if (stack.size () == 0) + return false; + function_instance *s = get_function_instance_by_inline_stack (stack); + if (s == NULL) + return false; + return s->get_count_info (stack[0].second, info); +} + +/* Mark LOC as annotated. */ + +void +autofdo_source_profile::mark_annotated (location_t loc) { + inline_stack stack; + get_inline_stack (loc, &stack); + if (stack.size () == 0) + return; + function_instance *s = get_function_instance_by_inline_stack (stack); + if (s == NULL) + return; + s->mark_annotated (stack[0].second); +} + +/* Update value profile INFO for STMT from the inlined indirect callsite. + Return true if INFO is updated. */ + +bool +autofdo_source_profile::update_inlined_ind_target ( + gimple stmt, count_info *info) +{ + if (LOCATION_LOCUS (gimple_location (stmt)) == cfun->function_end_locus) + return false; + + count_info old_info; + get_count_info (stmt, &old_info); + gcov_type total = 0; + for (icall_target_map::const_iterator iter = old_info.targets.begin(); + iter != old_info.targets.end(); ++iter) + total += iter->second; + + /* Program behavior changed, original promoted (and inlined) target is not + hot any more. Will avoid promote the original target. + + To check if original promoted target is still hot, we check the total + count of the unpromoted targets (stored in old_info). If it is no less + than half of the callsite count (stored in INFO), the original promoted + target is considered not hot any more. */ + if (total >= info->count / 2) + return false; + + inline_stack stack; + get_inline_stack (gimple_location (stmt), &stack); + if (stack.size () == 0) + return false; + function_instance *s = get_function_instance_by_inline_stack (stack); + if (s == NULL) + return false; + icall_target_map map; + if (s->find_icall_target_map (stmt, &map) == 0) + return false; + for (icall_target_map::const_iterator iter = map.begin(); + iter != map.end(); ++iter) + info->targets[iter->first] = iter->second; + return true; +} + +/* Find total count of the callee of EDGE. */ + +gcov_type +autofdo_source_profile::get_callsite_total_count ( + struct cgraph_edge *edge) const +{ + inline_stack stack; + stack.push_back (std::make_pair(edge->callee->decl, 0)); + get_inline_stack (gimple_location (edge->call_stmt), &stack); + + function_instance *s = get_function_instance_by_inline_stack (stack); + if (s == NULL || + afdo_string_table->get_index (IDENTIFIER_POINTER (DECL_ASSEMBLER_NAME (edge->callee->decl))) != s->name ()) + return 0; + else + return s->total_count (); +} + +/* Read AutoFDO profile and returns TRUE on success. */ + +/* source profile format: + + GCOV_TAG_AFDO_FUNCTION: 4 bytes + LENGTH: 4 bytes + NUM_FUNCTIONS: 4 bytes + FUNCTION_INSTANCE_1 + FUNCTION_INSTANCE_2 + ... + FUNCTION_INSTANCE_N. */ + +bool +autofdo_source_profile::read () +{ + if (gcov_read_unsigned () != GCOV_TAG_AFDO_FUNCTION) + { + inform (0, "Not expected TAG."); + return false; + } + + /* Skip the length of the section. */ + gcov_read_unsigned (); + + /* Read in the function/callsite profile, and store it in local + data structure. */ + unsigned function_num = gcov_read_unsigned (); + for (unsigned i = 0; i < function_num; i++) + { + function_instance::function_instance_stack stack; + function_instance *s = function_instance::read_function_instance ( + &stack, gcov_read_counter ()); + afdo_profile_info->sum_all += s->total_count (); + map_[s->name ()] = s; + } + return true; +} + +/* Return the function_instance in the profile that correspond to the + inline STACK. */ + +function_instance * +autofdo_source_profile::get_function_instance_by_inline_stack ( + const inline_stack &stack) const +{ + name_function_instance_map::const_iterator iter = map_.find ( + afdo_string_table->get_index_by_decl ( + stack[stack.size() - 1].first)); + return iter == map_.end() + ? NULL + : iter->second->get_function_instance (stack, stack.size() - 1); +} + +/* Module profile is only used by LIPO. Here we simply ignore it. */ + +static void +fake_read_autofdo_module_profile () +{ + /* Read in the module info. */ + gcov_read_unsigned (); + + /* Skip the length of the section. */ + gcov_read_unsigned (); + + /* Read in the file name table. */ + unsigned total_module_num = gcov_read_unsigned (); + gcc_assert (total_module_num == 0); +} + +static void +read_profile (void) +{ + if (gcov_open (auto_profile_file, 1) == 0) + error ("Cannot open profile file %s.", auto_profile_file); + + if (gcov_read_unsigned () != GCOV_DATA_MAGIC) + error ("AutoFDO profile magic number does not mathch."); + + /* Skip the version number. */ + gcov_read_unsigned (); + + /* Skip the empty integer. */ + gcov_read_unsigned (); + + /* string_table. */ + afdo_string_table = string_table::create (); + if (afdo_string_table == NULL) + error ("Cannot read string table from %s.", auto_profile_file); + + /* autofdo_source_profile. */ + afdo_source_profile = autofdo_source_profile::create (); + if (afdo_source_profile == NULL) + error ("Cannot read function profile from %s.", auto_profile_file); + + /* autofdo_module_profile. */ + fake_read_autofdo_module_profile (); + + /* Read in the working set. */ + if (gcov_read_unsigned () != GCOV_TAG_AFDO_WORKING_SET) + error ("Cannot read working set from %s.", auto_profile_file); + + /* Skip the length of the section. */ + gcov_read_unsigned (); + gcov_working_set_t set[128]; + for (unsigned i = 0; i < 128; i++) + { + set[i].num_counters = gcov_read_unsigned (); + set[i].min_counter = gcov_read_counter (); + } + add_working_set (set); +} + +/* From AutoFDO profiles, find values inside STMT for that we want to measure + histograms for indirect-call optimization. */ + +static void +afdo_indirect_call (gimple_stmt_iterator *gsi, const icall_target_map &map, + bool transform) +{ + gimple stmt = gsi_stmt (*gsi); + tree callee; + + if (map.size() == 0 || gimple_code (stmt) != GIMPLE_CALL + || gimple_call_fndecl (stmt) != NULL_TREE) + return; + + callee = gimple_call_fn (stmt); + + histogram_value hist = gimple_alloc_histogram_value ( + cfun, HIST_TYPE_INDIR_CALL, stmt, callee); + hist->n_counters = 3; + hist->hvalue.counters = XNEWVEC (gcov_type, hist->n_counters); + gimple_add_histogram_value (cfun, stmt, hist); + + gcov_type total = 0; + icall_target_map::const_iterator max_iter = map.end(); + + for (icall_target_map::const_iterator iter = map.begin(); + iter != map.end(); ++iter) + { + total += iter->second; + if (max_iter == map.end() || max_iter->second < iter->second) + max_iter = iter; + } + + hist->hvalue.counters[0] = (unsigned long long) + afdo_string_table->get_name (max_iter->first); + hist->hvalue.counters[1] = max_iter->second; + hist->hvalue.counters[2] = total; + + if (!transform) + return; + + if (gimple_ic_transform (gsi)) + { + struct cgraph_edge *indirect_edge = + cgraph_node::get (current_function_decl)->get_edge (stmt); + struct cgraph_node *direct_call = + find_func_by_profile_id ((int)hist->hvalue.counters[0]); + if (DECL_STRUCT_FUNCTION (direct_call->decl) == NULL) + return; + struct cgraph_edge *new_edge = + indirect_edge->make_speculative (direct_call, 0, 0); + new_edge->redirect_call_stmt_to_callee (); + gimple_remove_histogram_value (cfun, stmt, hist); + inline_call (new_edge, true, NULL, NULL, false); + return; + } + return; +} + +/* From AutoFDO profiles, find values inside STMT for that we want to measure + histograms and adds them to list VALUES. */ + +static void +afdo_vpt (gimple_stmt_iterator *gsi, const icall_target_map &map, + bool transform) +{ + afdo_indirect_call (gsi, map, transform); +} + +typedef std::set<basic_block> bb_set; +typedef std::set<edge> edge_set; + +static bool +is_bb_annotated (const basic_block bb, const bb_set &annotated) +{ + return annotated.find(bb) != annotated.end(); +} + +static void +set_bb_annotated (basic_block bb, bb_set *annotated) +{ + annotated->insert(bb); +} + +static bool +is_edge_annotated (const edge e, const edge_set &annotated) +{ + return annotated.find(e) != annotated.end(); +} + +static void +set_edge_annotated (edge e, edge_set *annotated) +{ + annotated->insert(e); +} + +/* For a given BB, set its execution count. Attach value profile if a stmt + is not in PROMOTED, because we only want to promot an indirect call once. + Return TRUE if BB is annotated. */ + +static bool +afdo_set_bb_count (basic_block bb, const stmt_set &promoted) +{ + gimple_stmt_iterator gsi; + edge e; + edge_iterator ei; + gcov_type max_count = 0; + bool has_annotated = false; + + for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi)) + { + count_info info; + gimple stmt = gsi_stmt (gsi); + if (gimple_clobber_p (stmt) || is_gimple_debug (stmt)) + continue; + if (afdo_source_profile->get_count_info (stmt, &info)) + { + if (info.count > max_count) + max_count = info.count; + has_annotated = true; + if (info.targets.size() > 0 && promoted.find (stmt) == promoted.end ()) + afdo_vpt (&gsi, info.targets, false); + } + } + + if (!has_annotated) + return false; + + for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi)) + afdo_source_profile->mark_annotated (gimple_location (gsi_stmt (gsi))); + for (gsi = gsi_start_phis (bb); !gsi_end_p (gsi); gsi_next (&gsi)) + { + gimple phi = gsi_stmt (gsi); + size_t i; + for (i = 0; i < gimple_phi_num_args (phi); i++) + afdo_source_profile->mark_annotated (gimple_phi_arg_location (phi, i)); + } + FOR_EACH_EDGE (e, ei, bb->succs) + afdo_source_profile->mark_annotated (e->goto_locus); + + bb->count = max_count; + return true; +} + +/* BB1 and BB2 are in an equivalent class iff: + 1. BB1 dominates BB2. + 2. BB2 post-dominates BB1. + 3. BB1 and BB2 are in the same loop nest. + This function finds the equivalent class for each basic block, and + stores a pointer to the first BB in its equivalent class. Meanwhile, + set bb counts for the same equivalent class to be idenical. Update + ANNOTATED_BB for the first BB in its equivalent class. */ + +static void +afdo_find_equiv_class (bb_set *annotated_bb) +{ + basic_block bb; + + FOR_ALL_BB_FN (bb, cfun) + bb->aux = NULL; + + FOR_ALL_BB_FN (bb, cfun) + { + vec<basic_block> dom_bbs; + basic_block bb1; + int i; + + if (bb->aux != NULL) + continue; + bb->aux = bb; + dom_bbs = get_dominated_by (CDI_DOMINATORS, bb); + FOR_EACH_VEC_ELT (dom_bbs, i, bb1) + if (bb1->aux == NULL + && dominated_by_p (CDI_POST_DOMINATORS, bb, bb1) + && bb1->loop_father == bb->loop_father) + { + bb1->aux = bb; + if (bb1->count > bb->count && is_bb_annotated (bb1, *annotated_bb)) + { + bb->count = MAX (bb->count, bb1->count); + set_bb_annotated (bb, annotated_bb); + } + } + dom_bbs = get_dominated_by (CDI_POST_DOMINATORS, bb); + FOR_EACH_VEC_ELT (dom_bbs, i, bb1) + if (bb1->aux == NULL + && dominated_by_p (CDI_DOMINATORS, bb, bb1) + && bb1->loop_father == bb->loop_father) + { + bb1->aux = bb; + if (bb1->count > bb->count && is_bb_annotated (bb1, *annotated_bb)) + { + bb->count = MAX (bb->count, bb1->count); + set_bb_annotated (bb, annotated_bb); + } + } + } +} + +/* If a basic block's count is known, and only one of its in/out edges' count + is unknown, its count can be calculated. Meanwhile, if all of the in/out + edges' counts are known, then the basic block's unknown count can also be + calculated. + IS_SUCC is true if out edges of a basic blocks are examined. + Update ANNOTATED_BB and ANNOTATED_EDGE accordingly. + Return TRUE if any basic block/edge count is changed. */ + +static bool +afdo_propagate_edge (bool is_succ, + bb_set *annotated_bb, edge_set *annotated_edge) +{ + basic_block bb; + bool changed = false; + + FOR_EACH_BB_FN (bb, cfun) + { + edge e, unknown_edge = NULL; + edge_iterator ei; + int num_unknown_edge = 0; + gcov_type total_known_count = 0; + + FOR_EACH_EDGE (e, ei, is_succ ? bb->succs : bb->preds) + if (!is_edge_annotated (e, *annotated_edge)) + num_unknown_edge ++, unknown_edge = e; + else + total_known_count += e->count; + + if (num_unknown_edge == 0) + { + if (total_known_count > bb->count) + { + bb->count = total_known_count; + changed = true; + } + if (!is_bb_annotated (bb, *annotated_bb)) + { + set_bb_annotated (bb, annotated_bb); + changed = true; + } + } + else if (num_unknown_edge == 1 + && is_bb_annotated (bb, *annotated_bb)) + { + if (bb->count >= total_known_count) + unknown_edge->count = bb->count - total_known_count; + else + unknown_edge->count = 0; + set_edge_annotated (unknown_edge, annotated_edge); + changed = true; + } + } + return changed; +} + +/* Special propagation for circuit expressions. Because GCC translates + control flow into data flow for circuit expressions. E.g. + BB1: + if (a && b) + BB2 + else + BB3 + + will be translated into: + + BB1: + if (a) + goto BB.t1 + else + goto BB.t3 + BB.t1: + if (b) + goto BB.t2 + else + goto BB.t3 + BB.t2: + goto BB.t3 + BB.t3: + tmp = PHI (0 (BB1), 0 (BB.t1), 1 (BB.t2) + if (tmp) + goto BB2 + else + goto BB3 + + In this case, we need to propagate through PHI to determine the edge + count of BB1->BB.t1, BB.t1->BB.t2. + Update ANNOTATED_EDGE accordingly. */ + +static void +afdo_propagate_circuit (const bb_set &annotated_bb, edge_set *annotated_edge) +{ + basic_block bb; + FOR_ALL_BB_FN (bb, cfun) + { + gimple phi_stmt; + tree cmp_rhs, cmp_lhs; + gimple cmp_stmt = last_stmt (bb); + edge e; + edge_iterator ei; + + if (!cmp_stmt || gimple_code (cmp_stmt) != GIMPLE_COND) + continue; + cmp_rhs = gimple_cond_rhs (cmp_stmt); + cmp_lhs = gimple_cond_lhs (cmp_stmt); + if (!TREE_CONSTANT (cmp_rhs) + || !(integer_zerop (cmp_rhs) || integer_onep (cmp_rhs))) + continue; + if (TREE_CODE (cmp_lhs) != SSA_NAME) + continue; + if (!is_bb_annotated (bb, annotated_bb)) + continue; + phi_stmt = SSA_NAME_DEF_STMT (cmp_lhs); + while (phi_stmt && gimple_code (phi_stmt) == GIMPLE_ASSIGN + && gimple_assign_single_p (phi_stmt) + && TREE_CODE (gimple_assign_rhs1 (phi_stmt)) == SSA_NAME) + phi_stmt = SSA_NAME_DEF_STMT (gimple_assign_rhs1 (phi_stmt)); + if (!phi_stmt || gimple_code (phi_stmt) != GIMPLE_PHI) + continue; + FOR_EACH_EDGE (e, ei, bb->succs) + { + unsigned i, total = 0; + edge only_one; + bool check_value_one = (((integer_onep (cmp_rhs)) + ^ (gimple_cond_code (cmp_stmt) == EQ_EXPR)) + ^ ((e->flags & EDGE_TRUE_VALUE) != 0)); + if (!is_edge_annotated (e, *annotated_edge)) + continue; + for (i = 0; i < gimple_phi_num_args (phi_stmt); i++) + { + tree val = gimple_phi_arg_def (phi_stmt, i); + edge ep = gimple_phi_arg_edge (phi_stmt, i); + + if (!TREE_CONSTANT (val) || !(integer_zerop (val) + || integer_onep (val))) + continue; + if (check_value_one ^ integer_onep (val)) + continue; + total++; + only_one = ep; + if (e->probability == 0 + && !is_edge_annotated (ep, *annotated_edge)) + { + ep->probability = 0; + ep->count = 0; + set_edge_annotated (ep, annotated_edge); + } + } + if (total == 1 && !is_edge_annotated (only_one, *annotated_edge)) + { + only_one->probability = e->probability; + only_one->count = e->count; + set_edge_annotated (only_one, annotated_edge); + } + } + } +} + +/* Propagate the basic block count and edge count on the control flow + graph. We do the propagation iteratively until stablize. */ + +static void +afdo_propagate (bb_set *annotated_bb, edge_set *annotated_edge) +{ + basic_block bb; + bool changed = true; + int i = 0; + + FOR_ALL_BB_FN (bb, cfun) + { + bb->count = ((basic_block) bb->aux)->count; + if (is_bb_annotated ((const basic_block) bb->aux, *annotated_bb)) + set_bb_annotated (bb, annotated_bb); + } + + while (changed && i++ < 10) + { + changed = false; + + if (afdo_propagate_edge (true, annotated_bb, annotated_edge)) + changed = true; + if (afdo_propagate_edge (false, annotated_bb, annotated_edge)) + changed = true; + afdo_propagate_circuit (*annotated_bb, annotated_edge); + } +} + +/* Propagate counts on control flow graph and calculate branch + probabilities. */ + +static void +afdo_calculate_branch_prob (bb_set *annotated_bb, edge_set *annotated_edge) +{ + basic_block bb; + bool has_sample = false; + + FOR_EACH_BB_FN (bb, cfun) + if (bb->count > 0) + has_sample = true; + + if (!has_sample) + return; + + calculate_dominance_info (CDI_POST_DOMINATORS); + calculate_dominance_info (CDI_DOMINATORS); + loop_optimizer_init (0); + + afdo_find_equiv_class (annotated_bb); + afdo_propagate (annotated_bb, annotated_edge); + + FOR_EACH_BB_FN (bb, cfun) + { + edge e; + edge_iterator ei; + int num_unknown_succ = 0; + gcov_type total_count = 0; + + FOR_EACH_EDGE (e, ei, bb->succs) + { + if (!is_edge_annotated (e, *annotated_edge)) + num_unknown_succ ++; + else + total_count += e->count; + } + if (num_unknown_succ == 0 && total_count > 0) + { + FOR_EACH_EDGE (e, ei, bb->succs) + e->probability = + (double) e->count * REG_BR_PROB_BASE / total_count; + } + } + FOR_ALL_BB_FN (bb, cfun) + { + edge e; + edge_iterator ei; + + FOR_EACH_EDGE (e, ei, bb->succs) + e->count = + (double) bb->count * e->probability / REG_BR_PROB_BASE; + bb->aux = NULL; + } + + loop_optimizer_finalize (); + free_dominance_info (CDI_DOMINATORS); + free_dominance_info (CDI_POST_DOMINATORS); +} + +/* Perform value profile transformation using AutoFDO profile. Add the + promoted stmts to PROMOTED_STMTS. Return TRUE if there is any + indirect call promoted. */ + +static bool +afdo_vpt_for_early_inline (stmt_set *promoted_stmts) +{ + basic_block bb; + if (afdo_source_profile->get_function_instance_by_decl ( + current_function_decl) == NULL) + return false; + + compute_inline_parameters (cgraph_node::get (current_function_decl), true); + + bool has_vpt = false; + FOR_EACH_BB_FN (bb, cfun) + { + if (!has_indirect_call (bb)) + continue; + gimple_stmt_iterator gsi; + + gcov_type bb_count = 0; + for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi)) + { + count_info info; + gimple stmt = gsi_stmt (gsi); + if (afdo_source_profile->get_count_info (stmt, &info)) + bb_count = MAX (bb_count, info.count); + } + + for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi)) + { + gimple stmt = gsi_stmt (gsi); + /* IC_promotion and early_inline_2 is done in multiple iterations. + No need to promoted the stmt if its in promoted_stmts (means + it is already been promoted in the previous iterations). */ + if (gimple_code (stmt) != GIMPLE_CALL + || gimple_call_fn (stmt) == NULL + || TREE_CODE (gimple_call_fn (stmt)) == FUNCTION_DECL + || promoted_stmts->find (stmt) != promoted_stmts->end ()) + continue; + + count_info info; + afdo_source_profile->get_count_info (stmt, &info); + info.count = bb_count; + if (afdo_source_profile->update_inlined_ind_target (stmt, &info)) + { + /* Promote the indirect call and update the promoted_stmts. */ + promoted_stmts->insert (stmt); + afdo_vpt (&gsi, info.targets, true); + has_vpt = true; + } + } + } + if (has_vpt) + { + optimize_inline_calls (current_function_decl); + return true; + } + else + return false; +} + +/* Annotate auto profile to the control flow graph. Do not annotate value + profile for stmts in PROMOTED_STMTS. */ + +static void +afdo_annotate_cfg (const stmt_set &promoted_stmts) +{ + basic_block bb; + bb_set annotated_bb; + edge_set annotated_edge; + const function_instance *s = + afdo_source_profile->get_function_instance_by_decl ( + current_function_decl); + + if (s == NULL) + return; + cgraph_node::get (current_function_decl)->count = s->head_count (); + ENTRY_BLOCK_PTR_FOR_FN (cfun)->count = s->head_count (); + gcov_type max_count = ENTRY_BLOCK_PTR_FOR_FN (cfun)->count; + + FOR_EACH_BB_FN (bb, cfun) + { + edge e; + edge_iterator ei; + + bb->count = 0; + FOR_EACH_EDGE (e, ei, bb->succs) + e->count = 0; + + if (afdo_set_bb_count (bb, promoted_stmts)) + set_bb_annotated (bb, &annotated_bb); + if (bb->count > max_count) + max_count = bb->count; + } + if (ENTRY_BLOCK_PTR_FOR_FN (cfun)->count > + ENTRY_BLOCK_PTR_FOR_FN (cfun)->next_bb->count) + { + ENTRY_BLOCK_PTR_FOR_FN (cfun)->next_bb->count = + ENTRY_BLOCK_PTR_FOR_FN (cfun)->count; + set_bb_annotated (ENTRY_BLOCK_PTR_FOR_FN (cfun)->next_bb, &annotated_bb); + } + if (ENTRY_BLOCK_PTR_FOR_FN (cfun)->count > + EXIT_BLOCK_PTR_FOR_FN (cfun)->prev_bb->count) + { + EXIT_BLOCK_PTR_FOR_FN (cfun)->prev_bb->count = + ENTRY_BLOCK_PTR_FOR_FN (cfun)->count; + set_bb_annotated (EXIT_BLOCK_PTR_FOR_FN (cfun)->prev_bb, &annotated_bb); + } + afdo_source_profile->mark_annotated ( + DECL_SOURCE_LOCATION (current_function_decl)); + afdo_source_profile->mark_annotated (cfun->function_start_locus); + afdo_source_profile->mark_annotated (cfun->function_end_locus); + if (max_count > 0) + { + afdo_calculate_branch_prob (&annotated_bb, &annotated_edge); + counts_to_freqs (); + profile_status_for_fn (cfun) = PROFILE_READ; + } + if (flag_value_profile_transformations) + gimple_value_profile_transformations (); +} + +/* Wrapper function to invoke early inliner. */ + +static void early_inline () +{ + compute_inline_parameters (cgraph_node::get (current_function_decl), true); + unsigned todo = early_inliner (cfun); + if (todo & TODO_update_ssa_any) + update_ssa (TODO_update_ssa); +} + +/* Use AutoFDO profile to annoate the control flow graph. + Return the todo flag. */ + +static unsigned int +auto_profile (void) +{ + struct cgraph_node *node; + + if (symtab->state == FINISHED) + return 0; + + init_node_map (true); + profile_info = autofdo::afdo_profile_info; + + FOR_EACH_FUNCTION (node) + { + if (!gimple_has_body_p (node->decl)) + continue; + + /* Don't profile functions produced for builtin stuff. */ + if (DECL_SOURCE_LOCATION (node->decl) == BUILTINS_LOCATION) + continue; + + push_cfun (DECL_STRUCT_FUNCTION (node->decl)); + + /* First do indirect call promotion and early inline to make the + IR match the profiled binary before actual annotation. + + This is needed because an indirect call might have been promoted + and inlined in the profiled binary. If we do not promote and + inline these indirect calls before annotation, the profile for + these promoted functions will be lost. + + e.g. foo() --indirect_call--> bar() + In profiled binary, the callsite is promoted and inlined, making + the profile look like: + + foo: { + loc_foo_1: count_1 + bar@loc_foo_2: { + loc_bar_1: count_2 + loc_bar_2: count_3 + } + } + + Before AutoFDO pass, loc_foo_2 is not promoted thus not inlined. + If we perform annotation on it, the profile inside bar@loc_foo2 + will be wasted. + + To avoid this, we promote loc_foo_2 and inline the promoted bar + function before annotation, so the profile inside bar@loc_foo2 + will be useful. */ + autofdo::stmt_set promoted_stmts; + for (int i = 0; i < PARAM_VALUE (PARAM_EARLY_INLINER_MAX_ITERATIONS); i++) + { + if (!flag_value_profile_transformations + || !autofdo::afdo_vpt_for_early_inline (&promoted_stmts)) + break; + early_inline (); + } + + early_inline (); + autofdo::afdo_annotate_cfg (promoted_stmts); + compute_function_frequency (); + update_ssa (TODO_update_ssa); + + /* Local pure-const may imply need to fixup the cfg. */ + if (execute_fixup_cfg () & TODO_cleanup_cfg) + cleanup_tree_cfg (); + + free_dominance_info (CDI_DOMINATORS); + free_dominance_info (CDI_POST_DOMINATORS); + cgraph_edge::rebuild_edges (); + pop_cfun (); + } + + return TODO_rebuild_cgraph_edges; +} +} /* namespace autofdo. */ + +/* Read the profile from the profile data file. */ + +void +read_autofdo_file (void) +{ + if (auto_profile_file == NULL) + auto_profile_file = DEFAULT_AUTO_PROFILE_FILE; + + autofdo::afdo_profile_info = (struct gcov_ctr_summary *) + xcalloc (1, sizeof (struct gcov_ctr_summary)); + autofdo::afdo_profile_info->runs = 1; + autofdo::afdo_profile_info->sum_max = 0; + autofdo::afdo_profile_info->sum_all = 0; + + /* Read the profile from the profile file. */ + autofdo::read_profile (); +} + +/* Free the resources. */ + +void +end_auto_profile (void) +{ + delete autofdo::afdo_source_profile; + delete autofdo::afdo_string_table; + profile_info = NULL; +} + +/* Returns TRUE if EDGE is hot enough to be inlined early. */ + +bool +afdo_callsite_hot_enough_for_early_inline (struct cgraph_edge *edge) +{ + gcov_type count = + autofdo::afdo_source_profile->get_callsite_total_count (edge); + if (count > 0) + { + bool is_hot; + const struct gcov_ctr_summary *saved_profile_info = profile_info; + /* At earling inline stage, profile_info is not set yet. We need to + temporarily set it to afdo_profile_info to calculate hotness. */ + profile_info = autofdo::afdo_profile_info; + is_hot = maybe_hot_count_p (NULL, count); + profile_info = saved_profile_info; + return is_hot; + } + else + return false; +} + +namespace { + +const pass_data pass_data_ipa_auto_profile = +{ + SIMPLE_IPA_PASS, + "afdo", /* name */ + OPTGROUP_NONE, /* optinfo_flags */ + TV_IPA_AUTOFDO, /* tv_id */ + 0, /* properties_required */ + 0, /* properties_provided */ + 0, /* properties_destroyed */ + 0, /* todo_flags_start */ + 0, /* todo_flags_finish */ +}; + +class pass_ipa_auto_profile : public simple_ipa_opt_pass +{ +public: + pass_ipa_auto_profile(gcc::context *ctxt) + : simple_ipa_opt_pass(pass_data_ipa_auto_profile, ctxt) + {} + + /* opt_pass methods: */ + virtual bool gate (function *) + { + return flag_auto_profile; + } + virtual unsigned int execute (function *) + { + return autofdo::auto_profile (); + } +}; // class pass_ipa_auto_profile + +} // anon namespace + +simple_ipa_opt_pass * +make_pass_ipa_auto_profile (gcc::context *ctxt) +{ + return new pass_ipa_auto_profile (ctxt); +} + Index: gcc/auto-profile.h =================================================================== --- gcc/auto-profile.h (revision 0) +++ gcc/auto-profile.h (revision 0) @@ -0,0 +1,31 @@ +/* auto-profile.h - Defines data exported from auto-profile.c + Copyright (C) 2014. Free Software Foundation, Inc. + Contributed by Dehao Chen (dehao@google.com) + +This file is part of GCC. + +GCC is free software; you can redistribute it and/or modify it under +the terms of the GNU General Public License as published by the Free +Software Foundation; either version 3, or (at your option) any later +version. + +GCC is distributed in the hope that it will be useful, but WITHOUT ANY +WARRANTY; without even the implied warranty of MERCHANTABILITY or +FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License +for more details. + +You should have received a copy of the GNU General Public License +along with GCC; see the file COPYING3. If not see +<http://www.gnu.org/licenses/>. */ + +#ifndef AUTO_PROFILE_H +#define AUTO_PROFILE_H + +/* Read, process, finalize AutoFDO data structures. */ +extern void read_autofdo_file (void); +extern void end_auto_profile (void); + +/* Returns TRUE if EDGE is hot enough to be inlined early. */ +extern bool afdo_callsite_hot_enough_for_early_inline (struct cgraph_edge *); + +#endif /* AUTO_PROFILE_H */ Index: gcc/passes.def =================================================================== --- gcc/passes.def (revision 215826) +++ gcc/passes.def (working copy) @@ -90,6 +90,7 @@ NEXT_PASS (pass_rebuild_cgraph_edges); NEXT_PASS (pass_inline_parameters); POP_INSERT_PASSES () + NEXT_PASS (pass_ipa_auto_profile); NEXT_PASS (pass_ipa_free_inline_summary); NEXT_PASS (pass_ipa_tree_profile); PUSH_INSERT_PASSES_WITHIN (pass_ipa_tree_profile) Index: gcc/tree-pass.h =================================================================== --- gcc/tree-pass.h (revision 215826) +++ gcc/tree-pass.h (working copy) @@ -449,6 +449,7 @@ extern simple_ipa_opt_pass *make_pass_ipa_function_and_variable_visibility (gcc::context *ctxt); extern simple_ipa_opt_pass *make_pass_ipa_tree_profile (gcc::context *ctxt); +extern simple_ipa_opt_pass *make_pass_ipa_auto_profile (gcc::context *ctxt); extern simple_ipa_opt_pass *make_pass_early_local_passes (gcc::context *ctxt); Index: gcc/profile.c =================================================================== --- gcc/profile.c (revision 215826) +++ gcc/profile.c (working copy) @@ -106,6 +106,14 @@ static int total_hist_br_prob[20]; static int total_num_branches; +/* Helper function to update gcov_working_sets. */ + +void add_working_set (gcov_working_set_t *set) { + int i = 0; + for (; i < NUM_GCOV_WORKING_SETS; i++) + gcov_working_sets[i] = set[i]; +} + /* Forward declarations. */ static void find_spanning_tree (struct edge_list *); Index: gcc/profile.h =================================================================== --- gcc/profile.h (revision 215826) +++ gcc/profile.h (working copy) @@ -47,6 +47,7 @@ extern void del_node_map (void); extern void get_working_sets (void); +extern void add_working_set (gcov_working_set_t *); /* In predict.c. */ extern gcov_type get_hot_bb_threshold (void); Index: gcc/basic-block.h =================================================================== --- gcc/basic-block.h (revision 215826) +++ gcc/basic-block.h (working copy) @@ -724,6 +724,7 @@ extern void compute_available (sbitmap *, sbitmap *, sbitmap *, sbitmap *); /* In predict.c */ +extern bool maybe_hot_count_p (struct function *, gcov_type); extern bool maybe_hot_bb_p (struct function *, const_basic_block); extern bool maybe_hot_edge_p (edge); extern bool probably_never_executed_bb_p (struct function *, const_basic_block); Index: gcc/value-prof.c =================================================================== --- gcc/value-prof.c (revision 215826) +++ gcc/value-prof.c (working copy) @@ -134,11 +134,10 @@ static bool gimple_mod_pow2_value_transform (gimple_stmt_iterator *); static bool gimple_mod_subtract_transform (gimple_stmt_iterator *); static bool gimple_stringops_transform (gimple_stmt_iterator *); -static bool gimple_ic_transform (gimple_stmt_iterator *); /* Allocate histogram value. */ -static histogram_value +histogram_value gimple_alloc_histogram_value (struct function *fun ATTRIBUTE_UNUSED, enum hist_type type, gimple stmt, tree value) { @@ -1307,6 +1306,9 @@ struct cgraph_node* find_func_by_profile_id (int profile_id) { + if (flag_auto_profile) + return cgraph_node::get_for_asmname ( + get_identifier ((const char *)(size_t)profile_id)); cgraph_node **val = cgraph_node_map->get (profile_id); if (val) return *val; @@ -1320,7 +1322,7 @@ may ICE. Here we only do very minimal sanity check just to make compiler happy. Returns true if TARGET is considered ok for call CALL_STMT. */ -static bool +bool check_ic_target (gimple call_stmt, struct cgraph_node *target) { location_t locus; @@ -1483,7 +1485,7 @@ this call to: */ -static bool +bool gimple_ic_transform (gimple_stmt_iterator *gsi) { gimple stmt = gsi_stmt (*gsi); Index: gcc/value-prof.h =================================================================== --- gcc/value-prof.h (revision 215826) +++ gcc/value-prof.h (working copy) @@ -75,6 +75,8 @@ extern void gimple_find_values_to_profile (histogram_values *); extern bool gimple_value_profile_transformations (void); +histogram_value gimple_alloc_histogram_value (struct function *, enum hist_type, + gimple stmt, tree); histogram_value gimple_histogram_value (struct function *, gimple); histogram_value gimple_histogram_value_of_type (struct function *, gimple, enum hist_type); @@ -89,6 +91,8 @@ void free_histograms (void); void stringop_block_profile (gimple, unsigned int *, HOST_WIDE_INT *); gimple gimple_ic (gimple, struct cgraph_node *, int, gcov_type, gcov_type); +bool check_ic_target (gimple, struct cgraph_node *); +bool gimple_ic_transform (gimple_stmt_iterator *); /* In tree-profile.c. */ Index: gcc/tree-profile.c =================================================================== --- gcc/tree-profile.c (revision 215826) +++ gcc/tree-profile.c (working copy) @@ -699,8 +699,10 @@ bool pass_ipa_tree_profile::gate (function *) { - /* When profile instrumentation, use or test coverage shall be performed. */ - return (!in_lto_p + /* When profile instrumentation, use or test coverage shall be performed. + But for AutoFDO, this there is no instrumentation, thus this pass is + diabled. */ + return (!in_lto_p && !flag_auto_profile && (flag_branch_probabilities || flag_test_coverage || profile_arc_flag)); } Index: gcc/tree-ssa-live.c =================================================================== --- gcc/tree-ssa-live.c (revision 215826) +++ gcc/tree-ssa-live.c (working copy) @@ -605,7 +605,7 @@ ; /* When not generating debug info we can eliminate info on unused variables. */ - else if (debug_info_level == DINFO_LEVEL_NONE) + else if (!flag_auto_profile && debug_info_level == DINFO_LEVEL_NONE) { /* Even for -g0 don't prune outer scopes from artificial functions, otherwise diagnostics using tree_nonartificial_location Index: gcc/timevar.def =================================================================== --- gcc/timevar.def (revision 215826) +++ gcc/timevar.def (working copy) @@ -89,6 +89,7 @@ DEFTIMEVAR (TV_WHOPR_LTRANS , "whopr ltrans") DEFTIMEVAR (TV_IPA_REFERENCE , "ipa reference") DEFTIMEVAR (TV_IPA_PROFILE , "ipa profile") +DEFTIMEVAR (TV_IPA_AUTOFDO , "auto profile") DEFTIMEVAR (TV_IPA_PURE_CONST , "ipa pure const") DEFTIMEVAR (TV_IPA_PTA , "ipa points-to") DEFTIMEVAR (TV_IPA_SRA , "ipa SRA") Index: gcc/cfgloop.c =================================================================== --- gcc/cfgloop.c (revision 215826) +++ gcc/cfgloop.c (working copy) @@ -1802,7 +1802,7 @@ } if (realistic && (!loop->any_estimate - || wi::ltu_p (i_bound, loop->nb_iterations_estimate))) + || (!flag_auto_profile && wi::ltu_p (i_bound, loop->nb_iterations_estimate)))) { loop->any_estimate = true; loop->nb_iterations_estimate = i_bound; Index: gcc/coverage.c =================================================================== --- gcc/coverage.c (revision 215826) +++ gcc/coverage.c (working copy) @@ -55,6 +55,7 @@ #include "filenames.h" #include "target.h" #include "params.h" +#include "auto-profile.h" #include "gcov-io.h" #include "gcov-io.c" @@ -1208,7 +1209,9 @@ bbg_file_stamp = local_tick; - if (flag_branch_probabilities) + if (flag_auto_profile) + read_autofdo_file (); + else if (flag_branch_probabilities) read_counts_file (); /* Name of bbg file. */ Index: gcc/opts.c =================================================================== --- gcc/opts.c (revision 215826) +++ gcc/opts.c (working copy) @@ -1279,6 +1279,55 @@ opts->x_help_columns, opts, lang_mask); } +static void +enable_fdo_optimizations (struct gcc_options *opts, + struct gcc_options *opts_set, + int value) +{ + if (!opts_set->x_flag_branch_probabilities) + opts->x_flag_branch_probabilities = value; + if (!opts_set->x_flag_profile_values) + opts->x_flag_profile_values = value; + if (!opts_set->x_flag_unroll_loops) + opts->x_flag_unroll_loops = value; + if (!opts_set->x_flag_peel_loops) + opts->x_flag_peel_loops = value; + if (!opts_set->x_flag_tracer) + opts->x_flag_tracer = value; + if (!opts_set->x_flag_value_profile_transformations) + opts->x_flag_value_profile_transformations = value; + if (!opts_set->x_flag_inline_functions) + opts->x_flag_inline_functions = value; + if (!opts_set->x_flag_ipa_cp) + opts->x_flag_ipa_cp = value; + if (!opts_set->x_flag_ipa_cp_clone + && value && opts->x_flag_ipa_cp) + opts->x_flag_ipa_cp_clone = value; + if (!opts_set->x_flag_predictive_commoning) + opts->x_flag_predictive_commoning = value; + if (!opts_set->x_flag_unswitch_loops) + opts->x_flag_unswitch_loops = value; + if (!opts_set->x_flag_gcse_after_reload) + opts->x_flag_gcse_after_reload = value; + if (!opts_set->x_flag_tree_loop_vectorize + && !opts_set->x_flag_tree_vectorize) + opts->x_flag_tree_loop_vectorize = value; + if (!opts_set->x_flag_tree_slp_vectorize + && !opts_set->x_flag_tree_vectorize) + opts->x_flag_tree_slp_vectorize = value; + if (!opts_set->x_flag_vect_cost_model) + opts->x_flag_vect_cost_model = VECT_COST_MODEL_DYNAMIC; + if (!opts_set->x_flag_tree_loop_distribute_patterns) + opts->x_flag_tree_loop_distribute_patterns = value; + if (!opts_set->x_flag_profile_reorder_functions) + opts->x_flag_profile_reorder_functions = value; + /* Indirect call profiling should do all useful transformations + speculative devirtualization does. */ + if (!opts_set->x_flag_devirtualize_speculatively + && opts->x_flag_value_profile_transformations) + opts->x_flag_devirtualize_speculatively = false; +} + /* Handle target- and language-independent options. Return zero to generate an "unknown option" message. Only options that need extra handling need to be listed here; if you simply want @@ -1746,50 +1795,23 @@ value = true; /* No break here - do -fprofile-use processing. */ case OPT_fprofile_use: - if (!opts_set->x_flag_branch_probabilities) - opts->x_flag_branch_probabilities = value; - if (!opts_set->x_flag_profile_values) - opts->x_flag_profile_values = value; - if (!opts_set->x_flag_unroll_loops) - opts->x_flag_unroll_loops = value; - if (!opts_set->x_flag_peel_loops) - opts->x_flag_peel_loops = value; - if (!opts_set->x_flag_tracer) - opts->x_flag_tracer = value; - if (!opts_set->x_flag_value_profile_transformations) - opts->x_flag_value_profile_transformations = value; - if (!opts_set->x_flag_inline_functions) - opts->x_flag_inline_functions = value; - if (!opts_set->x_flag_ipa_cp) - opts->x_flag_ipa_cp = value; - if (!opts_set->x_flag_ipa_cp_clone - && value && opts->x_flag_ipa_cp) - opts->x_flag_ipa_cp_clone = value; - if (!opts_set->x_flag_predictive_commoning) - opts->x_flag_predictive_commoning = value; - if (!opts_set->x_flag_unswitch_loops) - opts->x_flag_unswitch_loops = value; - if (!opts_set->x_flag_gcse_after_reload) - opts->x_flag_gcse_after_reload = value; - if (!opts_set->x_flag_tree_loop_vectorize - && !opts_set->x_flag_tree_vectorize) - opts->x_flag_tree_loop_vectorize = value; - if (!opts_set->x_flag_tree_slp_vectorize - && !opts_set->x_flag_tree_vectorize) - opts->x_flag_tree_slp_vectorize = value; - if (!opts_set->x_flag_vect_cost_model) - opts->x_flag_vect_cost_model = VECT_COST_MODEL_DYNAMIC; - if (!opts_set->x_flag_tree_loop_distribute_patterns) - opts->x_flag_tree_loop_distribute_patterns = value; - if (!opts_set->x_flag_profile_reorder_functions) - opts->x_flag_profile_reorder_functions = value; - /* Indirect call profiling should do all useful transformations - speculative devirtualization does. */ - if (!opts_set->x_flag_devirtualize_speculatively - && opts->x_flag_value_profile_transformations) - opts->x_flag_devirtualize_speculatively = false; + enable_fdo_optimizations (opts, opts_set, value); break; + case OPT_fauto_profile_: + opts->x_auto_profile_file = xstrdup (arg); + opts->x_flag_auto_profile = true; + value = true; + /* No break here - do -fauto-profile processing. */ + case OPT_fauto_profile: + enable_fdo_optimizations (opts, opts_set, value); + if (!opts_set->x_flag_profile_correction) + opts->x_flag_profile_correction = value; + maybe_set_param_value ( + PARAM_EARLY_INLINER_MAX_ITERATIONS, 10, + opts->x_param_values, opts_set->x_param_values); + break; + case OPT_fprofile_generate_: opts->x_profile_data_prefix = xstrdup (arg); value = true; Index: gcc/doc/invoke.texi =================================================================== --- gcc/doc/invoke.texi (revision 215826) +++ gcc/doc/invoke.texi (working copy) @@ -365,7 +365,8 @@ @gccoptlist{-faggressive-loop-optimizations -falign-functions[=@var{n}] @gol -falign-jumps[=@var{n}] @gol -falign-labels[=@var{n}] -falign-loops[=@var{n}] @gol --fassociative-math -fauto-inc-dec -fbranch-probabilities @gol +-fassociative-math -fauto-profile -fauto-profile[=@var{path}] @gol +-fauto-inc-dec -fbranch-probabilities @gol -fbranch-target-load-optimize -fbranch-target-load-optimize2 @gol -fbtr-bb-exclusive -fcaller-saves @gol -fcheck-data-deps -fcombine-stack-adjustments -fconserve-stack @gol @@ -9164,6 +9165,19 @@ If @var{path} is specified, GCC looks at the @var{path} to find the profile feedback data files. See @option{-fprofile-dir}. + +@item -fauto-profile +@itemx -fauto-profile=@var{path} +@opindex fauto-profile +Enable sampling based feedback directed optimizations, and optimizations +generally profitable only with profile feedback available. + +The following options are enabled: @code{-fbranch-probabilities}, @code{-fvpt}, +@code{-funroll-loops}, @code{-fpeel-loops}, @code{-ftracer}, @code{-ftree-vectorize}, +@code{ftree-loop-distribute-patterns} + +If @var{path} is specified, GCC looks at the @var{path} to find +the profile feedback data files. @end table The following options control compiler behavior regarding floating-point Index: gcc/common.opt =================================================================== --- gcc/common.opt (revision 215826) +++ gcc/common.opt (working copy) @@ -895,6 +895,16 @@ Common Report Var(flag_auto_inc_dec) Init(1) Generate auto-inc/dec instructions +fauto-profile +Common Report Var(flag_auto_profile) Optimization +Use sample profile information for call graph node weights. The default +profile file is fbdata.afdo in 'pwd'. + +fauto-profile= +Common Joined RejectNegative Var(auto_profile_file) +Use sample profile information for call graph node weights. The profile +file is specified in the argument. + ; -fcheck-bounds causes gcc to generate array bounds checks. ; For C, C++ and ObjC: defaults off. ; For Java: defaults to on. Index: gcc/ipa-inline.c =================================================================== --- gcc/ipa-inline.c (revision 215826) +++ gcc/ipa-inline.c (working copy) @@ -121,6 +121,7 @@ #include "ipa-inline.h" #include "ipa-utils.h" #include "sreal.h" +#include "auto-profile.h" #include "cilk.h" #include "builtins.h" @@ -449,6 +450,14 @@ if (DECL_DISREGARD_INLINE_LIMITS (callee->decl)) ; + /* For AutoFDO, we need to make sure that before profile annotation, all + hot paths' IR look exactly the same as profiled binary. As a result, + in einliner, we will disregard size limit and inline those callsites + that are: + * inlined in the profiled binary, and + * the cloned callee has enough samples to be considered "hot". */ + else if (flag_auto_profile && afdo_callsite_hot_enough_for_early_inline (e)) + ; else if (!DECL_DECLARED_INLINE_P (callee->decl) && !flag_inline_small_functions) { @@ -2366,39 +2375,8 @@ return inlined; } -/* Do inlining of small functions. Doing so early helps profiling and other - passes to be somewhat more effective and avoids some code duplication in - later real inlining pass for testcases with very many function calls. */ - -namespace { - -const pass_data pass_data_early_inline = -{ - GIMPLE_PASS, /* type */ - "einline", /* name */ - OPTGROUP_INLINE, /* optinfo_flags */ - TV_EARLY_INLINING, /* tv_id */ - PROP_ssa, /* properties_required */ - 0, /* properties_provided */ - 0, /* properties_destroyed */ - 0, /* todo_flags_start */ - 0, /* todo_flags_finish */ -}; - -class pass_early_inline : public gimple_opt_pass -{ -public: - pass_early_inline (gcc::context *ctxt) - : gimple_opt_pass (pass_data_early_inline, ctxt) - {} - - /* opt_pass methods: */ - virtual unsigned int execute (function *); - -}; // class pass_early_inline - unsigned int -pass_early_inline::execute (function *fun) +early_inliner (function *fun) { struct cgraph_node *node = cgraph_node::get (current_function_decl); struct cgraph_edge *edge; @@ -2499,6 +2477,43 @@ return todo; } +/* Do inlining of small functions. Doing so early helps profiling and other + passes to be somewhat more effective and avoids some code duplication in + later real inlining pass for testcases with very many function calls. */ + +namespace { + +const pass_data pass_data_early_inline = +{ + GIMPLE_PASS, /* type */ + "einline", /* name */ + OPTGROUP_INLINE, /* optinfo_flags */ + TV_EARLY_INLINING, /* tv_id */ + PROP_ssa, /* properties_required */ + 0, /* properties_provided */ + 0, /* properties_destroyed */ + 0, /* todo_flags_start */ + 0, /* todo_flags_finish */ +}; + +class pass_early_inline : public gimple_opt_pass +{ +public: + pass_early_inline (gcc::context *ctxt) + : gimple_opt_pass (pass_data_early_inline, ctxt) + {} + + /* opt_pass methods: */ + virtual unsigned int execute (function *); + +}; // class pass_early_inline + +unsigned int +pass_early_inline::execute (function *fun) +{ + return early_inliner (fun); +} + } // anon namespace gimple_opt_pass * Index: gcc/ipa-inline.h =================================================================== --- gcc/ipa-inline.h (revision 215826) +++ gcc/ipa-inline.h (working copy) @@ -238,6 +238,7 @@ void free_growth_caches (void); void compute_inline_parameters (struct cgraph_node *, bool); bool speculation_useful_p (struct cgraph_edge *e, bool anticipate_inlining); +unsigned int early_inliner (function *fun); /* In ipa-inline-transform.c */ bool inline_call (struct cgraph_edge *, bool, vec<cgraph_edge *> *, int *, bool, Index: gcc/toplev.c =================================================================== --- gcc/toplev.c (revision 215826) +++ gcc/toplev.c (working copy) @@ -79,6 +79,7 @@ #include "diagnostic-color.h" #include "context.h" #include "pass_manager.h" +#include "auto-profile.h" #include "optabs.h" #if defined(DBX_DEBUGGING_INFO) || defined(XCOFF_DEBUGGING_INFO) @@ -663,6 +664,10 @@ targetm.asm_out.output_ident (ident_str); } + /* Auto profile finalization. */ + if (flag_auto_profile) + end_auto_profile (); + /* Invoke registered plugin callbacks. */ invoke_plugin_callbacks (PLUGIN_FINISH_UNIT, NULL); ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH] AutoFDO patch for trunk 2014-10-09 0:24 ` Dehao Chen @ 2014-10-09 22:34 ` Jan Hubicka 2014-10-10 3:15 ` Dehao Chen 2014-10-14 15:08 ` Jan Hubicka 1 sibling, 1 reply; 25+ messages in thread From: Jan Hubicka @ 2014-10-09 22:34 UTC (permalink / raw) To: Dehao Chen; +Cc: Jan Hubicka, GCC Patches, David Li > >> Index: gcc/bb-reorder.c > >> =================================================================== > >> --- gcc/bb-reorder.c (revision 210180) > >> +++ gcc/bb-reorder.c (working copy) > >> @@ -2663,7 +2663,7 @@ pass_partition_blocks::gate (function *fun) > >> user defined section attributes. Don't call it if either case > >> arises. */ > >> return (flag_reorder_blocks_and_partition > >> - && optimize > >> + && optimize &&!flag_auto_profile > > > > Formatting error. Why we want !flag_auto_profile? Can't we just arrange > > flag_reorder_blocks_and_partition to default to false? > > > > In fact I would like to see flag_reorder_blocks_and_partition to offload obviously > > cold code (in partiuclar the EH cleanups) even without profile. That should work > > well with autofdo. > > This will cause bzip2 performance to degrade 6%. I haven't had time to > triage the problem. Will investigate this later. Still I would preffer to make this by default flag_reorder_blocks_and_partition to false with auto_profile. We could do that incrementally, lets just drop this from initial patch. > >> - if (flag_branch_probabilities) > >> + if (flag_auto_profile) > >> + init_auto_profile (); > > > > Perhaps initialization can happen from toplev.c instead? > > This is actually reading of afdo profile. I've renamed the function to > make it consistent. OK. > > >> Index: gcc/cgraphclones.c > >> =================================================================== > >> --- gcc/cgraphclones.c (revision 210180) > >> +++ gcc/cgraphclones.c (working copy) > >> @@ -435,6 +435,11 @@ cgraph_clone_node (struct cgraph_node *n, tree dec > >> } > >> else > >> count_scale = 0; > >> + /* In AutoFDO, if edge count is larger than callee's entry block > >> + count, we will not update the original callee because it may > >> + mistakenly mark some hot function as cold. */ > >> + if (flag_auto_profile && count >= n->count) > >> + update_original = false; > > > > This looks like a hack. Lets go with it later - I see what you are shooting for here > > and non-autoFDO has kind of same problem, too. Counts are not at all that reliable > > after some inlining. > > > > I wonder how this hack would fare for -fprofile-use > > It does not affect -fprofile-use performance in my testing. However, > for AutoFDO, this happens much more often. Yep, lets however drop it from the initial patch and make the tweaks go in later. > > >> @@ -1286,6 +1285,9 @@ del_node_map (void) > >> struct cgraph_node* > >> find_func_by_profile_id (int profile_id) > >> { > >> + if (flag_auto_profile) > >> + return cgraph_node_for_asm ( > >> + get_identifier ((const char *)(size_t)profile_id)); > > > > I think with LTO we will need to make this machinery safe for symbol mangling. Any plans on this? > > No plan yet. Let's get AutoFDO in first, and spend some separate > effort to make AutoFDO+LTO work? OK. > > >> Index: gcc/ipa-inline.h > >> =================================================================== > >> --- gcc/ipa-inline.h (revision 210180) > >> +++ gcc/ipa-inline.h (working copy) > >> @@ -345,3 +345,5 @@ reset_edge_growth_cache (struct cgraph_edge *edge) > >> edge_growth_cache[edge->uid] = zero; > >> } > >> } > >> + > >> +unsigned int early_inliner (function *fun); > > > > Can't this be handled by scheduling early inliner as subpass of autofdo or something similar? > > I would rather make this explicit in PM. > > See my comment below in auto-profile.c > > >> Index: gcc/auto-profile.c > >> =================================================================== > >> --- gcc/auto-profile.c (revision 0) > >> +++ gcc/auto-profile.c (revision 0) > >> + After the above 3 phases, all profile is readily annotated on the GCC IR. > >> + AutoFDO tries to reuse all FDO infrastructure as much as possible to make > >> + use of the profile. E.g. it uses existing mechanism to calculate the basic > >> + block/edge frequency, as well as the cgraph node/edge count. > >> +*/ > > > > I suppose the phases can be made explicit to the PM, so early inliner can stay > > a pass. > > > Unfortunately, this is more complicated: > > while(changed) { > vpt(); > einline(); > annotate(); > } Hmm, PM do not support iteration (though it wouldmake sense in some cases at least for expensive optimization). Why you need to iterate einline with the other two? > > > >> +/* Store a string array, indexed by string position in the array. */ > >> +class string_table { > >> +public: > >> + static string_table *create (); > >> + > >> + /* For a given string, returns its index. */ > >> + int get_index (const char *name) const; > >> + > >> + /* For a given decl, returns the index of the decl name. */ > >> + int get_index_by_decl (tree decl) const; > >> + > >> + /* For a given index, returns the string. */ > >> + const char *get_name (int index) const; > > > > I wonder how these should work with LTO. I suppose we can tell user to not LTO > > for train run, we should teach the table to ignore LTO generated suffixes as > > well as random seeds (like coverage code already does). > > > > What happens when two static functions have same name? > > Shall we first focus on non-LTO version. And there could be separate > effort to make AutoFDO+LTO work? > > For static functions, there will only be one copy in the profile. the > profile will be merged. I agree that this could potentially cause > problem. But OTOH, in a good practice, these functions should *not* be > hot functions. At least in our practice, we didn't see any hot > functions like this. You mean that static functions in differnt units that have same name tends to be not hot? I don't really follow it... > > I agree. My original intention is to make the function name handling > as simple as possible. The current impl is based on the asumption that > all important (hot) functions deserve a separate name (not just suffix > difference). We could definitely refine the function name handling in > the future patches. > > > I would definitely merge logic here with logic in gcov that already knows how to skip > > random seeds and other cases that makes filenames to diverge... > > Could you point me to the gcov logic that handles skipping function name suffix? See coverage_checksum_string in coverage.c > > >> + > >> +/* Return the combined location, which is a 32bit integer in which > >> + higher 16 bits stores the line offset of LOC to the start lineno > >> + of DECL, The lower 16 bits stores the discrimnator. */ > >> + > >> +static unsigned > >> +get_combined_location (location_t loc, tree decl) > >> +{ > >> + return ((LOCATION_LINE (loc) - DECL_SOURCE_LINE (decl)) << 16); > >> +} > > > > What happens on overflows here? I guess they are possible - 65536 lines is not infinity > > these days. > > If a function is longer than 65535 lines, then some lines may have > incorrect count attributed. But in pactice, we did not find any > function that is over 10K lines. In autogenerated code that may happen. I.e. insn-recog is 300k lines of code and it is pure luck that the generator artifically split the large decision tree into subfunctions (probably since early 90's) But yep, also something that can be dealt with incrementally. Peraps we could add a TODO comment to the begginign of file describing the main limitations. > >> + > >> +/* Special propagation for circuit expressions. Because GCC translates > >> + control flow into data flow for circuit expressions. E.g. > >> + BB1: > >> + if (a && b) > >> + BB2 > >> + else > >> + BB3 > >> + > >> + will be translated into: > >> + > >> + BB1: > >> + if (a) > >> + goto BB.t1 > >> + else > >> + goto BB.t3 > >> + BB.t1: > >> + if (b) > >> + goto BB.t2 > >> + else > >> + goto BB.t3 > >> + BB.t2: > >> + goto BB.t3 > >> + BB.t3: > >> + tmp = PHI (0 (BB1), 0 (BB.t1), 1 (BB.t2) > >> + if (tmp) > >> + goto BB2 > >> + else > >> + goto BB3 > >> + > >> + In this case, we need to propagate through PHI to determine the edge > >> + count of BB1->BB.t1, BB.t1->BB.t2. */ > > > > So here you have known probabilities of BB1, BB2 and BB3 and you need to guess all the rest? > > Why it need so much of gimple matching? > > This is basically because some basic blocks have no stmt so that we > can not get its count from source level profile. However, we can get > edge count through PHI nodes. > > >> + while (changed && i++ < 10) > >> + { > >> + changed = false; > >> + > >> + if (afdo_propagate_edge (true)) > >> + changed = true; > >> + if (afdo_propagate_edge (false)) > >> + changed = true; > >> + afdo_propagate_circuit (); > >> + } > > > > I wonder if there is any code to share with same (except for the circuit thing) code in profile.c > > and gcov.c? Profile.c also have the minimal flow algorithm for fixing broken profiles, perhaps > > it could be useful here? > > The initial SampleFDO implementation uses MCF algorithm to calculate > edge counts (the current mcf.c actually comes from that effort). > However, from our performance tuning effort, we found that MCF is an > overkill for AutoFDO purpose, and it's extremely slow and hard to > debug. Thus we implemented this ad-hoc heuristic for propagation. OK, I do not see however why we do not share one of these two solutions for both cases. Again someting that may be cleaned up later. How slow MCF is in practice? > > >> + FOR_ALL_BB_FN (bb, cfun) > >> + { > >> + edge e; > >> + edge_iterator ei; > >> + > >> + FOR_EACH_EDGE (e, ei, bb->succs) > >> + e->count = > >> + (double) bb->count * e->probability / REG_BR_PROB_BASE; > >> + bb->aux = NULL; > >> + } > >> + > >> + loop_optimizer_finalize (); > >> + free_dominance_info (CDI_DOMINATORS); > >> + free_dominance_info (CDI_POST_DOMINATORS); > >> +} > > > > I wonder if using the estimated profile to fill in the missing gaps would make sense? > > I.e. running the standard branch probabilities->freuqencies propagation code bug this > > time taking the determined counts as known values? > > Not sure if I follow. Could you elaborate? > > >> + > >> +/* Perform value profile transformation using AutoFDO profile. Add the > >> + promoted stmts to PROMOTED_STMTS. Return TRUE if there is any > >> + indirect call promoted. */ > >> + > >> +static bool > >> +afdo_vpt_for_early_inline (stmt_set *promoted_stmts) > > > > What exactly this function does? It inlines according to the inline decisions > > made at train run time and recorded to profile feedback? > > As replied above, we need iterative vpt-einline to make sure the IR > looks exactly the same as profiling binary before annotation. So the main reason is that by vpt you turn indirect calls into direct calls? Perhaps this can happen during the einline itself (so we do not go through the redundant inline transform passes)? I will look in detail into the patch soon. Honza ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH] AutoFDO patch for trunk 2014-10-09 22:34 ` Jan Hubicka @ 2014-10-10 3:15 ` Dehao Chen 0 siblings, 0 replies; 25+ messages in thread From: Dehao Chen @ 2014-10-10 3:15 UTC (permalink / raw) To: Jan Hubicka; +Cc: GCC Patches, David Li >> >> This will cause bzip2 performance to degrade 6%. I haven't had time to >> triage the problem. Will investigate this later. > > Still I would preffer to make this by default flag_reorder_blocks_and_partition > to false with auto_profile. We could do that incrementally, lets just drop > this from initial patch. Sure, I'll remove that for now. >> >> Index: gcc/cgraphclones.c >> >> =================================================================== >> >> --- gcc/cgraphclones.c (revision 210180) >> >> +++ gcc/cgraphclones.c (working copy) >> >> @@ -435,6 +435,11 @@ cgraph_clone_node (struct cgraph_node *n, tree dec >> >> } >> >> else >> >> count_scale = 0; >> >> + /* In AutoFDO, if edge count is larger than callee's entry block >> >> + count, we will not update the original callee because it may >> >> + mistakenly mark some hot function as cold. */ >> >> + if (flag_auto_profile && count >= n->count) >> >> + update_original = false; >> > >> > This looks like a hack. Lets go with it later - I see what you are shooting for here >> > and non-autoFDO has kind of same problem, too. Counts are not at all that reliable >> > after some inlining. >> > >> > I wonder how this hack would fare for -fprofile-use >> >> It does not affect -fprofile-use performance in my testing. However, >> for AutoFDO, this happens much more often. > > Yep, lets however drop it from the initial patch and make the tweaks go in later. OK >> >> Index: gcc/auto-profile.c >> >> =================================================================== >> >> --- gcc/auto-profile.c (revision 0) >> >> +++ gcc/auto-profile.c (revision 0) >> >> + After the above 3 phases, all profile is readily annotated on the GCC IR. >> >> + AutoFDO tries to reuse all FDO infrastructure as much as possible to make >> >> + use of the profile. E.g. it uses existing mechanism to calculate the basic >> >> + block/edge frequency, as well as the cgraph node/edge count. >> >> +*/ >> > >> > I suppose the phases can be made explicit to the PM, so early inliner can stay >> > a pass. >> >> >> Unfortunately, this is more complicated: >> >> while(changed) { >> vpt(); >> einline(); >> annotate(); >> } > > Hmm, PM do not support iteration (though it wouldmake sense in some cases at least for > expensive optimization). Why you need to iterate einline with the other two? It's for iterative AFDO. i.e. the profile could be collected from AFDO-optimized binary. E.g. main() ind_call foo() foo() ind_call bar() First, profile is collected from O2, thus foo will be promoted and inlined into main. Then we profile on this AFDO-optimized binary, before annotation, we need to make sure that foo is promoted and inlined main. And in the next iteration, before annotation, we need to make sure bar is promoted and inlined into foo, which is promoted and inlined into main. >> >> >> >> +/* Store a string array, indexed by string position in the array. */ >> >> +class string_table { >> >> +public: >> >> + static string_table *create (); >> >> + >> >> + /* For a given string, returns its index. */ >> >> + int get_index (const char *name) const; >> >> + >> >> + /* For a given decl, returns the index of the decl name. */ >> >> + int get_index_by_decl (tree decl) const; >> >> + >> >> + /* For a given index, returns the string. */ >> >> + const char *get_name (int index) const; >> > >> > I wonder how these should work with LTO. I suppose we can tell user to not LTO >> > for train run, we should teach the table to ignore LTO generated suffixes as >> > well as random seeds (like coverage code already does). >> > >> > What happens when two static functions have same name? >> >> Shall we first focus on non-LTO version. And there could be separate >> effort to make AutoFDO+LTO work? >> >> For static functions, there will only be one copy in the profile. the >> profile will be merged. I agree that this could potentially cause >> problem. But OTOH, in a good practice, these functions should *not* be >> hot functions. At least in our practice, we didn't see any hot >> functions like this. > > You mean that static functions in differnt units that have same name tends to be > not hot? I don't really follow it... I mean that if two static functions are both hot, it's better for them to have different names. Otherwise performance tuning/debuging would be super hard. >> >> I agree. My original intention is to make the function name handling >> as simple as possible. The current impl is based on the asumption that >> all important (hot) functions deserve a separate name (not just suffix >> difference). We could definitely refine the function name handling in >> the future patches. >> >> > I would definitely merge logic here with logic in gcov that already knows how to skip >> > random seeds and other cases that makes filenames to diverge... >> >> Could you point me to the gcov logic that handles skipping function name suffix? > > See coverage_checksum_string in coverage.c Looks like that function is filling in randomly generated numbers by "0". This was not what I want because compiler may not even generate these random numbers. E.g. for a given function, compiler may generate .part suffix in FDO compilation, but not in O2 build. I want to make sure that they are all mapped to the same function name. Thus I choose to strip the entire suffix. >> >> >> + >> >> +/* Return the combined location, which is a 32bit integer in which >> >> + higher 16 bits stores the line offset of LOC to the start lineno >> >> + of DECL, The lower 16 bits stores the discrimnator. */ >> >> + >> >> +static unsigned >> >> +get_combined_location (location_t loc, tree decl) >> >> +{ >> >> + return ((LOCATION_LINE (loc) - DECL_SOURCE_LINE (decl)) << 16); >> >> +} >> > >> > What happens on overflows here? I guess they are possible - 65536 lines is not infinity >> > these days. >> >> If a function is longer than 65535 lines, then some lines may have >> incorrect count attributed. But in pactice, we did not find any >> function that is over 10K lines. > > In autogenerated code that may happen. I.e. insn-recog is 300k lines of code and it is > pure luck that the generator artifically split the large decision tree into subfunctions > (probably since early 90's) > > But yep, also something that can be dealt with incrementally. > Peraps we could add a TODO comment to the begginign of file describing the main > limitations. Agree. Another option is to leave more bits for offset, and less bits for discriminator, which are usually small numbers. >> >> + >> >> +/* Special propagation for circuit expressions. Because GCC translates >> >> + control flow into data flow for circuit expressions. E.g. >> >> + BB1: >> >> + if (a && b) >> >> + BB2 >> >> + else >> >> + BB3 >> >> + >> >> + will be translated into: >> >> + >> >> + BB1: >> >> + if (a) >> >> + goto BB.t1 >> >> + else >> >> + goto BB.t3 >> >> + BB.t1: >> >> + if (b) >> >> + goto BB.t2 >> >> + else >> >> + goto BB.t3 >> >> + BB.t2: >> >> + goto BB.t3 >> >> + BB.t3: >> >> + tmp = PHI (0 (BB1), 0 (BB.t1), 1 (BB.t2) >> >> + if (tmp) >> >> + goto BB2 >> >> + else >> >> + goto BB3 >> >> + >> >> + In this case, we need to propagate through PHI to determine the edge >> >> + count of BB1->BB.t1, BB.t1->BB.t2. */ >> > >> > So here you have known probabilities of BB1, BB2 and BB3 and you need to guess all the rest? >> > Why it need so much of gimple matching? >> >> This is basically because some basic blocks have no stmt so that we >> can not get its count from source level profile. However, we can get >> edge count through PHI nodes. >> >> >> + while (changed && i++ < 10) >> >> + { >> >> + changed = false; >> >> + >> >> + if (afdo_propagate_edge (true)) >> >> + changed = true; >> >> + if (afdo_propagate_edge (false)) >> >> + changed = true; >> >> + afdo_propagate_circuit (); >> >> + } >> > >> > I wonder if there is any code to share with same (except for the circuit thing) code in profile.c >> > and gcov.c? Profile.c also have the minimal flow algorithm for fixing broken profiles, perhaps >> > it could be useful here? >> >> The initial SampleFDO implementation uses MCF algorithm to calculate >> edge counts (the current mcf.c actually comes from that effort). >> However, from our performance tuning effort, we found that MCF is an >> overkill for AutoFDO purpose, and it's extremely slow and hard to >> debug. Thus we implemented this ad-hoc heuristic for propagation. > > OK, I do not see however why we do not share one of these two solutions for both > cases. Again someting that may be cleaned up later. > How slow MCF is in practice? It depends, for large functions, it will be super slow, thus we need to limit the # of iterations. As a result, the graph after MCF may still be inconsistent. My adhoc algorithm is not aiming at making flow consistent, but guessing the right branch probability out of limited BB counts. MCF can do the same thing, but if we reached the max # iter threshold, the impact would be tragical. >> >> >> + >> >> +/* Perform value profile transformation using AutoFDO profile. Add the >> >> + promoted stmts to PROMOTED_STMTS. Return TRUE if there is any >> >> + indirect call promoted. */ >> >> + >> >> +static bool >> >> +afdo_vpt_for_early_inline (stmt_set *promoted_stmts) >> > >> > What exactly this function does? It inlines according to the inline decisions >> > made at train run time and recorded to profile feedback? >> >> As replied above, we need iterative vpt-einline to make sure the IR >> looks exactly the same as profiling binary before annotation. > > So the main reason is that by vpt you turn indirect calls into direct calls? > Perhaps this can happen during the einline itself (so we do not go through the > redundant inline transform passes)? I think it makes sense to add vpt into einline. In this way, we don't need to expose early_inliner to auto-profile.c Thanks, Dehao ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH] AutoFDO patch for trunk 2014-10-09 0:24 ` Dehao Chen 2014-10-09 22:34 ` Jan Hubicka @ 2014-10-14 15:08 ` Jan Hubicka 2014-10-14 21:10 ` Dehao Chen 1 sibling, 1 reply; 25+ messages in thread From: Jan Hubicka @ 2014-10-14 15:08 UTC (permalink / raw) To: Dehao Chen; +Cc: Jan Hubicka, GCC Patches, David Li > Index: gcc/cgraphclones.c > =================================================================== > --- gcc/cgraphclones.c (revision 215826) > +++ gcc/cgraphclones.c (working copy) > @@ -453,6 +453,11 @@ > } > else > count_scale = 0; > + /* In AutoFDO, if edge count is larger than callee's entry block > + count, we will not update the original callee because it may > + mistakenly mark some hot function as cold. */ > + if (flag_auto_profile && gcov_count >= count) > + update_original = false; lets drop this from initial patch. > Index: gcc/bb-reorder.c > =================================================================== > --- gcc/bb-reorder.c (revision 215826) > +++ gcc/bb-reorder.c (working copy) > @@ -1569,15 +1569,14 @@ > /* Mark which partition (hot/cold) each basic block belongs in. */ > FOR_EACH_BB_FN (bb, cfun) > { > - bool cold_bb = false; > + bool cold_bb = probably_never_executed_bb_p (cfun, bb); and this too (basically all the tweaks should IMO go in independently and ideally in a way that does not need flag_auto_profile test). > +/* Return true if BB contains indirect call. */ > + > +static bool > +has_indirect_call (basic_block bb) > +{ > + gimple_stmt_iterator gsi; > + > + for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi)) > + { > + gimple stmt = gsi_stmt (gsi); > + if (gimple_code (stmt) == GIMPLE_CALL > + && (gimple_call_fn (stmt) == NULL > + || TREE_CODE (gimple_call_fn (stmt)) != FUNCTION_DECL)) You probably want to skip gimple_call_internal_p calls here. > + > +/* From AutoFDO profiles, find values inside STMT for that we want to measure > + histograms for indirect-call optimization. */ > + > +static void > +afdo_indirect_call (gimple_stmt_iterator *gsi, const icall_target_map &map, > + bool transform) > +{ > + gimple stmt = gsi_stmt (*gsi); > + tree callee; > + > + if (map.size() == 0 || gimple_code (stmt) != GIMPLE_CALL > + || gimple_call_fndecl (stmt) != NULL_TREE) > + return; > + > + callee = gimple_call_fn (stmt); > + > + histogram_value hist = gimple_alloc_histogram_value ( > + cfun, HIST_TYPE_INDIR_CALL, stmt, callee); > + hist->n_counters = 3; > + hist->hvalue.counters = XNEWVEC (gcov_type, hist->n_counters); > + gimple_add_histogram_value (cfun, stmt, hist); > + > + gcov_type total = 0; > + icall_target_map::const_iterator max_iter = map.end(); > + > + for (icall_target_map::const_iterator iter = map.begin(); > + iter != map.end(); ++iter) > + { > + total += iter->second; > + if (max_iter == map.end() || max_iter->second < iter->second) > + max_iter = iter; > + } > + > + hist->hvalue.counters[0] = (unsigned long long) > + afdo_string_table->get_name (max_iter->first); > + hist->hvalue.counters[1] = max_iter->second; > + hist->hvalue.counters[2] = total; > + > + if (!transform) > + return; > + > + if (gimple_ic_transform (gsi)) > + { > + struct cgraph_edge *indirect_edge = > + cgraph_node::get (current_function_decl)->get_edge (stmt); > + struct cgraph_node *direct_call = > + find_func_by_profile_id ((int)hist->hvalue.counters[0]); > + if (DECL_STRUCT_FUNCTION (direct_call->decl) == NULL) > + return; > + struct cgraph_edge *new_edge = > + indirect_edge->make_speculative (direct_call, 0, 0); > + new_edge->redirect_call_stmt_to_callee (); > + gimple_remove_histogram_value (cfun, stmt, hist); > + inline_call (new_edge, true, NULL, NULL, false); > + return; > + } > + return; Is it necessary to go via histogram and gimple_ic_transform here? I would expect that all you need is to make the speculative edge and inline it. (bypassing the work of producing fake histogram value and calling igmple_ic_transofrm on it) Also it seems to me that you want to set direct_count nad frequency argument of make_speculative so the resulting function profile is not off. The rest of interfaces seems quite sane now. Can you please look into using speculative edges directly instead of hooking into the vpt infrastructure and fixing the formatting issues of the new pass? I will try to make another pass over the actual streaming logic that I find bit difficult to read, but I quite trust you it does the right thing ;) Honza ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH] AutoFDO patch for trunk 2014-10-14 15:08 ` Jan Hubicka @ 2014-10-14 21:10 ` Dehao Chen 2014-10-14 23:28 ` Dehao Chen 0 siblings, 1 reply; 25+ messages in thread From: Dehao Chen @ 2014-10-14 21:10 UTC (permalink / raw) To: Jan Hubicka; +Cc: GCC Patches, David Li [-- Attachment #1: Type: text/plain, Size: 5354 bytes --] On Tue, Oct 14, 2014 at 8:02 AM, Jan Hubicka <hubicka@ucw.cz> wrote: >> Index: gcc/cgraphclones.c >> =================================================================== >> --- gcc/cgraphclones.c (revision 215826) >> +++ gcc/cgraphclones.c (working copy) >> @@ -453,6 +453,11 @@ >> } >> else >> count_scale = 0; >> + /* In AutoFDO, if edge count is larger than callee's entry block >> + count, we will not update the original callee because it may >> + mistakenly mark some hot function as cold. */ >> + if (flag_auto_profile && gcov_count >= count) >> + update_original = false; > > lets drop this from initial patch. Done > >> Index: gcc/bb-reorder.c >> =================================================================== >> --- gcc/bb-reorder.c (revision 215826) >> +++ gcc/bb-reorder.c (working copy) >> @@ -1569,15 +1569,14 @@ >> /* Mark which partition (hot/cold) each basic block belongs in. */ >> FOR_EACH_BB_FN (bb, cfun) >> { >> - bool cold_bb = false; >> + bool cold_bb = probably_never_executed_bb_p (cfun, bb); > > and this too > (basically all the tweaks should IMO go in independently and ideally in > a way that does not need flag_auto_profile test). Done. >> +/* Return true if BB contains indirect call. */ >> + >> +static bool >> +has_indirect_call (basic_block bb) >> +{ >> + gimple_stmt_iterator gsi; >> + >> + for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi)) >> + { >> + gimple stmt = gsi_stmt (gsi); >> + if (gimple_code (stmt) == GIMPLE_CALL >> + && (gimple_call_fn (stmt) == NULL >> + || TREE_CODE (gimple_call_fn (stmt)) != FUNCTION_DECL)) > > You probably want to skip gimple_call_internal_p calls here. Done >> + >> +/* From AutoFDO profiles, find values inside STMT for that we want to measure >> + histograms for indirect-call optimization. */ >> + >> +static void >> +afdo_indirect_call (gimple_stmt_iterator *gsi, const icall_target_map &map, >> + bool transform) >> +{ >> + gimple stmt = gsi_stmt (*gsi); >> + tree callee; >> + >> + if (map.size() == 0 || gimple_code (stmt) != GIMPLE_CALL >> + || gimple_call_fndecl (stmt) != NULL_TREE) >> + return; >> + >> + callee = gimple_call_fn (stmt); >> + >> + histogram_value hist = gimple_alloc_histogram_value ( >> + cfun, HIST_TYPE_INDIR_CALL, stmt, callee); >> + hist->n_counters = 3; >> + hist->hvalue.counters = XNEWVEC (gcov_type, hist->n_counters); >> + gimple_add_histogram_value (cfun, stmt, hist); >> + >> + gcov_type total = 0; >> + icall_target_map::const_iterator max_iter = map.end(); >> + >> + for (icall_target_map::const_iterator iter = map.begin(); >> + iter != map.end(); ++iter) >> + { >> + total += iter->second; >> + if (max_iter == map.end() || max_iter->second < iter->second) >> + max_iter = iter; >> + } >> + >> + hist->hvalue.counters[0] = (unsigned long long) >> + afdo_string_table->get_name (max_iter->first); >> + hist->hvalue.counters[1] = max_iter->second; >> + hist->hvalue.counters[2] = total; >> + >> + if (!transform) >> + return; >> + >> + if (gimple_ic_transform (gsi)) >> + { >> + struct cgraph_edge *indirect_edge = >> + cgraph_node::get (current_function_decl)->get_edge (stmt); >> + struct cgraph_node *direct_call = >> + find_func_by_profile_id ((int)hist->hvalue.counters[0]); >> + if (DECL_STRUCT_FUNCTION (direct_call->decl) == NULL) >> + return; >> + struct cgraph_edge *new_edge = >> + indirect_edge->make_speculative (direct_call, 0, 0); >> + new_edge->redirect_call_stmt_to_callee (); >> + gimple_remove_histogram_value (cfun, stmt, hist); >> + inline_call (new_edge, true, NULL, NULL, false); >> + return; >> + } >> + return; > > Is it necessary to go via histogram and gimple_ic_transform here? I would expect that all you > need is to make the speculative edge and inline it. (bypassing the work of producing fake > histogram value and calling igmple_ic_transofrm on it) > > Also it seems to me that you want to set direct_count nad frequency argument of > make_speculative so the resulting function profile is not off. This function is actually served for 2 purposes: * before annotation, we need to mark histogram, promote and inline * after annotation, we just need to mark, and let follow-up logic to decide if it needs to promote and inline. And you are right, for the "before annotation" case, we can simply call "mark speculative" and "inline". But we still need the logic to fake histogram for "after annotation" case. As a result, I unified two cases into one function to reuse code as much as possible. Shall I separate it into two functions instead? > > The rest of interfaces seems quite sane now. Can you please look into > using speculative edges directly instead of hooking into the vpt infrastructure > and fixing the formatting issues of the new pass? I'll work on the formatting issues now (need to learn the format first ;-). The attached patch is up-to-date except for formatting changes. I'll upload the patch again once the format change is in. Thanks, Dehao > > I will try to make another pass over the actual streaming logic that I find bit difficult > to read, but I quite trust you it does the right thing ;) > > Honza [-- Attachment #2: patch.txt --] [-- Type: text/plain, Size: 72700 bytes --] Index: gcc/debug.h =================================================================== --- gcc/debug.h (revision 215826) +++ gcc/debug.h (working copy) @@ -169,6 +169,7 @@ extern const struct gcc_debug_hooks sdb_debug_hooks; extern const struct gcc_debug_hooks xcoff_debug_hooks; extern const struct gcc_debug_hooks dwarf2_debug_hooks; +extern const struct gcc_debug_hooks auto_profile_debug_hooks; extern const struct gcc_debug_hooks vmsdbg_debug_hooks; /* Dwarf2 frame information. */ Index: gcc/auto-profile.c =================================================================== --- gcc/auto-profile.c (revision 0) +++ gcc/auto-profile.c (revision 0) @@ -0,0 +1,1664 @@ +/* Read and annotate call graph profile from the auto profile data file. + Copyright (C) 2014. Free Software Foundation, Inc. + Contributed by Dehao Chen (dehao@google.com) + +This file is part of GCC. + +GCC is free software; you can redistribute it and/or modify it under +the terms of the GNU General Public License as published by the Free +Software Foundation; either version 3, or (at your option) any later +version. + +GCC is distributed in the hope that it will be useful, but WITHOUT ANY +WARRANTY; without even the implied warranty of MERCHANTABILITY or +FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License +for more details. + +You should have received a copy of the GNU General Public License +along with GCC; see the file COPYING3. If not see +<http://www.gnu.org/licenses/>. */ + +#include <string.h> +#include <map> +#include <vector> +#include <set> + +#include "config.h" +#include "system.h" +#include "coretypes.h" +#include "tree.h" +#include "tree-pass.h" +#include "flags.h" +#include "basic-block.h" +#include "diagnostic-core.h" +#include "gcov-io.h" +#include "input.h" +#include "profile.h" +#include "langhooks.h" +#include "opts.h" +#include "tree-pass.h" +#include "cfgloop.h" +#include "tree-ssa-alias.h" +#include "tree-cfg.h" +#include "tree-cfgcleanup.h" +#include "tree-ssa-operands.h" +#include "tree-into-ssa.h" +#include "internal-fn.h" +#include "is-a.h" +#include "gimple-expr.h" +#include "gimple.h" +#include "gimple-iterator.h" +#include "gimple-ssa.h" +#include "cgraph.h" +#include "value-prof.h" +#include "coverage.h" +#include "params.h" +#include "ipa-inline.h" +#include "tree-inline.h" +#include "auto-profile.h" + +/* The following routines implements AutoFDO optimization. + + This optimization uses sampling profiles to annotate basic block counts + and uses heuristics to estimate branch probabilities. + + There are three phases in AutoFDO: + + Phase 1: Read profile from the profile data file. + The following info is read from the profile datafile: + * string_table: a map between function name and its index. + * autofdo_source_profile: a map from function_instance name to + function_instance. This is represented as a forest of + function_instances. + * WorkingSet: a histogram of how many instructions are covered for a + given percentage of total cycles. + + Phase 2: Early inline + VPT. + Early inline uses autofdo_source_profile to find if a callsite is: + * inlined in the profiled binary. + * callee body is hot in the profiling run. + If both condition satisfies, early inline will inline the callsite + regardless of the code growth. + Phase 2 is an iterative process. During each iteration, we also check + if an indirect callsite is promoted and inlined in the profiling run. + If yes, vpt will happen to force promote it and in the next iteration, + einline will inline the promoted callsite in the next iteration. + + Phase 3: Annotate control flow graph. + AutoFDO uses a separate pass to: + * Annotate basic block count + * Estimate branch probability + + After the above 3 phases, all profile is readily annotated on the GCC IR. + AutoFDO tries to reuse all FDO infrastructure as much as possible to make + use of the profile. E.g. it uses existing mechanism to calculate the basic + block/edge frequency, as well as the cgraph node/edge count. +*/ + +#define DEFAULT_AUTO_PROFILE_FILE "fbdata.afdo" + +namespace autofdo { + +/* Represent a source location: (function_decl, lineno). */ +typedef std::pair<tree, unsigned> decl_lineno; + +/* Represent an inline stack. vector[0] is the leaf node. */ +typedef std::vector<decl_lineno> inline_stack; + +/* String array that stores function names. */ +typedef std::vector<const char *> string_vector; + +/* Map from function name's index in string_table to target's + execution count. */ +typedef std::map<unsigned, gcov_type> icall_target_map; + +/* Set of gimple stmts. Used to track if the stmt has already been promoted + to direct call. */ +typedef std::set<gimple> stmt_set; + +/* Represent count info of an inline stack. */ +struct count_info +{ + /* Sampled count of the inline stack. */ + gcov_type count; + + /* Map from indirect call target to its sample count. */ + icall_target_map targets; + + /* Whether this inline stack is already used in annotation. + + Each inline stack should only be used to annotate IR once. + This will be enforced when instruction-level discriminator + is supported. */ + bool annotated; +}; + +/* operator< for "const char *". */ +struct string_compare +{ + bool operator() (const char *a, const char *b) const + { return strcmp (a, b) < 0; } +}; + +/* Store a string array, indexed by string position in the array. */ +class string_table { +public: + static string_table *create (); + + /* For a given string, returns its index. */ + int get_index (const char *name) const; + + /* For a given decl, returns the index of the decl name. */ + int get_index_by_decl (tree decl) const; + + /* For a given index, returns the string. */ + const char *get_name (int index) const; + +private: + string_table () {} + bool read (); + + typedef std::map<const char *, unsigned, string_compare> string_index_map; + string_vector vector_; + string_index_map map_; +}; + +/* Profile of a function instance: + 1. total_count of the function. + 2. head_count of the function (only valid when function is a top-level + function_instance, i.e. it is the original copy instead of the + inlined copy). + 3. map from source location (decl_lineno) of the inlined callsite to + profile (count_info). + 4. map from callsite to callee function_instance. */ +class function_instance { +public: + typedef std::vector<function_instance *> function_instance_stack; + + /* Read the profile and return a function_instance with head count as + HEAD_COUNT. Recursively read callsites to create nested function_instances + too. STACK is used to track the recursive creation process. */ + static function_instance *read_function_instance ( + function_instance_stack *stack, gcov_type head_count); + + /* Recursively deallocate all callsites (nested function_instances). */ + ~function_instance (); + + /* Accessors. */ + int name () const { return name_; } + gcov_type total_count () const { return total_count_; } + gcov_type head_count () const { return head_count_; } + + /* Recursively traverse STACK starting from LEVEL to find the corresponding + function_instance. */ + function_instance *get_function_instance (const inline_stack &stack, + unsigned level); + + /* Store the profile info for LOC in INFO. Return TRUE if profile info + is found. */ + bool get_count_info (location_t loc, count_info *info) const; + + /* Read the inlined indirect call target profile for STMT and store it in + MAP, return the total count for all inlined indirect calls. */ + gcov_type find_icall_target_map (gimple stmt, icall_target_map *map) const; + + /* Sum of counts that is used during annotation. */ + gcov_type total_annotated_count () const; + + /* Mark LOC as annotated. */ + void mark_annotated (location_t loc); + +private: + /* Callsite, represented as (decl_lineno, callee_function_name_index). */ + typedef std::pair<unsigned, unsigned> callsite; + + /* Map from callsite to callee function_instance. */ + typedef std::map<callsite, function_instance *> callsite_map; + + function_instance (unsigned name, gcov_type head_count) + : name_(name), total_count_(0), head_count_(head_count) {} + + /* Traverse callsites of the current function_instance to find one at the + location of LINENO and callee name represented in DECL. */ + function_instance *get_function_instance_by_decl ( + unsigned lineno, tree decl) const; + + /* Map from source location (decl_lineno) to profile (count_info). */ + typedef std::map<unsigned, count_info> position_count_map; + + /* function_instance name index in the string_table. */ + unsigned name_; + + /* Total sample count. */ + gcov_type total_count_; + + /* Entry BB's sample count. */ + gcov_type head_count_; + + /* Map from callsite location to callee function_instance. */ + callsite_map callsites; + + /* Map from source location to count_info. */ + position_count_map pos_counts; +}; + +/* Profile for all functions. */ +class autofdo_source_profile { +public: + static autofdo_source_profile *create () + { + autofdo_source_profile *map = new autofdo_source_profile (); + + if (map->read ()) + return map; + delete map; + return NULL; + } + + ~autofdo_source_profile (); + + /* For a given DECL, returns the top-level function_instance. */ + function_instance *get_function_instance_by_decl (tree decl) const; + + /* Find count_info for a given gimple STMT. If found, store the count_info + in INFO and return true; otherwise return false. */ + bool get_count_info (gimple stmt, count_info *info) const; + + /* Find total count of the callee of EDGE. */ + gcov_type get_callsite_total_count (struct cgraph_edge *edge) const; + + /* Update value profile INFO for STMT from the inlined indirect callsite. + Return true if INFO is updated. */ + bool update_inlined_ind_target (gimple stmt, count_info *info); + + /* Mark LOC as annotated. */ + void mark_annotated (location_t loc); + +private: + /* Map from function_instance name index (in string_table) to + function_instance. */ + typedef std::map<unsigned, function_instance *> + name_function_instance_map; + + autofdo_source_profile () {} + + /* Read AutoFDO profile and returns TRUE on success. */ + bool read (); + + /* Return the function_instance in the profile that correspond to the + inline STACK. */ + function_instance *get_function_instance_by_inline_stack ( + const inline_stack &stack) const; + + name_function_instance_map map_; +}; + +/* Store the strings read from the profile data file. */ +static string_table *afdo_string_table; + +/* Store the AutoFDO source profile. */ +static autofdo_source_profile *afdo_source_profile; + +/* gcov_ctr_summary structure to store the profile_info. */ +static struct gcov_ctr_summary *afdo_profile_info; + +/* Helper functions. */ + +/* Return the original name of NAME: strip the suffix that starts + with '.' */ + +static const char *get_original_name (const char *name) +{ + char *ret = xstrdup (name); + char *find = strchr (ret, '.'); + if (find != NULL) + *find = 0; + return ret; +} + +/* Return the combined location, which is a 32bit integer in which + higher 16 bits stores the line offset of LOC to the start lineno + of DECL, The lower 16 bits stores the discrimnator. */ + +static unsigned +get_combined_location (location_t loc, tree decl) +{ + /* TODO: allow more bits for line and less bits for discriminator. */ + return ((LOCATION_LINE (loc) - DECL_SOURCE_LINE (decl)) << 16); +} + +/* Return the function decl of a given lexical BLOCK. */ + +static tree +get_function_decl_from_block (tree block) +{ + tree decl; + + if (LOCATION_LOCUS (BLOCK_SOURCE_LOCATION (block) == UNKNOWN_LOCATION)) + return NULL_TREE; + + for (decl = BLOCK_ABSTRACT_ORIGIN (block); + decl && (TREE_CODE (decl) == BLOCK); + decl = BLOCK_ABSTRACT_ORIGIN (decl)) + if (TREE_CODE (decl) == FUNCTION_DECL) + break; + return decl; +} + +/* Store inline stack for STMT in STACK. */ + +static void +get_inline_stack (location_t locus, inline_stack *stack) +{ + if (LOCATION_LOCUS (locus) == UNKNOWN_LOCATION) + return; + + tree block = LOCATION_BLOCK (locus); + if (block && TREE_CODE (block) == BLOCK) + { + int level = 0; + for (block = BLOCK_SUPERCONTEXT (block); + block && (TREE_CODE (block) == BLOCK); + block = BLOCK_SUPERCONTEXT (block)) + { + location_t tmp_locus = BLOCK_SOURCE_LOCATION (block); + if (LOCATION_LOCUS (tmp_locus) == UNKNOWN_LOCATION) + continue; + + tree decl = get_function_decl_from_block (block); + stack->push_back (std::make_pair ( + decl, get_combined_location (locus, decl))); + locus = tmp_locus; + level++; + } + } + stack->push_back (std::make_pair ( + current_function_decl, + get_combined_location (locus, current_function_decl))); +} + +/* Return STMT's combined location, which is a 32bit integer in which + higher 16 bits stores the line offset of LOC to the start lineno + of DECL, The lower 16 bits stores the discrimnator. */ + +static unsigned +get_relative_location_for_stmt (gimple stmt) +{ + location_t locus = gimple_location (stmt); + if (LOCATION_LOCUS (locus) == UNKNOWN_LOCATION) + return UNKNOWN_LOCATION; + + for (tree block = gimple_block (stmt); + block && (TREE_CODE (block) == BLOCK); + block = BLOCK_SUPERCONTEXT (block)) + if (LOCATION_LOCUS (BLOCK_SOURCE_LOCATION (block)) != UNKNOWN_LOCATION) + return get_combined_location ( + locus, get_function_decl_from_block (block)); + return get_combined_location (locus, current_function_decl); +} + +/* Return true if BB contains indirect call. */ + +static bool +has_indirect_call (basic_block bb) +{ + gimple_stmt_iterator gsi; + + for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi)) + { + gimple stmt = gsi_stmt (gsi); + if (gimple_code (stmt) == GIMPLE_CALL + && !gimple_call_internal_p (stmt) + && (gimple_call_fn (stmt) == NULL + || TREE_CODE (gimple_call_fn (stmt)) != FUNCTION_DECL)) + return true; + } + return false; +} + +/* Member functions for string_table. */ + +string_table * +string_table::create () +{ + string_table *map = new string_table(); + if (map->read ()) + return map; + delete map; + return NULL; +} + +/* Return the index of a given function NAME. */ + +int +string_table::get_index (const char *name) const +{ + if (name == NULL) + return -1; + string_index_map::const_iterator iter = map_.find (name); + if (iter == map_.end()) + return -1; + else + return iter->second; +} + +/* Return the index of a given function DECL. */ + +int +string_table::get_index_by_decl (tree decl) const +{ + const char *name = get_original_name ( + IDENTIFIER_POINTER (DECL_ASSEMBLER_NAME (decl))); + int ret = get_index (name); + if (ret != -1) + return ret; + ret = get_index (lang_hooks.dwarf_name (decl, 0)); + if (ret != -1) + return ret; + if (DECL_ABSTRACT_ORIGIN (decl)) + return get_index_by_decl (DECL_ABSTRACT_ORIGIN (decl)); + else + return -1; +} + +/* Return the function name of a given INDEX. */ + +const char * +string_table::get_name (int index) const +{ + gcc_assert (index > 0 && index < (int) vector_.size()); + return vector_[index]; +} + +/* Read the string table. Return TRUE if reading is successful. */ + +bool +string_table::read () +{ + if (gcov_read_unsigned () != GCOV_TAG_AFDO_FILE_NAMES) + return false; + /* Skip the length of the section. */ + gcov_read_unsigned (); + /* Read in the file name table. */ + unsigned string_num = gcov_read_unsigned (); + for (unsigned i = 0; i < string_num; i++) + { + vector_.push_back (get_original_name (gcov_read_string ())); + map_[vector_.back()] = i; + } + return true; +} + + +/* Member functions for function_instance. */ + +function_instance::~function_instance () +{ + for (callsite_map::iterator iter = callsites.begin(); + iter != callsites.end(); ++iter) + delete iter->second; +} + +/* Traverse callsites of the current function_instance to find one at the + location of LINENO and callee name represented in DECL. */ + +function_instance * +function_instance::get_function_instance_by_decl ( + unsigned lineno, tree decl) const +{ + int func_name_idx = afdo_string_table->get_index_by_decl (decl); + if (func_name_idx != -1) + { + callsite_map::const_iterator ret = callsites.find ( + std::make_pair (lineno, func_name_idx)); + if (ret != callsites.end ()) + return ret->second; + } + func_name_idx = afdo_string_table->get_index ( + lang_hooks.dwarf_name (decl, 0)); + if (func_name_idx != -1) + { + callsite_map::const_iterator ret = callsites.find ( + std::make_pair (lineno, func_name_idx)); + if (ret != callsites.end ()) + return ret->second; + } + if (DECL_ABSTRACT_ORIGIN (decl)) + return get_function_instance_by_decl (lineno, DECL_ABSTRACT_ORIGIN (decl)); + else + return NULL; +} + +/* Recursively traverse STACK starting from LEVEL to find the corresponding + function_instance. */ + +function_instance * +function_instance::get_function_instance ( + const inline_stack &stack, unsigned level) +{ + if (level == 0) + return this; + function_instance *s = + get_function_instance_by_decl (stack[level].second, stack[level - 1].first); + if (s) + return s->get_function_instance (stack, level - 1); + else + return NULL; +} + +/* Store the profile info for LOC in INFO. Return TRUE if profile info + is found. */ + +bool +function_instance::get_count_info (location_t loc, count_info *info) const +{ + position_count_map::const_iterator iter = pos_counts.find (loc); + if (iter == pos_counts.end ()) + return false; + *info = iter->second; + return true; +} + +/* Mark LOC as annotated. */ + +void +function_instance::mark_annotated (location_t loc) +{ + position_count_map::iterator iter = pos_counts.find (loc); + if (iter == pos_counts.end ()) + return; + iter->second.annotated = true; +} + +/* Read the inlinied indirect call target profile for STMT and store it in + MAP, return the total count for all inlined indirect calls. */ + +gcov_type +function_instance::find_icall_target_map ( + gimple stmt, icall_target_map *map) const +{ + gcov_type ret = 0; + unsigned stmt_offset = get_relative_location_for_stmt (stmt); + + for (callsite_map::const_iterator iter = callsites.begin(); + iter != callsites.end(); ++iter) + { + unsigned callee = iter->second->name(); + /* Check if callsite location match the stmt. */ + if (iter->first.first != stmt_offset) + continue; + struct cgraph_node *node = find_func_by_profile_id ( + (unsigned long long) afdo_string_table->get_name (callee)); + if (node == NULL) + continue; + if (!check_ic_target (stmt, node)) + continue; + (*map)[callee] = iter->second->total_count (); + ret += iter->second->total_count (); + } + return ret; +} + +/* Read the profile and create a function_instance with head count as + HEAD_COUNT. Recursively read callsites to create nested function_instances + too. STACK is used to track the recursive creation process. */ + +/* function instance profile format: + + ENTRY_COUNT: 8 bytes + NAME_INDEX: 4 bytes + NUM_POS_COUNTS: 4 bytes + NUM_CALLSITES: 4 byte + POS_COUNT_1: + POS_1_OFFSET: 4 bytes + NUM_TARGETS: 4 bytes + COUNT: 8 bytes + TARGET_1: + VALUE_PROFILE_TYPE: 4 bytes + TARGET_IDX: 8 bytes + COUNT: 8 bytes + TARGET_2 + ... + TARGET_n + POS_COUNT_2 + ... + POS_COUNT_N + CALLSITE_1: + CALLSITE_1_OFFSET: 4 bytes + FUNCTION_INSTANCE_PROFILE (nested) + CALLSITE_2 + ... + CALLSITE_n. */ + +function_instance * +function_instance::read_function_instance ( + function_instance_stack *stack, gcov_type head_count) +{ + unsigned name = gcov_read_unsigned (); + unsigned num_pos_counts = gcov_read_unsigned (); + unsigned num_callsites = gcov_read_unsigned (); + function_instance *s = new function_instance (name, head_count); + stack->push_back(s); + + for (unsigned i = 0; i < num_pos_counts; i++) + { + unsigned offset = gcov_read_unsigned () & 0xffff0000; + unsigned num_targets = gcov_read_unsigned (); + gcov_type count = gcov_read_counter (); + s->pos_counts[offset].count = count; + for (unsigned j = 0; j < stack->size(); j++) + (*stack)[j]->total_count_ += count; + for (unsigned j = 0; j < num_targets; j++) + { + /* Only indirect call target histogram is supported now. */ + gcov_read_unsigned (); + gcov_type target_idx = gcov_read_counter (); + s->pos_counts[offset].targets[target_idx] = + gcov_read_counter (); + } + } + for (unsigned i = 0; i < num_callsites; i++) { + unsigned offset = gcov_read_unsigned (); + function_instance *callee_function_instance = + read_function_instance (stack, 0); + s->callsites[std::make_pair (offset, callee_function_instance->name ())] = + callee_function_instance; + } + stack->pop_back(); + return s; +} + +/* Sum of counts that is used during annotation. */ + +gcov_type +function_instance::total_annotated_count () const +{ + gcov_type ret = 0; + for (callsite_map::const_iterator iter = callsites.begin(); + iter != callsites.end(); ++iter) + ret += iter->second->total_annotated_count (); + for (position_count_map::const_iterator iter = pos_counts.begin(); + iter != pos_counts.end(); ++iter) + if (iter->second.annotated) + ret += iter->second.count; + return ret; +} + + +/* Member functions for autofdo_source_profile. */ + +autofdo_source_profile::~autofdo_source_profile () +{ + for (name_function_instance_map::const_iterator iter = map_.begin (); + iter != map_.end (); ++iter) + delete iter->second; +} + +/* For a given DECL, returns the top-level function_instance. */ + +function_instance * +autofdo_source_profile::get_function_instance_by_decl (tree decl) const +{ + int index = afdo_string_table->get_index_by_decl (decl); + if (index == -1) + return NULL; + name_function_instance_map::const_iterator ret = map_.find (index); + return ret == map_.end() ? NULL : ret->second; +} + +/* Find count_info for a given gimple STMT. If found, store the count_info + in INFO and return true; otherwise return false. */ + +bool +autofdo_source_profile::get_count_info (gimple stmt, count_info *info) const +{ + if (LOCATION_LOCUS (gimple_location (stmt)) == cfun->function_end_locus) + return false; + + inline_stack stack; + get_inline_stack (gimple_location (stmt), &stack); + if (stack.size () == 0) + return false; + function_instance *s = get_function_instance_by_inline_stack (stack); + if (s == NULL) + return false; + return s->get_count_info (stack[0].second, info); +} + +/* Mark LOC as annotated. */ + +void +autofdo_source_profile::mark_annotated (location_t loc) { + inline_stack stack; + get_inline_stack (loc, &stack); + if (stack.size () == 0) + return; + function_instance *s = get_function_instance_by_inline_stack (stack); + if (s == NULL) + return; + s->mark_annotated (stack[0].second); +} + +/* Update value profile INFO for STMT from the inlined indirect callsite. + Return true if INFO is updated. */ + +bool +autofdo_source_profile::update_inlined_ind_target ( + gimple stmt, count_info *info) +{ + if (LOCATION_LOCUS (gimple_location (stmt)) == cfun->function_end_locus) + return false; + + count_info old_info; + get_count_info (stmt, &old_info); + gcov_type total = 0; + for (icall_target_map::const_iterator iter = old_info.targets.begin(); + iter != old_info.targets.end(); ++iter) + total += iter->second; + + /* Program behavior changed, original promoted (and inlined) target is not + hot any more. Will avoid promote the original target. + + To check if original promoted target is still hot, we check the total + count of the unpromoted targets (stored in old_info). If it is no less + than half of the callsite count (stored in INFO), the original promoted + target is considered not hot any more. */ + if (total >= info->count / 2) + return false; + + inline_stack stack; + get_inline_stack (gimple_location (stmt), &stack); + if (stack.size () == 0) + return false; + function_instance *s = get_function_instance_by_inline_stack (stack); + if (s == NULL) + return false; + icall_target_map map; + if (s->find_icall_target_map (stmt, &map) == 0) + return false; + for (icall_target_map::const_iterator iter = map.begin(); + iter != map.end(); ++iter) + info->targets[iter->first] = iter->second; + return true; +} + +/* Find total count of the callee of EDGE. */ + +gcov_type +autofdo_source_profile::get_callsite_total_count ( + struct cgraph_edge *edge) const +{ + inline_stack stack; + stack.push_back (std::make_pair(edge->callee->decl, 0)); + get_inline_stack (gimple_location (edge->call_stmt), &stack); + + function_instance *s = get_function_instance_by_inline_stack (stack); + if (s == NULL || + afdo_string_table->get_index (IDENTIFIER_POINTER (DECL_ASSEMBLER_NAME (edge->callee->decl))) != s->name ()) + return 0; + else + return s->total_count (); +} + +/* Read AutoFDO profile and returns TRUE on success. */ + +/* source profile format: + + GCOV_TAG_AFDO_FUNCTION: 4 bytes + LENGTH: 4 bytes + NUM_FUNCTIONS: 4 bytes + FUNCTION_INSTANCE_1 + FUNCTION_INSTANCE_2 + ... + FUNCTION_INSTANCE_N. */ + +bool +autofdo_source_profile::read () +{ + if (gcov_read_unsigned () != GCOV_TAG_AFDO_FUNCTION) + { + inform (0, "Not expected TAG."); + return false; + } + + /* Skip the length of the section. */ + gcov_read_unsigned (); + + /* Read in the function/callsite profile, and store it in local + data structure. */ + unsigned function_num = gcov_read_unsigned (); + for (unsigned i = 0; i < function_num; i++) + { + function_instance::function_instance_stack stack; + function_instance *s = function_instance::read_function_instance ( + &stack, gcov_read_counter ()); + afdo_profile_info->sum_all += s->total_count (); + map_[s->name ()] = s; + } + return true; +} + +/* Return the function_instance in the profile that correspond to the + inline STACK. */ + +function_instance * +autofdo_source_profile::get_function_instance_by_inline_stack ( + const inline_stack &stack) const +{ + name_function_instance_map::const_iterator iter = map_.find ( + afdo_string_table->get_index_by_decl ( + stack[stack.size() - 1].first)); + return iter == map_.end() + ? NULL + : iter->second->get_function_instance (stack, stack.size() - 1); +} + +/* Module profile is only used by LIPO. Here we simply ignore it. */ + +static void +fake_read_autofdo_module_profile () +{ + /* Read in the module info. */ + gcov_read_unsigned (); + + /* Skip the length of the section. */ + gcov_read_unsigned (); + + /* Read in the file name table. */ + unsigned total_module_num = gcov_read_unsigned (); + gcc_assert (total_module_num == 0); +} + +static void +read_profile (void) +{ + if (gcov_open (auto_profile_file, 1) == 0) + error ("Cannot open profile file %s.", auto_profile_file); + + if (gcov_read_unsigned () != GCOV_DATA_MAGIC) + error ("AutoFDO profile magic number does not mathch."); + + /* Skip the version number. */ + gcov_read_unsigned (); + + /* Skip the empty integer. */ + gcov_read_unsigned (); + + /* string_table. */ + afdo_string_table = string_table::create (); + if (afdo_string_table == NULL) + error ("Cannot read string table from %s.", auto_profile_file); + + /* autofdo_source_profile. */ + afdo_source_profile = autofdo_source_profile::create (); + if (afdo_source_profile == NULL) + error ("Cannot read function profile from %s.", auto_profile_file); + + /* autofdo_module_profile. */ + fake_read_autofdo_module_profile (); + + /* Read in the working set. */ + if (gcov_read_unsigned () != GCOV_TAG_AFDO_WORKING_SET) + error ("Cannot read working set from %s.", auto_profile_file); + + /* Skip the length of the section. */ + gcov_read_unsigned (); + gcov_working_set_t set[128]; + for (unsigned i = 0; i < 128; i++) + { + set[i].num_counters = gcov_read_unsigned (); + set[i].min_counter = gcov_read_counter (); + } + add_working_set (set); +} + +/* From AutoFDO profiles, find values inside STMT for that we want to measure + histograms for indirect-call optimization. */ + +static void +afdo_indirect_call (gimple_stmt_iterator *gsi, const icall_target_map &map, + bool transform) +{ + gimple stmt = gsi_stmt (*gsi); + tree callee; + + if (map.size() == 0 || gimple_code (stmt) != GIMPLE_CALL + || gimple_call_fndecl (stmt) != NULL_TREE) + return; + + callee = gimple_call_fn (stmt); + + histogram_value hist = gimple_alloc_histogram_value ( + cfun, HIST_TYPE_INDIR_CALL, stmt, callee); + hist->n_counters = 3; + hist->hvalue.counters = XNEWVEC (gcov_type, hist->n_counters); + gimple_add_histogram_value (cfun, stmt, hist); + + gcov_type total = 0; + icall_target_map::const_iterator max_iter = map.end(); + + for (icall_target_map::const_iterator iter = map.begin(); + iter != map.end(); ++iter) + { + total += iter->second; + if (max_iter == map.end() || max_iter->second < iter->second) + max_iter = iter; + } + + hist->hvalue.counters[0] = (unsigned long long) + afdo_string_table->get_name (max_iter->first); + hist->hvalue.counters[1] = max_iter->second; + hist->hvalue.counters[2] = total; + + if (!transform) + return; + + if (gimple_ic_transform (gsi)) + { + struct cgraph_edge *indirect_edge = + cgraph_node::get (current_function_decl)->get_edge (stmt); + struct cgraph_node *direct_call = + find_func_by_profile_id ((int)hist->hvalue.counters[0]); + if (DECL_STRUCT_FUNCTION (direct_call->decl) == NULL) + return; + struct cgraph_edge *new_edge = + indirect_edge->make_speculative (direct_call, 0, 0); + new_edge->redirect_call_stmt_to_callee (); + gimple_remove_histogram_value (cfun, stmt, hist); + inline_call (new_edge, true, NULL, NULL, false); + return; + } + return; +} + +/* From AutoFDO profiles, find values inside STMT for that we want to measure + histograms and adds them to list VALUES. */ + +static void +afdo_vpt (gimple_stmt_iterator *gsi, const icall_target_map &map, + bool transform) +{ + afdo_indirect_call (gsi, map, transform); +} + +typedef std::set<basic_block> bb_set; +typedef std::set<edge> edge_set; + +static bool +is_bb_annotated (const basic_block bb, const bb_set &annotated) +{ + return annotated.find(bb) != annotated.end(); +} + +static void +set_bb_annotated (basic_block bb, bb_set *annotated) +{ + annotated->insert(bb); +} + +static bool +is_edge_annotated (const edge e, const edge_set &annotated) +{ + return annotated.find(e) != annotated.end(); +} + +static void +set_edge_annotated (edge e, edge_set *annotated) +{ + annotated->insert(e); +} + +/* For a given BB, set its execution count. Attach value profile if a stmt + is not in PROMOTED, because we only want to promot an indirect call once. + Return TRUE if BB is annotated. */ + +static bool +afdo_set_bb_count (basic_block bb, const stmt_set &promoted) +{ + gimple_stmt_iterator gsi; + edge e; + edge_iterator ei; + gcov_type max_count = 0; + bool has_annotated = false; + + for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi)) + { + count_info info; + gimple stmt = gsi_stmt (gsi); + if (gimple_clobber_p (stmt) || is_gimple_debug (stmt)) + continue; + if (afdo_source_profile->get_count_info (stmt, &info)) + { + if (info.count > max_count) + max_count = info.count; + has_annotated = true; + if (info.targets.size() > 0 && promoted.find (stmt) == promoted.end ()) + afdo_vpt (&gsi, info.targets, false); + } + } + + if (!has_annotated) + return false; + + for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi)) + afdo_source_profile->mark_annotated (gimple_location (gsi_stmt (gsi))); + for (gsi = gsi_start_phis (bb); !gsi_end_p (gsi); gsi_next (&gsi)) + { + gimple phi = gsi_stmt (gsi); + size_t i; + for (i = 0; i < gimple_phi_num_args (phi); i++) + afdo_source_profile->mark_annotated (gimple_phi_arg_location (phi, i)); + } + FOR_EACH_EDGE (e, ei, bb->succs) + afdo_source_profile->mark_annotated (e->goto_locus); + + bb->count = max_count; + return true; +} + +/* BB1 and BB2 are in an equivalent class iff: + 1. BB1 dominates BB2. + 2. BB2 post-dominates BB1. + 3. BB1 and BB2 are in the same loop nest. + This function finds the equivalent class for each basic block, and + stores a pointer to the first BB in its equivalent class. Meanwhile, + set bb counts for the same equivalent class to be idenical. Update + ANNOTATED_BB for the first BB in its equivalent class. */ + +static void +afdo_find_equiv_class (bb_set *annotated_bb) +{ + basic_block bb; + + FOR_ALL_BB_FN (bb, cfun) + bb->aux = NULL; + + FOR_ALL_BB_FN (bb, cfun) + { + vec<basic_block> dom_bbs; + basic_block bb1; + int i; + + if (bb->aux != NULL) + continue; + bb->aux = bb; + dom_bbs = get_dominated_by (CDI_DOMINATORS, bb); + FOR_EACH_VEC_ELT (dom_bbs, i, bb1) + if (bb1->aux == NULL + && dominated_by_p (CDI_POST_DOMINATORS, bb, bb1) + && bb1->loop_father == bb->loop_father) + { + bb1->aux = bb; + if (bb1->count > bb->count && is_bb_annotated (bb1, *annotated_bb)) + { + bb->count = MAX (bb->count, bb1->count); + set_bb_annotated (bb, annotated_bb); + } + } + dom_bbs = get_dominated_by (CDI_POST_DOMINATORS, bb); + FOR_EACH_VEC_ELT (dom_bbs, i, bb1) + if (bb1->aux == NULL + && dominated_by_p (CDI_DOMINATORS, bb, bb1) + && bb1->loop_father == bb->loop_father) + { + bb1->aux = bb; + if (bb1->count > bb->count && is_bb_annotated (bb1, *annotated_bb)) + { + bb->count = MAX (bb->count, bb1->count); + set_bb_annotated (bb, annotated_bb); + } + } + } +} + +/* If a basic block's count is known, and only one of its in/out edges' count + is unknown, its count can be calculated. Meanwhile, if all of the in/out + edges' counts are known, then the basic block's unknown count can also be + calculated. + IS_SUCC is true if out edges of a basic blocks are examined. + Update ANNOTATED_BB and ANNOTATED_EDGE accordingly. + Return TRUE if any basic block/edge count is changed. */ + +static bool +afdo_propagate_edge (bool is_succ, + bb_set *annotated_bb, edge_set *annotated_edge) +{ + basic_block bb; + bool changed = false; + + FOR_EACH_BB_FN (bb, cfun) + { + edge e, unknown_edge = NULL; + edge_iterator ei; + int num_unknown_edge = 0; + gcov_type total_known_count = 0; + + FOR_EACH_EDGE (e, ei, is_succ ? bb->succs : bb->preds) + if (!is_edge_annotated (e, *annotated_edge)) + num_unknown_edge ++, unknown_edge = e; + else + total_known_count += e->count; + + if (num_unknown_edge == 0) + { + if (total_known_count > bb->count) + { + bb->count = total_known_count; + changed = true; + } + if (!is_bb_annotated (bb, *annotated_bb)) + { + set_bb_annotated (bb, annotated_bb); + changed = true; + } + } + else if (num_unknown_edge == 1 + && is_bb_annotated (bb, *annotated_bb)) + { + if (bb->count >= total_known_count) + unknown_edge->count = bb->count - total_known_count; + else + unknown_edge->count = 0; + set_edge_annotated (unknown_edge, annotated_edge); + changed = true; + } + } + return changed; +} + +/* Special propagation for circuit expressions. Because GCC translates + control flow into data flow for circuit expressions. E.g. + BB1: + if (a && b) + BB2 + else + BB3 + + will be translated into: + + BB1: + if (a) + goto BB.t1 + else + goto BB.t3 + BB.t1: + if (b) + goto BB.t2 + else + goto BB.t3 + BB.t2: + goto BB.t3 + BB.t3: + tmp = PHI (0 (BB1), 0 (BB.t1), 1 (BB.t2) + if (tmp) + goto BB2 + else + goto BB3 + + In this case, we need to propagate through PHI to determine the edge + count of BB1->BB.t1, BB.t1->BB.t2. + Update ANNOTATED_EDGE accordingly. */ + +static void +afdo_propagate_circuit (const bb_set &annotated_bb, edge_set *annotated_edge) +{ + basic_block bb; + FOR_ALL_BB_FN (bb, cfun) + { + gimple phi_stmt; + tree cmp_rhs, cmp_lhs; + gimple cmp_stmt = last_stmt (bb); + edge e; + edge_iterator ei; + + if (!cmp_stmt || gimple_code (cmp_stmt) != GIMPLE_COND) + continue; + cmp_rhs = gimple_cond_rhs (cmp_stmt); + cmp_lhs = gimple_cond_lhs (cmp_stmt); + if (!TREE_CONSTANT (cmp_rhs) + || !(integer_zerop (cmp_rhs) || integer_onep (cmp_rhs))) + continue; + if (TREE_CODE (cmp_lhs) != SSA_NAME) + continue; + if (!is_bb_annotated (bb, annotated_bb)) + continue; + phi_stmt = SSA_NAME_DEF_STMT (cmp_lhs); + while (phi_stmt && gimple_code (phi_stmt) == GIMPLE_ASSIGN + && gimple_assign_single_p (phi_stmt) + && TREE_CODE (gimple_assign_rhs1 (phi_stmt)) == SSA_NAME) + phi_stmt = SSA_NAME_DEF_STMT (gimple_assign_rhs1 (phi_stmt)); + if (!phi_stmt || gimple_code (phi_stmt) != GIMPLE_PHI) + continue; + FOR_EACH_EDGE (e, ei, bb->succs) + { + unsigned i, total = 0; + edge only_one; + bool check_value_one = (((integer_onep (cmp_rhs)) + ^ (gimple_cond_code (cmp_stmt) == EQ_EXPR)) + ^ ((e->flags & EDGE_TRUE_VALUE) != 0)); + if (!is_edge_annotated (e, *annotated_edge)) + continue; + for (i = 0; i < gimple_phi_num_args (phi_stmt); i++) + { + tree val = gimple_phi_arg_def (phi_stmt, i); + edge ep = gimple_phi_arg_edge (phi_stmt, i); + + if (!TREE_CONSTANT (val) || !(integer_zerop (val) + || integer_onep (val))) + continue; + if (check_value_one ^ integer_onep (val)) + continue; + total++; + only_one = ep; + if (e->probability == 0 + && !is_edge_annotated (ep, *annotated_edge)) + { + ep->probability = 0; + ep->count = 0; + set_edge_annotated (ep, annotated_edge); + } + } + if (total == 1 && !is_edge_annotated (only_one, *annotated_edge)) + { + only_one->probability = e->probability; + only_one->count = e->count; + set_edge_annotated (only_one, annotated_edge); + } + } + } +} + +/* Propagate the basic block count and edge count on the control flow + graph. We do the propagation iteratively until stablize. */ + +static void +afdo_propagate (bb_set *annotated_bb, edge_set *annotated_edge) +{ + basic_block bb; + bool changed = true; + int i = 0; + + FOR_ALL_BB_FN (bb, cfun) + { + bb->count = ((basic_block) bb->aux)->count; + if (is_bb_annotated ((const basic_block) bb->aux, *annotated_bb)) + set_bb_annotated (bb, annotated_bb); + } + + while (changed && i++ < 10) + { + changed = false; + + if (afdo_propagate_edge (true, annotated_bb, annotated_edge)) + changed = true; + if (afdo_propagate_edge (false, annotated_bb, annotated_edge)) + changed = true; + afdo_propagate_circuit (*annotated_bb, annotated_edge); + } +} + +/* Propagate counts on control flow graph and calculate branch + probabilities. */ + +static void +afdo_calculate_branch_prob (bb_set *annotated_bb, edge_set *annotated_edge) +{ + basic_block bb; + bool has_sample = false; + + FOR_EACH_BB_FN (bb, cfun) + if (bb->count > 0) + has_sample = true; + + if (!has_sample) + return; + + calculate_dominance_info (CDI_POST_DOMINATORS); + calculate_dominance_info (CDI_DOMINATORS); + loop_optimizer_init (0); + + afdo_find_equiv_class (annotated_bb); + afdo_propagate (annotated_bb, annotated_edge); + + FOR_EACH_BB_FN (bb, cfun) + { + edge e; + edge_iterator ei; + int num_unknown_succ = 0; + gcov_type total_count = 0; + + FOR_EACH_EDGE (e, ei, bb->succs) + { + if (!is_edge_annotated (e, *annotated_edge)) + num_unknown_succ ++; + else + total_count += e->count; + } + if (num_unknown_succ == 0 && total_count > 0) + { + FOR_EACH_EDGE (e, ei, bb->succs) + e->probability = + (double) e->count * REG_BR_PROB_BASE / total_count; + } + } + FOR_ALL_BB_FN (bb, cfun) + { + edge e; + edge_iterator ei; + + FOR_EACH_EDGE (e, ei, bb->succs) + e->count = + (double) bb->count * e->probability / REG_BR_PROB_BASE; + bb->aux = NULL; + } + + loop_optimizer_finalize (); + free_dominance_info (CDI_DOMINATORS); + free_dominance_info (CDI_POST_DOMINATORS); +} + +/* Perform value profile transformation using AutoFDO profile. Add the + promoted stmts to PROMOTED_STMTS. Return TRUE if there is any + indirect call promoted. */ + +static bool +afdo_vpt_for_early_inline (stmt_set *promoted_stmts) +{ + basic_block bb; + if (afdo_source_profile->get_function_instance_by_decl ( + current_function_decl) == NULL) + return false; + + compute_inline_parameters (cgraph_node::get (current_function_decl), true); + + bool has_vpt = false; + FOR_EACH_BB_FN (bb, cfun) + { + if (!has_indirect_call (bb)) + continue; + gimple_stmt_iterator gsi; + + gcov_type bb_count = 0; + for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi)) + { + count_info info; + gimple stmt = gsi_stmt (gsi); + if (afdo_source_profile->get_count_info (stmt, &info)) + bb_count = MAX (bb_count, info.count); + } + + for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi)) + { + gimple stmt = gsi_stmt (gsi); + /* IC_promotion and early_inline_2 is done in multiple iterations. + No need to promoted the stmt if its in promoted_stmts (means + it is already been promoted in the previous iterations). */ + if (gimple_code (stmt) != GIMPLE_CALL + || gimple_call_fn (stmt) == NULL + || TREE_CODE (gimple_call_fn (stmt)) == FUNCTION_DECL + || promoted_stmts->find (stmt) != promoted_stmts->end ()) + continue; + + count_info info; + afdo_source_profile->get_count_info (stmt, &info); + info.count = bb_count; + if (afdo_source_profile->update_inlined_ind_target (stmt, &info)) + { + /* Promote the indirect call and update the promoted_stmts. */ + promoted_stmts->insert (stmt); + afdo_vpt (&gsi, info.targets, true); + has_vpt = true; + } + } + } + if (has_vpt) + { + optimize_inline_calls (current_function_decl); + return true; + } + else + return false; +} + +/* Annotate auto profile to the control flow graph. Do not annotate value + profile for stmts in PROMOTED_STMTS. */ + +static void +afdo_annotate_cfg (const stmt_set &promoted_stmts) +{ + basic_block bb; + bb_set annotated_bb; + edge_set annotated_edge; + const function_instance *s = + afdo_source_profile->get_function_instance_by_decl ( + current_function_decl); + + if (s == NULL) + return; + cgraph_node::get (current_function_decl)->count = s->head_count (); + ENTRY_BLOCK_PTR_FOR_FN (cfun)->count = s->head_count (); + gcov_type max_count = ENTRY_BLOCK_PTR_FOR_FN (cfun)->count; + + FOR_EACH_BB_FN (bb, cfun) + { + edge e; + edge_iterator ei; + + bb->count = 0; + FOR_EACH_EDGE (e, ei, bb->succs) + e->count = 0; + + if (afdo_set_bb_count (bb, promoted_stmts)) + set_bb_annotated (bb, &annotated_bb); + if (bb->count > max_count) + max_count = bb->count; + } + if (ENTRY_BLOCK_PTR_FOR_FN (cfun)->count > + ENTRY_BLOCK_PTR_FOR_FN (cfun)->next_bb->count) + { + ENTRY_BLOCK_PTR_FOR_FN (cfun)->next_bb->count = + ENTRY_BLOCK_PTR_FOR_FN (cfun)->count; + set_bb_annotated (ENTRY_BLOCK_PTR_FOR_FN (cfun)->next_bb, &annotated_bb); + } + if (ENTRY_BLOCK_PTR_FOR_FN (cfun)->count > + EXIT_BLOCK_PTR_FOR_FN (cfun)->prev_bb->count) + { + EXIT_BLOCK_PTR_FOR_FN (cfun)->prev_bb->count = + ENTRY_BLOCK_PTR_FOR_FN (cfun)->count; + set_bb_annotated (EXIT_BLOCK_PTR_FOR_FN (cfun)->prev_bb, &annotated_bb); + } + afdo_source_profile->mark_annotated ( + DECL_SOURCE_LOCATION (current_function_decl)); + afdo_source_profile->mark_annotated (cfun->function_start_locus); + afdo_source_profile->mark_annotated (cfun->function_end_locus); + if (max_count > 0) + { + afdo_calculate_branch_prob (&annotated_bb, &annotated_edge); + counts_to_freqs (); + profile_status_for_fn (cfun) = PROFILE_READ; + } + if (flag_value_profile_transformations) + gimple_value_profile_transformations (); +} + +/* Wrapper function to invoke early inliner. */ + +static void early_inline () +{ + compute_inline_parameters (cgraph_node::get (current_function_decl), true); + unsigned todo = early_inliner (cfun); + if (todo & TODO_update_ssa_any) + update_ssa (TODO_update_ssa); +} + +/* Use AutoFDO profile to annoate the control flow graph. + Return the todo flag. */ + +static unsigned int +auto_profile (void) +{ + struct cgraph_node *node; + + if (symtab->state == FINISHED) + return 0; + + init_node_map (true); + profile_info = autofdo::afdo_profile_info; + + FOR_EACH_FUNCTION (node) + { + if (!gimple_has_body_p (node->decl)) + continue; + + /* Don't profile functions produced for builtin stuff. */ + if (DECL_SOURCE_LOCATION (node->decl) == BUILTINS_LOCATION) + continue; + + push_cfun (DECL_STRUCT_FUNCTION (node->decl)); + + /* First do indirect call promotion and early inline to make the + IR match the profiled binary before actual annotation. + + This is needed because an indirect call might have been promoted + and inlined in the profiled binary. If we do not promote and + inline these indirect calls before annotation, the profile for + these promoted functions will be lost. + + e.g. foo() --indirect_call--> bar() + In profiled binary, the callsite is promoted and inlined, making + the profile look like: + + foo: { + loc_foo_1: count_1 + bar@loc_foo_2: { + loc_bar_1: count_2 + loc_bar_2: count_3 + } + } + + Before AutoFDO pass, loc_foo_2 is not promoted thus not inlined. + If we perform annotation on it, the profile inside bar@loc_foo2 + will be wasted. + + To avoid this, we promote loc_foo_2 and inline the promoted bar + function before annotation, so the profile inside bar@loc_foo2 + will be useful. */ + autofdo::stmt_set promoted_stmts; + for (int i = 0; i < PARAM_VALUE (PARAM_EARLY_INLINER_MAX_ITERATIONS); i++) + { + if (!flag_value_profile_transformations + || !autofdo::afdo_vpt_for_early_inline (&promoted_stmts)) + break; + early_inline (); + } + + early_inline (); + autofdo::afdo_annotate_cfg (promoted_stmts); + compute_function_frequency (); + update_ssa (TODO_update_ssa); + + /* Local pure-const may imply need to fixup the cfg. */ + if (execute_fixup_cfg () & TODO_cleanup_cfg) + cleanup_tree_cfg (); + + free_dominance_info (CDI_DOMINATORS); + free_dominance_info (CDI_POST_DOMINATORS); + cgraph_edge::rebuild_edges (); + pop_cfun (); + } + + return TODO_rebuild_cgraph_edges; +} +} /* namespace autofdo. */ + +/* Read the profile from the profile data file. */ + +void +read_autofdo_file (void) +{ + if (auto_profile_file == NULL) + auto_profile_file = DEFAULT_AUTO_PROFILE_FILE; + + autofdo::afdo_profile_info = (struct gcov_ctr_summary *) + xcalloc (1, sizeof (struct gcov_ctr_summary)); + autofdo::afdo_profile_info->runs = 1; + autofdo::afdo_profile_info->sum_max = 0; + autofdo::afdo_profile_info->sum_all = 0; + + /* Read the profile from the profile file. */ + autofdo::read_profile (); +} + +/* Free the resources. */ + +void +end_auto_profile (void) +{ + delete autofdo::afdo_source_profile; + delete autofdo::afdo_string_table; + profile_info = NULL; +} + +/* Returns TRUE if EDGE is hot enough to be inlined early. */ + +bool +afdo_callsite_hot_enough_for_early_inline (struct cgraph_edge *edge) +{ + gcov_type count = + autofdo::afdo_source_profile->get_callsite_total_count (edge); + if (count > 0) + { + bool is_hot; + const struct gcov_ctr_summary *saved_profile_info = profile_info; + /* At earling inline stage, profile_info is not set yet. We need to + temporarily set it to afdo_profile_info to calculate hotness. */ + profile_info = autofdo::afdo_profile_info; + is_hot = maybe_hot_count_p (NULL, count); + profile_info = saved_profile_info; + return is_hot; + } + else + return false; +} + +namespace { + +const pass_data pass_data_ipa_auto_profile = +{ + SIMPLE_IPA_PASS, + "afdo", /* name */ + OPTGROUP_NONE, /* optinfo_flags */ + TV_IPA_AUTOFDO, /* tv_id */ + 0, /* properties_required */ + 0, /* properties_provided */ + 0, /* properties_destroyed */ + 0, /* todo_flags_start */ + 0, /* todo_flags_finish */ +}; + +class pass_ipa_auto_profile : public simple_ipa_opt_pass +{ +public: + pass_ipa_auto_profile(gcc::context *ctxt) + : simple_ipa_opt_pass(pass_data_ipa_auto_profile, ctxt) + {} + + /* opt_pass methods: */ + virtual bool gate (function *) + { + return flag_auto_profile; + } + virtual unsigned int execute (function *) + { + return autofdo::auto_profile (); + } +}; // class pass_ipa_auto_profile + +} // anon namespace + +simple_ipa_opt_pass * +make_pass_ipa_auto_profile (gcc::context *ctxt) +{ + return new pass_ipa_auto_profile (ctxt); +} + Index: gcc/auto-profile.h =================================================================== --- gcc/auto-profile.h (revision 0) +++ gcc/auto-profile.h (revision 0) @@ -0,0 +1,31 @@ +/* auto-profile.h - Defines data exported from auto-profile.c + Copyright (C) 2014. Free Software Foundation, Inc. + Contributed by Dehao Chen (dehao@google.com) + +This file is part of GCC. + +GCC is free software; you can redistribute it and/or modify it under +the terms of the GNU General Public License as published by the Free +Software Foundation; either version 3, or (at your option) any later +version. + +GCC is distributed in the hope that it will be useful, but WITHOUT ANY +WARRANTY; without even the implied warranty of MERCHANTABILITY or +FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License +for more details. + +You should have received a copy of the GNU General Public License +along with GCC; see the file COPYING3. If not see +<http://www.gnu.org/licenses/>. */ + +#ifndef AUTO_PROFILE_H +#define AUTO_PROFILE_H + +/* Read, process, finalize AutoFDO data structures. */ +extern void read_autofdo_file (void); +extern void end_auto_profile (void); + +/* Returns TRUE if EDGE is hot enough to be inlined early. */ +extern bool afdo_callsite_hot_enough_for_early_inline (struct cgraph_edge *); + +#endif /* AUTO_PROFILE_H */ Index: gcc/cfgloop.c =================================================================== --- gcc/cfgloop.c (revision 215826) +++ gcc/cfgloop.c (working copy) @@ -1802,7 +1802,7 @@ } if (realistic && (!loop->any_estimate - || wi::ltu_p (i_bound, loop->nb_iterations_estimate))) + || (!flag_auto_profile && wi::ltu_p (i_bound, loop->nb_iterations_estimate)))) { loop->any_estimate = true; loop->nb_iterations_estimate = i_bound; Index: gcc/basic-block.h =================================================================== --- gcc/basic-block.h (revision 215826) +++ gcc/basic-block.h (working copy) @@ -724,6 +724,7 @@ extern void compute_available (sbitmap *, sbitmap *, sbitmap *, sbitmap *); /* In predict.c */ +extern bool maybe_hot_count_p (struct function *, gcov_type); extern bool maybe_hot_bb_p (struct function *, const_basic_block); extern bool maybe_hot_edge_p (edge); extern bool probably_never_executed_bb_p (struct function *, const_basic_block); Index: gcc/value-prof.c =================================================================== --- gcc/value-prof.c (revision 215826) +++ gcc/value-prof.c (working copy) @@ -134,11 +134,10 @@ static bool gimple_mod_pow2_value_transform (gimple_stmt_iterator *); static bool gimple_mod_subtract_transform (gimple_stmt_iterator *); static bool gimple_stringops_transform (gimple_stmt_iterator *); -static bool gimple_ic_transform (gimple_stmt_iterator *); /* Allocate histogram value. */ -static histogram_value +histogram_value gimple_alloc_histogram_value (struct function *fun ATTRIBUTE_UNUSED, enum hist_type type, gimple stmt, tree value) { @@ -1307,6 +1306,9 @@ struct cgraph_node* find_func_by_profile_id (int profile_id) { + if (flag_auto_profile) + return cgraph_node::get_for_asmname ( + get_identifier ((const char *)(size_t)profile_id)); cgraph_node **val = cgraph_node_map->get (profile_id); if (val) return *val; @@ -1320,7 +1322,7 @@ may ICE. Here we only do very minimal sanity check just to make compiler happy. Returns true if TARGET is considered ok for call CALL_STMT. */ -static bool +bool check_ic_target (gimple call_stmt, struct cgraph_node *target) { location_t locus; @@ -1483,7 +1485,7 @@ this call to: */ -static bool +bool gimple_ic_transform (gimple_stmt_iterator *gsi) { gimple stmt = gsi_stmt (*gsi); Index: gcc/value-prof.h =================================================================== --- gcc/value-prof.h (revision 215826) +++ gcc/value-prof.h (working copy) @@ -75,6 +75,8 @@ extern void gimple_find_values_to_profile (histogram_values *); extern bool gimple_value_profile_transformations (void); +histogram_value gimple_alloc_histogram_value (struct function *, enum hist_type, + gimple stmt, tree); histogram_value gimple_histogram_value (struct function *, gimple); histogram_value gimple_histogram_value_of_type (struct function *, gimple, enum hist_type); @@ -89,6 +91,8 @@ void free_histograms (void); void stringop_block_profile (gimple, unsigned int *, HOST_WIDE_INT *); gimple gimple_ic (gimple, struct cgraph_node *, int, gcov_type, gcov_type); +bool check_ic_target (gimple, struct cgraph_node *); +bool gimple_ic_transform (gimple_stmt_iterator *); /* In tree-profile.c. */ Index: gcc/coverage.c =================================================================== --- gcc/coverage.c (revision 215826) +++ gcc/coverage.c (working copy) @@ -55,6 +55,7 @@ #include "filenames.h" #include "target.h" #include "params.h" +#include "auto-profile.h" #include "gcov-io.h" #include "gcov-io.c" @@ -1208,7 +1209,9 @@ bbg_file_stamp = local_tick; - if (flag_branch_probabilities) + if (flag_auto_profile) + read_autofdo_file (); + else if (flag_branch_probabilities) read_counts_file (); /* Name of bbg file. */ Index: gcc/timevar.def =================================================================== --- gcc/timevar.def (revision 215826) +++ gcc/timevar.def (working copy) @@ -89,6 +89,7 @@ DEFTIMEVAR (TV_WHOPR_LTRANS , "whopr ltrans") DEFTIMEVAR (TV_IPA_REFERENCE , "ipa reference") DEFTIMEVAR (TV_IPA_PROFILE , "ipa profile") +DEFTIMEVAR (TV_IPA_AUTOFDO , "auto profile") DEFTIMEVAR (TV_IPA_PURE_CONST , "ipa pure const") DEFTIMEVAR (TV_IPA_PTA , "ipa points-to") DEFTIMEVAR (TV_IPA_SRA , "ipa SRA") Index: gcc/common.opt =================================================================== --- gcc/common.opt (revision 215826) +++ gcc/common.opt (working copy) @@ -895,6 +895,16 @@ Common Report Var(flag_auto_inc_dec) Init(1) Generate auto-inc/dec instructions +fauto-profile +Common Report Var(flag_auto_profile) Optimization +Use sample profile information for call graph node weights. The default +profile file is fbdata.afdo in 'pwd'. + +fauto-profile= +Common Joined RejectNegative Var(auto_profile_file) +Use sample profile information for call graph node weights. The profile +file is specified in the argument. + ; -fcheck-bounds causes gcc to generate array bounds checks. ; For C, C++ and ObjC: defaults off. ; For Java: defaults to on. Index: gcc/toplev.c =================================================================== --- gcc/toplev.c (revision 215826) +++ gcc/toplev.c (working copy) @@ -79,6 +79,7 @@ #include "diagnostic-color.h" #include "context.h" #include "pass_manager.h" +#include "auto-profile.h" #include "optabs.h" #if defined(DBX_DEBUGGING_INFO) || defined(XCOFF_DEBUGGING_INFO) @@ -663,6 +664,10 @@ targetm.asm_out.output_ident (ident_str); } + /* Auto profile finalization. */ + if (flag_auto_profile) + end_auto_profile (); + /* Invoke registered plugin callbacks. */ invoke_plugin_callbacks (PLUGIN_FINISH_UNIT, NULL); Index: gcc/tree-profile.c =================================================================== --- gcc/tree-profile.c (revision 215826) +++ gcc/tree-profile.c (working copy) @@ -699,8 +699,10 @@ bool pass_ipa_tree_profile::gate (function *) { - /* When profile instrumentation, use or test coverage shall be performed. */ - return (!in_lto_p + /* When profile instrumentation, use or test coverage shall be performed. + But for AutoFDO, this there is no instrumentation, thus this pass is + diabled. */ + return (!in_lto_p && !flag_auto_profile && (flag_branch_probabilities || flag_test_coverage || profile_arc_flag)); } Index: gcc/ipa-inline.c =================================================================== --- gcc/ipa-inline.c (revision 215826) +++ gcc/ipa-inline.c (working copy) @@ -121,6 +121,7 @@ #include "ipa-inline.h" #include "ipa-utils.h" #include "sreal.h" +#include "auto-profile.h" #include "cilk.h" #include "builtins.h" @@ -449,6 +450,14 @@ if (DECL_DISREGARD_INLINE_LIMITS (callee->decl)) ; + /* For AutoFDO, we need to make sure that before profile annotation, all + hot paths' IR look exactly the same as profiled binary. As a result, + in einliner, we will disregard size limit and inline those callsites + that are: + * inlined in the profiled binary, and + * the cloned callee has enough samples to be considered "hot". */ + else if (flag_auto_profile && afdo_callsite_hot_enough_for_early_inline (e)) + ; else if (!DECL_DECLARED_INLINE_P (callee->decl) && !flag_inline_small_functions) { @@ -2366,39 +2375,8 @@ return inlined; } -/* Do inlining of small functions. Doing so early helps profiling and other - passes to be somewhat more effective and avoids some code duplication in - later real inlining pass for testcases with very many function calls. */ - -namespace { - -const pass_data pass_data_early_inline = -{ - GIMPLE_PASS, /* type */ - "einline", /* name */ - OPTGROUP_INLINE, /* optinfo_flags */ - TV_EARLY_INLINING, /* tv_id */ - PROP_ssa, /* properties_required */ - 0, /* properties_provided */ - 0, /* properties_destroyed */ - 0, /* todo_flags_start */ - 0, /* todo_flags_finish */ -}; - -class pass_early_inline : public gimple_opt_pass -{ -public: - pass_early_inline (gcc::context *ctxt) - : gimple_opt_pass (pass_data_early_inline, ctxt) - {} - - /* opt_pass methods: */ - virtual unsigned int execute (function *); - -}; // class pass_early_inline - unsigned int -pass_early_inline::execute (function *fun) +early_inliner (function *fun) { struct cgraph_node *node = cgraph_node::get (current_function_decl); struct cgraph_edge *edge; @@ -2499,6 +2477,43 @@ return todo; } +/* Do inlining of small functions. Doing so early helps profiling and other + passes to be somewhat more effective and avoids some code duplication in + later real inlining pass for testcases with very many function calls. */ + +namespace { + +const pass_data pass_data_early_inline = +{ + GIMPLE_PASS, /* type */ + "einline", /* name */ + OPTGROUP_INLINE, /* optinfo_flags */ + TV_EARLY_INLINING, /* tv_id */ + PROP_ssa, /* properties_required */ + 0, /* properties_provided */ + 0, /* properties_destroyed */ + 0, /* todo_flags_start */ + 0, /* todo_flags_finish */ +}; + +class pass_early_inline : public gimple_opt_pass +{ +public: + pass_early_inline (gcc::context *ctxt) + : gimple_opt_pass (pass_data_early_inline, ctxt) + {} + + /* opt_pass methods: */ + virtual unsigned int execute (function *); + +}; // class pass_early_inline + +unsigned int +pass_early_inline::execute (function *fun) +{ + return early_inliner (fun); +} + } // anon namespace gimple_opt_pass * Index: gcc/ipa-inline.h =================================================================== --- gcc/ipa-inline.h (revision 215826) +++ gcc/ipa-inline.h (working copy) @@ -238,6 +238,7 @@ void free_growth_caches (void); void compute_inline_parameters (struct cgraph_node *, bool); bool speculation_useful_p (struct cgraph_edge *e, bool anticipate_inlining); +unsigned int early_inliner (function *fun); /* In ipa-inline-transform.c */ bool inline_call (struct cgraph_edge *, bool, vec<cgraph_edge *> *, int *, bool, Index: gcc/tree-ssa-live.c =================================================================== --- gcc/tree-ssa-live.c (revision 215826) +++ gcc/tree-ssa-live.c (working copy) @@ -605,7 +605,7 @@ ; /* When not generating debug info we can eliminate info on unused variables. */ - else if (debug_info_level == DINFO_LEVEL_NONE) + else if (!flag_auto_profile && debug_info_level == DINFO_LEVEL_NONE) { /* Even for -g0 don't prune outer scopes from artificial functions, otherwise diagnostics using tree_nonartificial_location Index: gcc/gcov-io.h =================================================================== --- gcc/gcov-io.h (revision 215826) +++ gcc/gcov-io.h (working copy) @@ -244,6 +244,9 @@ #define GCOV_TAG_PROGRAM_SUMMARY ((gcov_unsigned_t)0xa3000000) #define GCOV_TAG_SUMMARY_LENGTH(NUM) \ (1 + GCOV_COUNTERS_SUMMABLE * (10 + 3 * 2) + (NUM) * 5) +#define GCOV_TAG_AFDO_FILE_NAMES ((gcov_unsigned_t)0xaa000000) +#define GCOV_TAG_AFDO_FUNCTION ((gcov_unsigned_t)0xac000000) +#define GCOV_TAG_AFDO_WORKING_SET ((gcov_unsigned_t)0xaf000000) /* Counters that are collected. */ Index: gcc/Makefile.in =================================================================== --- gcc/Makefile.in (revision 215826) +++ gcc/Makefile.in (working copy) @@ -1153,6 +1153,7 @@ alias.o \ alloc-pool.o \ auto-inc-dec.o \ + auto-profile.o \ bb-reorder.o \ bitmap.o \ bt-load.o \ Index: gcc/predict.c =================================================================== --- gcc/predict.c (revision 215826) +++ gcc/predict.c (working copy) @@ -161,7 +161,7 @@ /* Return TRUE if frequency FREQ is considered to be hot. */ -static inline bool +bool maybe_hot_count_p (struct function *fun, gcov_type count) { if (fun && profile_status_for_fn (fun) != PROFILE_READ) @@ -2852,7 +2852,7 @@ /* Don't overwrite the estimated frequencies when the profile for the function is missing. We may drop this function PROFILE_GUESSED later in drop_profile (). */ - if (!ENTRY_BLOCK_PTR_FOR_FN (cfun)->count) + if (!flag_auto_profile && !ENTRY_BLOCK_PTR_FOR_FN (cfun)->count) return 0; FOR_BB_BETWEEN (bb, ENTRY_BLOCK_PTR_FOR_FN (cfun), NULL, next_bb) @@ -3223,7 +3223,8 @@ count_max = MAX (bb->count, count_max); if (profile_status_for_fn (cfun) == PROFILE_GUESSED - || (profile_status_for_fn (cfun) == PROFILE_READ && count_max < REG_BR_PROB_BASE/10)) + || (!flag_auto_profile && profile_status_for_fn (cfun) == PROFILE_READ + && count_max < REG_BR_PROB_BASE/10)) { loop_optimizer_init (0); add_noreturn_fake_exit_edges (); Index: gcc/doc/invoke.texi =================================================================== --- gcc/doc/invoke.texi (revision 215826) +++ gcc/doc/invoke.texi (working copy) @@ -365,7 +365,8 @@ @gccoptlist{-faggressive-loop-optimizations -falign-functions[=@var{n}] @gol -falign-jumps[=@var{n}] @gol -falign-labels[=@var{n}] -falign-loops[=@var{n}] @gol --fassociative-math -fauto-inc-dec -fbranch-probabilities @gol +-fassociative-math -fauto-profile -fauto-profile[=@var{path}] @gol +-fauto-inc-dec -fbranch-probabilities @gol -fbranch-target-load-optimize -fbranch-target-load-optimize2 @gol -fbtr-bb-exclusive -fcaller-saves @gol -fcheck-data-deps -fcombine-stack-adjustments -fconserve-stack @gol @@ -9164,6 +9165,19 @@ If @var{path} is specified, GCC looks at the @var{path} to find the profile feedback data files. See @option{-fprofile-dir}. + +@item -fauto-profile +@itemx -fauto-profile=@var{path} +@opindex fauto-profile +Enable sampling based feedback directed optimizations, and optimizations +generally profitable only with profile feedback available. + +The following options are enabled: @code{-fbranch-probabilities}, @code{-fvpt}, +@code{-funroll-loops}, @code{-fpeel-loops}, @code{-ftracer}, @code{-ftree-vectorize}, +@code{ftree-loop-distribute-patterns} + +If @var{path} is specified, GCC looks at the @var{path} to find +the profile feedback data files. @end table The following options control compiler behavior regarding floating-point Index: gcc/profile.c =================================================================== --- gcc/profile.c (revision 215826) +++ gcc/profile.c (working copy) @@ -106,6 +106,14 @@ static int total_hist_br_prob[20]; static int total_num_branches; +/* Helper function to update gcov_working_sets. */ + +void add_working_set (gcov_working_set_t *set) { + int i = 0; + for (; i < NUM_GCOV_WORKING_SETS; i++) + gcov_working_sets[i] = set[i]; +} + /* Forward declarations. */ static void find_spanning_tree (struct edge_list *); Index: gcc/profile.h =================================================================== --- gcc/profile.h (revision 215826) +++ gcc/profile.h (working copy) @@ -47,6 +47,7 @@ extern void del_node_map (void); extern void get_working_sets (void); +extern void add_working_set (gcov_working_set_t *); /* In predict.c. */ extern gcov_type get_hot_bb_threshold (void); Index: gcc/passes.def =================================================================== --- gcc/passes.def (revision 215826) +++ gcc/passes.def (working copy) @@ -90,6 +90,7 @@ NEXT_PASS (pass_rebuild_cgraph_edges); NEXT_PASS (pass_inline_parameters); POP_INSERT_PASSES () + NEXT_PASS (pass_ipa_auto_profile); NEXT_PASS (pass_ipa_free_inline_summary); NEXT_PASS (pass_ipa_tree_profile); PUSH_INSERT_PASSES_WITHIN (pass_ipa_tree_profile) Index: gcc/tree-pass.h =================================================================== --- gcc/tree-pass.h (revision 215826) +++ gcc/tree-pass.h (working copy) @@ -449,6 +449,7 @@ extern simple_ipa_opt_pass *make_pass_ipa_function_and_variable_visibility (gcc::context *ctxt); extern simple_ipa_opt_pass *make_pass_ipa_tree_profile (gcc::context *ctxt); +extern simple_ipa_opt_pass *make_pass_ipa_auto_profile (gcc::context *ctxt); extern simple_ipa_opt_pass *make_pass_early_local_passes (gcc::context *ctxt); Index: gcc/opts.c =================================================================== --- gcc/opts.c (revision 215826) +++ gcc/opts.c (working copy) @@ -1279,6 +1279,55 @@ opts->x_help_columns, opts, lang_mask); } +static void +enable_fdo_optimizations (struct gcc_options *opts, + struct gcc_options *opts_set, + int value) +{ + if (!opts_set->x_flag_branch_probabilities) + opts->x_flag_branch_probabilities = value; + if (!opts_set->x_flag_profile_values) + opts->x_flag_profile_values = value; + if (!opts_set->x_flag_unroll_loops) + opts->x_flag_unroll_loops = value; + if (!opts_set->x_flag_peel_loops) + opts->x_flag_peel_loops = value; + if (!opts_set->x_flag_tracer) + opts->x_flag_tracer = value; + if (!opts_set->x_flag_value_profile_transformations) + opts->x_flag_value_profile_transformations = value; + if (!opts_set->x_flag_inline_functions) + opts->x_flag_inline_functions = value; + if (!opts_set->x_flag_ipa_cp) + opts->x_flag_ipa_cp = value; + if (!opts_set->x_flag_ipa_cp_clone + && value && opts->x_flag_ipa_cp) + opts->x_flag_ipa_cp_clone = value; + if (!opts_set->x_flag_predictive_commoning) + opts->x_flag_predictive_commoning = value; + if (!opts_set->x_flag_unswitch_loops) + opts->x_flag_unswitch_loops = value; + if (!opts_set->x_flag_gcse_after_reload) + opts->x_flag_gcse_after_reload = value; + if (!opts_set->x_flag_tree_loop_vectorize + && !opts_set->x_flag_tree_vectorize) + opts->x_flag_tree_loop_vectorize = value; + if (!opts_set->x_flag_tree_slp_vectorize + && !opts_set->x_flag_tree_vectorize) + opts->x_flag_tree_slp_vectorize = value; + if (!opts_set->x_flag_vect_cost_model) + opts->x_flag_vect_cost_model = VECT_COST_MODEL_DYNAMIC; + if (!opts_set->x_flag_tree_loop_distribute_patterns) + opts->x_flag_tree_loop_distribute_patterns = value; + if (!opts_set->x_flag_profile_reorder_functions) + opts->x_flag_profile_reorder_functions = value; + /* Indirect call profiling should do all useful transformations + speculative devirtualization does. */ + if (!opts_set->x_flag_devirtualize_speculatively + && opts->x_flag_value_profile_transformations) + opts->x_flag_devirtualize_speculatively = false; +} + /* Handle target- and language-independent options. Return zero to generate an "unknown option" message. Only options that need extra handling need to be listed here; if you simply want @@ -1746,50 +1795,23 @@ value = true; /* No break here - do -fprofile-use processing. */ case OPT_fprofile_use: - if (!opts_set->x_flag_branch_probabilities) - opts->x_flag_branch_probabilities = value; - if (!opts_set->x_flag_profile_values) - opts->x_flag_profile_values = value; - if (!opts_set->x_flag_unroll_loops) - opts->x_flag_unroll_loops = value; - if (!opts_set->x_flag_peel_loops) - opts->x_flag_peel_loops = value; - if (!opts_set->x_flag_tracer) - opts->x_flag_tracer = value; - if (!opts_set->x_flag_value_profile_transformations) - opts->x_flag_value_profile_transformations = value; - if (!opts_set->x_flag_inline_functions) - opts->x_flag_inline_functions = value; - if (!opts_set->x_flag_ipa_cp) - opts->x_flag_ipa_cp = value; - if (!opts_set->x_flag_ipa_cp_clone - && value && opts->x_flag_ipa_cp) - opts->x_flag_ipa_cp_clone = value; - if (!opts_set->x_flag_predictive_commoning) - opts->x_flag_predictive_commoning = value; - if (!opts_set->x_flag_unswitch_loops) - opts->x_flag_unswitch_loops = value; - if (!opts_set->x_flag_gcse_after_reload) - opts->x_flag_gcse_after_reload = value; - if (!opts_set->x_flag_tree_loop_vectorize - && !opts_set->x_flag_tree_vectorize) - opts->x_flag_tree_loop_vectorize = value; - if (!opts_set->x_flag_tree_slp_vectorize - && !opts_set->x_flag_tree_vectorize) - opts->x_flag_tree_slp_vectorize = value; - if (!opts_set->x_flag_vect_cost_model) - opts->x_flag_vect_cost_model = VECT_COST_MODEL_DYNAMIC; - if (!opts_set->x_flag_tree_loop_distribute_patterns) - opts->x_flag_tree_loop_distribute_patterns = value; - if (!opts_set->x_flag_profile_reorder_functions) - opts->x_flag_profile_reorder_functions = value; - /* Indirect call profiling should do all useful transformations - speculative devirtualization does. */ - if (!opts_set->x_flag_devirtualize_speculatively - && opts->x_flag_value_profile_transformations) - opts->x_flag_devirtualize_speculatively = false; + enable_fdo_optimizations (opts, opts_set, value); break; + case OPT_fauto_profile_: + opts->x_auto_profile_file = xstrdup (arg); + opts->x_flag_auto_profile = true; + value = true; + /* No break here - do -fauto-profile processing. */ + case OPT_fauto_profile: + enable_fdo_optimizations (opts, opts_set, value); + if (!opts_set->x_flag_profile_correction) + opts->x_flag_profile_correction = value; + maybe_set_param_value ( + PARAM_EARLY_INLINER_MAX_ITERATIONS, 10, + opts->x_param_values, opts_set->x_param_values); + break; + case OPT_fprofile_generate_: opts->x_profile_data_prefix = xstrdup (arg); value = true; ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH] AutoFDO patch for trunk 2014-10-14 21:10 ` Dehao Chen @ 2014-10-14 23:28 ` Dehao Chen 2014-10-15 2:57 ` Andi Kleen 2014-10-15 14:36 ` Jan Hubicka 0 siblings, 2 replies; 25+ messages in thread From: Dehao Chen @ 2014-10-14 23:28 UTC (permalink / raw) To: Jan Hubicka; +Cc: GCC Patches, David Li [-- Attachment #1: Type: text/plain, Size: 5684 bytes --] The new patch is attached. I used clang-format for format auto-profile.{c|h} Thanks, Dehao On Tue, Oct 14, 2014 at 2:05 PM, Dehao Chen <dehao@google.com> wrote: > On Tue, Oct 14, 2014 at 8:02 AM, Jan Hubicka <hubicka@ucw.cz> wrote: >>> Index: gcc/cgraphclones.c >>> =================================================================== >>> --- gcc/cgraphclones.c (revision 215826) >>> +++ gcc/cgraphclones.c (working copy) >>> @@ -453,6 +453,11 @@ >>> } >>> else >>> count_scale = 0; >>> + /* In AutoFDO, if edge count is larger than callee's entry block >>> + count, we will not update the original callee because it may >>> + mistakenly mark some hot function as cold. */ >>> + if (flag_auto_profile && gcov_count >= count) >>> + update_original = false; >> >> lets drop this from initial patch. > > Done >> >>> Index: gcc/bb-reorder.c >>> =================================================================== >>> --- gcc/bb-reorder.c (revision 215826) >>> +++ gcc/bb-reorder.c (working copy) >>> @@ -1569,15 +1569,14 @@ >>> /* Mark which partition (hot/cold) each basic block belongs in. */ >>> FOR_EACH_BB_FN (bb, cfun) >>> { >>> - bool cold_bb = false; >>> + bool cold_bb = probably_never_executed_bb_p (cfun, bb); >> >> and this too >> (basically all the tweaks should IMO go in independently and ideally in >> a way that does not need flag_auto_profile test). > > Done. > >>> +/* Return true if BB contains indirect call. */ >>> + >>> +static bool >>> +has_indirect_call (basic_block bb) >>> +{ >>> + gimple_stmt_iterator gsi; >>> + >>> + for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi)) >>> + { >>> + gimple stmt = gsi_stmt (gsi); >>> + if (gimple_code (stmt) == GIMPLE_CALL >>> + && (gimple_call_fn (stmt) == NULL >>> + || TREE_CODE (gimple_call_fn (stmt)) != FUNCTION_DECL)) >> >> You probably want to skip gimple_call_internal_p calls here. > > Done > >>> + >>> +/* From AutoFDO profiles, find values inside STMT for that we want to measure >>> + histograms for indirect-call optimization. */ >>> + >>> +static void >>> +afdo_indirect_call (gimple_stmt_iterator *gsi, const icall_target_map &map, >>> + bool transform) >>> +{ >>> + gimple stmt = gsi_stmt (*gsi); >>> + tree callee; >>> + >>> + if (map.size() == 0 || gimple_code (stmt) != GIMPLE_CALL >>> + || gimple_call_fndecl (stmt) != NULL_TREE) >>> + return; >>> + >>> + callee = gimple_call_fn (stmt); >>> + >>> + histogram_value hist = gimple_alloc_histogram_value ( >>> + cfun, HIST_TYPE_INDIR_CALL, stmt, callee); >>> + hist->n_counters = 3; >>> + hist->hvalue.counters = XNEWVEC (gcov_type, hist->n_counters); >>> + gimple_add_histogram_value (cfun, stmt, hist); >>> + >>> + gcov_type total = 0; >>> + icall_target_map::const_iterator max_iter = map.end(); >>> + >>> + for (icall_target_map::const_iterator iter = map.begin(); >>> + iter != map.end(); ++iter) >>> + { >>> + total += iter->second; >>> + if (max_iter == map.end() || max_iter->second < iter->second) >>> + max_iter = iter; >>> + } >>> + >>> + hist->hvalue.counters[0] = (unsigned long long) >>> + afdo_string_table->get_name (max_iter->first); >>> + hist->hvalue.counters[1] = max_iter->second; >>> + hist->hvalue.counters[2] = total; >>> + >>> + if (!transform) >>> + return; >>> + >>> + if (gimple_ic_transform (gsi)) >>> + { >>> + struct cgraph_edge *indirect_edge = >>> + cgraph_node::get (current_function_decl)->get_edge (stmt); >>> + struct cgraph_node *direct_call = >>> + find_func_by_profile_id ((int)hist->hvalue.counters[0]); >>> + if (DECL_STRUCT_FUNCTION (direct_call->decl) == NULL) >>> + return; >>> + struct cgraph_edge *new_edge = >>> + indirect_edge->make_speculative (direct_call, 0, 0); >>> + new_edge->redirect_call_stmt_to_callee (); >>> + gimple_remove_histogram_value (cfun, stmt, hist); >>> + inline_call (new_edge, true, NULL, NULL, false); >>> + return; >>> + } >>> + return; >> >> Is it necessary to go via histogram and gimple_ic_transform here? I would expect that all you >> need is to make the speculative edge and inline it. (bypassing the work of producing fake >> histogram value and calling igmple_ic_transofrm on it) >> >> Also it seems to me that you want to set direct_count nad frequency argument of >> make_speculative so the resulting function profile is not off. > > This function is actually served for 2 purposes: > > * before annotation, we need to mark histogram, promote and inline > * after annotation, we just need to mark, and let follow-up logic to > decide if it needs to promote and inline. > > And you are right, for the "before annotation" case, we can simply > call "mark speculative" and "inline". But we still need the logic to > fake histogram for "after annotation" case. As a result, I unified two > cases into one function to reuse code as much as possible. Shall I > separate it into two functions instead? > >> >> The rest of interfaces seems quite sane now. Can you please look into >> using speculative edges directly instead of hooking into the vpt infrastructure >> and fixing the formatting issues of the new pass? > > I'll work on the formatting issues now (need to learn the format first > ;-). The attached patch is up-to-date except for formatting changes. > I'll upload the patch again once the format change is in. > > Thanks, > Dehao > >> >> I will try to make another pass over the actual streaming logic that I find bit difficult >> to read, but I quite trust you it does the right thing ;) >> >> Honza [-- Attachment #2: patch_formatted.txt --] [-- Type: text/plain, Size: 75708 bytes --] Index: gcc/tree-ssa-live.c =================================================================== --- gcc/tree-ssa-live.c (revision 215826) +++ gcc/tree-ssa-live.c (working copy) @@ -605,7 +605,7 @@ remove_unused_scope_block_p (tree scope) ; /* When not generating debug info we can eliminate info on unused variables. */ - else if (debug_info_level == DINFO_LEVEL_NONE) + else if (!flag_auto_profile && debug_info_level == DINFO_LEVEL_NONE) { /* Even for -g0 don't prune outer scopes from artificial functions, otherwise diagnostics using tree_nonartificial_location Index: gcc/cfgloop.c =================================================================== --- gcc/cfgloop.c (revision 215826) +++ gcc/cfgloop.c (working copy) @@ -1802,7 +1802,7 @@ record_niter_bound (struct loop *loop, const wides } if (realistic && (!loop->any_estimate - || wi::ltu_p (i_bound, loop->nb_iterations_estimate))) + || (!flag_auto_profile && wi::ltu_p (i_bound, loop->nb_iterations_estimate)))) { loop->any_estimate = true; loop->nb_iterations_estimate = i_bound; Index: gcc/tree-pass.h =================================================================== --- gcc/tree-pass.h (revision 215826) +++ gcc/tree-pass.h (working copy) @@ -449,6 +449,7 @@ extern simple_ipa_opt_pass *make_pass_ipa_lower_em extern simple_ipa_opt_pass *make_pass_ipa_function_and_variable_visibility (gcc::context *ctxt); extern simple_ipa_opt_pass *make_pass_ipa_tree_profile (gcc::context *ctxt); +extern simple_ipa_opt_pass *make_pass_ipa_auto_profile (gcc::context *ctxt); extern simple_ipa_opt_pass *make_pass_early_local_passes (gcc::context *ctxt); Index: gcc/toplev.c =================================================================== --- gcc/toplev.c (revision 215826) +++ gcc/toplev.c (working copy) @@ -79,6 +79,7 @@ along with GCC; see the file COPYING3. If not see #include "diagnostic-color.h" #include "context.h" #include "pass_manager.h" +#include "auto-profile.h" #include "optabs.h" #if defined(DBX_DEBUGGING_INFO) || defined(XCOFF_DEBUGGING_INFO) @@ -663,6 +664,10 @@ compile_file (void) targetm.asm_out.output_ident (ident_str); } + /* Auto profile finalization. */ + if (flag_auto_profile) + end_auto_profile (); + /* Invoke registered plugin callbacks. */ invoke_plugin_callbacks (PLUGIN_FINISH_UNIT, NULL); Index: gcc/doc/invoke.texi =================================================================== --- gcc/doc/invoke.texi (revision 215826) +++ gcc/doc/invoke.texi (working copy) @@ -365,7 +365,8 @@ Objective-C and Objective-C++ Dialects}. @gccoptlist{-faggressive-loop-optimizations -falign-functions[=@var{n}] @gol -falign-jumps[=@var{n}] @gol -falign-labels[=@var{n}] -falign-loops[=@var{n}] @gol --fassociative-math -fauto-inc-dec -fbranch-probabilities @gol +-fassociative-math -fauto-profile -fauto-profile[=@var{path}] @gol +-fauto-inc-dec -fbranch-probabilities @gol -fbranch-target-load-optimize -fbranch-target-load-optimize2 @gol -fbtr-bb-exclusive -fcaller-saves @gol -fcheck-data-deps -fcombine-stack-adjustments -fconserve-stack @gol @@ -9164,6 +9165,19 @@ code. If @var{path} is specified, GCC looks at the @var{path} to find the profile feedback data files. See @option{-fprofile-dir}. + +@item -fauto-profile +@itemx -fauto-profile=@var{path} +@opindex fauto-profile +Enable sampling based feedback directed optimizations, and optimizations +generally profitable only with profile feedback available. + +The following options are enabled: @code{-fbranch-probabilities}, @code{-fvpt}, +@code{-funroll-loops}, @code{-fpeel-loops}, @code{-ftracer}, @code{-ftree-vectorize}, +@code{ftree-loop-distribute-patterns} + +If @var{path} is specified, GCC looks at the @var{path} to find +the profile feedback data files. @end table The following options control compiler behavior regarding floating-point Index: gcc/coverage.c =================================================================== --- gcc/coverage.c (revision 215826) +++ gcc/coverage.c (working copy) @@ -55,6 +55,7 @@ along with GCC; see the file COPYING3. If not see #include "filenames.h" #include "target.h" #include "params.h" +#include "auto-profile.h" #include "gcov-io.h" #include "gcov-io.c" @@ -1208,7 +1209,9 @@ coverage_init (const char *filename) bbg_file_stamp = local_tick; - if (flag_branch_probabilities) + if (flag_auto_profile) + read_autofdo_file (); + else if (flag_branch_probabilities) read_counts_file (); /* Name of bbg file. */ Index: gcc/common.opt =================================================================== --- gcc/common.opt (revision 215826) +++ gcc/common.opt (working copy) @@ -895,6 +895,16 @@ fauto-inc-dec Common Report Var(flag_auto_inc_dec) Init(1) Generate auto-inc/dec instructions +fauto-profile +Common Report Var(flag_auto_profile) Optimization +Use sample profile information for call graph node weights. The default +profile file is fbdata.afdo in 'pwd'. + +fauto-profile= +Common Joined RejectNegative Var(auto_profile_file) +Use sample profile information for call graph node weights. The profile +file is specified in the argument. + ; -fcheck-bounds causes gcc to generate array bounds checks. ; For C, C++ and ObjC: defaults off. ; For Java: defaults to on. Index: gcc/timevar.def =================================================================== --- gcc/timevar.def (revision 215826) +++ gcc/timevar.def (working copy) @@ -89,6 +89,7 @@ DEFTIMEVAR (TV_WHOPR_PARTITIONING , "whopr part DEFTIMEVAR (TV_WHOPR_LTRANS , "whopr ltrans") DEFTIMEVAR (TV_IPA_REFERENCE , "ipa reference") DEFTIMEVAR (TV_IPA_PROFILE , "ipa profile") +DEFTIMEVAR (TV_IPA_AUTOFDO , "auto profile") DEFTIMEVAR (TV_IPA_PURE_CONST , "ipa pure const") DEFTIMEVAR (TV_IPA_PTA , "ipa points-to") DEFTIMEVAR (TV_IPA_SRA , "ipa SRA") Index: gcc/value-prof.c =================================================================== --- gcc/value-prof.c (revision 215826) +++ gcc/value-prof.c (working copy) @@ -134,11 +134,10 @@ static bool gimple_divmod_fixed_value_transform (g static bool gimple_mod_pow2_value_transform (gimple_stmt_iterator *); static bool gimple_mod_subtract_transform (gimple_stmt_iterator *); static bool gimple_stringops_transform (gimple_stmt_iterator *); -static bool gimple_ic_transform (gimple_stmt_iterator *); /* Allocate histogram value. */ -static histogram_value +histogram_value gimple_alloc_histogram_value (struct function *fun ATTRIBUTE_UNUSED, enum hist_type type, gimple stmt, tree value) { @@ -1307,6 +1306,9 @@ del_node_map (void) struct cgraph_node* find_func_by_profile_id (int profile_id) { + if (flag_auto_profile) + return cgraph_node::get_for_asmname ( + get_identifier ((const char *)(size_t)profile_id)); cgraph_node **val = cgraph_node_map->get (profile_id); if (val) return *val; @@ -1320,7 +1322,7 @@ find_func_by_profile_id (int profile_id) may ICE. Here we only do very minimal sanity check just to make compiler happy. Returns true if TARGET is considered ok for call CALL_STMT. */ -static bool +bool check_ic_target (gimple call_stmt, struct cgraph_node *target) { location_t locus; @@ -1483,7 +1485,7 @@ gimple_ic (gimple icall_stmt, struct cgraph_node * this call to: */ -static bool +bool gimple_ic_transform (gimple_stmt_iterator *gsi) { gimple stmt = gsi_stmt (*gsi); Index: gcc/value-prof.h =================================================================== --- gcc/value-prof.h (revision 215826) +++ gcc/value-prof.h (working copy) @@ -75,6 +75,8 @@ typedef vec<histogram_value> histogram_values; extern void gimple_find_values_to_profile (histogram_values *); extern bool gimple_value_profile_transformations (void); +histogram_value gimple_alloc_histogram_value (struct function *, enum hist_type, + gimple stmt, tree); histogram_value gimple_histogram_value (struct function *, gimple); histogram_value gimple_histogram_value_of_type (struct function *, gimple, enum hist_type); @@ -89,6 +91,8 @@ void verify_histograms (void); void free_histograms (void); void stringop_block_profile (gimple, unsigned int *, HOST_WIDE_INT *); gimple gimple_ic (gimple, struct cgraph_node *, int, gcov_type, gcov_type); +bool check_ic_target (gimple, struct cgraph_node *); +bool gimple_ic_transform (gimple_stmt_iterator *); /* In tree-profile.c. */ Index: gcc/auto-profile.c =================================================================== --- gcc/auto-profile.c (revision 0) +++ gcc/auto-profile.c (revision 0) @@ -0,0 +1,1679 @@ +/* Read and annotate call graph profile from the auto profile data file. + Copyright (C) 2014. Free Software Foundation, Inc. + Contributed by Dehao Chen (dehao@google.com) + +This file is part of GCC. + +GCC is free software; you can redistribute it and/or modify it under +the terms of the GNU General Public License as published by the Free +Software Foundation; either version 3, or (at your option) any later +version. + +GCC is distributed in the hope that it will be useful, but WITHOUT ANY +WARRANTY; without even the implied warranty of MERCHANTABILITY or +FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License +for more details. + +You should have received a copy of the GNU General Public License +along with GCC; see the file COPYING3. If not see +<http://www.gnu.org/licenses/>. */ + +#include <string.h> +#include <map> +#include <vector> +#include <set> + +#include "config.h" +#include "system.h" +#include "coretypes.h" +#include "tree.h" +#include "tree-pass.h" +#include "flags.h" +#include "basic-block.h" +#include "diagnostic-core.h" +#include "gcov-io.h" +#include "input.h" +#include "profile.h" +#include "langhooks.h" +#include "opts.h" +#include "tree-pass.h" +#include "cfgloop.h" +#include "tree-ssa-alias.h" +#include "tree-cfg.h" +#include "tree-cfgcleanup.h" +#include "tree-ssa-operands.h" +#include "tree-into-ssa.h" +#include "internal-fn.h" +#include "is-a.h" +#include "gimple-expr.h" +#include "gimple.h" +#include "gimple-iterator.h" +#include "gimple-ssa.h" +#include "cgraph.h" +#include "value-prof.h" +#include "coverage.h" +#include "params.h" +#include "ipa-inline.h" +#include "tree-inline.h" +#include "auto-profile.h" + +/* The following routines implements AutoFDO optimization. + + This optimization uses sampling profiles to annotate basic block counts + and uses heuristics to estimate branch probabilities. + + There are three phases in AutoFDO: + + Phase 1: Read profile from the profile data file. + The following info is read from the profile datafile: + * string_table: a map between function name and its index. + * autofdo_source_profile: a map from function_instance name to + function_instance. This is represented as a forest of + function_instances. + * WorkingSet: a histogram of how many instructions are covered for a + given percentage of total cycles. + + Phase 2: Early inline + VPT. + Early inline uses autofdo_source_profile to find if a callsite is: + * inlined in the profiled binary. + * callee body is hot in the profiling run. + If both condition satisfies, early inline will inline the callsite + regardless of the code growth. + Phase 2 is an iterative process. During each iteration, we also check + if an indirect callsite is promoted and inlined in the profiling run. + If yes, vpt will happen to force promote it and in the next iteration, + einline will inline the promoted callsite in the next iteration. + + Phase 3: Annotate control flow graph. + AutoFDO uses a separate pass to: + * Annotate basic block count + * Estimate branch probability + + After the above 3 phases, all profile is readily annotated on the GCC IR. + AutoFDO tries to reuse all FDO infrastructure as much as possible to make + use of the profile. E.g. it uses existing mechanism to calculate the basic + block/edge frequency, as well as the cgraph node/edge count. +*/ + +#define DEFAULT_AUTO_PROFILE_FILE "fbdata.afdo" + +namespace autofdo +{ + +/* Represent a source location: (function_decl, lineno). */ +typedef std::pair<tree, unsigned> decl_lineno; + +/* Represent an inline stack. vector[0] is the leaf node. */ +typedef std::vector<decl_lineno> inline_stack; + +/* String array that stores function names. */ +typedef std::vector<const char *> string_vector; + +/* Map from function name's index in string_table to target's + execution count. */ +typedef std::map<unsigned, gcov_type> icall_target_map; + +/* Set of gimple stmts. Used to track if the stmt has already been promoted + to direct call. */ +typedef std::set<gimple> stmt_set; + +/* Represent count info of an inline stack. */ +struct count_info +{ + /* Sampled count of the inline stack. */ + gcov_type count; + + /* Map from indirect call target to its sample count. */ + icall_target_map targets; + + /* Whether this inline stack is already used in annotation. + + Each inline stack should only be used to annotate IR once. + This will be enforced when instruction-level discriminator + is supported. */ + bool annotated; +}; + +/* operator< for "const char *". */ +struct string_compare +{ + bool operator()(const char *a, const char *b) const + { + return strcmp (a, b) < 0; + } +}; + +/* Store a string array, indexed by string position in the array. */ +class string_table +{ +public: + static string_table *create (); + + /* For a given string, returns its index. */ + int get_index (const char *name) const; + + /* For a given decl, returns the index of the decl name. */ + int get_index_by_decl (tree decl) const; + + /* For a given index, returns the string. */ + const char *get_name (int index) const; + +private: + string_table () {} + bool read (); + + typedef std::map<const char *, unsigned, string_compare> string_index_map; + string_vector vector_; + string_index_map map_; +}; + +/* Profile of a function instance: + 1. total_count of the function. + 2. head_count of the function (only valid when function is a top-level + function_instance, i.e. it is the original copy instead of the + inlined copy). + 3. map from source location (decl_lineno) of the inlined callsite to + profile (count_info). + 4. map from callsite to callee function_instance. */ +class function_instance +{ +public: + typedef std::vector<function_instance *> function_instance_stack; + + /* Read the profile and return a function_instance with head count as + HEAD_COUNT. Recursively read callsites to create nested function_instances + too. STACK is used to track the recursive creation process. */ + static function_instance * + read_function_instance (function_instance_stack *stack, + gcov_type head_count); + + /* Recursively deallocate all callsites (nested function_instances). */ + ~function_instance (); + + /* Accessors. */ + int + name () const + { + return name_; + } + gcov_type + total_count () const + { + return total_count_; + } + gcov_type + head_count () const + { + return head_count_; + } + + /* Recursively traverse STACK starting from LEVEL to find the corresponding + function_instance. */ + function_instance *get_function_instance (const inline_stack &stack, + unsigned level); + + /* Store the profile info for LOC in INFO. Return TRUE if profile info + is found. */ + bool get_count_info (location_t loc, count_info *info) const; + + /* Read the inlined indirect call target profile for STMT and store it in + MAP, return the total count for all inlined indirect calls. */ + gcov_type find_icall_target_map (gimple stmt, icall_target_map *map) const; + + /* Sum of counts that is used during annotation. */ + gcov_type total_annotated_count () const; + + /* Mark LOC as annotated. */ + void mark_annotated (location_t loc); + +private: + /* Callsite, represented as (decl_lineno, callee_function_name_index). */ + typedef std::pair<unsigned, unsigned> callsite; + + /* Map from callsite to callee function_instance. */ + typedef std::map<callsite, function_instance *> callsite_map; + + function_instance (unsigned name, gcov_type head_count) + : name_ (name), total_count_ (0), head_count_ (head_count) + { + } + + /* Traverse callsites of the current function_instance to find one at the + location of LINENO and callee name represented in DECL. */ + function_instance *get_function_instance_by_decl (unsigned lineno, + tree decl) const; + + /* Map from source location (decl_lineno) to profile (count_info). */ + typedef std::map<unsigned, count_info> position_count_map; + + /* function_instance name index in the string_table. */ + unsigned name_; + + /* Total sample count. */ + gcov_type total_count_; + + /* Entry BB's sample count. */ + gcov_type head_count_; + + /* Map from callsite location to callee function_instance. */ + callsite_map callsites; + + /* Map from source location to count_info. */ + position_count_map pos_counts; +}; + +/* Profile for all functions. */ +class autofdo_source_profile +{ +public: + static autofdo_source_profile * + create () + { + autofdo_source_profile *map = new autofdo_source_profile (); + + if (map->read ()) + return map; + delete map; + return NULL; + } + + ~autofdo_source_profile (); + + /* For a given DECL, returns the top-level function_instance. */ + function_instance *get_function_instance_by_decl (tree decl) const; + + /* Find count_info for a given gimple STMT. If found, store the count_info + in INFO and return true; otherwise return false. */ + bool get_count_info (gimple stmt, count_info *info) const; + + /* Find total count of the callee of EDGE. */ + gcov_type get_callsite_total_count (struct cgraph_edge *edge) const; + + /* Update value profile INFO for STMT from the inlined indirect callsite. + Return true if INFO is updated. */ + bool update_inlined_ind_target (gimple stmt, count_info *info); + + /* Mark LOC as annotated. */ + void mark_annotated (location_t loc); + +private: + /* Map from function_instance name index (in string_table) to + function_instance. */ + typedef std::map<unsigned, function_instance *> name_function_instance_map; + + autofdo_source_profile () {} + + /* Read AutoFDO profile and returns TRUE on success. */ + bool read (); + + /* Return the function_instance in the profile that correspond to the + inline STACK. */ + function_instance * + get_function_instance_by_inline_stack (const inline_stack &stack) const; + + name_function_instance_map map_; +}; + +/* Store the strings read from the profile data file. */ +static string_table *afdo_string_table; + +/* Store the AutoFDO source profile. */ +static autofdo_source_profile *afdo_source_profile; + +/* gcov_ctr_summary structure to store the profile_info. */ +static struct gcov_ctr_summary *afdo_profile_info; + +/* Helper functions. */ + +/* Return the original name of NAME: strip the suffix that starts + with '.' */ + +static const char * +get_original_name (const char *name) +{ + char *ret = xstrdup (name); + char *find = strchr (ret, '.'); + if (find != NULL) + *find = 0; + return ret; +} + +/* Return the combined location, which is a 32bit integer in which + higher 16 bits stores the line offset of LOC to the start lineno + of DECL, The lower 16 bits stores the discrimnator. */ + +static unsigned +get_combined_location (location_t loc, tree decl) +{ + /* TODO: allow more bits for line and less bits for discriminator. */ + return ((LOCATION_LINE (loc) - DECL_SOURCE_LINE (decl)) << 16); +} + +/* Return the function decl of a given lexical BLOCK. */ + +static tree +get_function_decl_from_block (tree block) +{ + tree decl; + + if (LOCATION_LOCUS (BLOCK_SOURCE_LOCATION (block) == UNKNOWN_LOCATION)) + return NULL_TREE; + + for (decl = BLOCK_ABSTRACT_ORIGIN (block); + decl && (TREE_CODE (decl) == BLOCK); + decl = BLOCK_ABSTRACT_ORIGIN (decl)) + if (TREE_CODE (decl) == FUNCTION_DECL) + break; + return decl; +} + +/* Store inline stack for STMT in STACK. */ + +static void +get_inline_stack (location_t locus, inline_stack *stack) +{ + if (LOCATION_LOCUS (locus) == UNKNOWN_LOCATION) + return; + + tree block = LOCATION_BLOCK (locus); + if (block && TREE_CODE (block) == BLOCK) + { + int level = 0; + for (block = BLOCK_SUPERCONTEXT (block); + block && (TREE_CODE (block) == BLOCK); + block = BLOCK_SUPERCONTEXT (block)) + { + location_t tmp_locus = BLOCK_SOURCE_LOCATION (block); + if (LOCATION_LOCUS (tmp_locus) == UNKNOWN_LOCATION) + continue; + + tree decl = get_function_decl_from_block (block); + stack->push_back ( + std::make_pair (decl, get_combined_location (locus, decl))); + locus = tmp_locus; + level++; + } + } + stack->push_back ( + std::make_pair (current_function_decl, + get_combined_location (locus, current_function_decl))); +} + +/* Return STMT's combined location, which is a 32bit integer in which + higher 16 bits stores the line offset of LOC to the start lineno + of DECL, The lower 16 bits stores the discrimnator. */ + +static unsigned +get_relative_location_for_stmt (gimple stmt) +{ + location_t locus = gimple_location (stmt); + if (LOCATION_LOCUS (locus) == UNKNOWN_LOCATION) + return UNKNOWN_LOCATION; + + for (tree block = gimple_block (stmt); block && (TREE_CODE (block) == BLOCK); + block = BLOCK_SUPERCONTEXT (block)) + if (LOCATION_LOCUS (BLOCK_SOURCE_LOCATION (block)) != UNKNOWN_LOCATION) + return get_combined_location (locus, + get_function_decl_from_block (block)); + return get_combined_location (locus, current_function_decl); +} + +/* Return true if BB contains indirect call. */ + +static bool +has_indirect_call (basic_block bb) +{ + gimple_stmt_iterator gsi; + + for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi)) + { + gimple stmt = gsi_stmt (gsi); + if (gimple_code (stmt) == GIMPLE_CALL && !gimple_call_internal_p (stmt) + && (gimple_call_fn (stmt) == NULL + || TREE_CODE (gimple_call_fn (stmt)) != FUNCTION_DECL)) + return true; + } + return false; +} + +/* Member functions for string_table. */ + +string_table * +string_table::create () +{ + string_table *map = new string_table (); + if (map->read ()) + return map; + delete map; + return NULL; +} + +/* Return the index of a given function NAME. */ + +int +string_table::get_index (const char *name) const +{ + if (name == NULL) + return -1; + string_index_map::const_iterator iter = map_.find (name); + if (iter == map_.end ()) + return -1; + else + return iter->second; +} + +/* Return the index of a given function DECL. */ + +int +string_table::get_index_by_decl (tree decl) const +{ + const char *name + = get_original_name (IDENTIFIER_POINTER (DECL_ASSEMBLER_NAME (decl))); + int ret = get_index (name); + if (ret != -1) + return ret; + ret = get_index (lang_hooks.dwarf_name (decl, 0)); + if (ret != -1) + return ret; + if (DECL_ABSTRACT_ORIGIN (decl)) + return get_index_by_decl (DECL_ABSTRACT_ORIGIN (decl)); + else + return -1; +} + +/* Return the function name of a given INDEX. */ + +const char * +string_table::get_name (int index) const +{ + gcc_assert (index > 0 && index < (int)vector_.size ()); + return vector_[index]; +} + +/* Read the string table. Return TRUE if reading is successful. */ + +bool +string_table::read () +{ + if (gcov_read_unsigned () != GCOV_TAG_AFDO_FILE_NAMES) + return false; + /* Skip the length of the section. */ + gcov_read_unsigned (); + /* Read in the file name table. */ + unsigned string_num = gcov_read_unsigned (); + for (unsigned i = 0; i < string_num; i++) + { + vector_.push_back (get_original_name (gcov_read_string ())); + map_[vector_.back ()] = i; + } + return true; +} + +/* Member functions for function_instance. */ + +function_instance::~function_instance () +{ + for (callsite_map::iterator iter = callsites.begin (); + iter != callsites.end (); ++iter) + delete iter->second; +} + +/* Traverse callsites of the current function_instance to find one at the + location of LINENO and callee name represented in DECL. */ + +function_instance * +function_instance::get_function_instance_by_decl (unsigned lineno, + tree decl) const +{ + int func_name_idx = afdo_string_table->get_index_by_decl (decl); + if (func_name_idx != -1) + { + callsite_map::const_iterator ret + = callsites.find (std::make_pair (lineno, func_name_idx)); + if (ret != callsites.end ()) + return ret->second; + } + func_name_idx + = afdo_string_table->get_index (lang_hooks.dwarf_name (decl, 0)); + if (func_name_idx != -1) + { + callsite_map::const_iterator ret + = callsites.find (std::make_pair (lineno, func_name_idx)); + if (ret != callsites.end ()) + return ret->second; + } + if (DECL_ABSTRACT_ORIGIN (decl)) + return get_function_instance_by_decl (lineno, DECL_ABSTRACT_ORIGIN (decl)); + else + return NULL; +} + +/* Recursively traverse STACK starting from LEVEL to find the corresponding + function_instance. */ + +function_instance * +function_instance::get_function_instance (const inline_stack &stack, + unsigned level) +{ + if (level == 0) + return this; + function_instance *s = get_function_instance_by_decl ( + stack[level].second, stack[level - 1].first); + if (s) + return s->get_function_instance (stack, level - 1); + else + return NULL; +} + +/* Store the profile info for LOC in INFO. Return TRUE if profile info + is found. */ + +bool +function_instance::get_count_info (location_t loc, count_info *info) const +{ + position_count_map::const_iterator iter = pos_counts.find (loc); + if (iter == pos_counts.end ()) + return false; + *info = iter->second; + return true; +} + +/* Mark LOC as annotated. */ + +void +function_instance::mark_annotated (location_t loc) +{ + position_count_map::iterator iter = pos_counts.find (loc); + if (iter == pos_counts.end ()) + return; + iter->second.annotated = true; +} + +/* Read the inlinied indirect call target profile for STMT and store it in + MAP, return the total count for all inlined indirect calls. */ + +gcov_type +function_instance::find_icall_target_map (gimple stmt, + icall_target_map *map) const +{ + gcov_type ret = 0; + unsigned stmt_offset = get_relative_location_for_stmt (stmt); + + for (callsite_map::const_iterator iter = callsites.begin (); + iter != callsites.end (); ++iter) + { + unsigned callee = iter->second->name (); + /* Check if callsite location match the stmt. */ + if (iter->first.first != stmt_offset) + continue; + struct cgraph_node *node = find_func_by_profile_id ( + (unsigned long long)afdo_string_table->get_name (callee)); + if (node == NULL) + continue; + if (!check_ic_target (stmt, node)) + continue; + (*map)[callee] = iter->second->total_count (); + ret += iter->second->total_count (); + } + return ret; +} + +/* Read the profile and create a function_instance with head count as + HEAD_COUNT. Recursively read callsites to create nested function_instances + too. STACK is used to track the recursive creation process. */ + +/* function instance profile format: + + ENTRY_COUNT: 8 bytes + NAME_INDEX: 4 bytes + NUM_POS_COUNTS: 4 bytes + NUM_CALLSITES: 4 byte + POS_COUNT_1: + POS_1_OFFSET: 4 bytes + NUM_TARGETS: 4 bytes + COUNT: 8 bytes + TARGET_1: + VALUE_PROFILE_TYPE: 4 bytes + TARGET_IDX: 8 bytes + COUNT: 8 bytes + TARGET_2 + ... + TARGET_n + POS_COUNT_2 + ... + POS_COUNT_N + CALLSITE_1: + CALLSITE_1_OFFSET: 4 bytes + FUNCTION_INSTANCE_PROFILE (nested) + CALLSITE_2 + ... + CALLSITE_n. */ + +function_instance * +function_instance::read_function_instance (function_instance_stack *stack, + gcov_type head_count) +{ + unsigned name = gcov_read_unsigned (); + unsigned num_pos_counts = gcov_read_unsigned (); + unsigned num_callsites = gcov_read_unsigned (); + function_instance *s = new function_instance (name, head_count); + stack->push_back (s); + + for (unsigned i = 0; i < num_pos_counts; i++) + { + unsigned offset = gcov_read_unsigned () & 0xffff0000; + unsigned num_targets = gcov_read_unsigned (); + gcov_type count = gcov_read_counter (); + s->pos_counts[offset].count = count; + for (unsigned j = 0; j < stack->size (); j++) + (*stack)[j]->total_count_ += count; + for (unsigned j = 0; j < num_targets; j++) + { + /* Only indirect call target histogram is supported now. */ + gcov_read_unsigned (); + gcov_type target_idx = gcov_read_counter (); + s->pos_counts[offset].targets[target_idx] = gcov_read_counter (); + } + } + for (unsigned i = 0; i < num_callsites; i++) + { + unsigned offset = gcov_read_unsigned (); + function_instance *callee_function_instance + = read_function_instance (stack, 0); + s->callsites[std::make_pair (offset, callee_function_instance->name ())] + = callee_function_instance; + } + stack->pop_back (); + return s; +} + +/* Sum of counts that is used during annotation. */ + +gcov_type +function_instance::total_annotated_count () const +{ + gcov_type ret = 0; + for (callsite_map::const_iterator iter = callsites.begin (); + iter != callsites.end (); ++iter) + ret += iter->second->total_annotated_count (); + for (position_count_map::const_iterator iter = pos_counts.begin (); + iter != pos_counts.end (); ++iter) + if (iter->second.annotated) + ret += iter->second.count; + return ret; +} + +/* Member functions for autofdo_source_profile. */ + +autofdo_source_profile::~autofdo_source_profile () +{ + for (name_function_instance_map::const_iterator iter = map_.begin (); + iter != map_.end (); ++iter) + delete iter->second; +} + +/* For a given DECL, returns the top-level function_instance. */ + +function_instance * +autofdo_source_profile::get_function_instance_by_decl (tree decl) const +{ + int index = afdo_string_table->get_index_by_decl (decl); + if (index == -1) + return NULL; + name_function_instance_map::const_iterator ret = map_.find (index); + return ret == map_.end () ? NULL : ret->second; +} + +/* Find count_info for a given gimple STMT. If found, store the count_info + in INFO and return true; otherwise return false. */ + +bool +autofdo_source_profile::get_count_info (gimple stmt, count_info *info) const +{ + if (LOCATION_LOCUS (gimple_location (stmt)) == cfun->function_end_locus) + return false; + + inline_stack stack; + get_inline_stack (gimple_location (stmt), &stack); + if (stack.size () == 0) + return false; + function_instance *s = get_function_instance_by_inline_stack (stack); + if (s == NULL) + return false; + return s->get_count_info (stack[0].second, info); +} + +/* Mark LOC as annotated. */ + +void +autofdo_source_profile::mark_annotated (location_t loc) +{ + inline_stack stack; + get_inline_stack (loc, &stack); + if (stack.size () == 0) + return; + function_instance *s = get_function_instance_by_inline_stack (stack); + if (s == NULL) + return; + s->mark_annotated (stack[0].second); +} + +/* Update value profile INFO for STMT from the inlined indirect callsite. + Return true if INFO is updated. */ + +bool +autofdo_source_profile::update_inlined_ind_target (gimple stmt, + count_info *info) +{ + if (LOCATION_LOCUS (gimple_location (stmt)) == cfun->function_end_locus) + return false; + + count_info old_info; + get_count_info (stmt, &old_info); + gcov_type total = 0; + for (icall_target_map::const_iterator iter = old_info.targets.begin (); + iter != old_info.targets.end (); ++iter) + total += iter->second; + + /* Program behavior changed, original promoted (and inlined) target is not + hot any more. Will avoid promote the original target. + + To check if original promoted target is still hot, we check the total + count of the unpromoted targets (stored in old_info). If it is no less + than half of the callsite count (stored in INFO), the original promoted + target is considered not hot any more. */ + if (total >= info->count / 2) + return false; + + inline_stack stack; + get_inline_stack (gimple_location (stmt), &stack); + if (stack.size () == 0) + return false; + function_instance *s = get_function_instance_by_inline_stack (stack); + if (s == NULL) + return false; + icall_target_map map; + if (s->find_icall_target_map (stmt, &map) == 0) + return false; + for (icall_target_map::const_iterator iter = map.begin (); + iter != map.end (); ++iter) + info->targets[iter->first] = iter->second; + return true; +} + +/* Find total count of the callee of EDGE. */ + +gcov_type +autofdo_source_profile::get_callsite_total_count ( + struct cgraph_edge *edge) const +{ + inline_stack stack; + stack.push_back (std::make_pair (edge->callee->decl, 0)); + get_inline_stack (gimple_location (edge->call_stmt), &stack); + + function_instance *s = get_function_instance_by_inline_stack (stack); + if (s == NULL + || afdo_string_table->get_index (IDENTIFIER_POINTER ( + DECL_ASSEMBLER_NAME (edge->callee->decl))) != s->name ()) + return 0; + else + return s->total_count (); +} + +/* Read AutoFDO profile and returns TRUE on success. */ + +/* source profile format: + + GCOV_TAG_AFDO_FUNCTION: 4 bytes + LENGTH: 4 bytes + NUM_FUNCTIONS: 4 bytes + FUNCTION_INSTANCE_1 + FUNCTION_INSTANCE_2 + ... + FUNCTION_INSTANCE_N. */ + +bool +autofdo_source_profile::read () +{ + if (gcov_read_unsigned () != GCOV_TAG_AFDO_FUNCTION) + { + inform (0, "Not expected TAG."); + return false; + } + + /* Skip the length of the section. */ + gcov_read_unsigned (); + + /* Read in the function/callsite profile, and store it in local + data structure. */ + unsigned function_num = gcov_read_unsigned (); + for (unsigned i = 0; i < function_num; i++) + { + function_instance::function_instance_stack stack; + function_instance *s = function_instance::read_function_instance ( + &stack, gcov_read_counter ()); + afdo_profile_info->sum_all += s->total_count (); + map_[s->name ()] = s; + } + return true; +} + +/* Return the function_instance in the profile that correspond to the + inline STACK. */ + +function_instance * +autofdo_source_profile::get_function_instance_by_inline_stack ( + const inline_stack &stack) const +{ + name_function_instance_map::const_iterator iter = map_.find ( + afdo_string_table->get_index_by_decl (stack[stack.size () - 1].first)); + return iter == map_.end () + ? NULL + : iter->second->get_function_instance (stack, stack.size () - 1); +} + +/* Module profile is only used by LIPO. Here we simply ignore it. */ + +static void +fake_read_autofdo_module_profile () +{ + /* Read in the module info. */ + gcov_read_unsigned (); + + /* Skip the length of the section. */ + gcov_read_unsigned (); + + /* Read in the file name table. */ + unsigned total_module_num = gcov_read_unsigned (); + gcc_assert (total_module_num == 0); +} + +static void +read_profile (void) +{ + if (gcov_open (auto_profile_file, 1) == 0) + error ("Cannot open profile file %s.", auto_profile_file); + + if (gcov_read_unsigned () != GCOV_DATA_MAGIC) + error ("AutoFDO profile magic number does not mathch."); + + /* Skip the version number. */ + gcov_read_unsigned (); + + /* Skip the empty integer. */ + gcov_read_unsigned (); + + /* string_table. */ + afdo_string_table = string_table::create (); + if (afdo_string_table == NULL) + error ("Cannot read string table from %s.", auto_profile_file); + + /* autofdo_source_profile. */ + afdo_source_profile = autofdo_source_profile::create (); + if (afdo_source_profile == NULL) + error ("Cannot read function profile from %s.", auto_profile_file); + + /* autofdo_module_profile. */ + fake_read_autofdo_module_profile (); + + /* Read in the working set. */ + if (gcov_read_unsigned () != GCOV_TAG_AFDO_WORKING_SET) + error ("Cannot read working set from %s.", auto_profile_file); + + /* Skip the length of the section. */ + gcov_read_unsigned (); + gcov_working_set_t set[128]; + for (unsigned i = 0; i < 128; i++) + { + set[i].num_counters = gcov_read_unsigned (); + set[i].min_counter = gcov_read_counter (); + } + add_working_set (set); +} + +/* From AutoFDO profiles, find values inside STMT for that we want to measure + histograms for indirect-call optimization. */ + +static void +afdo_indirect_call (gimple_stmt_iterator *gsi, const icall_target_map &map, + bool transform) +{ + gimple stmt = gsi_stmt (*gsi); + tree callee; + + if (map.size () == 0 || gimple_code (stmt) != GIMPLE_CALL + || gimple_call_fndecl (stmt) != NULL_TREE) + return; + + callee = gimple_call_fn (stmt); + + histogram_value hist = gimple_alloc_histogram_value ( + cfun, HIST_TYPE_INDIR_CALL, stmt, callee); + hist->n_counters = 3; + hist->hvalue.counters = XNEWVEC (gcov_type, hist->n_counters); + gimple_add_histogram_value (cfun, stmt, hist); + + gcov_type total = 0; + icall_target_map::const_iterator max_iter = map.end (); + + for (icall_target_map::const_iterator iter = map.begin (); + iter != map.end (); ++iter) + { + total += iter->second; + if (max_iter == map.end () || max_iter->second < iter->second) + max_iter = iter; + } + + hist->hvalue.counters[0] + = (unsigned long long)afdo_string_table->get_name (max_iter->first); + hist->hvalue.counters[1] = max_iter->second; + hist->hvalue.counters[2] = total; + + if (!transform) + return; + + if (gimple_ic_transform (gsi)) + { + struct cgraph_edge *indirect_edge + = cgraph_node::get (current_function_decl)->get_edge (stmt); + struct cgraph_node *direct_call + = find_func_by_profile_id ((int)hist->hvalue.counters[0]); + if (DECL_STRUCT_FUNCTION (direct_call->decl) == NULL) + return; + struct cgraph_edge *new_edge + = indirect_edge->make_speculative (direct_call, 0, 0); + new_edge->redirect_call_stmt_to_callee (); + gimple_remove_histogram_value (cfun, stmt, hist); + inline_call (new_edge, true, NULL, NULL, false); + return; + } + return; +} + +/* From AutoFDO profiles, find values inside STMT for that we want to measure + histograms and adds them to list VALUES. */ + +static void +afdo_vpt (gimple_stmt_iterator *gsi, const icall_target_map &map, + bool transform) +{ + afdo_indirect_call (gsi, map, transform); +} + +typedef std::set<basic_block> bb_set; +typedef std::set<edge> edge_set; + +static bool +is_bb_annotated (const basic_block bb, const bb_set &annotated) +{ + return annotated.find (bb) != annotated.end (); +} + +static void +set_bb_annotated (basic_block bb, bb_set *annotated) +{ + annotated->insert (bb); +} + +static bool +is_edge_annotated (const edge e, const edge_set &annotated) +{ + return annotated.find (e) != annotated.end (); +} + +static void +set_edge_annotated (edge e, edge_set *annotated) +{ + annotated->insert (e); +} + +/* For a given BB, set its execution count. Attach value profile if a stmt + is not in PROMOTED, because we only want to promot an indirect call once. + Return TRUE if BB is annotated. */ + +static bool +afdo_set_bb_count (basic_block bb, const stmt_set &promoted) +{ + gimple_stmt_iterator gsi; + edge e; + edge_iterator ei; + gcov_type max_count = 0; + bool has_annotated = false; + + for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi)) + { + count_info info; + gimple stmt = gsi_stmt (gsi); + if (gimple_clobber_p (stmt) || is_gimple_debug (stmt)) + continue; + if (afdo_source_profile->get_count_info (stmt, &info)) + { + if (info.count > max_count) + max_count = info.count; + has_annotated = true; + if (info.targets.size () > 0 + && promoted.find (stmt) == promoted.end ()) + afdo_vpt (&gsi, info.targets, false); + } + } + + if (!has_annotated) + return false; + + for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi)) + afdo_source_profile->mark_annotated (gimple_location (gsi_stmt (gsi))); + for (gsi = gsi_start_phis (bb); !gsi_end_p (gsi); gsi_next (&gsi)) + { + gimple phi = gsi_stmt (gsi); + size_t i; + for (i = 0; i < gimple_phi_num_args (phi); i++) + afdo_source_profile->mark_annotated (gimple_phi_arg_location (phi, i)); + } + FOR_EACH_EDGE (e, ei, bb->succs) + afdo_source_profile->mark_annotated (e->goto_locus); + + bb->count = max_count; + return true; +} + +/* BB1 and BB2 are in an equivalent class iff: + 1. BB1 dominates BB2. + 2. BB2 post-dominates BB1. + 3. BB1 and BB2 are in the same loop nest. + This function finds the equivalent class for each basic block, and + stores a pointer to the first BB in its equivalent class. Meanwhile, + set bb counts for the same equivalent class to be idenical. Update + ANNOTATED_BB for the first BB in its equivalent class. */ + +static void +afdo_find_equiv_class (bb_set *annotated_bb) +{ + basic_block bb; + + FOR_ALL_BB_FN (bb, cfun) + bb->aux = NULL; + + FOR_ALL_BB_FN (bb, cfun) + { + vec<basic_block> dom_bbs; + basic_block bb1; + int i; + + if (bb->aux != NULL) + continue; + bb->aux = bb; + dom_bbs = get_dominated_by (CDI_DOMINATORS, bb); + FOR_EACH_VEC_ELT (dom_bbs, i, bb1) + if (bb1->aux == NULL && dominated_by_p (CDI_POST_DOMINATORS, bb, bb1) + && bb1->loop_father == bb->loop_father) + { + bb1->aux = bb; + if (bb1->count > bb->count && is_bb_annotated (bb1, *annotated_bb)) + { + bb->count = MAX (bb->count, bb1->count); + set_bb_annotated (bb, annotated_bb); + } + } + dom_bbs = get_dominated_by (CDI_POST_DOMINATORS, bb); + FOR_EACH_VEC_ELT (dom_bbs, i, bb1) + if (bb1->aux == NULL && dominated_by_p (CDI_DOMINATORS, bb, bb1) + && bb1->loop_father == bb->loop_father) + { + bb1->aux = bb; + if (bb1->count > bb->count && is_bb_annotated (bb1, *annotated_bb)) + { + bb->count = MAX (bb->count, bb1->count); + set_bb_annotated (bb, annotated_bb); + } + } + } +} + +/* If a basic block's count is known, and only one of its in/out edges' count + is unknown, its count can be calculated. Meanwhile, if all of the in/out + edges' counts are known, then the basic block's unknown count can also be + calculated. + IS_SUCC is true if out edges of a basic blocks are examined. + Update ANNOTATED_BB and ANNOTATED_EDGE accordingly. + Return TRUE if any basic block/edge count is changed. */ + +static bool +afdo_propagate_edge (bool is_succ, bb_set *annotated_bb, + edge_set *annotated_edge) +{ + basic_block bb; + bool changed = false; + + FOR_EACH_BB_FN (bb, cfun) + { + edge e, unknown_edge = NULL; + edge_iterator ei; + int num_unknown_edge = 0; + gcov_type total_known_count = 0; + + FOR_EACH_EDGE (e, ei, is_succ ? bb->succs : bb->preds) + if (!is_edge_annotated (e, *annotated_edge)) + num_unknown_edge++, unknown_edge = e; + else + total_known_count += e->count; + + if (num_unknown_edge == 0) + { + if (total_known_count > bb->count) + { + bb->count = total_known_count; + changed = true; + } + if (!is_bb_annotated (bb, *annotated_bb)) + { + set_bb_annotated (bb, annotated_bb); + changed = true; + } + } + else if (num_unknown_edge == 1 && is_bb_annotated (bb, *annotated_bb)) + { + if (bb->count >= total_known_count) + unknown_edge->count = bb->count - total_known_count; + else + unknown_edge->count = 0; + set_edge_annotated (unknown_edge, annotated_edge); + changed = true; + } + } + return changed; +} + +/* Special propagation for circuit expressions. Because GCC translates + control flow into data flow for circuit expressions. E.g. + BB1: + if (a && b) + BB2 + else + BB3 + + will be translated into: + + BB1: + if (a) + goto BB.t1 + else + goto BB.t3 + BB.t1: + if (b) + goto BB.t2 + else + goto BB.t3 + BB.t2: + goto BB.t3 + BB.t3: + tmp = PHI (0 (BB1), 0 (BB.t1), 1 (BB.t2) + if (tmp) + goto BB2 + else + goto BB3 + + In this case, we need to propagate through PHI to determine the edge + count of BB1->BB.t1, BB.t1->BB.t2. + Update ANNOTATED_EDGE accordingly. */ + +static void +afdo_propagate_circuit (const bb_set &annotated_bb, edge_set *annotated_edge) +{ + basic_block bb; + FOR_ALL_BB_FN (bb, cfun) + { + gimple phi_stmt; + tree cmp_rhs, cmp_lhs; + gimple cmp_stmt = last_stmt (bb); + edge e; + edge_iterator ei; + + if (!cmp_stmt || gimple_code (cmp_stmt) != GIMPLE_COND) + continue; + cmp_rhs = gimple_cond_rhs (cmp_stmt); + cmp_lhs = gimple_cond_lhs (cmp_stmt); + if (!TREE_CONSTANT (cmp_rhs) + || !(integer_zerop (cmp_rhs) || integer_onep (cmp_rhs))) + continue; + if (TREE_CODE (cmp_lhs) != SSA_NAME) + continue; + if (!is_bb_annotated (bb, annotated_bb)) + continue; + phi_stmt = SSA_NAME_DEF_STMT (cmp_lhs); + while (phi_stmt && gimple_code (phi_stmt) == GIMPLE_ASSIGN + && gimple_assign_single_p (phi_stmt) + && TREE_CODE (gimple_assign_rhs1 (phi_stmt)) == SSA_NAME) + phi_stmt = SSA_NAME_DEF_STMT (gimple_assign_rhs1 (phi_stmt)); + if (!phi_stmt || gimple_code (phi_stmt) != GIMPLE_PHI) + continue; + FOR_EACH_EDGE (e, ei, bb->succs) + { + unsigned i, total = 0; + edge only_one; + bool check_value_one = (((integer_onep (cmp_rhs)) + ^ (gimple_cond_code (cmp_stmt) == EQ_EXPR)) + ^ ((e->flags & EDGE_TRUE_VALUE) != 0)); + if (!is_edge_annotated (e, *annotated_edge)) + continue; + for (i = 0; i < gimple_phi_num_args (phi_stmt); i++) + { + tree val = gimple_phi_arg_def (phi_stmt, i); + edge ep = gimple_phi_arg_edge (phi_stmt, i); + + if (!TREE_CONSTANT (val) + || !(integer_zerop (val) || integer_onep (val))) + continue; + if (check_value_one ^ integer_onep (val)) + continue; + total++; + only_one = ep; + if (e->probability == 0 && !is_edge_annotated (ep, *annotated_edge)) + { + ep->probability = 0; + ep->count = 0; + set_edge_annotated (ep, annotated_edge); + } + } + if (total == 1 && !is_edge_annotated (only_one, *annotated_edge)) + { + only_one->probability = e->probability; + only_one->count = e->count; + set_edge_annotated (only_one, annotated_edge); + } + } + } +} + +/* Propagate the basic block count and edge count on the control flow + graph. We do the propagation iteratively until stablize. */ + +static void +afdo_propagate (bb_set *annotated_bb, edge_set *annotated_edge) +{ + basic_block bb; + bool changed = true; + int i = 0; + + FOR_ALL_BB_FN (bb, cfun) + { + bb->count = ((basic_block)bb->aux)->count; + if (is_bb_annotated ((const basic_block)bb->aux, *annotated_bb)) + set_bb_annotated (bb, annotated_bb); + } + + while (changed && i++ < 10) + { + changed = false; + + if (afdo_propagate_edge (true, annotated_bb, annotated_edge)) + changed = true; + if (afdo_propagate_edge (false, annotated_bb, annotated_edge)) + changed = true; + afdo_propagate_circuit (*annotated_bb, annotated_edge); + } +} + +/* Propagate counts on control flow graph and calculate branch + probabilities. */ + +static void +afdo_calculate_branch_prob (bb_set *annotated_bb, edge_set *annotated_edge) +{ + basic_block bb; + bool has_sample = false; + + FOR_EACH_BB_FN (bb, cfun) + if (bb->count > 0) + has_sample = true; + + if (!has_sample) + return; + + calculate_dominance_info (CDI_POST_DOMINATORS); + calculate_dominance_info (CDI_DOMINATORS); + loop_optimizer_init (0); + + afdo_find_equiv_class (annotated_bb); + afdo_propagate (annotated_bb, annotated_edge); + + FOR_EACH_BB_FN (bb, cfun) + { + edge e; + edge_iterator ei; + int num_unknown_succ = 0; + gcov_type total_count = 0; + + FOR_EACH_EDGE (e, ei, bb->succs) + { + if (!is_edge_annotated (e, *annotated_edge)) + num_unknown_succ++; + else + total_count += e->count; + } + if (num_unknown_succ == 0 && total_count > 0) + { + FOR_EACH_EDGE (e, ei, bb->succs) + e->probability = (double)e->count * REG_BR_PROB_BASE / total_count; + } + } + FOR_ALL_BB_FN (bb, cfun) + { + edge e; + edge_iterator ei; + + FOR_EACH_EDGE (e, ei, bb->succs) + e->count = (double)bb->count * e->probability / REG_BR_PROB_BASE; + bb->aux = NULL; + } + + loop_optimizer_finalize (); + free_dominance_info (CDI_DOMINATORS); + free_dominance_info (CDI_POST_DOMINATORS); +} + +/* Perform value profile transformation using AutoFDO profile. Add the + promoted stmts to PROMOTED_STMTS. Return TRUE if there is any + indirect call promoted. */ + +static bool +afdo_vpt_for_early_inline (stmt_set *promoted_stmts) +{ + basic_block bb; + if (afdo_source_profile->get_function_instance_by_decl ( + current_function_decl) == NULL) + return false; + + compute_inline_parameters (cgraph_node::get (current_function_decl), true); + + bool has_vpt = false; + FOR_EACH_BB_FN (bb, cfun) + { + if (!has_indirect_call (bb)) + continue; + gimple_stmt_iterator gsi; + + gcov_type bb_count = 0; + for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi)) + { + count_info info; + gimple stmt = gsi_stmt (gsi); + if (afdo_source_profile->get_count_info (stmt, &info)) + bb_count = MAX (bb_count, info.count); + } + + for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi)) + { + gimple stmt = gsi_stmt (gsi); + /* IC_promotion and early_inline_2 is done in multiple iterations. + No need to promoted the stmt if its in promoted_stmts (means + it is already been promoted in the previous iterations). */ + if (gimple_code (stmt) != GIMPLE_CALL || gimple_call_fn (stmt) == NULL + || TREE_CODE (gimple_call_fn (stmt)) == FUNCTION_DECL + || promoted_stmts->find (stmt) != promoted_stmts->end ()) + continue; + + count_info info; + afdo_source_profile->get_count_info (stmt, &info); + info.count = bb_count; + if (afdo_source_profile->update_inlined_ind_target (stmt, &info)) + { + /* Promote the indirect call and update the promoted_stmts. */ + promoted_stmts->insert (stmt); + afdo_vpt (&gsi, info.targets, true); + has_vpt = true; + } + } + } + if (has_vpt) + { + optimize_inline_calls (current_function_decl); + return true; + } + else + return false; +} + +/* Annotate auto profile to the control flow graph. Do not annotate value + profile for stmts in PROMOTED_STMTS. */ + +static void +afdo_annotate_cfg (const stmt_set &promoted_stmts) +{ + basic_block bb; + bb_set annotated_bb; + edge_set annotated_edge; + const function_instance *s + = afdo_source_profile->get_function_instance_by_decl ( + current_function_decl); + + if (s == NULL) + return; + cgraph_node::get (current_function_decl)->count = s->head_count (); + ENTRY_BLOCK_PTR_FOR_FN (cfun)->count = s->head_count (); + gcov_type max_count = ENTRY_BLOCK_PTR_FOR_FN (cfun)->count; + + FOR_EACH_BB_FN (bb, cfun) + { + edge e; + edge_iterator ei; + + bb->count = 0; + FOR_EACH_EDGE (e, ei, bb->succs) + e->count = 0; + + if (afdo_set_bb_count (bb, promoted_stmts)) + set_bb_annotated (bb, &annotated_bb); + if (bb->count > max_count) + max_count = bb->count; + } + if (ENTRY_BLOCK_PTR_FOR_FN (cfun)->count + > ENTRY_BLOCK_PTR_FOR_FN (cfun)->next_bb->count) + { + ENTRY_BLOCK_PTR_FOR_FN (cfun)->next_bb->count + = ENTRY_BLOCK_PTR_FOR_FN (cfun)->count; + set_bb_annotated (ENTRY_BLOCK_PTR_FOR_FN (cfun)->next_bb, &annotated_bb); + } + if (ENTRY_BLOCK_PTR_FOR_FN (cfun)->count + > EXIT_BLOCK_PTR_FOR_FN (cfun)->prev_bb->count) + { + EXIT_BLOCK_PTR_FOR_FN (cfun)->prev_bb->count + = ENTRY_BLOCK_PTR_FOR_FN (cfun)->count; + set_bb_annotated (EXIT_BLOCK_PTR_FOR_FN (cfun)->prev_bb, &annotated_bb); + } + afdo_source_profile->mark_annotated ( + DECL_SOURCE_LOCATION (current_function_decl)); + afdo_source_profile->mark_annotated (cfun->function_start_locus); + afdo_source_profile->mark_annotated (cfun->function_end_locus); + if (max_count > 0) + { + afdo_calculate_branch_prob (&annotated_bb, &annotated_edge); + counts_to_freqs (); + profile_status_for_fn (cfun) = PROFILE_READ; + } + if (flag_value_profile_transformations) + gimple_value_profile_transformations (); +} + +/* Wrapper function to invoke early inliner. */ + +static void +early_inline () +{ + compute_inline_parameters (cgraph_node::get (current_function_decl), true); + unsigned todo = early_inliner (cfun); + if (todo & TODO_update_ssa_any) + update_ssa (TODO_update_ssa); +} + +/* Use AutoFDO profile to annoate the control flow graph. + Return the todo flag. */ + +static unsigned int +auto_profile (void) +{ + struct cgraph_node *node; + + if (symtab->state == FINISHED) + return 0; + + init_node_map (true); + profile_info = autofdo::afdo_profile_info; + + FOR_EACH_FUNCTION (node) + { + if (!gimple_has_body_p (node->decl)) + continue; + + /* Don't profile functions produced for builtin stuff. */ + if (DECL_SOURCE_LOCATION (node->decl) == BUILTINS_LOCATION) + continue; + + push_cfun (DECL_STRUCT_FUNCTION (node->decl)); + + /* First do indirect call promotion and early inline to make the + IR match the profiled binary before actual annotation. + + This is needed because an indirect call might have been promoted + and inlined in the profiled binary. If we do not promote and + inline these indirect calls before annotation, the profile for + these promoted functions will be lost. + + e.g. foo() --indirect_call--> bar() + In profiled binary, the callsite is promoted and inlined, making + the profile look like: + + foo: { + loc_foo_1: count_1 + bar@loc_foo_2: { + loc_bar_1: count_2 + loc_bar_2: count_3 + } + } + + Before AutoFDO pass, loc_foo_2 is not promoted thus not inlined. + If we perform annotation on it, the profile inside bar@loc_foo2 + will be wasted. + + To avoid this, we promote loc_foo_2 and inline the promoted bar + function before annotation, so the profile inside bar@loc_foo2 + will be useful. */ + autofdo::stmt_set promoted_stmts; + for (int i = 0; i < PARAM_VALUE (PARAM_EARLY_INLINER_MAX_ITERATIONS); i++) + { + if (!flag_value_profile_transformations + || !autofdo::afdo_vpt_for_early_inline (&promoted_stmts)) + break; + early_inline (); + } + + early_inline (); + autofdo::afdo_annotate_cfg (promoted_stmts); + compute_function_frequency (); + update_ssa (TODO_update_ssa); + + /* Local pure-const may imply need to fixup the cfg. */ + if (execute_fixup_cfg () & TODO_cleanup_cfg) + cleanup_tree_cfg (); + + free_dominance_info (CDI_DOMINATORS); + free_dominance_info (CDI_POST_DOMINATORS); + cgraph_edge::rebuild_edges (); + pop_cfun (); + } + + return TODO_rebuild_cgraph_edges; +} +} /* namespace autofdo. */ + +/* Read the profile from the profile data file. */ + +void +read_autofdo_file (void) +{ + if (auto_profile_file == NULL) + auto_profile_file = DEFAULT_AUTO_PROFILE_FILE; + + autofdo::afdo_profile_info = (struct gcov_ctr_summary *)xcalloc ( + 1, sizeof (struct gcov_ctr_summary)); + autofdo::afdo_profile_info->runs = 1; + autofdo::afdo_profile_info->sum_max = 0; + autofdo::afdo_profile_info->sum_all = 0; + + /* Read the profile from the profile file. */ + autofdo::read_profile (); +} + +/* Free the resources. */ + +void +end_auto_profile (void) +{ + delete autofdo::afdo_source_profile; + delete autofdo::afdo_string_table; + profile_info = NULL; +} + +/* Returns TRUE if EDGE is hot enough to be inlined early. */ + +bool +afdo_callsite_hot_enough_for_early_inline (struct cgraph_edge *edge) +{ + gcov_type count + = autofdo::afdo_source_profile->get_callsite_total_count (edge); + if (count > 0) + { + bool is_hot; + const struct gcov_ctr_summary *saved_profile_info = profile_info; + /* At earling inline stage, profile_info is not set yet. We need to + temporarily set it to afdo_profile_info to calculate hotness. */ + profile_info = autofdo::afdo_profile_info; + is_hot = maybe_hot_count_p (NULL, count); + profile_info = saved_profile_info; + return is_hot; + } + else + return false; +} + +namespace +{ + +const pass_data pass_data_ipa_auto_profile = { + SIMPLE_IPA_PASS, "afdo", /* name */ + OPTGROUP_NONE, /* optinfo_flags */ + TV_IPA_AUTOFDO, /* tv_id */ + 0, /* properties_required */ + 0, /* properties_provided */ + 0, /* properties_destroyed */ + 0, /* todo_flags_start */ + 0, /* todo_flags_finish */ +}; + +class pass_ipa_auto_profile : public simple_ipa_opt_pass +{ +public: + pass_ipa_auto_profile (gcc::context *ctxt) + : simple_ipa_opt_pass (pass_data_ipa_auto_profile, ctxt) + { + } + + /* opt_pass methods: */ + virtual bool + gate (function *) + { + return flag_auto_profile; + } + virtual unsigned int + execute (function *) + { + return autofdo::auto_profile (); + } +}; // class pass_ipa_auto_profile + +} // anon namespace + +simple_ipa_opt_pass * +make_pass_ipa_auto_profile (gcc::context *ctxt) +{ + return new pass_ipa_auto_profile (ctxt); +} Index: gcc/auto-profile.h =================================================================== --- gcc/auto-profile.h (revision 0) +++ gcc/auto-profile.h (revision 0) @@ -0,0 +1,31 @@ +/* auto-profile.h - Defines data exported from auto-profile.c + Copyright (C) 2014. Free Software Foundation, Inc. + Contributed by Dehao Chen (dehao@google.com) + +This file is part of GCC. + +GCC is free software; you can redistribute it and/or modify it under +the terms of the GNU General Public License as published by the Free +Software Foundation; either version 3, or (at your option) any later +version. + +GCC is distributed in the hope that it will be useful, but WITHOUT ANY +WARRANTY; without even the implied warranty of MERCHANTABILITY or +FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License +for more details. + +You should have received a copy of the GNU General Public License +along with GCC; see the file COPYING3. If not see +<http://www.gnu.org/licenses/>. */ + +#ifndef AUTO_PROFILE_H +#define AUTO_PROFILE_H + +/* Read, process, finalize AutoFDO data structures. */ +extern void read_autofdo_file (void); +extern void end_auto_profile (void); + +/* Returns TRUE if EDGE is hot enough to be inlined early. */ +extern bool afdo_callsite_hot_enough_for_early_inline (struct cgraph_edge *); + +#endif /* AUTO_PROFILE_H */ Index: gcc/basic-block.h =================================================================== --- gcc/basic-block.h (revision 215826) +++ gcc/basic-block.h (working copy) @@ -724,6 +724,7 @@ extern struct edge_list *pre_edge_rev_lcm (int, sb extern void compute_available (sbitmap *, sbitmap *, sbitmap *, sbitmap *); /* In predict.c */ +extern bool maybe_hot_count_p (struct function *, gcov_type); extern bool maybe_hot_bb_p (struct function *, const_basic_block); extern bool maybe_hot_edge_p (edge); extern bool probably_never_executed_bb_p (struct function *, const_basic_block); Index: gcc/gcov-io.h =================================================================== --- gcc/gcov-io.h (revision 215826) +++ gcc/gcov-io.h (working copy) @@ -244,6 +244,9 @@ typedef uint64_t gcov_type_unsigned; #define GCOV_TAG_PROGRAM_SUMMARY ((gcov_unsigned_t)0xa3000000) #define GCOV_TAG_SUMMARY_LENGTH(NUM) \ (1 + GCOV_COUNTERS_SUMMABLE * (10 + 3 * 2) + (NUM) * 5) +#define GCOV_TAG_AFDO_FILE_NAMES ((gcov_unsigned_t)0xaa000000) +#define GCOV_TAG_AFDO_FUNCTION ((gcov_unsigned_t)0xac000000) +#define GCOV_TAG_AFDO_WORKING_SET ((gcov_unsigned_t)0xaf000000) /* Counters that are collected. */ Index: gcc/opts.c =================================================================== --- gcc/opts.c (revision 215826) +++ gcc/opts.c (working copy) @@ -1279,6 +1279,55 @@ print_specific_help (unsigned int include_flags, opts->x_help_columns, opts, lang_mask); } +static void +enable_fdo_optimizations (struct gcc_options *opts, + struct gcc_options *opts_set, + int value) +{ + if (!opts_set->x_flag_branch_probabilities) + opts->x_flag_branch_probabilities = value; + if (!opts_set->x_flag_profile_values) + opts->x_flag_profile_values = value; + if (!opts_set->x_flag_unroll_loops) + opts->x_flag_unroll_loops = value; + if (!opts_set->x_flag_peel_loops) + opts->x_flag_peel_loops = value; + if (!opts_set->x_flag_tracer) + opts->x_flag_tracer = value; + if (!opts_set->x_flag_value_profile_transformations) + opts->x_flag_value_profile_transformations = value; + if (!opts_set->x_flag_inline_functions) + opts->x_flag_inline_functions = value; + if (!opts_set->x_flag_ipa_cp) + opts->x_flag_ipa_cp = value; + if (!opts_set->x_flag_ipa_cp_clone + && value && opts->x_flag_ipa_cp) + opts->x_flag_ipa_cp_clone = value; + if (!opts_set->x_flag_predictive_commoning) + opts->x_flag_predictive_commoning = value; + if (!opts_set->x_flag_unswitch_loops) + opts->x_flag_unswitch_loops = value; + if (!opts_set->x_flag_gcse_after_reload) + opts->x_flag_gcse_after_reload = value; + if (!opts_set->x_flag_tree_loop_vectorize + && !opts_set->x_flag_tree_vectorize) + opts->x_flag_tree_loop_vectorize = value; + if (!opts_set->x_flag_tree_slp_vectorize + && !opts_set->x_flag_tree_vectorize) + opts->x_flag_tree_slp_vectorize = value; + if (!opts_set->x_flag_vect_cost_model) + opts->x_flag_vect_cost_model = VECT_COST_MODEL_DYNAMIC; + if (!opts_set->x_flag_tree_loop_distribute_patterns) + opts->x_flag_tree_loop_distribute_patterns = value; + if (!opts_set->x_flag_profile_reorder_functions) + opts->x_flag_profile_reorder_functions = value; + /* Indirect call profiling should do all useful transformations + speculative devirtualization does. */ + if (!opts_set->x_flag_devirtualize_speculatively + && opts->x_flag_value_profile_transformations) + opts->x_flag_devirtualize_speculatively = false; +} + /* Handle target- and language-independent options. Return zero to generate an "unknown option" message. Only options that need extra handling need to be listed here; if you simply want @@ -1746,50 +1795,23 @@ common_handle_option (struct gcc_options *opts, value = true; /* No break here - do -fprofile-use processing. */ case OPT_fprofile_use: - if (!opts_set->x_flag_branch_probabilities) - opts->x_flag_branch_probabilities = value; - if (!opts_set->x_flag_profile_values) - opts->x_flag_profile_values = value; - if (!opts_set->x_flag_unroll_loops) - opts->x_flag_unroll_loops = value; - if (!opts_set->x_flag_peel_loops) - opts->x_flag_peel_loops = value; - if (!opts_set->x_flag_tracer) - opts->x_flag_tracer = value; - if (!opts_set->x_flag_value_profile_transformations) - opts->x_flag_value_profile_transformations = value; - if (!opts_set->x_flag_inline_functions) - opts->x_flag_inline_functions = value; - if (!opts_set->x_flag_ipa_cp) - opts->x_flag_ipa_cp = value; - if (!opts_set->x_flag_ipa_cp_clone - && value && opts->x_flag_ipa_cp) - opts->x_flag_ipa_cp_clone = value; - if (!opts_set->x_flag_predictive_commoning) - opts->x_flag_predictive_commoning = value; - if (!opts_set->x_flag_unswitch_loops) - opts->x_flag_unswitch_loops = value; - if (!opts_set->x_flag_gcse_after_reload) - opts->x_flag_gcse_after_reload = value; - if (!opts_set->x_flag_tree_loop_vectorize - && !opts_set->x_flag_tree_vectorize) - opts->x_flag_tree_loop_vectorize = value; - if (!opts_set->x_flag_tree_slp_vectorize - && !opts_set->x_flag_tree_vectorize) - opts->x_flag_tree_slp_vectorize = value; - if (!opts_set->x_flag_vect_cost_model) - opts->x_flag_vect_cost_model = VECT_COST_MODEL_DYNAMIC; - if (!opts_set->x_flag_tree_loop_distribute_patterns) - opts->x_flag_tree_loop_distribute_patterns = value; - if (!opts_set->x_flag_profile_reorder_functions) - opts->x_flag_profile_reorder_functions = value; - /* Indirect call profiling should do all useful transformations - speculative devirtualization does. */ - if (!opts_set->x_flag_devirtualize_speculatively - && opts->x_flag_value_profile_transformations) - opts->x_flag_devirtualize_speculatively = false; + enable_fdo_optimizations (opts, opts_set, value); break; + case OPT_fauto_profile_: + opts->x_auto_profile_file = xstrdup (arg); + opts->x_flag_auto_profile = true; + value = true; + /* No break here - do -fauto-profile processing. */ + case OPT_fauto_profile: + enable_fdo_optimizations (opts, opts_set, value); + if (!opts_set->x_flag_profile_correction) + opts->x_flag_profile_correction = value; + maybe_set_param_value ( + PARAM_EARLY_INLINER_MAX_ITERATIONS, 10, + opts->x_param_values, opts_set->x_param_values); + break; + case OPT_fprofile_generate_: opts->x_profile_data_prefix = xstrdup (arg); value = true; Index: gcc/ipa-inline.c =================================================================== --- gcc/ipa-inline.c (revision 215826) +++ gcc/ipa-inline.c (working copy) @@ -121,6 +121,7 @@ along with GCC; see the file COPYING3. If not see #include "ipa-inline.h" #include "ipa-utils.h" #include "sreal.h" +#include "auto-profile.h" #include "cilk.h" #include "builtins.h" @@ -449,6 +450,14 @@ want_early_inline_function_p (struct cgraph_edge * if (DECL_DISREGARD_INLINE_LIMITS (callee->decl)) ; + /* For AutoFDO, we need to make sure that before profile annotation, all + hot paths' IR look exactly the same as profiled binary. As a result, + in einliner, we will disregard size limit and inline those callsites + that are: + * inlined in the profiled binary, and + * the cloned callee has enough samples to be considered "hot". */ + else if (flag_auto_profile && afdo_callsite_hot_enough_for_early_inline (e)) + ; else if (!DECL_DECLARED_INLINE_P (callee->decl) && !flag_inline_small_functions) { @@ -2366,39 +2375,8 @@ early_inline_small_functions (struct cgraph_node * return inlined; } -/* Do inlining of small functions. Doing so early helps profiling and other - passes to be somewhat more effective and avoids some code duplication in - later real inlining pass for testcases with very many function calls. */ - -namespace { - -const pass_data pass_data_early_inline = -{ - GIMPLE_PASS, /* type */ - "einline", /* name */ - OPTGROUP_INLINE, /* optinfo_flags */ - TV_EARLY_INLINING, /* tv_id */ - PROP_ssa, /* properties_required */ - 0, /* properties_provided */ - 0, /* properties_destroyed */ - 0, /* todo_flags_start */ - 0, /* todo_flags_finish */ -}; - -class pass_early_inline : public gimple_opt_pass -{ -public: - pass_early_inline (gcc::context *ctxt) - : gimple_opt_pass (pass_data_early_inline, ctxt) - {} - - /* opt_pass methods: */ - virtual unsigned int execute (function *); - -}; // class pass_early_inline - unsigned int -pass_early_inline::execute (function *fun) +early_inliner (function *fun) { struct cgraph_node *node = cgraph_node::get (current_function_decl); struct cgraph_edge *edge; @@ -2499,6 +2477,43 @@ unsigned int return todo; } +/* Do inlining of small functions. Doing so early helps profiling and other + passes to be somewhat more effective and avoids some code duplication in + later real inlining pass for testcases with very many function calls. */ + +namespace { + +const pass_data pass_data_early_inline = +{ + GIMPLE_PASS, /* type */ + "einline", /* name */ + OPTGROUP_INLINE, /* optinfo_flags */ + TV_EARLY_INLINING, /* tv_id */ + PROP_ssa, /* properties_required */ + 0, /* properties_provided */ + 0, /* properties_destroyed */ + 0, /* todo_flags_start */ + 0, /* todo_flags_finish */ +}; + +class pass_early_inline : public gimple_opt_pass +{ +public: + pass_early_inline (gcc::context *ctxt) + : gimple_opt_pass (pass_data_early_inline, ctxt) + {} + + /* opt_pass methods: */ + virtual unsigned int execute (function *); + +}; // class pass_early_inline + +unsigned int +pass_early_inline::execute (function *fun) +{ + return early_inliner (fun); +} + } // anon namespace gimple_opt_pass * Index: gcc/ipa-inline.h =================================================================== --- gcc/ipa-inline.h (revision 215826) +++ gcc/ipa-inline.h (working copy) @@ -238,6 +238,7 @@ void initialize_growth_caches (void); void free_growth_caches (void); void compute_inline_parameters (struct cgraph_node *, bool); bool speculation_useful_p (struct cgraph_edge *e, bool anticipate_inlining); +unsigned int early_inliner (function *fun); /* In ipa-inline-transform.c */ bool inline_call (struct cgraph_edge *, bool, vec<cgraph_edge *> *, int *, bool, Index: gcc/predict.c =================================================================== --- gcc/predict.c (revision 215826) +++ gcc/predict.c (working copy) @@ -161,7 +161,7 @@ set_hot_bb_threshold (gcov_type min) /* Return TRUE if frequency FREQ is considered to be hot. */ -static inline bool +bool maybe_hot_count_p (struct function *fun, gcov_type count) { if (fun && profile_status_for_fn (fun) != PROFILE_READ) @@ -2852,7 +2852,7 @@ counts_to_freqs (void) /* Don't overwrite the estimated frequencies when the profile for the function is missing. We may drop this function PROFILE_GUESSED later in drop_profile (). */ - if (!ENTRY_BLOCK_PTR_FOR_FN (cfun)->count) + if (!flag_auto_profile && !ENTRY_BLOCK_PTR_FOR_FN (cfun)->count) return 0; FOR_BB_BETWEEN (bb, ENTRY_BLOCK_PTR_FOR_FN (cfun), NULL, next_bb) @@ -3223,7 +3223,8 @@ rebuild_frequencies (void) count_max = MAX (bb->count, count_max); if (profile_status_for_fn (cfun) == PROFILE_GUESSED - || (profile_status_for_fn (cfun) == PROFILE_READ && count_max < REG_BR_PROB_BASE/10)) + || (!flag_auto_profile && profile_status_for_fn (cfun) == PROFILE_READ + && count_max < REG_BR_PROB_BASE/10)) { loop_optimizer_init (0); add_noreturn_fake_exit_edges (); Index: gcc/tree-profile.c =================================================================== --- gcc/tree-profile.c (revision 215826) +++ gcc/tree-profile.c (working copy) @@ -699,8 +699,10 @@ class pass_ipa_tree_profile : public simple_ipa_op bool pass_ipa_tree_profile::gate (function *) { - /* When profile instrumentation, use or test coverage shall be performed. */ - return (!in_lto_p + /* When profile instrumentation, use or test coverage shall be performed. + But for AutoFDO, this there is no instrumentation, thus this pass is + diabled. */ + return (!in_lto_p && !flag_auto_profile && (flag_branch_probabilities || flag_test_coverage || profile_arc_flag)); } Index: gcc/profile.c =================================================================== --- gcc/profile.c (revision 215826) +++ gcc/profile.c (working copy) @@ -106,6 +106,14 @@ static int total_num_times_called; static int total_hist_br_prob[20]; static int total_num_branches; +/* Helper function to update gcov_working_sets. */ + +void add_working_set (gcov_working_set_t *set) { + int i = 0; + for (; i < NUM_GCOV_WORKING_SETS; i++) + gcov_working_sets[i] = set[i]; +} + /* Forward declarations. */ static void find_spanning_tree (struct edge_list *); Index: gcc/Makefile.in =================================================================== --- gcc/Makefile.in (revision 215826) +++ gcc/Makefile.in (working copy) @@ -1153,6 +1153,7 @@ OBJS = \ alias.o \ alloc-pool.o \ auto-inc-dec.o \ + auto-profile.o \ bb-reorder.o \ bitmap.o \ bt-load.o \ Index: gcc/profile.h =================================================================== --- gcc/profile.h (revision 215826) +++ gcc/profile.h (working copy) @@ -47,6 +47,7 @@ extern void init_node_map (bool); extern void del_node_map (void); extern void get_working_sets (void); +extern void add_working_set (gcov_working_set_t *); /* In predict.c. */ extern gcov_type get_hot_bb_threshold (void); Index: gcc/debug.h =================================================================== --- gcc/debug.h (revision 215826) +++ gcc/debug.h (working copy) @@ -169,6 +169,7 @@ extern const struct gcc_debug_hooks dbx_debug_hook extern const struct gcc_debug_hooks sdb_debug_hooks; extern const struct gcc_debug_hooks xcoff_debug_hooks; extern const struct gcc_debug_hooks dwarf2_debug_hooks; +extern const struct gcc_debug_hooks auto_profile_debug_hooks; extern const struct gcc_debug_hooks vmsdbg_debug_hooks; /* Dwarf2 frame information. */ Index: gcc/passes.def =================================================================== --- gcc/passes.def (revision 215826) +++ gcc/passes.def (working copy) @@ -90,6 +90,7 @@ along with GCC; see the file COPYING3. If not see NEXT_PASS (pass_rebuild_cgraph_edges); NEXT_PASS (pass_inline_parameters); POP_INSERT_PASSES () + NEXT_PASS (pass_ipa_auto_profile); NEXT_PASS (pass_ipa_free_inline_summary); NEXT_PASS (pass_ipa_tree_profile); PUSH_INSERT_PASSES_WITHIN (pass_ipa_tree_profile) ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH] AutoFDO patch for trunk 2014-10-14 23:28 ` Dehao Chen @ 2014-10-15 2:57 ` Andi Kleen 2014-10-15 14:36 ` Jan Hubicka 1 sibling, 0 replies; 25+ messages in thread From: Andi Kleen @ 2014-10-15 2:57 UTC (permalink / raw) To: Dehao Chen; +Cc: Jan Hubicka, GCC Patches, David Li Dehao Chen <dehao@google.com> writes: > + > +@item -fauto-profile > +@itemx -fauto-profile=@var{path} > +@opindex fauto-profile > +Enable sampling based feedback directed optimizations, and optimizations > +generally profitable only with profile feedback available. > + > +The following options are enabled: @code{-fbranch-probabilities}, @code{-fvpt}, > +@code{-funroll-loops}, @code{-fpeel-loops}, @code{-ftracer}, @code{-ftree-vectorize}, > +@code{ftree-loop-distribute-patterns} This needs more description aimed end-users, what it is good for and why, and a pointer to the needed utilities and a short summary what steps they need to take. -Andi -- ak@linux.intel.com -- Speaking for myself only ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH] AutoFDO patch for trunk 2014-10-14 23:28 ` Dehao Chen 2014-10-15 2:57 ` Andi Kleen @ 2014-10-15 14:36 ` Jan Hubicka 2014-10-16 18:57 ` Dehao Chen 1 sibling, 1 reply; 25+ messages in thread From: Jan Hubicka @ 2014-10-15 14:36 UTC (permalink / raw) To: Dehao Chen; +Cc: Jan Hubicka, GCC Patches, David Li > Index: gcc/cfgloop.c > =================================================================== > --- gcc/cfgloop.c (revision 215826) > +++ gcc/cfgloop.c (working copy) > @@ -1802,7 +1802,7 @@ record_niter_bound (struct loop *loop, const wides > } > if (realistic > && (!loop->any_estimate > - || wi::ltu_p (i_bound, loop->nb_iterations_estimate))) > + || (!flag_auto_profile && wi::ltu_p (i_bound, loop->nb_iterations_estimate)))) I do not remember this hunk from previous patches. So you make autofdo to drop the conditional that new estimate must be smaller than preivously derived. I suppose it is because you want autofdo estimates to be overwritten by anything determined later. Lets drop this hunk now, too. I think we really want to have way to track if the estimate is from autofdo and thus "not too reliable but better than nothing" and we may want to introduce some kind of reliability metric on way estimates are derived. Lets handle this incrementally and drop this change from this patch. > Index: gcc/doc/invoke.texi > =================================================================== > --- gcc/doc/invoke.texi (revision 215826) > +++ gcc/doc/invoke.texi (working copy) > @@ -365,7 +365,8 @@ Objective-C and Objective-C++ Dialects}. > @gccoptlist{-faggressive-loop-optimizations -falign-functions[=@var{n}] @gol > -falign-jumps[=@var{n}] @gol > -falign-labels[=@var{n}] -falign-loops[=@var{n}] @gol > --fassociative-math -fauto-inc-dec -fbranch-probabilities @gol > +-fassociative-math -fauto-profile -fauto-profile[=@var{path}] @gol > +-fauto-inc-dec -fbranch-probabilities @gol > -fbranch-target-load-optimize -fbranch-target-load-optimize2 @gol > -fbtr-bb-exclusive -fcaller-saves @gol > -fcheck-data-deps -fcombine-stack-adjustments -fconserve-stack @gol > @@ -9164,6 +9165,19 @@ code. > > If @var{path} is specified, GCC looks at the @var{path} to find > the profile feedback data files. See @option{-fprofile-dir}. > + > +@item -fauto-profile > +@itemx -fauto-profile=@var{path} > +@opindex fauto-profile > +Enable sampling based feedback directed optimizations, and optimizations > +generally profitable only with profile feedback available. > + > +The following options are enabled: @code{-fbranch-probabilities}, @code{-fvpt}, > +@code{-funroll-loops}, @code{-fpeel-loops}, @code{-ftracer}, @code{-ftree-vectorize}, > +@code{ftree-loop-distribute-patterns} > + > +If @var{path} is specified, GCC looks at the @var{path} to find > +the profile feedback data files. I agree with Andi that you need to write more here - basically it should be clear how to profile the application and use the data. We do use external tools here (oprofile/gperf/non-Linux tools I suppose) plus an conversion tool. So you need to provide some references to those. Actually I think it would be really nice to add tutorial-like chapter into GCC manual about using FDO/autoFDO to improve your code or even more general aimed covering advanced topics like correct LTO use, profile feedback and other fun. > Index: gcc/value-prof.c > =================================================================== > --- gcc/value-prof.c (revision 215826) > +++ gcc/value-prof.c (working copy) > @@ -134,11 +134,10 @@ static bool gimple_divmod_fixed_value_transform (g > static bool gimple_mod_pow2_value_transform (gimple_stmt_iterator *); > static bool gimple_mod_subtract_transform (gimple_stmt_iterator *); > static bool gimple_stringops_transform (gimple_stmt_iterator *); > -static bool gimple_ic_transform (gimple_stmt_iterator *); > > /* Allocate histogram value. */ > > -static histogram_value > +histogram_value > gimple_alloc_histogram_value (struct function *fun ATTRIBUTE_UNUSED, > enum hist_type type, gimple stmt, tree value) Do we still need to expor those? > { > @@ -1307,6 +1306,9 @@ del_node_map (void) > struct cgraph_node* > find_func_by_profile_id (int profile_id) > { > + if (flag_auto_profile) > + return cgraph_node::get_for_asmname ( > + get_identifier ((const char *)(size_t)profile_id)); This definitly needs an comment. Why you need to use find_func_by_profile_id on both gcov profiling and auto profiling paths? Perhaps you can use your own fnuction here? Or perhaps this is not necessary if you implement directly the indirect call optimiization? We may try to head towards of having way to read both coverage and auto profile into compiler eventually. It may be useful to get extra precision from gcov profile and to measure data like cache misses with auto profile. > cgraph_node **val = cgraph_node_map->get (profile_id); > if (val) > return *val; > @@ -1320,7 +1322,7 @@ find_func_by_profile_id (int profile_id) > may ICE. Here we only do very minimal sanity check just to make compiler happy. > Returns true if TARGET is considered ok for call CALL_STMT. */ > > -static bool > +bool > check_ic_target (gimple call_stmt, struct cgraph_node *target) > { > location_t locus; > @@ -1483,7 +1485,7 @@ gimple_ic (gimple icall_stmt, struct cgraph_node * > this call to: > */ > > -static bool > +bool > gimple_ic_transform (gimple_stmt_iterator *gsi) I do not think those needs to be exported. > { > gimple stmt = gsi_stmt (*gsi); > + > +/* The following routines implements AutoFDO optimization. > + > + This optimization uses sampling profiles to annotate basic block counts > + and uses heuristics to estimate branch probabilities. > + > + There are three phases in AutoFDO: > + > + Phase 1: Read profile from the profile data file. > + The following info is read from the profile datafile: > + * string_table: a map between function name and its index. > + * autofdo_source_profile: a map from function_instance name to > + function_instance. This is represented as a forest of > + function_instances. This description does not seem to make it very clear what autofdo_source_profile is. Perhaps expand a bit more about what function instance is? > + * WorkingSet: a histogram of how many instructions are covered for a > + given percentage of total cycles. Here I think a bit extra explanation would be nice. Working sets are per-line information? > + > + Phase 2: Early inline + VPT. Probably expand VPT - it requires one to be quite familiar with GCC sources to follow 3 letter acronyms ;) > + Early inline uses autofdo_source_profile to find if a callsite is: > + * inlined in the profiled binary. > + * callee body is hot in the profiling run. > + If both condition satisfies, early inline will inline the callsite > + regardless of the code growth. > + Phase 2 is an iterative process. During each iteration, we also check > + if an indirect callsite is promoted and inlined in the profiling run. > + If yes, vpt will happen to force promote it and in the next iteration, > + einline will inline the promoted callsite in the next iteration. Perhaps "If yes, direct speculative call is introduced" given that speculative edges are now kind of first class players here. > + > + Phase 3: Annotate control flow graph. > + AutoFDO uses a separate pass to: > + * Annotate basic block count * Annotate basic blocks with count? > + * Estimate branch probability * Propagate counts across callgraph to fill in information missing in the profile. You probably could expand a bit how profile is driven by source lines and thus may completely miss some basic blocks. > + > + After the above 3 phases, all profile is readily annotated on the GCC IR. > + AutoFDO tries to reuse all FDO infrastructure as much as possible to make > + use of the profile. E.g. it uses existing mechanism to calculate the basic > + block/edge frequency, as well as the cgraph node/edge count. > +*/ > + > +#define DEFAULT_AUTO_PROFILE_FILE "fbdata.afdo" > + > +namespace autofdo > +{ > + > +/* Represent a source location: (function_decl, lineno). */ > +typedef std::pair<tree, unsigned> decl_lineno; > + > +/* Represent an inline stack. vector[0] is the leaf node. */ > +typedef std::vector<decl_lineno> inline_stack; > + > +/* String array that stores function names. */ > +typedef std::vector<const char *> string_vector; I am not sure about use of std:: stuff in GCC. Basically I do not see anything really bad about it, just in case we have our own equivalents, we may stick of using those. So perhaps using our vec would make more sense. > + > +/* Profile of a function instance: > + 1. total_count of the function. > + 2. head_count of the function (only valid when function is a top-level > + function_instance, i.e. it is the original copy instead of the > + inlined copy). Perhaps explain what is head count? > + 3. map from source location (decl_lineno) of the inlined callsite to > + profile (count_info). Of inline callsite? I would expect this to be actual line profile of the function. > +/* Helper functions. */ > + > +/* Return the original name of NAME: strip the suffix that starts > + with '.' */ > + > +static const char * > +get_original_name (const char *name) > +{ > + char *ret = xstrdup (name); > + char *find = strchr (ret, '.'); > + if (find != NULL) > + *find = 0; > + return ret; > +} Caller is actuallly responsible for freeing RET, right? Perhaps it should be mentioned in the comment and actually done in uses. > + > +/* Return the combined location, which is a 32bit integer in which > + higher 16 bits stores the line offset of LOC to the start lineno > + of DECL, The lower 16 bits stores the discrimnator. */ > + > +static unsigned > +get_combined_location (location_t loc, tree decl) > +{ > + /* TODO: allow more bits for line and less bits for discriminator. */ > + return ((LOCATION_LINE (loc) - DECL_SOURCE_LINE (decl)) << 16); > +} We may consider dropping a message into dump file making clear that overflow happened. > + > +/* Return the function decl of a given lexical BLOCK. */ > + > +static tree > +get_function_decl_from_block (tree block) > +{ > + tree decl; > + > + if (LOCATION_LOCUS (BLOCK_SOURCE_LOCATION (block) == UNKNOWN_LOCATION)) > + return NULL_TREE; > + > + for (decl = BLOCK_ABSTRACT_ORIGIN (block); > + decl && (TREE_CODE (decl) == BLOCK); > + decl = BLOCK_ABSTRACT_ORIGIN (decl)) > + if (TREE_CODE (decl) == FUNCTION_DECL) > + break; > + return decl; > +} > + > +/* Store inline stack for STMT in STACK. */ > + > +static void > +get_inline_stack (location_t locus, inline_stack *stack) > +{ > + if (LOCATION_LOCUS (locus) == UNKNOWN_LOCATION) > + return; > + > + tree block = LOCATION_BLOCK (locus); > + if (block && TREE_CODE (block) == BLOCK) > + { > + int level = 0; > + for (block = BLOCK_SUPERCONTEXT (block); > + block && (TREE_CODE (block) == BLOCK); > + block = BLOCK_SUPERCONTEXT (block)) > + { > + location_t tmp_locus = BLOCK_SOURCE_LOCATION (block); > + if (LOCATION_LOCUS (tmp_locus) == UNKNOWN_LOCATION) > + continue; > + > + tree decl = get_function_decl_from_block (block); > + stack->push_back ( > + std::make_pair (decl, get_combined_location (locus, decl))); OK, you walk block tree up and for each block you want to lookup the function context it originate from and push it to stack. Won't you get extra NULL entries for inner blocks within the functions? > + > +/* Member functions for string_table. */ > + > +string_table * > +string_table::create () Why this is not a constructor? > +{ > + string_table *map = new string_table (); > + if (map->read ()) > + return map; > + delete map; > + return NULL; > +} > + > +/* Return the index of a given function NAME. */ Perhaps document that function returns -1 when index is not found. > + > +int > +string_table::get_index (const char *name) const > +{ > + if (name == NULL) > + return -1; > + string_index_map::const_iterator iter = map_.find (name); > + if (iter == map_.end ()) > + return -1; > + else > + return iter->second; > +} > + > +/* Return the index of a given function DECL. */ Similarly here. > + > +/* Traverse callsites of the current function_instance to find one at the > + location of LINENO and callee name represented in DECL. */ > + > +function_instance * > +function_instance::get_function_instance_by_decl (unsigned lineno, > + tree decl) const > +{ > + int func_name_idx = afdo_string_table->get_index_by_decl (decl); > + if (func_name_idx != -1) > + { > + callsite_map::const_iterator ret > + = callsites.find (std::make_pair (lineno, func_name_idx)); > + if (ret != callsites.end ()) > + return ret->second; > + } > + func_name_idx > + = afdo_string_table->get_index (lang_hooks.dwarf_name (decl, 0)); > + if (func_name_idx != -1) > + { > + callsite_map::const_iterator ret > + = callsites.find (std::make_pair (lineno, func_name_idx)); > + if (ret != callsites.end ()) > + return ret->second; > + } The two hunks probably can be unified. Why get_index_by_decl does not already use dwarf name and why do you need to consider both? > + if (DECL_ABSTRACT_ORIGIN (decl)) > + return get_function_instance_by_decl (lineno, DECL_ABSTRACT_ORIGIN (decl)); What really happens for ipa-cp function clones and for split functions? > + else > + return NULL; > +} > + > +/* Recursively traverse STACK starting from LEVEL to find the corresponding > + function_instance. */ > + > +function_instance * > +function_instance::get_function_instance (const inline_stack &stack, > + unsigned level) > +{ > + if (level == 0) > + return this; > + function_instance *s = get_function_instance_by_decl ( > + stack[level].second, stack[level - 1].first); > + if (s) > + return s->get_function_instance (stack, level - 1); > + else > + return NULL; Someting that probably could be written by iteration ;) > + > +/* Read the profile and create a function_instance with head count as > + HEAD_COUNT. Recursively read callsites to create nested function_instances > + too. STACK is used to track the recursive creation process. */ > + > +/* function instance profile format: > + > + ENTRY_COUNT: 8 bytes > + NAME_INDEX: 4 bytes > + NUM_POS_COUNTS: 4 bytes > + NUM_CALLSITES: 4 byte > + POS_COUNT_1: > + POS_1_OFFSET: 4 bytes > + NUM_TARGETS: 4 bytes > + COUNT: 8 bytes > + TARGET_1: > + VALUE_PROFILE_TYPE: 4 bytes > + TARGET_IDX: 8 bytes > + COUNT: 8 bytes So basically you have list of statements identified by offsets that have counts with them. If the statement is indirect call, then you have numb targets and you have the value profiles associated with them, right? I am not sure it is a win to have the things represented as VPT histograms when you shoot for quite special purpose problem. But lets deal with these incrementally, too. (in general I do not see why you try to share so much with the gcov based VTP - it seems easier to do the speculation by hand) > + > +static void > +read_profile (void) Block comment here, even though it is quite obvious ;) > +{ > + if (gcov_open (auto_profile_file, 1) == 0) > + error ("Cannot open profile file %s.", auto_profile_file); > + > + if (gcov_read_unsigned () != GCOV_DATA_MAGIC) > + error ("AutoFDO profile magic number does not mathch."); > + > + /* Skip the version number. */ > + gcov_read_unsigned (); > + > + /* Skip the empty integer. */ > + gcov_read_unsigned (); Perhaps we can just check the values? > + > + /* string_table. */ > + afdo_string_table = string_table::create (); > + if (afdo_string_table == NULL) > + error ("Cannot read string table from %s.", auto_profile_file); > + > + /* autofdo_source_profile. */ > + afdo_source_profile = autofdo_source_profile::create (); > + if (afdo_source_profile == NULL) > + error ("Cannot read function profile from %s.", auto_profile_file); > + > + /* autofdo_module_profile. */ > + fake_read_autofdo_module_profile (); > + > + /* Read in the working set. */ > + if (gcov_read_unsigned () != GCOV_TAG_AFDO_WORKING_SET) > + error ("Cannot read working set from %s.", auto_profile_file); > + > + /* Skip the length of the section. */ > + gcov_read_unsigned (); > + gcov_working_set_t set[128]; > + for (unsigned i = 0; i < 128; i++) > + { > + set[i].num_counters = gcov_read_unsigned (); > + set[i].min_counter = gcov_read_counter (); > + } > + add_working_set (set); > +} > + > +/* From AutoFDO profiles, find values inside STMT for that we want to measure > + histograms for indirect-call optimization. */ This function is actually served for 2 purposes: * before annotation, we need to mark histogram, promote and inline * after annotation, we just need to mark, and let follow-up logic to decide if it needs to promote and inline. Perhaps add it to the block comment? And you are right, for the "before annotation" case, we can simply call "mark speculative" and "inline". But we still need the logic to fake histogram for "after annotation" case. As a result, I unified two cases into one function to reuse code as much as possible. Shall I separate it into two functions instead? Who is using the histogram later? Is it the normal ipa-profile code? In that case it probably makes sense to produce the histogram indeed. You sitll can probably avoid using gimple_ic_transform. Because you fake the histogram, the early checks in function will not lead to anything useful. You probably want to check yourself check_ic_target Does this have chance to work inter-module? > Index: gcc/opts.c > =================================================================== > --- gcc/opts.c (revision 215826) > +++ gcc/opts.c (working copy) > @@ -1279,6 +1279,55 @@ print_specific_help (unsigned int include_flags, > opts->x_help_columns, opts, lang_mask); > } > > +static void > +enable_fdo_optimizations (struct gcc_options *opts, > + struct gcc_options *opts_set, > + int value) Block comment here. > +{ > + if (!opts_set->x_flag_branch_probabilities) > + opts->x_flag_branch_probabilities = value; > + if (!opts_set->x_flag_profile_values) > + opts->x_flag_profile_values = value; > + if (!opts_set->x_flag_unroll_loops) > + opts->x_flag_unroll_loops = value; > + if (!opts_set->x_flag_peel_loops) > + opts->x_flag_peel_loops = value; > + if (!opts_set->x_flag_tracer) > + opts->x_flag_tracer = value; > + if (!opts_set->x_flag_value_profile_transformations) > + opts->x_flag_value_profile_transformations = value; > + if (!opts_set->x_flag_inline_functions) > + opts->x_flag_inline_functions = value; > + if (!opts_set->x_flag_ipa_cp) > + opts->x_flag_ipa_cp = value; > + if (!opts_set->x_flag_ipa_cp_clone > + && value && opts->x_flag_ipa_cp) > + opts->x_flag_ipa_cp_clone = value; > + if (!opts_set->x_flag_predictive_commoning) > + opts->x_flag_predictive_commoning = value; > + if (!opts_set->x_flag_unswitch_loops) > + opts->x_flag_unswitch_loops = value; > + if (!opts_set->x_flag_gcse_after_reload) > + opts->x_flag_gcse_after_reload = value; > + if (!opts_set->x_flag_tree_loop_vectorize > + && !opts_set->x_flag_tree_vectorize) > + opts->x_flag_tree_loop_vectorize = value; > + if (!opts_set->x_flag_tree_slp_vectorize > + && !opts_set->x_flag_tree_vectorize) > + opts->x_flag_tree_slp_vectorize = value; > + if (!opts_set->x_flag_vect_cost_model) > + opts->x_flag_vect_cost_model = VECT_COST_MODEL_DYNAMIC; > + if (!opts_set->x_flag_tree_loop_distribute_patterns) > + opts->x_flag_tree_loop_distribute_patterns = value; > + if (!opts_set->x_flag_profile_reorder_functions) > + opts->x_flag_profile_reorder_functions = value; > + /* Indirect call profiling should do all useful transformations > + speculative devirtualization does. */ > + if (!opts_set->x_flag_devirtualize_speculatively > + && opts->x_flag_value_profile_transformations) > + opts->x_flag_devirtualize_speculatively = false; > +} > + > /* Handle target- and language-independent options. Return zero to > generate an "unknown option" message. Only options that need > extra handling need to be listed here; if you simply want > @@ -1746,50 +1795,23 @@ common_handle_option (struct gcc_options *opts, > value = true; > /* No break here - do -fprofile-use processing. */ > case OPT_fprofile_use: > - if (!opts_set->x_flag_branch_probabilities) > - opts->x_flag_branch_probabilities = value; > - if (!opts_set->x_flag_profile_values) > - opts->x_flag_profile_values = value; > - if (!opts_set->x_flag_unroll_loops) > - opts->x_flag_unroll_loops = value; > - if (!opts_set->x_flag_peel_loops) > - opts->x_flag_peel_loops = value; > - if (!opts_set->x_flag_tracer) > - opts->x_flag_tracer = value; > - if (!opts_set->x_flag_value_profile_transformations) > - opts->x_flag_value_profile_transformations = value; > - if (!opts_set->x_flag_inline_functions) > - opts->x_flag_inline_functions = value; > - if (!opts_set->x_flag_ipa_cp) > - opts->x_flag_ipa_cp = value; > - if (!opts_set->x_flag_ipa_cp_clone > - && value && opts->x_flag_ipa_cp) > - opts->x_flag_ipa_cp_clone = value; > - if (!opts_set->x_flag_predictive_commoning) > - opts->x_flag_predictive_commoning = value; > - if (!opts_set->x_flag_unswitch_loops) > - opts->x_flag_unswitch_loops = value; > - if (!opts_set->x_flag_gcse_after_reload) > - opts->x_flag_gcse_after_reload = value; > - if (!opts_set->x_flag_tree_loop_vectorize > - && !opts_set->x_flag_tree_vectorize) > - opts->x_flag_tree_loop_vectorize = value; > - if (!opts_set->x_flag_tree_slp_vectorize > - && !opts_set->x_flag_tree_vectorize) > - opts->x_flag_tree_slp_vectorize = value; > - if (!opts_set->x_flag_vect_cost_model) > - opts->x_flag_vect_cost_model = VECT_COST_MODEL_DYNAMIC; > - if (!opts_set->x_flag_tree_loop_distribute_patterns) > - opts->x_flag_tree_loop_distribute_patterns = value; > - if (!opts_set->x_flag_profile_reorder_functions) > - opts->x_flag_profile_reorder_functions = value; > - /* Indirect call profiling should do all useful transformations > - speculative devirtualization does. */ > - if (!opts_set->x_flag_devirtualize_speculatively > - && opts->x_flag_value_profile_transformations) > - opts->x_flag_devirtualize_speculatively = false; > + enable_fdo_optimizations (opts, opts_set, value); > break; > > + case OPT_fauto_profile_: > + opts->x_auto_profile_file = xstrdup (arg); > + opts->x_flag_auto_profile = true; > + value = true; > + /* No break here - do -fauto-profile processing. */ > + case OPT_fauto_profile: > + enable_fdo_optimizations (opts, opts_set, value); > + if (!opts_set->x_flag_profile_correction) > + opts->x_flag_profile_correction = value; > + maybe_set_param_value ( > + PARAM_EARLY_INLINER_MAX_ITERATIONS, 10, > + opts->x_param_values, opts_set->x_param_values); > + break; > + > case OPT_fprofile_generate_: > opts->x_profile_data_prefix = xstrdup (arg); > value = true; > Index: gcc/ipa-inline.c > =================================================================== > --- gcc/ipa-inline.c (revision 215826) > +++ gcc/ipa-inline.c (working copy) > @@ -121,6 +121,7 @@ along with GCC; see the file COPYING3. If not see > #include "ipa-inline.h" > #include "ipa-utils.h" > #include "sreal.h" > +#include "auto-profile.h" > #include "cilk.h" > #include "builtins.h" > > @@ -449,6 +450,14 @@ want_early_inline_function_p (struct cgraph_edge * > > if (DECL_DISREGARD_INLINE_LIMITS (callee->decl)) > ; > + /* For AutoFDO, we need to make sure that before profile annotation, all > + hot paths' IR look exactly the same as profiled binary. As a result, > + in einliner, we will disregard size limit and inline those callsites > + that are: > + * inlined in the profiled binary, and > + * the cloned callee has enough samples to be considered "hot". */ > + else if (flag_auto_profile && afdo_callsite_hot_enough_for_early_inline (e)) > + ; I wonder if it won't make more sense to have special autofdo inliner instead of sharing the outer loop with early inliner, but we can clean this up incrementally. I think the patch looks quite good now. Please do the changes above and send updated verison. I think it may be last iteration over it. Honza ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH] AutoFDO patch for trunk 2014-10-15 14:36 ` Jan Hubicka @ 2014-10-16 18:57 ` Dehao Chen 2014-10-19 10:02 ` Jan Hubicka 0 siblings, 1 reply; 25+ messages in thread From: Dehao Chen @ 2014-10-16 18:57 UTC (permalink / raw) To: Jan Hubicka; +Cc: GCC Patches, David Li [-- Attachment #1: Type: text/plain, Size: 27705 bytes --] Hi, Honza, I've integrated all your comments to the patch. New patch attached. Thanks, Dehao On Wed, Oct 15, 2014 at 7:28 AM, Jan Hubicka <hubicka@ucw.cz> wrote: >> Index: gcc/cfgloop.c >> =================================================================== >> --- gcc/cfgloop.c (revision 215826) >> +++ gcc/cfgloop.c (working copy) >> @@ -1802,7 +1802,7 @@ record_niter_bound (struct loop *loop, const wides >> } >> if (realistic >> && (!loop->any_estimate >> - || wi::ltu_p (i_bound, loop->nb_iterations_estimate))) >> + || (!flag_auto_profile && wi::ltu_p (i_bound, loop->nb_iterations_estimate)))) > > I do not remember this hunk from previous patches. So you make autofdo to drop > the conditional that new estimate must be smaller than preivously derived. I > suppose it is because you want autofdo estimates to be overwritten by anything > determined later. > > Lets drop this hunk now, too. I think we really want to have way to track if > the estimate is from autofdo and thus "not too reliable but better than > nothing" and we may want to introduce some kind of reliability metric on way > estimates are derived. > > Lets handle this incrementally and drop this change from this patch. Done. > >> Index: gcc/doc/invoke.texi >> =================================================================== >> --- gcc/doc/invoke.texi (revision 215826) >> +++ gcc/doc/invoke.texi (working copy) >> @@ -365,7 +365,8 @@ Objective-C and Objective-C++ Dialects}. >> @gccoptlist{-faggressive-loop-optimizations -falign-functions[=@var{n}] @gol >> -falign-jumps[=@var{n}] @gol >> -falign-labels[=@var{n}] -falign-loops[=@var{n}] @gol >> --fassociative-math -fauto-inc-dec -fbranch-probabilities @gol >> +-fassociative-math -fauto-profile -fauto-profile[=@var{path}] @gol >> +-fauto-inc-dec -fbranch-probabilities @gol >> -fbranch-target-load-optimize -fbranch-target-load-optimize2 @gol >> -fbtr-bb-exclusive -fcaller-saves @gol >> -fcheck-data-deps -fcombine-stack-adjustments -fconserve-stack @gol >> @@ -9164,6 +9165,19 @@ code. >> >> If @var{path} is specified, GCC looks at the @var{path} to find >> the profile feedback data files. See @option{-fprofile-dir}. >> + >> +@item -fauto-profile >> +@itemx -fauto-profile=@var{path} >> +@opindex fauto-profile >> +Enable sampling based feedback directed optimizations, and optimizations >> +generally profitable only with profile feedback available. >> + >> +The following options are enabled: @code{-fbranch-probabilities}, @code{-fvpt}, >> +@code{-funroll-loops}, @code{-fpeel-loops}, @code{-ftracer}, @code{-ftree-vectorize}, >> +@code{ftree-loop-distribute-patterns} >> + >> +If @var{path} is specified, GCC looks at the @var{path} to find >> +the profile feedback data files. > > I agree with Andi that you need to write more here - basically it should be clear how to > profile the application and use the data. > > We do use external tools here (oprofile/gperf/non-Linux tools I suppose) plus an > conversion tool. So you need to provide some references to those. > > Actually I think it would be really nice to add tutorial-like chapter into GCC > manual about using FDO/autoFDO to improve your code or even more general aimed > covering advanced topics like correct LTO use, profile feedback and other fun. Done, added some little example to demonstrate how to collect profile. >> Index: gcc/value-prof.c >> =================================================================== >> --- gcc/value-prof.c (revision 215826) >> +++ gcc/value-prof.c (working copy) >> @@ -134,11 +134,10 @@ static bool gimple_divmod_fixed_value_transform (g >> static bool gimple_mod_pow2_value_transform (gimple_stmt_iterator *); >> static bool gimple_mod_subtract_transform (gimple_stmt_iterator *); >> static bool gimple_stringops_transform (gimple_stmt_iterator *); >> -static bool gimple_ic_transform (gimple_stmt_iterator *); >> >> /* Allocate histogram value. */ >> >> -static histogram_value >> +histogram_value >> gimple_alloc_histogram_value (struct function *fun ATTRIBUTE_UNUSED, >> enum hist_type type, gimple stmt, tree value) > > Do we still need to expor those? Yes, as soon as we are faking histogram. > >> { >> @@ -1307,6 +1306,9 @@ del_node_map (void) >> struct cgraph_node* >> find_func_by_profile_id (int profile_id) >> { >> + if (flag_auto_profile) >> + return cgraph_node::get_for_asmname ( >> + get_identifier ((const char *)(size_t)profile_id)); > > This definitly needs an comment. Why you need to use find_func_by_profile_id on both > gcov profiling and auto profiling paths? > Perhaps you can use your own fnuction here? > > Or perhaps this is not necessary if you implement directly the indirect call optimiization? Done > > We may try to head towards of having way to read both coverage and auto profile into compiler > eventually. It may be useful to get extra precision from gcov profile and to measure data > like cache misses with auto profile. >> cgraph_node **val = cgraph_node_map->get (profile_id); >> if (val) >> return *val; >> @@ -1320,7 +1322,7 @@ find_func_by_profile_id (int profile_id) >> may ICE. Here we only do very minimal sanity check just to make compiler happy. >> Returns true if TARGET is considered ok for call CALL_STMT. */ >> >> -static bool >> +bool >> check_ic_target (gimple call_stmt, struct cgraph_node *target) >> { >> location_t locus; >> @@ -1483,7 +1485,7 @@ gimple_ic (gimple icall_stmt, struct cgraph_node * >> this call to: >> */ >> >> -static bool >> +bool >> gimple_ic_transform (gimple_stmt_iterator *gsi) > > I do not think those needs to be exported. Done >> { >> gimple stmt = gsi_stmt (*gsi); >> + >> +/* The following routines implements AutoFDO optimization. >> + >> + This optimization uses sampling profiles to annotate basic block counts >> + and uses heuristics to estimate branch probabilities. >> + >> + There are three phases in AutoFDO: >> + >> + Phase 1: Read profile from the profile data file. >> + The following info is read from the profile datafile: >> + * string_table: a map between function name and its index. >> + * autofdo_source_profile: a map from function_instance name to >> + function_instance. This is represented as a forest of >> + function_instances. > > This description does not seem to make it very clear what autofdo_source_profile is. > Perhaps expand a bit more about what function instance is? Done. >> + * WorkingSet: a histogram of how many instructions are covered for a >> + given percentage of total cycles. > > Here I think a bit extra explanation would be nice. Working sets are per-line information? Done. >> + >> + Phase 2: Early inline + VPT. > > Probably expand VPT - it requires one to be quite familiar with GCC sources to > follow 3 letter acronyms ;) Done. >> + Early inline uses autofdo_source_profile to find if a callsite is: >> + * inlined in the profiled binary. >> + * callee body is hot in the profiling run. >> + If both condition satisfies, early inline will inline the callsite >> + regardless of the code growth. >> + Phase 2 is an iterative process. During each iteration, we also check >> + if an indirect callsite is promoted and inlined in the profiling run. >> + If yes, vpt will happen to force promote it and in the next iteration, >> + einline will inline the promoted callsite in the next iteration. > Perhaps "If yes, direct speculative call is introduced" > given that speculative edges are now kind of first class players here. >> + >> + Phase 3: Annotate control flow graph. >> + AutoFDO uses a separate pass to: >> + * Annotate basic block count > > * Annotate basic blocks with count? > >> + * Estimate branch probability > > * Propagate counts across callgraph to fill in information > missing in the profile. > > You probably could expand a bit how profile is driven by source lines and thus > may completely miss some basic blocks. >> + >> + After the above 3 phases, all profile is readily annotated on the GCC IR. >> + AutoFDO tries to reuse all FDO infrastructure as much as possible to make >> + use of the profile. E.g. it uses existing mechanism to calculate the basic >> + block/edge frequency, as well as the cgraph node/edge count. >> +*/ >> + >> +#define DEFAULT_AUTO_PROFILE_FILE "fbdata.afdo" >> + >> +namespace autofdo >> +{ >> + >> +/* Represent a source location: (function_decl, lineno). */ >> +typedef std::pair<tree, unsigned> decl_lineno; >> + >> +/* Represent an inline stack. vector[0] is the leaf node. */ >> +typedef std::vector<decl_lineno> inline_stack; >> + >> +/* String array that stores function names. */ >> +typedef std::vector<const char *> string_vector; > > I am not sure about use of std:: stuff in GCC. Basically I do not see anything > really bad about it, just in case we have our own equivalents, we may stick > of using those. So perhaps using our vec would make more sense. Done >> + >> +/* Profile of a function instance: >> + 1. total_count of the function. >> + 2. head_count of the function (only valid when function is a top-level >> + function_instance, i.e. it is the original copy instead of the >> + inlined copy). > > Perhaps explain what is head count? Done > >> + 3. map from source location (decl_lineno) of the inlined callsite to >> + profile (count_info). > > Of inline callsite? I would expect this to be actual line profile of the > function. You are right, updated the comment. > >> +/* Helper functions. */ >> + >> +/* Return the original name of NAME: strip the suffix that starts >> + with '.' */ >> + >> +static const char * >> +get_original_name (const char *name) >> +{ >> + char *ret = xstrdup (name); >> + char *find = strchr (ret, '.'); >> + if (find != NULL) >> + *find = 0; >> + return ret; >> +} > > Caller is actuallly responsible for freeing RET, right? Perhaps it should be mentioned > in the comment and actually done in uses. Done. >> + >> +/* Return the combined location, which is a 32bit integer in which >> + higher 16 bits stores the line offset of LOC to the start lineno >> + of DECL, The lower 16 bits stores the discrimnator. */ >> + >> +static unsigned >> +get_combined_location (location_t loc, tree decl) >> +{ >> + /* TODO: allow more bits for line and less bits for discriminator. */ >> + return ((LOCATION_LINE (loc) - DECL_SOURCE_LINE (decl)) << 16); >> +} > > We may consider dropping a message into dump file making clear that overflow happened. Done. >> + >> +/* Return the function decl of a given lexical BLOCK. */ >> + >> +static tree >> +get_function_decl_from_block (tree block) >> +{ >> + tree decl; >> + >> + if (LOCATION_LOCUS (BLOCK_SOURCE_LOCATION (block) == UNKNOWN_LOCATION)) >> + return NULL_TREE; >> + >> + for (decl = BLOCK_ABSTRACT_ORIGIN (block); >> + decl && (TREE_CODE (decl) == BLOCK); >> + decl = BLOCK_ABSTRACT_ORIGIN (decl)) >> + if (TREE_CODE (decl) == FUNCTION_DECL) >> + break; >> + return decl; >> +} >> + >> +/* Store inline stack for STMT in STACK. */ >> + >> +static void >> +get_inline_stack (location_t locus, inline_stack *stack) >> +{ >> + if (LOCATION_LOCUS (locus) == UNKNOWN_LOCATION) >> + return; >> + >> + tree block = LOCATION_BLOCK (locus); >> + if (block && TREE_CODE (block) == BLOCK) >> + { >> + int level = 0; >> + for (block = BLOCK_SUPERCONTEXT (block); >> + block && (TREE_CODE (block) == BLOCK); >> + block = BLOCK_SUPERCONTEXT (block)) >> + { >> + location_t tmp_locus = BLOCK_SOURCE_LOCATION (block); >> + if (LOCATION_LOCUS (tmp_locus) == UNKNOWN_LOCATION) >> + continue; >> + >> + tree decl = get_function_decl_from_block (block); >> + stack->push_back ( >> + std::make_pair (decl, get_combined_location (locus, decl))); > > OK, you walk block tree up and for each block you want to lookup the function context > it originate from and push it to stack. > Won't you get extra NULL entries for inner blocks within the functions? We only take those BLOCK_SOURCE_LOCATION is not UNKNOWN_LOCATION. > >> + >> +/* Member functions for string_table. */ >> + >> +string_table * >> +string_table::create () > > Why this is not a constructor? We use static initializer because it's not suggested to put too much logic in constructor. >> +{ >> + string_table *map = new string_table (); >> + if (map->read ()) >> + return map; >> + delete map; >> + return NULL; >> +} >> + >> +/* Return the index of a given function NAME. */ > Perhaps document that function returns -1 when index is not found. Done. >> + >> +int >> +string_table::get_index (const char *name) const >> +{ >> + if (name == NULL) >> + return -1; >> + string_index_map::const_iterator iter = map_.find (name); >> + if (iter == map_.end ()) >> + return -1; >> + else >> + return iter->second; >> +} >> + >> +/* Return the index of a given function DECL. */ > Similarly here. Done. >> + >> +/* Traverse callsites of the current function_instance to find one at the >> + location of LINENO and callee name represented in DECL. */ >> + >> +function_instance * >> +function_instance::get_function_instance_by_decl (unsigned lineno, >> + tree decl) const >> +{ >> + int func_name_idx = afdo_string_table->get_index_by_decl (decl); >> + if (func_name_idx != -1) >> + { >> + callsite_map::const_iterator ret >> + = callsites.find (std::make_pair (lineno, func_name_idx)); >> + if (ret != callsites.end ()) >> + return ret->second; >> + } >> + func_name_idx >> + = afdo_string_table->get_index (lang_hooks.dwarf_name (decl, 0)); >> + if (func_name_idx != -1) >> + { >> + callsite_map::const_iterator ret >> + = callsites.find (std::make_pair (lineno, func_name_idx)); >> + if (ret != callsites.end ()) >> + return ret->second; >> + } > > The two hunks probably can be unified. > Why get_index_by_decl does not already use dwarf name and why do you need to consider both? get_index_by_decl is actually a wrapper of get_index. It tolerates error when assembler name is not emitted in the debug binary (mostly for functions that are fully inlined). In this case, we will first try to find the index of the assembler name, if not found, then bfd name. If still not found, then we try to find name from its abstract location. > >> + if (DECL_ABSTRACT_ORIGIN (decl)) >> + return get_function_instance_by_decl (lineno, DECL_ABSTRACT_ORIGIN (decl)); > > What really happens for ipa-cp function clones and for split functions? Their name suffix will be stripped before matching names. > >> + else >> + return NULL; >> +} >> + >> +/* Recursively traverse STACK starting from LEVEL to find the corresponding >> + function_instance. */ >> + >> +function_instance * >> +function_instance::get_function_instance (const inline_stack &stack, >> + unsigned level) >> +{ >> + if (level == 0) >> + return this; >> + function_instance *s = get_function_instance_by_decl ( >> + stack[level].second, stack[level - 1].first); >> + if (s) >> + return s->get_function_instance (stack, level - 1); >> + else >> + return NULL; > > Someting that probably could be written by iteration ;) Done (removed the function, and use iteration in its caller. >> + >> +/* Read the profile and create a function_instance with head count as >> + HEAD_COUNT. Recursively read callsites to create nested function_instances >> + too. STACK is used to track the recursive creation process. */ >> + >> +/* function instance profile format: >> + >> + ENTRY_COUNT: 8 bytes >> + NAME_INDEX: 4 bytes >> + NUM_POS_COUNTS: 4 bytes >> + NUM_CALLSITES: 4 byte >> + POS_COUNT_1: >> + POS_1_OFFSET: 4 bytes >> + NUM_TARGETS: 4 bytes >> + COUNT: 8 bytes >> + TARGET_1: >> + VALUE_PROFILE_TYPE: 4 bytes >> + TARGET_IDX: 8 bytes >> + COUNT: 8 bytes > > So basically you have list of statements identified by offsets that have counts with them. > If the statement is indirect call, then you have numb targets and you have the value profiles > associated with them, right? yes > > I am not sure it is a win to have the things represented as VPT histograms when you shoot > for quite special purpose problem. But lets deal with these incrementally, too. > (in general I do not see why you try to share so much with the gcov based VTP - it seems > easier to do the speculation by hand) Basically there are 2 problem: * before annotation. I agree this is a special purpose problem * after annotation. This should be the same problem as VPT. > >> + >> +static void >> +read_profile (void) > > Block comment here, even though it is quite obvious ;) Done. >> +{ >> + if (gcov_open (auto_profile_file, 1) == 0) >> + error ("Cannot open profile file %s.", auto_profile_file); >> + >> + if (gcov_read_unsigned () != GCOV_DATA_MAGIC) >> + error ("AutoFDO profile magic number does not mathch."); >> + >> + /* Skip the version number. */ >> + gcov_read_unsigned (); >> + >> + /* Skip the empty integer. */ >> + gcov_read_unsigned (); > > Perhaps we can just check the values? What do you mean? Check the value against 0? >> + >> + /* string_table. */ >> + afdo_string_table = string_table::create (); >> + if (afdo_string_table == NULL) >> + error ("Cannot read string table from %s.", auto_profile_file); >> + >> + /* autofdo_source_profile. */ >> + afdo_source_profile = autofdo_source_profile::create (); >> + if (afdo_source_profile == NULL) >> + error ("Cannot read function profile from %s.", auto_profile_file); >> + >> + /* autofdo_module_profile. */ >> + fake_read_autofdo_module_profile (); >> + >> + /* Read in the working set. */ >> + if (gcov_read_unsigned () != GCOV_TAG_AFDO_WORKING_SET) >> + error ("Cannot read working set from %s.", auto_profile_file); >> + >> + /* Skip the length of the section. */ >> + gcov_read_unsigned (); >> + gcov_working_set_t set[128]; >> + for (unsigned i = 0; i < 128; i++) >> + { >> + set[i].num_counters = gcov_read_unsigned (); >> + set[i].min_counter = gcov_read_counter (); >> + } >> + add_working_set (set); >> +} >> + >> +/* From AutoFDO profiles, find values inside STMT for that we want to measure >> + histograms for indirect-call optimization. */ > > This function is actually served for 2 purposes: > > * before annotation, we need to mark histogram, promote and inline > * after annotation, we just need to mark, and let follow-up logic to > decide if it needs to promote and inline. > > Perhaps add it to the block comment? Done. > > And you are right, for the "before annotation" case, we can simply > call "mark speculative" and "inline". But we still need the logic to > fake histogram for "after annotation" case. As a result, I unified two > cases into one function to reuse code as much as possible. Shall I > separate it into two functions instead? > > Who is using the histogram later? Is it the normal ipa-profile code? > In that case it probably makes sense to produce the histogram indeed. At the end of afdo_annotate_cfg, we call gimple_value_profile_transformations, which reuse all the vpt logic to mark speculative/edges and ipa-inline the edges. > > You sitll can probably avoid using gimple_ic_transform. Because you fake the > histogram, the early checks in function will not lead to anything useful. > You probably want to check yourself check_ic_target Done. > > Does this have chance to work inter-module? Yes, it works fine with LIPO. > >> Index: gcc/opts.c >> =================================================================== >> --- gcc/opts.c (revision 215826) >> +++ gcc/opts.c (working copy) >> @@ -1279,6 +1279,55 @@ print_specific_help (unsigned int include_flags, >> opts->x_help_columns, opts, lang_mask); >> } >> >> +static void >> +enable_fdo_optimizations (struct gcc_options *opts, >> + struct gcc_options *opts_set, >> + int value) > > Block comment here. Done. >> +{ >> + if (!opts_set->x_flag_branch_probabilities) >> + opts->x_flag_branch_probabilities = value; >> + if (!opts_set->x_flag_profile_values) >> + opts->x_flag_profile_values = value; >> + if (!opts_set->x_flag_unroll_loops) >> + opts->x_flag_unroll_loops = value; >> + if (!opts_set->x_flag_peel_loops) >> + opts->x_flag_peel_loops = value; >> + if (!opts_set->x_flag_tracer) >> + opts->x_flag_tracer = value; >> + if (!opts_set->x_flag_value_profile_transformations) >> + opts->x_flag_value_profile_transformations = value; >> + if (!opts_set->x_flag_inline_functions) >> + opts->x_flag_inline_functions = value; >> + if (!opts_set->x_flag_ipa_cp) >> + opts->x_flag_ipa_cp = value; >> + if (!opts_set->x_flag_ipa_cp_clone >> + && value && opts->x_flag_ipa_cp) >> + opts->x_flag_ipa_cp_clone = value; >> + if (!opts_set->x_flag_predictive_commoning) >> + opts->x_flag_predictive_commoning = value; >> + if (!opts_set->x_flag_unswitch_loops) >> + opts->x_flag_unswitch_loops = value; >> + if (!opts_set->x_flag_gcse_after_reload) >> + opts->x_flag_gcse_after_reload = value; >> + if (!opts_set->x_flag_tree_loop_vectorize >> + && !opts_set->x_flag_tree_vectorize) >> + opts->x_flag_tree_loop_vectorize = value; >> + if (!opts_set->x_flag_tree_slp_vectorize >> + && !opts_set->x_flag_tree_vectorize) >> + opts->x_flag_tree_slp_vectorize = value; >> + if (!opts_set->x_flag_vect_cost_model) >> + opts->x_flag_vect_cost_model = VECT_COST_MODEL_DYNAMIC; >> + if (!opts_set->x_flag_tree_loop_distribute_patterns) >> + opts->x_flag_tree_loop_distribute_patterns = value; >> + if (!opts_set->x_flag_profile_reorder_functions) >> + opts->x_flag_profile_reorder_functions = value; >> + /* Indirect call profiling should do all useful transformations >> + speculative devirtualization does. */ >> + if (!opts_set->x_flag_devirtualize_speculatively >> + && opts->x_flag_value_profile_transformations) >> + opts->x_flag_devirtualize_speculatively = false; >> +} >> + >> /* Handle target- and language-independent options. Return zero to >> generate an "unknown option" message. Only options that need >> extra handling need to be listed here; if you simply want >> @@ -1746,50 +1795,23 @@ common_handle_option (struct gcc_options *opts, >> value = true; >> /* No break here - do -fprofile-use processing. */ >> case OPT_fprofile_use: >> - if (!opts_set->x_flag_branch_probabilities) >> - opts->x_flag_branch_probabilities = value; >> - if (!opts_set->x_flag_profile_values) >> - opts->x_flag_profile_values = value; >> - if (!opts_set->x_flag_unroll_loops) >> - opts->x_flag_unroll_loops = value; >> - if (!opts_set->x_flag_peel_loops) >> - opts->x_flag_peel_loops = value; >> - if (!opts_set->x_flag_tracer) >> - opts->x_flag_tracer = value; >> - if (!opts_set->x_flag_value_profile_transformations) >> - opts->x_flag_value_profile_transformations = value; >> - if (!opts_set->x_flag_inline_functions) >> - opts->x_flag_inline_functions = value; >> - if (!opts_set->x_flag_ipa_cp) >> - opts->x_flag_ipa_cp = value; >> - if (!opts_set->x_flag_ipa_cp_clone >> - && value && opts->x_flag_ipa_cp) >> - opts->x_flag_ipa_cp_clone = value; >> - if (!opts_set->x_flag_predictive_commoning) >> - opts->x_flag_predictive_commoning = value; >> - if (!opts_set->x_flag_unswitch_loops) >> - opts->x_flag_unswitch_loops = value; >> - if (!opts_set->x_flag_gcse_after_reload) >> - opts->x_flag_gcse_after_reload = value; >> - if (!opts_set->x_flag_tree_loop_vectorize >> - && !opts_set->x_flag_tree_vectorize) >> - opts->x_flag_tree_loop_vectorize = value; >> - if (!opts_set->x_flag_tree_slp_vectorize >> - && !opts_set->x_flag_tree_vectorize) >> - opts->x_flag_tree_slp_vectorize = value; >> - if (!opts_set->x_flag_vect_cost_model) >> - opts->x_flag_vect_cost_model = VECT_COST_MODEL_DYNAMIC; >> - if (!opts_set->x_flag_tree_loop_distribute_patterns) >> - opts->x_flag_tree_loop_distribute_patterns = value; >> - if (!opts_set->x_flag_profile_reorder_functions) >> - opts->x_flag_profile_reorder_functions = value; >> - /* Indirect call profiling should do all useful transformations >> - speculative devirtualization does. */ >> - if (!opts_set->x_flag_devirtualize_speculatively >> - && opts->x_flag_value_profile_transformations) >> - opts->x_flag_devirtualize_speculatively = false; >> + enable_fdo_optimizations (opts, opts_set, value); >> break; >> >> + case OPT_fauto_profile_: >> + opts->x_auto_profile_file = xstrdup (arg); >> + opts->x_flag_auto_profile = true; >> + value = true; >> + /* No break here - do -fauto-profile processing. */ >> + case OPT_fauto_profile: >> + enable_fdo_optimizations (opts, opts_set, value); >> + if (!opts_set->x_flag_profile_correction) >> + opts->x_flag_profile_correction = value; >> + maybe_set_param_value ( >> + PARAM_EARLY_INLINER_MAX_ITERATIONS, 10, >> + opts->x_param_values, opts_set->x_param_values); >> + break; >> + >> case OPT_fprofile_generate_: >> opts->x_profile_data_prefix = xstrdup (arg); >> value = true; >> Index: gcc/ipa-inline.c >> =================================================================== >> --- gcc/ipa-inline.c (revision 215826) >> +++ gcc/ipa-inline.c (working copy) >> @@ -121,6 +121,7 @@ along with GCC; see the file COPYING3. If not see >> #include "ipa-inline.h" >> #include "ipa-utils.h" >> #include "sreal.h" >> +#include "auto-profile.h" >> #include "cilk.h" >> #include "builtins.h" >> >> @@ -449,6 +450,14 @@ want_early_inline_function_p (struct cgraph_edge * >> >> if (DECL_DISREGARD_INLINE_LIMITS (callee->decl)) >> ; >> + /* For AutoFDO, we need to make sure that before profile annotation, all >> + hot paths' IR look exactly the same as profiled binary. As a result, >> + in einliner, we will disregard size limit and inline those callsites >> + that are: >> + * inlined in the profiled binary, and >> + * the cloned callee has enough samples to be considered "hot". */ >> + else if (flag_auto_profile && afdo_callsite_hot_enough_for_early_inline (e)) >> + ; > > I wonder if it won't make more sense to have special autofdo inliner instead of > sharing the outer loop with early inliner, but we can clean this up incrementally. Yes, this could be done. After a second thought, I think we actually need to expose einliner interface to autofdo (instead of doing vpt in early inliner). This is because we need to mark icall as "promoted", so that later vpt passes will not try to promote it again. Currently in AutoFDO, we use a self-contained way (use a "set" to mark all promoted stmts). If we do vpt in einline, then we will need some flag attached to gimple to indicate whether it's already promoted. > > I think the patch looks quite good now. Please do the changes above and send updated verison. > I think it may be last iteration over it. > > Honza [-- Attachment #2: patch_new.txt --] [-- Type: text/plain, Size: 76134 bytes --] Index: gcc/timevar.def =================================================================== --- gcc/timevar.def (revision 215826) +++ gcc/timevar.def (working copy) @@ -89,6 +89,7 @@ DEFTIMEVAR (TV_WHOPR_PARTITIONING , "whopr part DEFTIMEVAR (TV_WHOPR_LTRANS , "whopr ltrans") DEFTIMEVAR (TV_IPA_REFERENCE , "ipa reference") DEFTIMEVAR (TV_IPA_PROFILE , "ipa profile") +DEFTIMEVAR (TV_IPA_AUTOFDO , "auto profile") DEFTIMEVAR (TV_IPA_PURE_CONST , "ipa pure const") DEFTIMEVAR (TV_IPA_PTA , "ipa points-to") DEFTIMEVAR (TV_IPA_SRA , "ipa SRA") Index: gcc/passes.def =================================================================== --- gcc/passes.def (revision 215826) +++ gcc/passes.def (working copy) @@ -90,6 +90,7 @@ along with GCC; see the file COPYING3. If not see NEXT_PASS (pass_rebuild_cgraph_edges); NEXT_PASS (pass_inline_parameters); POP_INSERT_PASSES () + NEXT_PASS (pass_ipa_auto_profile); NEXT_PASS (pass_ipa_free_inline_summary); NEXT_PASS (pass_ipa_tree_profile); PUSH_INSERT_PASSES_WITHIN (pass_ipa_tree_profile) Index: gcc/basic-block.h =================================================================== --- gcc/basic-block.h (revision 215826) +++ gcc/basic-block.h (working copy) @@ -724,6 +724,7 @@ extern struct edge_list *pre_edge_rev_lcm (int, sb extern void compute_available (sbitmap *, sbitmap *, sbitmap *, sbitmap *); /* In predict.c */ +extern bool maybe_hot_count_p (struct function *, gcov_type); extern bool maybe_hot_bb_p (struct function *, const_basic_block); extern bool maybe_hot_edge_p (edge); extern bool probably_never_executed_bb_p (struct function *, const_basic_block); Index: gcc/doc/invoke.texi =================================================================== --- gcc/doc/invoke.texi (revision 215826) +++ gcc/doc/invoke.texi (working copy) @@ -365,7 +365,8 @@ Objective-C and Objective-C++ Dialects}. @gccoptlist{-faggressive-loop-optimizations -falign-functions[=@var{n}] @gol -falign-jumps[=@var{n}] @gol -falign-labels[=@var{n}] -falign-loops[=@var{n}] @gol --fassociative-math -fauto-inc-dec -fbranch-probabilities @gol +-fassociative-math -fauto-profile -fauto-profile[=@var{path}] @gol +-fauto-inc-dec -fbranch-probabilities @gol -fbranch-target-load-optimize -fbranch-target-load-optimize2 @gol -fbtr-bb-exclusive -fcaller-saves @gol -fcheck-data-deps -fcombine-stack-adjustments -fconserve-stack @gol @@ -9164,6 +9165,37 @@ code. If @var{path} is specified, GCC looks at the @var{path} to find the profile feedback data files. See @option{-fprofile-dir}. + +@item -fauto-profile +@itemx -fauto-profile=@var{path} +@opindex fauto-profile +Enable sampling based feedback directed optimizations, and optimizations +generally profitable only with profile feedback available. + +The following options are enabled: @code{-fbranch-probabilities}, @code{-fvpt}, +@code{-funroll-loops}, @code{-fpeel-loops}, @code{-ftracer}, @code{-ftree-vectorize}, +@code{ftree-loop-distribute-patterns} + +If @var{path} is specified, GCC looks at the @var{path} to find +the profile feedback data files. + +In order to collect AutoFDO profile, you need to have: + +1. A linux system with linux perf support +2. An Intel processor with last branch record (LBR) support + +To collect the profile, first use linux perf to collect raw profile +(see @uref{https://perf.wiki.kernel.org/}). + +E.g. +perf record -e br_inst_retired:near_taken -b -o perf.data -- your_program + +Then use create_gcov tool, which takes raw profile and unstripped binary to +generate AutoFDO profile that can be used by GCC. +(see @uref{https://github.com/google/autofdo}). + +E.g. +create_gcov --binary=your_program.unstripped --profile=perf.data --gcov=profile.afdo @end table The following options control compiler behavior regarding floating-point Index: gcc/gcov-io.h =================================================================== --- gcc/gcov-io.h (revision 215826) +++ gcc/gcov-io.h (working copy) @@ -244,6 +244,9 @@ typedef uint64_t gcov_type_unsigned; #define GCOV_TAG_PROGRAM_SUMMARY ((gcov_unsigned_t)0xa3000000) #define GCOV_TAG_SUMMARY_LENGTH(NUM) \ (1 + GCOV_COUNTERS_SUMMABLE * (10 + 3 * 2) + (NUM) * 5) +#define GCOV_TAG_AFDO_FILE_NAMES ((gcov_unsigned_t)0xaa000000) +#define GCOV_TAG_AFDO_FUNCTION ((gcov_unsigned_t)0xac000000) +#define GCOV_TAG_AFDO_WORKING_SET ((gcov_unsigned_t)0xaf000000) /* Counters that are collected. */ Index: gcc/toplev.c =================================================================== --- gcc/toplev.c (revision 215826) +++ gcc/toplev.c (working copy) @@ -79,6 +79,7 @@ along with GCC; see the file COPYING3. If not see #include "diagnostic-color.h" #include "context.h" #include "pass_manager.h" +#include "auto-profile.h" #include "optabs.h" #if defined(DBX_DEBUGGING_INFO) || defined(XCOFF_DEBUGGING_INFO) @@ -663,6 +664,10 @@ compile_file (void) targetm.asm_out.output_ident (ident_str); } + /* Auto profile finalization. */ + if (flag_auto_profile) + end_auto_profile (); + /* Invoke registered plugin callbacks. */ invoke_plugin_callbacks (PLUGIN_FINISH_UNIT, NULL); Index: gcc/predict.c =================================================================== --- gcc/predict.c (revision 215826) +++ gcc/predict.c (working copy) @@ -161,7 +161,7 @@ set_hot_bb_threshold (gcov_type min) /* Return TRUE if frequency FREQ is considered to be hot. */ -static inline bool +bool maybe_hot_count_p (struct function *fun, gcov_type count) { if (fun && profile_status_for_fn (fun) != PROFILE_READ) @@ -2852,7 +2852,7 @@ counts_to_freqs (void) /* Don't overwrite the estimated frequencies when the profile for the function is missing. We may drop this function PROFILE_GUESSED later in drop_profile (). */ - if (!ENTRY_BLOCK_PTR_FOR_FN (cfun)->count) + if (!flag_auto_profile && !ENTRY_BLOCK_PTR_FOR_FN (cfun)->count) return 0; FOR_BB_BETWEEN (bb, ENTRY_BLOCK_PTR_FOR_FN (cfun), NULL, next_bb) @@ -3223,7 +3223,8 @@ rebuild_frequencies (void) count_max = MAX (bb->count, count_max); if (profile_status_for_fn (cfun) == PROFILE_GUESSED - || (profile_status_for_fn (cfun) == PROFILE_READ && count_max < REG_BR_PROB_BASE/10)) + || (!flag_auto_profile && profile_status_for_fn (cfun) == PROFILE_READ + && count_max < REG_BR_PROB_BASE/10)) { loop_optimizer_init (0); add_noreturn_fake_exit_edges (); Index: gcc/coverage.c =================================================================== --- gcc/coverage.c (revision 215826) +++ gcc/coverage.c (working copy) @@ -55,6 +55,7 @@ along with GCC; see the file COPYING3. If not see #include "filenames.h" #include "target.h" #include "params.h" +#include "auto-profile.h" #include "gcov-io.h" #include "gcov-io.c" @@ -1208,7 +1209,9 @@ coverage_init (const char *filename) bbg_file_stamp = local_tick; - if (flag_branch_probabilities) + if (flag_auto_profile) + read_autofdo_file (); + else if (flag_branch_probabilities) read_counts_file (); /* Name of bbg file. */ Index: gcc/Makefile.in =================================================================== --- gcc/Makefile.in (revision 215826) +++ gcc/Makefile.in (working copy) @@ -1153,6 +1153,7 @@ OBJS = \ alias.o \ alloc-pool.o \ auto-inc-dec.o \ + auto-profile.o \ bb-reorder.o \ bitmap.o \ bt-load.o \ Index: gcc/common.opt =================================================================== --- gcc/common.opt (revision 215826) +++ gcc/common.opt (working copy) @@ -895,6 +895,16 @@ fauto-inc-dec Common Report Var(flag_auto_inc_dec) Init(1) Generate auto-inc/dec instructions +fauto-profile +Common Report Var(flag_auto_profile) Optimization +Use sample profile information for call graph node weights. The default +profile file is fbdata.afdo in 'pwd'. + +fauto-profile= +Common Joined RejectNegative Var(auto_profile_file) +Use sample profile information for call graph node weights. The profile +file is specified in the argument. + ; -fcheck-bounds causes gcc to generate array bounds checks. ; For C, C++ and ObjC: defaults off. ; For Java: defaults to on. Index: gcc/tree-ssa-live.c =================================================================== --- gcc/tree-ssa-live.c (revision 215826) +++ gcc/tree-ssa-live.c (working copy) @@ -605,7 +605,7 @@ remove_unused_scope_block_p (tree scope) ; /* When not generating debug info we can eliminate info on unused variables. */ - else if (debug_info_level == DINFO_LEVEL_NONE) + else if (!flag_auto_profile && debug_info_level == DINFO_LEVEL_NONE) { /* Even for -g0 don't prune outer scopes from artificial functions, otherwise diagnostics using tree_nonartificial_location Index: gcc/profile.c =================================================================== --- gcc/profile.c (revision 215826) +++ gcc/profile.c (working copy) @@ -106,6 +106,14 @@ static int total_num_times_called; static int total_hist_br_prob[20]; static int total_num_branches; +/* Helper function to update gcov_working_sets. */ + +void add_working_set (gcov_working_set_t *set) { + int i = 0; + for (; i < NUM_GCOV_WORKING_SETS; i++) + gcov_working_sets[i] = set[i]; +} + /* Forward declarations. */ static void find_spanning_tree (struct edge_list *); Index: gcc/profile.h =================================================================== --- gcc/profile.h (revision 215826) +++ gcc/profile.h (working copy) @@ -47,6 +47,7 @@ extern void init_node_map (bool); extern void del_node_map (void); extern void get_working_sets (void); +extern void add_working_set (gcov_working_set_t *); /* In predict.c. */ extern gcov_type get_hot_bb_threshold (void); Index: gcc/value-prof.c =================================================================== --- gcc/value-prof.c (revision 215826) +++ gcc/value-prof.c (working copy) @@ -138,7 +138,7 @@ static bool gimple_ic_transform (gimple_stmt_itera /* Allocate histogram value. */ -static histogram_value +histogram_value gimple_alloc_histogram_value (struct function *fun ATTRIBUTE_UNUSED, enum hist_type type, gimple stmt, tree value) { @@ -1320,7 +1320,7 @@ find_func_by_profile_id (int profile_id) may ICE. Here we only do very minimal sanity check just to make compiler happy. Returns true if TARGET is considered ok for call CALL_STMT. */ -static bool +bool check_ic_target (gimple call_stmt, struct cgraph_node *target) { location_t locus; Index: gcc/value-prof.h =================================================================== --- gcc/value-prof.h (revision 215826) +++ gcc/value-prof.h (working copy) @@ -75,6 +75,8 @@ typedef vec<histogram_value> histogram_values; extern void gimple_find_values_to_profile (histogram_values *); extern bool gimple_value_profile_transformations (void); +histogram_value gimple_alloc_histogram_value (struct function *, enum hist_type, + gimple stmt, tree); histogram_value gimple_histogram_value (struct function *, gimple); histogram_value gimple_histogram_value_of_type (struct function *, gimple, enum hist_type); @@ -89,6 +91,7 @@ void verify_histograms (void); void free_histograms (void); void stringop_block_profile (gimple, unsigned int *, HOST_WIDE_INT *); gimple gimple_ic (gimple, struct cgraph_node *, int, gcov_type, gcov_type); +bool check_ic_target (gimple, struct cgraph_node *); /* In tree-profile.c. */ Index: gcc/auto-profile.c =================================================================== --- gcc/auto-profile.c (revision 0) +++ gcc/auto-profile.c (revision 0) @@ -0,0 +1,1692 @@ +/* Read and annotate call graph profile from the auto profile data file. + Copyright (C) 2014. Free Software Foundation, Inc. + Contributed by Dehao Chen (dehao@google.com) + +This file is part of GCC. + +GCC is free software; you can redistribute it and/or modify it under +the terms of the GNU General Public License as published by the Free +Software Foundation; either version 3, or (at your option) any later +version. + +GCC is distributed in the hope that it will be useful, but WITHOUT ANY +WARRANTY; without even the implied warranty of MERCHANTABILITY or +FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License +for more details. + +You should have received a copy of the GNU General Public License +along with GCC; see the file COPYING3. If not see +<http://www.gnu.org/licenses/>. */ + +#include <string.h> +#include <map> +#include <set> + +#include "config.h" +#include "system.h" +#include "coretypes.h" +#include "tree.h" +#include "tree-pass.h" +#include "flags.h" +#include "basic-block.h" +#include "diagnostic-core.h" +#include "gcov-io.h" +#include "input.h" +#include "profile.h" +#include "langhooks.h" +#include "opts.h" +#include "tree-pass.h" +#include "cfgloop.h" +#include "tree-ssa-alias.h" +#include "tree-cfg.h" +#include "tree-cfgcleanup.h" +#include "tree-ssa-operands.h" +#include "tree-into-ssa.h" +#include "internal-fn.h" +#include "is-a.h" +#include "gimple-expr.h" +#include "gimple.h" +#include "gimple-iterator.h" +#include "gimple-ssa.h" +#include "cgraph.h" +#include "value-prof.h" +#include "coverage.h" +#include "params.h" +#include "ipa-inline.h" +#include "tree-inline.h" +#include "stringpool.h" +#include "auto-profile.h" +#include "vec.h" + +/* The following routines implements AutoFDO optimization. + + This optimization uses sampling profiles to annotate basic block counts + and uses heuristics to estimate branch probabilities. + + There are three phases in AutoFDO: + + Phase 1: Read profile from the profile data file. + The following info is read from the profile datafile: + * string_table: a map between function name and its index. + * autofdo_source_profile: a map from function_instance name to + function_instance. This is represented as a forest of + function_instances. + * WorkingSet: a histogram of how many instructions are covered for a + given percentage of total cycles. This is describing the binary + level information (not source level). This info is used to help + decide if we want aggressive optimizations that could increase + code footprint (e.g. loop unroll etc.) + A function instance is an instance of function that could either be a + standalone symbol, or a clone of a function that is inlined into another + function. + + Phase 2: Early inline + valur profile transformation. + Early inline uses autofdo_source_profile to find if a callsite is: + * inlined in the profiled binary. + * callee body is hot in the profiling run. + If both condition satisfies, early inline will inline the callsite + regardless of the code growth. + Phase 2 is an iterative process. During each iteration, we also check + if an indirect callsite is promoted and inlined in the profiling run. + If yes, vpt will happen to force promote it and in the next iteration, + einline will inline the promoted callsite in the next iteration. + + Phase 3: Annotate control flow graph. + AutoFDO uses a separate pass to: + * Annotate basic block count + * Estimate branch probability + + After the above 3 phases, all profile is readily annotated on the GCC IR. + AutoFDO tries to reuse all FDO infrastructure as much as possible to make + use of the profile. E.g. it uses existing mechanism to calculate the basic + block/edge frequency, as well as the cgraph node/edge count. +*/ + +#define DEFAULT_AUTO_PROFILE_FILE "fbdata.afdo" + +namespace autofdo +{ + +/* Represent a source location: (function_decl, lineno). */ +typedef std::pair<tree, unsigned> decl_lineno; + +/* Represent an inline stack. vector[0] is the leaf node. */ +typedef vec<decl_lineno> inline_stack; + +/* String array that stores function names. */ +typedef vec<char *> string_vector; + +/* Map from function name's index in string_table to target's + execution count. */ +typedef std::map<unsigned, gcov_type> icall_target_map; + +/* Set of gimple stmts. Used to track if the stmt has already been promoted + to direct call. */ +typedef std::set<gimple> stmt_set; + +/* Represent count info of an inline stack. */ +struct count_info +{ + /* Sampled count of the inline stack. */ + gcov_type count; + + /* Map from indirect call target to its sample count. */ + icall_target_map targets; + + /* Whether this inline stack is already used in annotation. + + Each inline stack should only be used to annotate IR once. + This will be enforced when instruction-level discriminator + is supported. */ + bool annotated; +}; + +/* operator< for "const char *". */ +struct string_compare +{ + bool operator()(const char *a, const char *b) const + { + return strcmp (a, b) < 0; + } +}; + +/* Store a string array, indexed by string position in the array. */ +class string_table +{ +public: + static string_table *create (); + + ~string_table (); + + /* For a given string, returns its index. */ + int get_index (const char *name) const; + + /* For a given decl, returns the index of the decl name. */ + int get_index_by_decl (tree decl) const; + + /* For a given index, returns the string. */ + const char *get_name (int index) const; + +private: + string_table () {} + bool read (); + + typedef std::map<const char *, unsigned, string_compare> string_index_map; + string_vector vector_; + string_index_map map_; +}; + +/* Profile of a function instance: + 1. total_count of the function. + 2. head_count (entry basic block count) of the function (only valid when + function is a top-level function_instance, i.e. it is the original copy + instead of the inlined copy). + 3. map from source location (decl_lineno) to profile (count_info). + 4. map from callsite to callee function_instance. */ +class function_instance +{ +public: + typedef vec<function_instance *> function_instance_stack; + + /* Read the profile and return a function_instance with head count as + HEAD_COUNT. Recursively read callsites to create nested function_instances + too. STACK is used to track the recursive creation process. */ + static function_instance * + read_function_instance (function_instance_stack *stack, + gcov_type head_count); + + /* Recursively deallocate all callsites (nested function_instances). */ + ~function_instance (); + + /* Accessors. */ + int + name () const + { + return name_; + } + gcov_type + total_count () const + { + return total_count_; + } + gcov_type + head_count () const + { + return head_count_; + } + + /* Traverse callsites of the current function_instance to find one at the + location of LINENO and callee name represented in DECL. */ + function_instance *get_function_instance_by_decl (unsigned lineno, + tree decl) const; + + /* Store the profile info for LOC in INFO. Return TRUE if profile info + is found. */ + bool get_count_info (location_t loc, count_info *info) const; + + /* Read the inlined indirect call target profile for STMT and store it in + MAP, return the total count for all inlined indirect calls. */ + gcov_type find_icall_target_map (gimple stmt, icall_target_map *map) const; + + /* Sum of counts that is used during annotation. */ + gcov_type total_annotated_count () const; + + /* Mark LOC as annotated. */ + void mark_annotated (location_t loc); + +private: + /* Callsite, represented as (decl_lineno, callee_function_name_index). */ + typedef std::pair<unsigned, unsigned> callsite; + + /* Map from callsite to callee function_instance. */ + typedef std::map<callsite, function_instance *> callsite_map; + + function_instance (unsigned name, gcov_type head_count) + : name_ (name), total_count_ (0), head_count_ (head_count) + { + } + + /* Map from source location (decl_lineno) to profile (count_info). */ + typedef std::map<unsigned, count_info> position_count_map; + + /* function_instance name index in the string_table. */ + unsigned name_; + + /* Total sample count. */ + gcov_type total_count_; + + /* Entry BB's sample count. */ + gcov_type head_count_; + + /* Map from callsite location to callee function_instance. */ + callsite_map callsites; + + /* Map from source location to count_info. */ + position_count_map pos_counts; +}; + +/* Profile for all functions. */ +class autofdo_source_profile +{ +public: + static autofdo_source_profile * + create () + { + autofdo_source_profile *map = new autofdo_source_profile (); + + if (map->read ()) + return map; + delete map; + return NULL; + } + + ~autofdo_source_profile (); + + /* For a given DECL, returns the top-level function_instance. */ + function_instance *get_function_instance_by_decl (tree decl) const; + + /* Find count_info for a given gimple STMT. If found, store the count_info + in INFO and return true; otherwise return false. */ + bool get_count_info (gimple stmt, count_info *info) const; + + /* Find total count of the callee of EDGE. */ + gcov_type get_callsite_total_count (struct cgraph_edge *edge) const; + + /* Update value profile INFO for STMT from the inlined indirect callsite. + Return true if INFO is updated. */ + bool update_inlined_ind_target (gimple stmt, count_info *info); + + /* Mark LOC as annotated. */ + void mark_annotated (location_t loc); + +private: + /* Map from function_instance name index (in string_table) to + function_instance. */ + typedef std::map<unsigned, function_instance *> name_function_instance_map; + + autofdo_source_profile () {} + + /* Read AutoFDO profile and returns TRUE on success. */ + bool read (); + + /* Return the function_instance in the profile that correspond to the + inline STACK. */ + function_instance * + get_function_instance_by_inline_stack (const inline_stack &stack) const; + + name_function_instance_map map_; +}; + +/* Store the strings read from the profile data file. */ +static string_table *afdo_string_table; + +/* Store the AutoFDO source profile. */ +static autofdo_source_profile *afdo_source_profile; + +/* gcov_ctr_summary structure to store the profile_info. */ +static struct gcov_ctr_summary *afdo_profile_info; + +/* Helper functions. */ + +/* Return the original name of NAME: strip the suffix that starts + with '.' Caller is responsible for freeing RET. */ + +static char * +get_original_name (const char *name) +{ + char *ret = xstrdup (name); + char *find = strchr (ret, '.'); + if (find != NULL) + *find = 0; + return ret; +} + +/* Return the combined location, which is a 32bit integer in which + higher 16 bits stores the line offset of LOC to the start lineno + of DECL, The lower 16 bits stores the discrimnator. */ + +static unsigned +get_combined_location (location_t loc, tree decl) +{ + /* TODO: allow more bits for line and less bits for discriminator. */ + if (LOCATION_LINE (loc) - DECL_SOURCE_LINE (decl) >= (1<<16)) + warning_at (loc, OPT_Woverflow, "Offset exceeds 16 bytes."); + return ((LOCATION_LINE (loc) - DECL_SOURCE_LINE (decl)) << 16); +} + +/* Return the function decl of a given lexical BLOCK. */ + +static tree +get_function_decl_from_block (tree block) +{ + tree decl; + + if (LOCATION_LOCUS (BLOCK_SOURCE_LOCATION (block) == UNKNOWN_LOCATION)) + return NULL_TREE; + + for (decl = BLOCK_ABSTRACT_ORIGIN (block); + decl && (TREE_CODE (decl) == BLOCK); + decl = BLOCK_ABSTRACT_ORIGIN (decl)) + if (TREE_CODE (decl) == FUNCTION_DECL) + break; + return decl; +} + +/* Store inline stack for STMT in STACK. */ + +static void +get_inline_stack (location_t locus, inline_stack *stack) +{ + if (LOCATION_LOCUS (locus) == UNKNOWN_LOCATION) + return; + + tree block = LOCATION_BLOCK (locus); + if (block && TREE_CODE (block) == BLOCK) + { + int level = 0; + for (block = BLOCK_SUPERCONTEXT (block); + block && (TREE_CODE (block) == BLOCK); + block = BLOCK_SUPERCONTEXT (block)) + { + location_t tmp_locus = BLOCK_SOURCE_LOCATION (block); + if (LOCATION_LOCUS (tmp_locus) == UNKNOWN_LOCATION) + continue; + + tree decl = get_function_decl_from_block (block); + stack->safe_push ( + std::make_pair (decl, get_combined_location (locus, decl))); + locus = tmp_locus; + level++; + } + } + stack->safe_push ( + std::make_pair (current_function_decl, + get_combined_location (locus, current_function_decl))); +} + +/* Return STMT's combined location, which is a 32bit integer in which + higher 16 bits stores the line offset of LOC to the start lineno + of DECL, The lower 16 bits stores the discrimnator. */ + +static unsigned +get_relative_location_for_stmt (gimple stmt) +{ + location_t locus = gimple_location (stmt); + if (LOCATION_LOCUS (locus) == UNKNOWN_LOCATION) + return UNKNOWN_LOCATION; + + for (tree block = gimple_block (stmt); block && (TREE_CODE (block) == BLOCK); + block = BLOCK_SUPERCONTEXT (block)) + if (LOCATION_LOCUS (BLOCK_SOURCE_LOCATION (block)) != UNKNOWN_LOCATION) + return get_combined_location (locus, + get_function_decl_from_block (block)); + return get_combined_location (locus, current_function_decl); +} + +/* Return true if BB contains indirect call. */ + +static bool +has_indirect_call (basic_block bb) +{ + gimple_stmt_iterator gsi; + + for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi)) + { + gimple stmt = gsi_stmt (gsi); + if (gimple_code (stmt) == GIMPLE_CALL && !gimple_call_internal_p (stmt) + && (gimple_call_fn (stmt) == NULL + || TREE_CODE (gimple_call_fn (stmt)) != FUNCTION_DECL)) + return true; + } + return false; +} + +/* Member functions for string_table. */ + +string_table * +string_table::create () +{ + string_table *map = new string_table (); + if (map->read ()) + return map; + delete map; + return NULL; +} + +/* Deconstructor. */ + +string_table::~string_table () +{ + for (unsigned i = 0; i < vector_.length (); i++) + free (vector_[i]); +} + + +/* Return the index of a given function NAME. Return -1 if NAME is not + found in string table. */ + +int +string_table::get_index (const char *name) const +{ + if (name == NULL) + return -1; + string_index_map::const_iterator iter = map_.find (name); + if (iter == map_.end ()) + return -1; + else + return iter->second; +} + +/* Return the index of a given function DECL. Return -1 if DECL is not + found in string table. */ + +int +string_table::get_index_by_decl (tree decl) const +{ + char *name + = get_original_name (IDENTIFIER_POINTER (DECL_ASSEMBLER_NAME (decl))); + int ret = get_index (name); + free (name); + if (ret != -1) + return ret; + ret = get_index (lang_hooks.dwarf_name (decl, 0)); + if (ret != -1) + return ret; + if (DECL_ABSTRACT_ORIGIN (decl)) + return get_index_by_decl (DECL_ABSTRACT_ORIGIN (decl)); + else + return -1; +} + +/* Return the function name of a given INDEX. */ + +const char * +string_table::get_name (int index) const +{ + gcc_assert (index > 0 && index < (int)vector_.length ()); + return vector_[index]; +} + +/* Read the string table. Return TRUE if reading is successful. */ + +bool +string_table::read () +{ + if (gcov_read_unsigned () != GCOV_TAG_AFDO_FILE_NAMES) + return false; + /* Skip the length of the section. */ + gcov_read_unsigned (); + /* Read in the file name table. */ + unsigned string_num = gcov_read_unsigned (); + for (unsigned i = 0; i < string_num; i++) + { + vector_.safe_push (get_original_name (gcov_read_string ())); + map_[vector_.last ()] = i; + } + return true; +} + +/* Member functions for function_instance. */ + +function_instance::~function_instance () +{ + for (callsite_map::iterator iter = callsites.begin (); + iter != callsites.end (); ++iter) + delete iter->second; +} + +/* Traverse callsites of the current function_instance to find one at the + location of LINENO and callee name represented in DECL. */ + +function_instance * +function_instance::get_function_instance_by_decl (unsigned lineno, + tree decl) const +{ + int func_name_idx = afdo_string_table->get_index_by_decl (decl); + if (func_name_idx != -1) + { + callsite_map::const_iterator ret + = callsites.find (std::make_pair (lineno, func_name_idx)); + if (ret != callsites.end ()) + return ret->second; + } + func_name_idx + = afdo_string_table->get_index (lang_hooks.dwarf_name (decl, 0)); + if (func_name_idx != -1) + { + callsite_map::const_iterator ret + = callsites.find (std::make_pair (lineno, func_name_idx)); + if (ret != callsites.end ()) + return ret->second; + } + if (DECL_ABSTRACT_ORIGIN (decl)) + return get_function_instance_by_decl (lineno, DECL_ABSTRACT_ORIGIN (decl)); + else + return NULL; +} + +/* Store the profile info for LOC in INFO. Return TRUE if profile info + is found. */ + +bool +function_instance::get_count_info (location_t loc, count_info *info) const +{ + position_count_map::const_iterator iter = pos_counts.find (loc); + if (iter == pos_counts.end ()) + return false; + *info = iter->second; + return true; +} + +/* Mark LOC as annotated. */ + +void +function_instance::mark_annotated (location_t loc) +{ + position_count_map::iterator iter = pos_counts.find (loc); + if (iter == pos_counts.end ()) + return; + iter->second.annotated = true; +} + +/* Read the inlinied indirect call target profile for STMT and store it in + MAP, return the total count for all inlined indirect calls. */ + +gcov_type +function_instance::find_icall_target_map (gimple stmt, + icall_target_map *map) const +{ + gcov_type ret = 0; + unsigned stmt_offset = get_relative_location_for_stmt (stmt); + + for (callsite_map::const_iterator iter = callsites.begin (); + iter != callsites.end (); ++iter) + { + unsigned callee = iter->second->name (); + /* Check if callsite location match the stmt. */ + if (iter->first.first != stmt_offset) + continue; + struct cgraph_node *node = cgraph_node::get_for_asmname ( + get_identifier (afdo_string_table->get_name (callee))); + if (node == NULL) + continue; + if (!check_ic_target (stmt, node)) + continue; + (*map)[callee] = iter->second->total_count (); + ret += iter->second->total_count (); + } + return ret; +} + +/* Read the profile and create a function_instance with head count as + HEAD_COUNT. Recursively read callsites to create nested function_instances + too. STACK is used to track the recursive creation process. */ + +/* function instance profile format: + + ENTRY_COUNT: 8 bytes + NAME_INDEX: 4 bytes + NUM_POS_COUNTS: 4 bytes + NUM_CALLSITES: 4 byte + POS_COUNT_1: + POS_1_OFFSET: 4 bytes + NUM_TARGETS: 4 bytes + COUNT: 8 bytes + TARGET_1: + VALUE_PROFILE_TYPE: 4 bytes + TARGET_IDX: 8 bytes + COUNT: 8 bytes + TARGET_2 + ... + TARGET_n + POS_COUNT_2 + ... + POS_COUNT_N + CALLSITE_1: + CALLSITE_1_OFFSET: 4 bytes + FUNCTION_INSTANCE_PROFILE (nested) + CALLSITE_2 + ... + CALLSITE_n. */ + +function_instance * +function_instance::read_function_instance (function_instance_stack *stack, + gcov_type head_count) +{ + unsigned name = gcov_read_unsigned (); + unsigned num_pos_counts = gcov_read_unsigned (); + unsigned num_callsites = gcov_read_unsigned (); + function_instance *s = new function_instance (name, head_count); + stack->safe_push (s); + + for (unsigned i = 0; i < num_pos_counts; i++) + { + unsigned offset = gcov_read_unsigned () & 0xffff0000; + unsigned num_targets = gcov_read_unsigned (); + gcov_type count = gcov_read_counter (); + s->pos_counts[offset].count = count; + for (unsigned j = 0; j < stack->length (); j++) + (*stack)[j]->total_count_ += count; + for (unsigned j = 0; j < num_targets; j++) + { + /* Only indirect call target histogram is supported now. */ + gcov_read_unsigned (); + gcov_type target_idx = gcov_read_counter (); + s->pos_counts[offset].targets[target_idx] = gcov_read_counter (); + } + } + for (unsigned i = 0; i < num_callsites; i++) + { + unsigned offset = gcov_read_unsigned (); + function_instance *callee_function_instance + = read_function_instance (stack, 0); + s->callsites[std::make_pair (offset, callee_function_instance->name ())] + = callee_function_instance; + } + stack->pop (); + return s; +} + +/* Sum of counts that is used during annotation. */ + +gcov_type +function_instance::total_annotated_count () const +{ + gcov_type ret = 0; + for (callsite_map::const_iterator iter = callsites.begin (); + iter != callsites.end (); ++iter) + ret += iter->second->total_annotated_count (); + for (position_count_map::const_iterator iter = pos_counts.begin (); + iter != pos_counts.end (); ++iter) + if (iter->second.annotated) + ret += iter->second.count; + return ret; +} + +/* Member functions for autofdo_source_profile. */ + +autofdo_source_profile::~autofdo_source_profile () +{ + for (name_function_instance_map::const_iterator iter = map_.begin (); + iter != map_.end (); ++iter) + delete iter->second; +} + +/* For a given DECL, returns the top-level function_instance. */ + +function_instance * +autofdo_source_profile::get_function_instance_by_decl (tree decl) const +{ + int index = afdo_string_table->get_index_by_decl (decl); + if (index == -1) + return NULL; + name_function_instance_map::const_iterator ret = map_.find (index); + return ret == map_.end () ? NULL : ret->second; +} + +/* Find count_info for a given gimple STMT. If found, store the count_info + in INFO and return true; otherwise return false. */ + +bool +autofdo_source_profile::get_count_info (gimple stmt, count_info *info) const +{ + if (LOCATION_LOCUS (gimple_location (stmt)) == cfun->function_end_locus) + return false; + + inline_stack stack; + get_inline_stack (gimple_location (stmt), &stack); + if (stack.length () == 0) + return false; + function_instance *s = get_function_instance_by_inline_stack (stack); + if (s == NULL) + return false; + return s->get_count_info (stack[0].second, info); +} + +/* Mark LOC as annotated. */ + +void +autofdo_source_profile::mark_annotated (location_t loc) +{ + inline_stack stack; + get_inline_stack (loc, &stack); + if (stack.length () == 0) + return; + function_instance *s = get_function_instance_by_inline_stack (stack); + if (s == NULL) + return; + s->mark_annotated (stack[0].second); +} + +/* Update value profile INFO for STMT from the inlined indirect callsite. + Return true if INFO is updated. */ + +bool +autofdo_source_profile::update_inlined_ind_target (gimple stmt, + count_info *info) +{ + if (LOCATION_LOCUS (gimple_location (stmt)) == cfun->function_end_locus) + return false; + + count_info old_info; + get_count_info (stmt, &old_info); + gcov_type total = 0; + for (icall_target_map::const_iterator iter = old_info.targets.begin (); + iter != old_info.targets.end (); ++iter) + total += iter->second; + + /* Program behavior changed, original promoted (and inlined) target is not + hot any more. Will avoid promote the original target. + + To check if original promoted target is still hot, we check the total + count of the unpromoted targets (stored in old_info). If it is no less + than half of the callsite count (stored in INFO), the original promoted + target is considered not hot any more. */ + if (total >= info->count / 2) + return false; + + inline_stack stack; + get_inline_stack (gimple_location (stmt), &stack); + if (stack.length () == 0) + return false; + function_instance *s = get_function_instance_by_inline_stack (stack); + if (s == NULL) + return false; + icall_target_map map; + if (s->find_icall_target_map (stmt, &map) == 0) + return false; + for (icall_target_map::const_iterator iter = map.begin (); + iter != map.end (); ++iter) + info->targets[iter->first] = iter->second; + return true; +} + +/* Find total count of the callee of EDGE. */ + +gcov_type +autofdo_source_profile::get_callsite_total_count ( + struct cgraph_edge *edge) const +{ + inline_stack stack; + stack.safe_push (std::make_pair (edge->callee->decl, 0)); + get_inline_stack (gimple_location (edge->call_stmt), &stack); + + function_instance *s = get_function_instance_by_inline_stack (stack); + if (s == NULL + || afdo_string_table->get_index (IDENTIFIER_POINTER ( + DECL_ASSEMBLER_NAME (edge->callee->decl))) != s->name ()) + return 0; + else + return s->total_count (); +} + +/* Read AutoFDO profile and returns TRUE on success. */ + +/* source profile format: + + GCOV_TAG_AFDO_FUNCTION: 4 bytes + LENGTH: 4 bytes + NUM_FUNCTIONS: 4 bytes + FUNCTION_INSTANCE_1 + FUNCTION_INSTANCE_2 + ... + FUNCTION_INSTANCE_N. */ + +bool +autofdo_source_profile::read () +{ + if (gcov_read_unsigned () != GCOV_TAG_AFDO_FUNCTION) + { + inform (0, "Not expected TAG."); + return false; + } + + /* Skip the length of the section. */ + gcov_read_unsigned (); + + /* Read in the function/callsite profile, and store it in local + data structure. */ + unsigned function_num = gcov_read_unsigned (); + for (unsigned i = 0; i < function_num; i++) + { + function_instance::function_instance_stack stack; + function_instance *s = function_instance::read_function_instance ( + &stack, gcov_read_counter ()); + afdo_profile_info->sum_all += s->total_count (); + map_[s->name ()] = s; + } + return true; +} + +/* Return the function_instance in the profile that correspond to the + inline STACK. */ + +function_instance * +autofdo_source_profile::get_function_instance_by_inline_stack ( + const inline_stack &stack) const +{ + name_function_instance_map::const_iterator iter = map_.find ( + afdo_string_table->get_index_by_decl (stack[stack.length () - 1].first)); + if (iter == map_.end()) + return NULL; + function_instance *s = iter->second; + for (unsigned i = stack.length() - 1; i > 0; i--) + { + s = s->get_function_instance_by_decl ( + stack[i].second, stack[i - 1].first); + if (s == NULL) + return NULL; + } + return s; +} + +/* Module profile is only used by LIPO. Here we simply ignore it. */ + +static void +fake_read_autofdo_module_profile () +{ + /* Read in the module info. */ + gcov_read_unsigned (); + + /* Skip the length of the section. */ + gcov_read_unsigned (); + + /* Read in the file name table. */ + unsigned total_module_num = gcov_read_unsigned (); + gcc_assert (total_module_num == 0); +} + +/* Read data from profile data file. */ + +static void +read_profile (void) +{ + if (gcov_open (auto_profile_file, 1) == 0) + error ("Cannot open profile file %s.", auto_profile_file); + + if (gcov_read_unsigned () != GCOV_DATA_MAGIC) + error ("AutoFDO profile magic number does not mathch."); + + /* Skip the version number. */ + gcov_read_unsigned (); + + /* Skip the empty integer. */ + gcov_read_unsigned (); + + /* string_table. */ + afdo_string_table = string_table::create (); + if (afdo_string_table == NULL) + error ("Cannot read string table from %s.", auto_profile_file); + + /* autofdo_source_profile. */ + afdo_source_profile = autofdo_source_profile::create (); + if (afdo_source_profile == NULL) + error ("Cannot read function profile from %s.", auto_profile_file); + + /* autofdo_module_profile. */ + fake_read_autofdo_module_profile (); + + /* Read in the working set. */ + if (gcov_read_unsigned () != GCOV_TAG_AFDO_WORKING_SET) + error ("Cannot read working set from %s.", auto_profile_file); + + /* Skip the length of the section. */ + gcov_read_unsigned (); + gcov_working_set_t set[128]; + for (unsigned i = 0; i < 128; i++) + { + set[i].num_counters = gcov_read_unsigned (); + set[i].min_counter = gcov_read_counter (); + } + add_working_set (set); +} + +/* From AutoFDO profiles, find values inside STMT for that we want to measure + histograms for indirect-call optimization. + + This function is actually served for 2 purposes: + * before annotation, we need to mark histogram, promote and inline + * after annotation, we just need to mark, and let follow-up logic to + decide if it needs to promote and inline. */ + +static void +afdo_indirect_call (gimple_stmt_iterator *gsi, const icall_target_map &map, + bool transform) +{ + gimple stmt = gsi_stmt (*gsi); + tree callee; + + if (map.size () == 0 || gimple_code (stmt) != GIMPLE_CALL + || gimple_call_fndecl (stmt) != NULL_TREE) + return; + + callee = gimple_call_fn (stmt); + + histogram_value hist = gimple_alloc_histogram_value ( + cfun, HIST_TYPE_INDIR_CALL, stmt, callee); + hist->n_counters = 3; + hist->hvalue.counters = XNEWVEC (gcov_type, hist->n_counters); + gimple_add_histogram_value (cfun, stmt, hist); + + gcov_type total = 0; + icall_target_map::const_iterator max_iter = map.end (); + + for (icall_target_map::const_iterator iter = map.begin (); + iter != map.end (); ++iter) + { + total += iter->second; + if (max_iter == map.end () || max_iter->second < iter->second) + max_iter = iter; + } + + hist->hvalue.counters[0] + = (unsigned long long)afdo_string_table->get_name (max_iter->first); + hist->hvalue.counters[1] = max_iter->second; + hist->hvalue.counters[2] = total; + + if (!transform) + return; + + struct cgraph_edge *indirect_edge + = cgraph_node::get (current_function_decl)->get_edge (stmt); + struct cgraph_node *direct_call = cgraph_node::get_for_asmname ( + get_identifier ((const char *) hist->hvalue.counters[0])); + + if (direct_call == NULL || !check_ic_target (stmt, direct_call)) + return; + if (DECL_STRUCT_FUNCTION (direct_call->decl) == NULL) + return; + struct cgraph_edge *new_edge + = indirect_edge->make_speculative (direct_call, 0, 0); + new_edge->redirect_call_stmt_to_callee (); + gimple_remove_histogram_value (cfun, stmt, hist); + inline_call (new_edge, true, NULL, NULL, false); +} + +/* From AutoFDO profiles, find values inside STMT for that we want to measure + histograms and adds them to list VALUES. */ + +static void +afdo_vpt (gimple_stmt_iterator *gsi, const icall_target_map &map, + bool transform) +{ + afdo_indirect_call (gsi, map, transform); +} + +typedef std::set<basic_block> bb_set; +typedef std::set<edge> edge_set; + +static bool +is_bb_annotated (const basic_block bb, const bb_set &annotated) +{ + return annotated.find (bb) != annotated.end (); +} + +static void +set_bb_annotated (basic_block bb, bb_set *annotated) +{ + annotated->insert (bb); +} + +static bool +is_edge_annotated (const edge e, const edge_set &annotated) +{ + return annotated.find (e) != annotated.end (); +} + +static void +set_edge_annotated (edge e, edge_set *annotated) +{ + annotated->insert (e); +} + +/* For a given BB, set its execution count. Attach value profile if a stmt + is not in PROMOTED, because we only want to promot an indirect call once. + Return TRUE if BB is annotated. */ + +static bool +afdo_set_bb_count (basic_block bb, const stmt_set &promoted) +{ + gimple_stmt_iterator gsi; + edge e; + edge_iterator ei; + gcov_type max_count = 0; + bool has_annotated = false; + + for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi)) + { + count_info info; + gimple stmt = gsi_stmt (gsi); + if (gimple_clobber_p (stmt) || is_gimple_debug (stmt)) + continue; + if (afdo_source_profile->get_count_info (stmt, &info)) + { + if (info.count > max_count) + max_count = info.count; + has_annotated = true; + if (info.targets.size () > 0 + && promoted.find (stmt) == promoted.end ()) + afdo_vpt (&gsi, info.targets, false); + } + } + + if (!has_annotated) + return false; + + for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi)) + afdo_source_profile->mark_annotated (gimple_location (gsi_stmt (gsi))); + for (gsi = gsi_start_phis (bb); !gsi_end_p (gsi); gsi_next (&gsi)) + { + gimple phi = gsi_stmt (gsi); + size_t i; + for (i = 0; i < gimple_phi_num_args (phi); i++) + afdo_source_profile->mark_annotated (gimple_phi_arg_location (phi, i)); + } + FOR_EACH_EDGE (e, ei, bb->succs) + afdo_source_profile->mark_annotated (e->goto_locus); + + bb->count = max_count; + return true; +} + +/* BB1 and BB2 are in an equivalent class iff: + 1. BB1 dominates BB2. + 2. BB2 post-dominates BB1. + 3. BB1 and BB2 are in the same loop nest. + This function finds the equivalent class for each basic block, and + stores a pointer to the first BB in its equivalent class. Meanwhile, + set bb counts for the same equivalent class to be idenical. Update + ANNOTATED_BB for the first BB in its equivalent class. */ + +static void +afdo_find_equiv_class (bb_set *annotated_bb) +{ + basic_block bb; + + FOR_ALL_BB_FN (bb, cfun) + bb->aux = NULL; + + FOR_ALL_BB_FN (bb, cfun) + { + vec<basic_block> dom_bbs; + basic_block bb1; + int i; + + if (bb->aux != NULL) + continue; + bb->aux = bb; + dom_bbs = get_dominated_by (CDI_DOMINATORS, bb); + FOR_EACH_VEC_ELT (dom_bbs, i, bb1) + if (bb1->aux == NULL && dominated_by_p (CDI_POST_DOMINATORS, bb, bb1) + && bb1->loop_father == bb->loop_father) + { + bb1->aux = bb; + if (bb1->count > bb->count && is_bb_annotated (bb1, *annotated_bb)) + { + bb->count = MAX (bb->count, bb1->count); + set_bb_annotated (bb, annotated_bb); + } + } + dom_bbs = get_dominated_by (CDI_POST_DOMINATORS, bb); + FOR_EACH_VEC_ELT (dom_bbs, i, bb1) + if (bb1->aux == NULL && dominated_by_p (CDI_DOMINATORS, bb, bb1) + && bb1->loop_father == bb->loop_father) + { + bb1->aux = bb; + if (bb1->count > bb->count && is_bb_annotated (bb1, *annotated_bb)) + { + bb->count = MAX (bb->count, bb1->count); + set_bb_annotated (bb, annotated_bb); + } + } + } +} + +/* If a basic block's count is known, and only one of its in/out edges' count + is unknown, its count can be calculated. Meanwhile, if all of the in/out + edges' counts are known, then the basic block's unknown count can also be + calculated. + IS_SUCC is true if out edges of a basic blocks are examined. + Update ANNOTATED_BB and ANNOTATED_EDGE accordingly. + Return TRUE if any basic block/edge count is changed. */ + +static bool +afdo_propagate_edge (bool is_succ, bb_set *annotated_bb, + edge_set *annotated_edge) +{ + basic_block bb; + bool changed = false; + + FOR_EACH_BB_FN (bb, cfun) + { + edge e, unknown_edge = NULL; + edge_iterator ei; + int num_unknown_edge = 0; + gcov_type total_known_count = 0; + + FOR_EACH_EDGE (e, ei, is_succ ? bb->succs : bb->preds) + if (!is_edge_annotated (e, *annotated_edge)) + num_unknown_edge++, unknown_edge = e; + else + total_known_count += e->count; + + if (num_unknown_edge == 0) + { + if (total_known_count > bb->count) + { + bb->count = total_known_count; + changed = true; + } + if (!is_bb_annotated (bb, *annotated_bb)) + { + set_bb_annotated (bb, annotated_bb); + changed = true; + } + } + else if (num_unknown_edge == 1 && is_bb_annotated (bb, *annotated_bb)) + { + if (bb->count >= total_known_count) + unknown_edge->count = bb->count - total_known_count; + else + unknown_edge->count = 0; + set_edge_annotated (unknown_edge, annotated_edge); + changed = true; + } + } + return changed; +} + +/* Special propagation for circuit expressions. Because GCC translates + control flow into data flow for circuit expressions. E.g. + BB1: + if (a && b) + BB2 + else + BB3 + + will be translated into: + + BB1: + if (a) + goto BB.t1 + else + goto BB.t3 + BB.t1: + if (b) + goto BB.t2 + else + goto BB.t3 + BB.t2: + goto BB.t3 + BB.t3: + tmp = PHI (0 (BB1), 0 (BB.t1), 1 (BB.t2) + if (tmp) + goto BB2 + else + goto BB3 + + In this case, we need to propagate through PHI to determine the edge + count of BB1->BB.t1, BB.t1->BB.t2. + Update ANNOTATED_EDGE accordingly. */ + +static void +afdo_propagate_circuit (const bb_set &annotated_bb, edge_set *annotated_edge) +{ + basic_block bb; + FOR_ALL_BB_FN (bb, cfun) + { + gimple phi_stmt; + tree cmp_rhs, cmp_lhs; + gimple cmp_stmt = last_stmt (bb); + edge e; + edge_iterator ei; + + if (!cmp_stmt || gimple_code (cmp_stmt) != GIMPLE_COND) + continue; + cmp_rhs = gimple_cond_rhs (cmp_stmt); + cmp_lhs = gimple_cond_lhs (cmp_stmt); + if (!TREE_CONSTANT (cmp_rhs) + || !(integer_zerop (cmp_rhs) || integer_onep (cmp_rhs))) + continue; + if (TREE_CODE (cmp_lhs) != SSA_NAME) + continue; + if (!is_bb_annotated (bb, annotated_bb)) + continue; + phi_stmt = SSA_NAME_DEF_STMT (cmp_lhs); + while (phi_stmt && gimple_code (phi_stmt) == GIMPLE_ASSIGN + && gimple_assign_single_p (phi_stmt) + && TREE_CODE (gimple_assign_rhs1 (phi_stmt)) == SSA_NAME) + phi_stmt = SSA_NAME_DEF_STMT (gimple_assign_rhs1 (phi_stmt)); + if (!phi_stmt || gimple_code (phi_stmt) != GIMPLE_PHI) + continue; + FOR_EACH_EDGE (e, ei, bb->succs) + { + unsigned i, total = 0; + edge only_one; + bool check_value_one = (((integer_onep (cmp_rhs)) + ^ (gimple_cond_code (cmp_stmt) == EQ_EXPR)) + ^ ((e->flags & EDGE_TRUE_VALUE) != 0)); + if (!is_edge_annotated (e, *annotated_edge)) + continue; + for (i = 0; i < gimple_phi_num_args (phi_stmt); i++) + { + tree val = gimple_phi_arg_def (phi_stmt, i); + edge ep = gimple_phi_arg_edge (phi_stmt, i); + + if (!TREE_CONSTANT (val) + || !(integer_zerop (val) || integer_onep (val))) + continue; + if (check_value_one ^ integer_onep (val)) + continue; + total++; + only_one = ep; + if (e->probability == 0 && !is_edge_annotated (ep, *annotated_edge)) + { + ep->probability = 0; + ep->count = 0; + set_edge_annotated (ep, annotated_edge); + } + } + if (total == 1 && !is_edge_annotated (only_one, *annotated_edge)) + { + only_one->probability = e->probability; + only_one->count = e->count; + set_edge_annotated (only_one, annotated_edge); + } + } + } +} + +/* Propagate the basic block count and edge count on the control flow + graph. We do the propagation iteratively until stablize. */ + +static void +afdo_propagate (bb_set *annotated_bb, edge_set *annotated_edge) +{ + basic_block bb; + bool changed = true; + int i = 0; + + FOR_ALL_BB_FN (bb, cfun) + { + bb->count = ((basic_block)bb->aux)->count; + if (is_bb_annotated ((const basic_block)bb->aux, *annotated_bb)) + set_bb_annotated (bb, annotated_bb); + } + + while (changed && i++ < 10) + { + changed = false; + + if (afdo_propagate_edge (true, annotated_bb, annotated_edge)) + changed = true; + if (afdo_propagate_edge (false, annotated_bb, annotated_edge)) + changed = true; + afdo_propagate_circuit (*annotated_bb, annotated_edge); + } +} + +/* Propagate counts on control flow graph and calculate branch + probabilities. */ + +static void +afdo_calculate_branch_prob (bb_set *annotated_bb, edge_set *annotated_edge) +{ + basic_block bb; + bool has_sample = false; + + FOR_EACH_BB_FN (bb, cfun) + if (bb->count > 0) + has_sample = true; + + if (!has_sample) + return; + + calculate_dominance_info (CDI_POST_DOMINATORS); + calculate_dominance_info (CDI_DOMINATORS); + loop_optimizer_init (0); + + afdo_find_equiv_class (annotated_bb); + afdo_propagate (annotated_bb, annotated_edge); + + FOR_EACH_BB_FN (bb, cfun) + { + edge e; + edge_iterator ei; + int num_unknown_succ = 0; + gcov_type total_count = 0; + + FOR_EACH_EDGE (e, ei, bb->succs) + { + if (!is_edge_annotated (e, *annotated_edge)) + num_unknown_succ++; + else + total_count += e->count; + } + if (num_unknown_succ == 0 && total_count > 0) + { + FOR_EACH_EDGE (e, ei, bb->succs) + e->probability = (double)e->count * REG_BR_PROB_BASE / total_count; + } + } + FOR_ALL_BB_FN (bb, cfun) + { + edge e; + edge_iterator ei; + + FOR_EACH_EDGE (e, ei, bb->succs) + e->count = (double)bb->count * e->probability / REG_BR_PROB_BASE; + bb->aux = NULL; + } + + loop_optimizer_finalize (); + free_dominance_info (CDI_DOMINATORS); + free_dominance_info (CDI_POST_DOMINATORS); +} + +/* Perform value profile transformation using AutoFDO profile. Add the + promoted stmts to PROMOTED_STMTS. Return TRUE if there is any + indirect call promoted. */ + +static bool +afdo_vpt_for_early_inline (stmt_set *promoted_stmts) +{ + basic_block bb; + if (afdo_source_profile->get_function_instance_by_decl ( + current_function_decl) == NULL) + return false; + + compute_inline_parameters (cgraph_node::get (current_function_decl), true); + + bool has_vpt = false; + FOR_EACH_BB_FN (bb, cfun) + { + if (!has_indirect_call (bb)) + continue; + gimple_stmt_iterator gsi; + + gcov_type bb_count = 0; + for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi)) + { + count_info info; + gimple stmt = gsi_stmt (gsi); + if (afdo_source_profile->get_count_info (stmt, &info)) + bb_count = MAX (bb_count, info.count); + } + + for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi)) + { + gimple stmt = gsi_stmt (gsi); + /* IC_promotion and early_inline_2 is done in multiple iterations. + No need to promoted the stmt if its in promoted_stmts (means + it is already been promoted in the previous iterations). */ + if (gimple_code (stmt) != GIMPLE_CALL || gimple_call_fn (stmt) == NULL + || TREE_CODE (gimple_call_fn (stmt)) == FUNCTION_DECL + || promoted_stmts->find (stmt) != promoted_stmts->end ()) + continue; + + count_info info; + afdo_source_profile->get_count_info (stmt, &info); + info.count = bb_count; + if (afdo_source_profile->update_inlined_ind_target (stmt, &info)) + { + /* Promote the indirect call and update the promoted_stmts. */ + promoted_stmts->insert (stmt); + afdo_vpt (&gsi, info.targets, true); + has_vpt = true; + } + } + } + if (has_vpt) + { + optimize_inline_calls (current_function_decl); + return true; + } + else + return false; +} + +/* Annotate auto profile to the control flow graph. Do not annotate value + profile for stmts in PROMOTED_STMTS. */ + +static void +afdo_annotate_cfg (const stmt_set &promoted_stmts) +{ + basic_block bb; + bb_set annotated_bb; + edge_set annotated_edge; + const function_instance *s + = afdo_source_profile->get_function_instance_by_decl ( + current_function_decl); + + if (s == NULL) + return; + cgraph_node::get (current_function_decl)->count = s->head_count (); + ENTRY_BLOCK_PTR_FOR_FN (cfun)->count = s->head_count (); + gcov_type max_count = ENTRY_BLOCK_PTR_FOR_FN (cfun)->count; + + FOR_EACH_BB_FN (bb, cfun) + { + edge e; + edge_iterator ei; + + bb->count = 0; + FOR_EACH_EDGE (e, ei, bb->succs) + e->count = 0; + + if (afdo_set_bb_count (bb, promoted_stmts)) + set_bb_annotated (bb, &annotated_bb); + if (bb->count > max_count) + max_count = bb->count; + } + if (ENTRY_BLOCK_PTR_FOR_FN (cfun)->count + > ENTRY_BLOCK_PTR_FOR_FN (cfun)->next_bb->count) + { + ENTRY_BLOCK_PTR_FOR_FN (cfun)->next_bb->count + = ENTRY_BLOCK_PTR_FOR_FN (cfun)->count; + set_bb_annotated (ENTRY_BLOCK_PTR_FOR_FN (cfun)->next_bb, &annotated_bb); + } + if (ENTRY_BLOCK_PTR_FOR_FN (cfun)->count + > EXIT_BLOCK_PTR_FOR_FN (cfun)->prev_bb->count) + { + EXIT_BLOCK_PTR_FOR_FN (cfun)->prev_bb->count + = ENTRY_BLOCK_PTR_FOR_FN (cfun)->count; + set_bb_annotated (EXIT_BLOCK_PTR_FOR_FN (cfun)->prev_bb, &annotated_bb); + } + afdo_source_profile->mark_annotated ( + DECL_SOURCE_LOCATION (current_function_decl)); + afdo_source_profile->mark_annotated (cfun->function_start_locus); + afdo_source_profile->mark_annotated (cfun->function_end_locus); + if (max_count > 0) + { + afdo_calculate_branch_prob (&annotated_bb, &annotated_edge); + counts_to_freqs (); + profile_status_for_fn (cfun) = PROFILE_READ; + } + if (flag_value_profile_transformations) + gimple_value_profile_transformations (); +} + +/* Wrapper function to invoke early inliner. */ + +static void +early_inline () +{ + compute_inline_parameters (cgraph_node::get (current_function_decl), true); + unsigned todo = early_inliner (cfun); + if (todo & TODO_update_ssa_any) + update_ssa (TODO_update_ssa); +} + +/* Use AutoFDO profile to annoate the control flow graph. + Return the todo flag. */ + +static unsigned int +auto_profile (void) +{ + struct cgraph_node *node; + + if (symtab->state == FINISHED) + return 0; + + init_node_map (true); + profile_info = autofdo::afdo_profile_info; + + FOR_EACH_FUNCTION (node) + { + if (!gimple_has_body_p (node->decl)) + continue; + + /* Don't profile functions produced for builtin stuff. */ + if (DECL_SOURCE_LOCATION (node->decl) == BUILTINS_LOCATION) + continue; + + push_cfun (DECL_STRUCT_FUNCTION (node->decl)); + + /* First do indirect call promotion and early inline to make the + IR match the profiled binary before actual annotation. + + This is needed because an indirect call might have been promoted + and inlined in the profiled binary. If we do not promote and + inline these indirect calls before annotation, the profile for + these promoted functions will be lost. + + e.g. foo() --indirect_call--> bar() + In profiled binary, the callsite is promoted and inlined, making + the profile look like: + + foo: { + loc_foo_1: count_1 + bar@loc_foo_2: { + loc_bar_1: count_2 + loc_bar_2: count_3 + } + } + + Before AutoFDO pass, loc_foo_2 is not promoted thus not inlined. + If we perform annotation on it, the profile inside bar@loc_foo2 + will be wasted. + + To avoid this, we promote loc_foo_2 and inline the promoted bar + function before annotation, so the profile inside bar@loc_foo2 + will be useful. */ + autofdo::stmt_set promoted_stmts; + for (int i = 0; i < PARAM_VALUE (PARAM_EARLY_INLINER_MAX_ITERATIONS); i++) + { + if (!flag_value_profile_transformations + || !autofdo::afdo_vpt_for_early_inline (&promoted_stmts)) + break; + early_inline (); + } + + early_inline (); + autofdo::afdo_annotate_cfg (promoted_stmts); + compute_function_frequency (); + update_ssa (TODO_update_ssa); + + /* Local pure-const may imply need to fixup the cfg. */ + if (execute_fixup_cfg () & TODO_cleanup_cfg) + cleanup_tree_cfg (); + + free_dominance_info (CDI_DOMINATORS); + free_dominance_info (CDI_POST_DOMINATORS); + cgraph_edge::rebuild_edges (); + pop_cfun (); + } + + return TODO_rebuild_cgraph_edges; +} +} /* namespace autofdo. */ + +/* Read the profile from the profile data file. */ + +void +read_autofdo_file (void) +{ + if (auto_profile_file == NULL) + auto_profile_file = DEFAULT_AUTO_PROFILE_FILE; + + autofdo::afdo_profile_info = (struct gcov_ctr_summary *)xcalloc ( + 1, sizeof (struct gcov_ctr_summary)); + autofdo::afdo_profile_info->runs = 1; + autofdo::afdo_profile_info->sum_max = 0; + autofdo::afdo_profile_info->sum_all = 0; + + /* Read the profile from the profile file. */ + autofdo::read_profile (); +} + +/* Free the resources. */ + +void +end_auto_profile (void) +{ + delete autofdo::afdo_source_profile; + delete autofdo::afdo_string_table; + profile_info = NULL; +} + +/* Returns TRUE if EDGE is hot enough to be inlined early. */ + +bool +afdo_callsite_hot_enough_for_early_inline (struct cgraph_edge *edge) +{ + gcov_type count + = autofdo::afdo_source_profile->get_callsite_total_count (edge); + if (count > 0) + { + bool is_hot; + const struct gcov_ctr_summary *saved_profile_info = profile_info; + /* At earling inline stage, profile_info is not set yet. We need to + temporarily set it to afdo_profile_info to calculate hotness. */ + profile_info = autofdo::afdo_profile_info; + is_hot = maybe_hot_count_p (NULL, count); + profile_info = saved_profile_info; + return is_hot; + } + else + return false; +} + +namespace +{ + +const pass_data pass_data_ipa_auto_profile = { + SIMPLE_IPA_PASS, "afdo", /* name */ + OPTGROUP_NONE, /* optinfo_flags */ + TV_IPA_AUTOFDO, /* tv_id */ + 0, /* properties_required */ + 0, /* properties_provided */ + 0, /* properties_destroyed */ + 0, /* todo_flags_start */ + 0, /* todo_flags_finish */ +}; + +class pass_ipa_auto_profile : public simple_ipa_opt_pass +{ +public: + pass_ipa_auto_profile (gcc::context *ctxt) + : simple_ipa_opt_pass (pass_data_ipa_auto_profile, ctxt) + { + } + + /* opt_pass methods: */ + virtual bool + gate (function *) + { + return flag_auto_profile; + } + virtual unsigned int + execute (function *) + { + return autofdo::auto_profile (); + } +}; // class pass_ipa_auto_profile + +} // anon namespace + +simple_ipa_opt_pass * +make_pass_ipa_auto_profile (gcc::context *ctxt) +{ + return new pass_ipa_auto_profile (ctxt); +} Index: gcc/auto-profile.h =================================================================== --- gcc/auto-profile.h (revision 0) +++ gcc/auto-profile.h (revision 0) @@ -0,0 +1,31 @@ +/* auto-profile.h - Defines data exported from auto-profile.c + Copyright (C) 2014. Free Software Foundation, Inc. + Contributed by Dehao Chen (dehao@google.com) + +This file is part of GCC. + +GCC is free software; you can redistribute it and/or modify it under +the terms of the GNU General Public License as published by the Free +Software Foundation; either version 3, or (at your option) any later +version. + +GCC is distributed in the hope that it will be useful, but WITHOUT ANY +WARRANTY; without even the implied warranty of MERCHANTABILITY or +FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License +for more details. + +You should have received a copy of the GNU General Public License +along with GCC; see the file COPYING3. If not see +<http://www.gnu.org/licenses/>. */ + +#ifndef AUTO_PROFILE_H +#define AUTO_PROFILE_H + +/* Read, process, finalize AutoFDO data structures. */ +extern void read_autofdo_file (void); +extern void end_auto_profile (void); + +/* Returns TRUE if EDGE is hot enough to be inlined early. */ +extern bool afdo_callsite_hot_enough_for_early_inline (struct cgraph_edge *); + +#endif /* AUTO_PROFILE_H */ Index: gcc/debug.h =================================================================== --- gcc/debug.h (revision 215826) +++ gcc/debug.h (working copy) @@ -169,6 +169,7 @@ extern const struct gcc_debug_hooks dbx_debug_hook extern const struct gcc_debug_hooks sdb_debug_hooks; extern const struct gcc_debug_hooks xcoff_debug_hooks; extern const struct gcc_debug_hooks dwarf2_debug_hooks; +extern const struct gcc_debug_hooks auto_profile_debug_hooks; extern const struct gcc_debug_hooks vmsdbg_debug_hooks; /* Dwarf2 frame information. */ Index: gcc/tree-pass.h =================================================================== --- gcc/tree-pass.h (revision 215826) +++ gcc/tree-pass.h (working copy) @@ -449,6 +449,7 @@ extern simple_ipa_opt_pass *make_pass_ipa_lower_em extern simple_ipa_opt_pass *make_pass_ipa_function_and_variable_visibility (gcc::context *ctxt); extern simple_ipa_opt_pass *make_pass_ipa_tree_profile (gcc::context *ctxt); +extern simple_ipa_opt_pass *make_pass_ipa_auto_profile (gcc::context *ctxt); extern simple_ipa_opt_pass *make_pass_early_local_passes (gcc::context *ctxt); Index: gcc/ipa-inline.c =================================================================== --- gcc/ipa-inline.c (revision 215826) +++ gcc/ipa-inline.c (working copy) @@ -121,6 +121,7 @@ along with GCC; see the file COPYING3. If not see #include "ipa-inline.h" #include "ipa-utils.h" #include "sreal.h" +#include "auto-profile.h" #include "cilk.h" #include "builtins.h" @@ -449,6 +450,14 @@ want_early_inline_function_p (struct cgraph_edge * if (DECL_DISREGARD_INLINE_LIMITS (callee->decl)) ; + /* For AutoFDO, we need to make sure that before profile annotation, all + hot paths' IR look exactly the same as profiled binary. As a result, + in einliner, we will disregard size limit and inline those callsites + that are: + * inlined in the profiled binary, and + * the cloned callee has enough samples to be considered "hot". */ + else if (flag_auto_profile && afdo_callsite_hot_enough_for_early_inline (e)) + ; else if (!DECL_DECLARED_INLINE_P (callee->decl) && !flag_inline_small_functions) { @@ -2366,39 +2375,8 @@ early_inline_small_functions (struct cgraph_node * return inlined; } -/* Do inlining of small functions. Doing so early helps profiling and other - passes to be somewhat more effective and avoids some code duplication in - later real inlining pass for testcases with very many function calls. */ - -namespace { - -const pass_data pass_data_early_inline = -{ - GIMPLE_PASS, /* type */ - "einline", /* name */ - OPTGROUP_INLINE, /* optinfo_flags */ - TV_EARLY_INLINING, /* tv_id */ - PROP_ssa, /* properties_required */ - 0, /* properties_provided */ - 0, /* properties_destroyed */ - 0, /* todo_flags_start */ - 0, /* todo_flags_finish */ -}; - -class pass_early_inline : public gimple_opt_pass -{ -public: - pass_early_inline (gcc::context *ctxt) - : gimple_opt_pass (pass_data_early_inline, ctxt) - {} - - /* opt_pass methods: */ - virtual unsigned int execute (function *); - -}; // class pass_early_inline - unsigned int -pass_early_inline::execute (function *fun) +early_inliner (function *fun) { struct cgraph_node *node = cgraph_node::get (current_function_decl); struct cgraph_edge *edge; @@ -2499,6 +2477,43 @@ unsigned int return todo; } +/* Do inlining of small functions. Doing so early helps profiling and other + passes to be somewhat more effective and avoids some code duplication in + later real inlining pass for testcases with very many function calls. */ + +namespace { + +const pass_data pass_data_early_inline = +{ + GIMPLE_PASS, /* type */ + "einline", /* name */ + OPTGROUP_INLINE, /* optinfo_flags */ + TV_EARLY_INLINING, /* tv_id */ + PROP_ssa, /* properties_required */ + 0, /* properties_provided */ + 0, /* properties_destroyed */ + 0, /* todo_flags_start */ + 0, /* todo_flags_finish */ +}; + +class pass_early_inline : public gimple_opt_pass +{ +public: + pass_early_inline (gcc::context *ctxt) + : gimple_opt_pass (pass_data_early_inline, ctxt) + {} + + /* opt_pass methods: */ + virtual unsigned int execute (function *); + +}; // class pass_early_inline + +unsigned int +pass_early_inline::execute (function *fun) +{ + return early_inliner (fun); +} + } // anon namespace gimple_opt_pass * Index: gcc/cfgloop.c =================================================================== --- gcc/cfgloop.c (revision 215826) +++ gcc/cfgloop.c (working copy) @@ -1802,7 +1802,7 @@ record_niter_bound (struct loop *loop, const wides } if (realistic && (!loop->any_estimate - || wi::ltu_p (i_bound, loop->nb_iterations_estimate))) + || (!flag_auto_profile && wi::ltu_p (i_bound, loop->nb_iterations_estimate)))) { loop->any_estimate = true; loop->nb_iterations_estimate = i_bound; Index: gcc/ipa-inline.h =================================================================== --- gcc/ipa-inline.h (revision 215826) +++ gcc/ipa-inline.h (working copy) @@ -238,6 +238,7 @@ void initialize_growth_caches (void); void free_growth_caches (void); void compute_inline_parameters (struct cgraph_node *, bool); bool speculation_useful_p (struct cgraph_edge *e, bool anticipate_inlining); +unsigned int early_inliner (function *fun); /* In ipa-inline-transform.c */ bool inline_call (struct cgraph_edge *, bool, vec<cgraph_edge *> *, int *, bool, Index: gcc/tree-profile.c =================================================================== --- gcc/tree-profile.c (revision 215826) +++ gcc/tree-profile.c (working copy) @@ -699,8 +699,10 @@ class pass_ipa_tree_profile : public simple_ipa_op bool pass_ipa_tree_profile::gate (function *) { - /* When profile instrumentation, use or test coverage shall be performed. */ - return (!in_lto_p + /* When profile instrumentation, use or test coverage shall be performed. + But for AutoFDO, this there is no instrumentation, thus this pass is + diabled. */ + return (!in_lto_p && !flag_auto_profile && (flag_branch_probabilities || flag_test_coverage || profile_arc_flag)); } Index: gcc/opts.c =================================================================== --- gcc/opts.c (revision 215826) +++ gcc/opts.c (working copy) @@ -1279,6 +1279,57 @@ print_specific_help (unsigned int include_flags, opts->x_help_columns, opts, lang_mask); } +/* Enable FDO-related flags. */ + +static void +enable_fdo_optimizations (struct gcc_options *opts, + struct gcc_options *opts_set, + int value) +{ + if (!opts_set->x_flag_branch_probabilities) + opts->x_flag_branch_probabilities = value; + if (!opts_set->x_flag_profile_values) + opts->x_flag_profile_values = value; + if (!opts_set->x_flag_unroll_loops) + opts->x_flag_unroll_loops = value; + if (!opts_set->x_flag_peel_loops) + opts->x_flag_peel_loops = value; + if (!opts_set->x_flag_tracer) + opts->x_flag_tracer = value; + if (!opts_set->x_flag_value_profile_transformations) + opts->x_flag_value_profile_transformations = value; + if (!opts_set->x_flag_inline_functions) + opts->x_flag_inline_functions = value; + if (!opts_set->x_flag_ipa_cp) + opts->x_flag_ipa_cp = value; + if (!opts_set->x_flag_ipa_cp_clone + && value && opts->x_flag_ipa_cp) + opts->x_flag_ipa_cp_clone = value; + if (!opts_set->x_flag_predictive_commoning) + opts->x_flag_predictive_commoning = value; + if (!opts_set->x_flag_unswitch_loops) + opts->x_flag_unswitch_loops = value; + if (!opts_set->x_flag_gcse_after_reload) + opts->x_flag_gcse_after_reload = value; + if (!opts_set->x_flag_tree_loop_vectorize + && !opts_set->x_flag_tree_vectorize) + opts->x_flag_tree_loop_vectorize = value; + if (!opts_set->x_flag_tree_slp_vectorize + && !opts_set->x_flag_tree_vectorize) + opts->x_flag_tree_slp_vectorize = value; + if (!opts_set->x_flag_vect_cost_model) + opts->x_flag_vect_cost_model = VECT_COST_MODEL_DYNAMIC; + if (!opts_set->x_flag_tree_loop_distribute_patterns) + opts->x_flag_tree_loop_distribute_patterns = value; + if (!opts_set->x_flag_profile_reorder_functions) + opts->x_flag_profile_reorder_functions = value; + /* Indirect call profiling should do all useful transformations + speculative devirtualization does. */ + if (!opts_set->x_flag_devirtualize_speculatively + && opts->x_flag_value_profile_transformations) + opts->x_flag_devirtualize_speculatively = false; +} + /* Handle target- and language-independent options. Return zero to generate an "unknown option" message. Only options that need extra handling need to be listed here; if you simply want @@ -1746,50 +1797,23 @@ common_handle_option (struct gcc_options *opts, value = true; /* No break here - do -fprofile-use processing. */ case OPT_fprofile_use: - if (!opts_set->x_flag_branch_probabilities) - opts->x_flag_branch_probabilities = value; - if (!opts_set->x_flag_profile_values) - opts->x_flag_profile_values = value; - if (!opts_set->x_flag_unroll_loops) - opts->x_flag_unroll_loops = value; - if (!opts_set->x_flag_peel_loops) - opts->x_flag_peel_loops = value; - if (!opts_set->x_flag_tracer) - opts->x_flag_tracer = value; - if (!opts_set->x_flag_value_profile_transformations) - opts->x_flag_value_profile_transformations = value; - if (!opts_set->x_flag_inline_functions) - opts->x_flag_inline_functions = value; - if (!opts_set->x_flag_ipa_cp) - opts->x_flag_ipa_cp = value; - if (!opts_set->x_flag_ipa_cp_clone - && value && opts->x_flag_ipa_cp) - opts->x_flag_ipa_cp_clone = value; - if (!opts_set->x_flag_predictive_commoning) - opts->x_flag_predictive_commoning = value; - if (!opts_set->x_flag_unswitch_loops) - opts->x_flag_unswitch_loops = value; - if (!opts_set->x_flag_gcse_after_reload) - opts->x_flag_gcse_after_reload = value; - if (!opts_set->x_flag_tree_loop_vectorize - && !opts_set->x_flag_tree_vectorize) - opts->x_flag_tree_loop_vectorize = value; - if (!opts_set->x_flag_tree_slp_vectorize - && !opts_set->x_flag_tree_vectorize) - opts->x_flag_tree_slp_vectorize = value; - if (!opts_set->x_flag_vect_cost_model) - opts->x_flag_vect_cost_model = VECT_COST_MODEL_DYNAMIC; - if (!opts_set->x_flag_tree_loop_distribute_patterns) - opts->x_flag_tree_loop_distribute_patterns = value; - if (!opts_set->x_flag_profile_reorder_functions) - opts->x_flag_profile_reorder_functions = value; - /* Indirect call profiling should do all useful transformations - speculative devirtualization does. */ - if (!opts_set->x_flag_devirtualize_speculatively - && opts->x_flag_value_profile_transformations) - opts->x_flag_devirtualize_speculatively = false; + enable_fdo_optimizations (opts, opts_set, value); break; + case OPT_fauto_profile_: + opts->x_auto_profile_file = xstrdup (arg); + opts->x_flag_auto_profile = true; + value = true; + /* No break here - do -fauto-profile processing. */ + case OPT_fauto_profile: + enable_fdo_optimizations (opts, opts_set, value); + if (!opts_set->x_flag_profile_correction) + opts->x_flag_profile_correction = value; + maybe_set_param_value ( + PARAM_EARLY_INLINER_MAX_ITERATIONS, 10, + opts->x_param_values, opts_set->x_param_values); + break; + case OPT_fprofile_generate_: opts->x_profile_data_prefix = xstrdup (arg); value = true; ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH] AutoFDO patch for trunk 2014-10-16 18:57 ` Dehao Chen @ 2014-10-19 10:02 ` Jan Hubicka 2014-10-20 21:39 ` Dehao Chen 0 siblings, 1 reply; 25+ messages in thread From: Jan Hubicka @ 2014-10-19 10:02 UTC (permalink / raw) To: Dehao Chen; +Cc: Jan Hubicka, GCC Patches, David Li > >> +/* Member functions for string_table. */ > >> + > >> +string_table * > >> +string_table::create () > > > > Why this is not a constructor? > > We use static initializer because it's not suggested to put too much > logic in constructor. Why not? :) > >> + } > > > > The two hunks probably can be unified. > > Why get_index_by_decl does not already use dwarf name and why do you need to consider both? > > get_index_by_decl is actually a wrapper of get_index. It tolerates > error when assembler name is not emitted in the debug binary (mostly > for functions that are fully inlined). In this case, we will first try > to find the index of the assembler name, if not found, then bfd name. > If still not found, then we try to find name from its abstract > location. Yep, I am just somewhat concerned that you need to try different variations of the name. I would expect the assembler name (minus the random seed mangling) to be sufficient. > > > > >> + if (DECL_ABSTRACT_ORIGIN (decl)) > >> + return get_function_instance_by_decl (lineno, DECL_ABSTRACT_ORIGIN (decl)); > > > > What really happens for ipa-cp function clones and for split functions? > > Their name suffix will be stripped before matching names. I see and profile merged, that seems resonable. > > I am not sure it is a win to have the things represented as VPT histograms when you shoot > > for quite special purpose problem. But lets deal with these incrementally, too. > > (in general I do not see why you try to share so much with the gcov based VTP - it seems > > easier to do the speculation by hand) > > Basically there are 2 problem: > > * before annotation. I agree this is a special purpose problem > * after annotation. This should be the same problem as VPT. Yep, I think adding VPT histograms to get devirtualization happen inter-procedurally with LTO is a good idea. You will need (incrementally I guess) to translate your indexes into the profile-id used or you will need to initialize profile-ids accordingly, so the IPA pass is able to identify the target cross-module. > >> +{ > >> + if (gcov_open (auto_profile_file, 1) == 0) > >> + error ("Cannot open profile file %s.", auto_profile_file); > >> + > >> + if (gcov_read_unsigned () != GCOV_DATA_MAGIC) > >> + error ("AutoFDO profile magic number does not mathch."); > >> + > >> + /* Skip the version number. */ > >> + gcov_read_unsigned (); > >> + > >> + /* Skip the empty integer. */ > >> + gcov_read_unsigned (); > > > > Perhaps we can just check the values? > > What do you mean? Check the value against 0? Yes, if you have version in there it seems to make sense to check it ;) I guess it is there so you can add extra stuff into the file format that will make it incompatible with current one, so probably we should reject all versions greater than 0. (especially because the tool is off-tree) > > > > Does this have chance to work inter-module? > > Yes, it works fine with LIPO. Important (for me) is to make it work with LTO, because LIPO is apparently not landing to 5.0. > > Yes, this could be done. > > After a second thought, I think we actually need to expose einliner > interface to autofdo (instead of doing vpt in early inliner). This is > because we need to mark icall as "promoted", so that later vpt passes > will not try to promote it again. Currently in AutoFDO, we use a > self-contained way (use a "set" to mark all promoted stmts). If we do > vpt in einline, then we will need some flag attached to gimple to > indicate whether it's already promoted. If we manage to remove the iteration loop that repeativly applies the inline plan, you only need way to mark edge as promoted. I guess you can just test the speculative flag on the edge to skip ones you dealt with earlier. Lets do that incrementally though. > + > +@item -fauto-profile > +@itemx -fauto-profile=@var{path} > +@opindex fauto-profile > +Enable sampling based feedback directed optimizations, and optimizations > +generally profitable only with profile feedback available. > + > +The following options are enabled: @code{-fbranch-probabilities}, @code{-fvpt}, > +@code{-funroll-loops}, @code{-fpeel-loops}, @code{-ftracer}, @code{-ftree-vectorize}, > +@code{ftree-loop-distribute-patterns} I wonder about loop peeling - we do not enale it by default because we do not believe our static branch predictor can well identify loops that have low expected trip count (except ones that can be fully inlined). I am not sure auto-FDO is much better here - how reliable the info about number of iteratoins is? But this is again something that can be handled separately. Documentation seems out of date though - it seems to enable also -finline, -fipa-cp and cloning, predictive commoning, unswitching, gcse-after-reload, You do not want to enable reorder-functions because autoFDO does not include the time profiler. Also I think speculative devirtualization should not be disabled. > + > +If @var{path} is specified, GCC looks at the @var{path} to find > +the profile feedback data files. > + > +In order to collect AutoFDO profile, you need to have: > + > +1. A linux system with linux perf support > +2. An Intel processor with last branch record (LBR) support Is it really a requirement or will it only make profiles more accurate? > + > +To collect the profile, first use linux perf to collect raw profile > +(see @uref{https://perf.wiki.kernel.org/}). > + > +E.g. > +perf record -e br_inst_retired:near_taken -b -o perf.data -- your_program You need some @ stuff to mark the example. > + > +Then use create_gcov tool, which takes raw profile and unstripped binary to > +generate AutoFDO profile that can be used by GCC. > +(see @uref{https://github.com/google/autofdo}). > + > +E.g. > +create_gcov --binary=your_program.unstripped --profile=perf.data --gcov=profile.afdo Same here. I see the AutoFDO tool is under apache licence. I wonder if we can not share it in tree like we do for some other dependencies (asan) for easier use. > Index: gcc/opts.c > =================================================================== > --- gcc/opts.c (revision 215826) > +++ gcc/opts.c (working copy) > @@ -1279,6 +1279,57 @@ print_specific_help (unsigned int include_flags, > opts->x_help_columns, opts, lang_mask); > } > > +/* Enable FDO-related flags. */ > + > +static void > +enable_fdo_optimizations (struct gcc_options *opts, > + struct gcc_options *opts_set, > + int value) > +{ > + if (!opts_set->x_flag_branch_probabilities) > + opts->x_flag_branch_probabilities = value; > + if (!opts_set->x_flag_profile_values) > + opts->x_flag_profile_values = value; > + if (!opts_set->x_flag_unroll_loops) > + opts->x_flag_unroll_loops = value; > + if (!opts_set->x_flag_peel_loops) > + opts->x_flag_peel_loops = value; > + if (!opts_set->x_flag_tracer) > + opts->x_flag_tracer = value; > + if (!opts_set->x_flag_value_profile_transformations) > + opts->x_flag_value_profile_transformations = value; > + if (!opts_set->x_flag_inline_functions) > + opts->x_flag_inline_functions = value; > + if (!opts_set->x_flag_ipa_cp) > + opts->x_flag_ipa_cp = value; > + if (!opts_set->x_flag_ipa_cp_clone > + && value && opts->x_flag_ipa_cp) > + opts->x_flag_ipa_cp_clone = value; > + if (!opts_set->x_flag_predictive_commoning) > + opts->x_flag_predictive_commoning = value; > + if (!opts_set->x_flag_unswitch_loops) > + opts->x_flag_unswitch_loops = value; > + if (!opts_set->x_flag_gcse_after_reload) > + opts->x_flag_gcse_after_reload = value; > + if (!opts_set->x_flag_tree_loop_vectorize > + && !opts_set->x_flag_tree_vectorize) > + opts->x_flag_tree_loop_vectorize = value; > + if (!opts_set->x_flag_tree_slp_vectorize > + && !opts_set->x_flag_tree_vectorize) > + opts->x_flag_tree_slp_vectorize = value; > + if (!opts_set->x_flag_vect_cost_model) > + opts->x_flag_vect_cost_model = VECT_COST_MODEL_DYNAMIC; > + if (!opts_set->x_flag_tree_loop_distribute_patterns) > + opts->x_flag_tree_loop_distribute_patterns = value; > + if (!opts_set->x_flag_profile_reorder_functions) > + opts->x_flag_profile_reorder_functions = value; > + /* Indirect call profiling should do all useful transformations > + speculative devirtualization does. */ > + if (!opts_set->x_flag_devirtualize_speculatively > + && opts->x_flag_value_profile_transformations) > + opts->x_flag_devirtualize_speculatively = false; Please move x_flag_devirtualize_speculatively and x_flag_profile_reorder_functions to -fprofile-use only. > +} > + > /* Handle target- and language-independent options. Return zero to > generate an "unknown option" message. Only options that need > extra handling need to be listed here; if you simply want > @@ -1746,50 +1797,23 @@ common_handle_option (struct gcc_options *opts, > value = true; > /* No break here - do -fprofile-use processing. */ > case OPT_fprofile_use: > - if (!opts_set->x_flag_branch_probabilities) > - opts->x_flag_branch_probabilities = value; > - if (!opts_set->x_flag_profile_values) > - opts->x_flag_profile_values = value; > - if (!opts_set->x_flag_unroll_loops) > - opts->x_flag_unroll_loops = value; > - if (!opts_set->x_flag_peel_loops) > - opts->x_flag_peel_loops = value; > - if (!opts_set->x_flag_tracer) > - opts->x_flag_tracer = value; > - if (!opts_set->x_flag_value_profile_transformations) > - opts->x_flag_value_profile_transformations = value; > - if (!opts_set->x_flag_inline_functions) > - opts->x_flag_inline_functions = value; > - if (!opts_set->x_flag_ipa_cp) > - opts->x_flag_ipa_cp = value; > - if (!opts_set->x_flag_ipa_cp_clone > - && value && opts->x_flag_ipa_cp) > - opts->x_flag_ipa_cp_clone = value; > - if (!opts_set->x_flag_predictive_commoning) > - opts->x_flag_predictive_commoning = value; > - if (!opts_set->x_flag_unswitch_loops) > - opts->x_flag_unswitch_loops = value; > - if (!opts_set->x_flag_gcse_after_reload) > - opts->x_flag_gcse_after_reload = value; > - if (!opts_set->x_flag_tree_loop_vectorize > - && !opts_set->x_flag_tree_vectorize) > - opts->x_flag_tree_loop_vectorize = value; > - if (!opts_set->x_flag_tree_slp_vectorize > - && !opts_set->x_flag_tree_vectorize) > - opts->x_flag_tree_slp_vectorize = value; > - if (!opts_set->x_flag_vect_cost_model) > - opts->x_flag_vect_cost_model = VECT_COST_MODEL_DYNAMIC; > - if (!opts_set->x_flag_tree_loop_distribute_patterns) > - opts->x_flag_tree_loop_distribute_patterns = value; > - if (!opts_set->x_flag_profile_reorder_functions) > - opts->x_flag_profile_reorder_functions = value; > - /* Indirect call profiling should do all useful transformations > - speculative devirtualization does. */ > - if (!opts_set->x_flag_devirtualize_speculatively > - && opts->x_flag_value_profile_transformations) > - opts->x_flag_devirtualize_speculatively = false; > + enable_fdo_optimizations (opts, opts_set, value); > break; > > + case OPT_fauto_profile_: > + opts->x_auto_profile_file = xstrdup (arg); > + opts->x_flag_auto_profile = true; > + value = true; > + /* No break here - do -fauto-profile processing. */ > + case OPT_fauto_profile: > + enable_fdo_optimizations (opts, opts_set, value); > + if (!opts_set->x_flag_profile_correction) > + opts->x_flag_profile_correction = value; > + maybe_set_param_value ( > + PARAM_EARLY_INLINER_MAX_ITERATIONS, 10, > + opts->x_param_values, opts_set->x_param_values); > + break; > + > case OPT_fprofile_generate_: > opts->x_profile_data_prefix = xstrdup (arg); > value = true; Patch is OK with these changes. Please also update news.html and perhaps even news on the main page? It is quite nice user visible change. Honza ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH] AutoFDO patch for trunk 2014-10-19 10:02 ` Jan Hubicka @ 2014-10-20 21:39 ` Dehao Chen 2014-10-21 19:43 ` Markus Trippelsdorf ` (2 more replies) 0 siblings, 3 replies; 25+ messages in thread From: Dehao Chen @ 2014-10-20 21:39 UTC (permalink / raw) To: Jan Hubicka; +Cc: GCC Patches, David Li [-- Attachment #1: Type: text/plain, Size: 13682 bytes --] The updated patch attached. Will commit the patch in 2~3 hours if no objection is received. Thanks, Dehao On Sun, Oct 19, 2014 at 2:58 AM, Jan Hubicka <hubicka@ucw.cz> wrote: >> >> +/* Member functions for string_table. */ >> >> + >> >> +string_table * >> >> +string_table::create () >> > >> > Why this is not a constructor? >> >> We use static initializer because it's not suggested to put too much >> logic in constructor. > > Why not? :) You're right. Updated to use constructor instead. >> >> + } >> > >> > The two hunks probably can be unified. >> > Why get_index_by_decl does not already use dwarf name and why do you need to consider both? >> >> get_index_by_decl is actually a wrapper of get_index. It tolerates >> error when assembler name is not emitted in the debug binary (mostly >> for functions that are fully inlined). In this case, we will first try >> to find the index of the assembler name, if not found, then bfd name. >> If still not found, then we try to find name from its abstract >> location. > > Yep, I am just somewhat concerned that you need to try different variations of the name. > I would expect the assembler name (minus the random seed mangling) to be sufficient. In theory, assembler name should be good enough. But in practice, we still see that for some functions, assembler name is not available. This function purely exists to tolerate faults like this. And should be simplified once debug info becomes perfect. >> >> > >> >> + if (DECL_ABSTRACT_ORIGIN (decl)) >> >> + return get_function_instance_by_decl (lineno, DECL_ABSTRACT_ORIGIN (decl)); >> > >> > What really happens for ipa-cp function clones and for split functions? >> >> Their name suffix will be stripped before matching names. > > I see and profile merged, that seems resonable. >> > I am not sure it is a win to have the things represented as VPT histograms when you shoot >> > for quite special purpose problem. But lets deal with these incrementally, too. >> > (in general I do not see why you try to share so much with the gcov based VTP - it seems >> > easier to do the speculation by hand) >> >> Basically there are 2 problem: >> >> * before annotation. I agree this is a special purpose problem >> * after annotation. This should be the same problem as VPT. > > Yep, I think adding VPT histograms to get devirtualization happen inter-procedurally > with LTO is a good idea. > You will need (incrementally I guess) to translate your indexes into the profile-id used > or you will need to initialize profile-ids accordingly, so the IPA pass is able to identify > the target cross-module. Sure, let's do this incrementally. But for the symbol look up, we actually use func_name instead of profile-id. So I guess we don't have the problem you mentioned here? (except that function name my collide). >> >> +{ >> >> + if (gcov_open (auto_profile_file, 1) == 0) >> >> + error ("Cannot open profile file %s.", auto_profile_file); >> >> + >> >> + if (gcov_read_unsigned () != GCOV_DATA_MAGIC) >> >> + error ("AutoFDO profile magic number does not mathch."); >> >> + >> >> + /* Skip the version number. */ >> >> + gcov_read_unsigned (); >> >> + >> >> + /* Skip the empty integer. */ >> >> + gcov_read_unsigned (); >> > >> > Perhaps we can just check the values? >> >> What do you mean? Check the value against 0? > > Yes, if you have version in there it seems to make sense to check it ;) Done > I guess it is there so you can add extra stuff into the file format that will make > it incompatible with current one, so probably we should reject all versions greater > than 0. (especially because the tool is off-tree) >> > >> > Does this have chance to work inter-module? >> >> Yes, it works fine with LIPO. > > Important (for me) is to make it work with LTO, because LIPO is apparently not landing to 5.0. I think we only need to handle function name colliding case. This could be simply be resolved by coupling function name with file name. >> >> Yes, this could be done. >> >> After a second thought, I think we actually need to expose einliner >> interface to autofdo (instead of doing vpt in early inliner). This is >> because we need to mark icall as "promoted", so that later vpt passes >> will not try to promote it again. Currently in AutoFDO, we use a >> self-contained way (use a "set" to mark all promoted stmts). If we do >> vpt in einline, then we will need some flag attached to gimple to >> indicate whether it's already promoted. > > If we manage to remove the iteration loop that repeativly applies the inline plan, you only > need way to mark edge as promoted. I guess you can just test the speculative flag on the edge > to skip ones you dealt with earlier. > > Lets do that incrementally though. Agree, let's do it incrementally. >> + >> +@item -fauto-profile >> +@itemx -fauto-profile=@var{path} >> +@opindex fauto-profile >> +Enable sampling based feedback directed optimizations, and optimizations >> +generally profitable only with profile feedback available. >> + >> +The following options are enabled: @code{-fbranch-probabilities}, @code{-fvpt}, >> +@code{-funroll-loops}, @code{-fpeel-loops}, @code{-ftracer}, @code{-ftree-vectorize}, >> +@code{ftree-loop-distribute-patterns} > > I wonder about loop peeling - we do not enale it by default because we do not believe > our static branch predictor can well identify loops that have low expected trip count > (except ones that can be fully inlined). > I am not sure auto-FDO is much better here - how reliable the info about number of iteratoins > is? With discriminator, the loop iter estimate is quite accurate. so loop-peeling should be turned on. > > But this is again something that can be handled separately. > > Documentation seems out of date though - it seems to enable also > -finline, -fipa-cp and cloning, predictive commoning, unswitching, gcse-after-reload, > > You do not want to enable reorder-functions because autoFDO does not include the time profiler. > Also I think speculative devirtualization should not be disabled. Why reorder function should be disabled? In google branch reorder function actually buys us a lot. We use both prefix based reordering and call graph based reordering. >> + >> +If @var{path} is specified, GCC looks at the @var{path} to find >> +the profile feedback data files. >> + >> +In order to collect AutoFDO profile, you need to have: >> + >> +1. A linux system with linux perf support >> +2. An Intel processor with last branch record (LBR) support > > Is it really a requirement or will it only make profiles more accurate? It's optional, updated the doc. >> + >> +To collect the profile, first use linux perf to collect raw profile >> +(see @uref{https://perf.wiki.kernel.org/}). >> + >> +E.g. >> +perf record -e br_inst_retired:near_taken -b -o perf.data -- your_program > > You need some @ stuff to mark the example. Done. >> + >> +Then use create_gcov tool, which takes raw profile and unstripped binary to >> +generate AutoFDO profile that can be used by GCC. >> +(see @uref{https://github.com/google/autofdo}). >> + >> +E.g. >> +create_gcov --binary=your_program.unstripped --profile=perf.data --gcov=profile.afdo > > Same here. Done. > > I see the AutoFDO tool is under apache licence. I wonder if we can not share it in tree > like we do for some other dependencies (asan) for easier use. I'm not an expert on license. But it'd be great if someone can volunteer to move the tool into GCC. > >> Index: gcc/opts.c >> =================================================================== >> --- gcc/opts.c (revision 215826) >> +++ gcc/opts.c (working copy) >> @@ -1279,6 +1279,57 @@ print_specific_help (unsigned int include_flags, >> opts->x_help_columns, opts, lang_mask); >> } >> >> +/* Enable FDO-related flags. */ >> + >> +static void >> +enable_fdo_optimizations (struct gcc_options *opts, >> + struct gcc_options *opts_set, >> + int value) >> +{ >> + if (!opts_set->x_flag_branch_probabilities) >> + opts->x_flag_branch_probabilities = value; >> + if (!opts_set->x_flag_profile_values) >> + opts->x_flag_profile_values = value; >> + if (!opts_set->x_flag_unroll_loops) >> + opts->x_flag_unroll_loops = value; >> + if (!opts_set->x_flag_peel_loops) >> + opts->x_flag_peel_loops = value; >> + if (!opts_set->x_flag_tracer) >> + opts->x_flag_tracer = value; >> + if (!opts_set->x_flag_value_profile_transformations) >> + opts->x_flag_value_profile_transformations = value; >> + if (!opts_set->x_flag_inline_functions) >> + opts->x_flag_inline_functions = value; >> + if (!opts_set->x_flag_ipa_cp) >> + opts->x_flag_ipa_cp = value; >> + if (!opts_set->x_flag_ipa_cp_clone >> + && value && opts->x_flag_ipa_cp) >> + opts->x_flag_ipa_cp_clone = value; >> + if (!opts_set->x_flag_predictive_commoning) >> + opts->x_flag_predictive_commoning = value; >> + if (!opts_set->x_flag_unswitch_loops) >> + opts->x_flag_unswitch_loops = value; >> + if (!opts_set->x_flag_gcse_after_reload) >> + opts->x_flag_gcse_after_reload = value; >> + if (!opts_set->x_flag_tree_loop_vectorize >> + && !opts_set->x_flag_tree_vectorize) >> + opts->x_flag_tree_loop_vectorize = value; >> + if (!opts_set->x_flag_tree_slp_vectorize >> + && !opts_set->x_flag_tree_vectorize) >> + opts->x_flag_tree_slp_vectorize = value; >> + if (!opts_set->x_flag_vect_cost_model) >> + opts->x_flag_vect_cost_model = VECT_COST_MODEL_DYNAMIC; >> + if (!opts_set->x_flag_tree_loop_distribute_patterns) >> + opts->x_flag_tree_loop_distribute_patterns = value; >> + if (!opts_set->x_flag_profile_reorder_functions) >> + opts->x_flag_profile_reorder_functions = value; >> + /* Indirect call profiling should do all useful transformations >> + speculative devirtualization does. */ >> + if (!opts_set->x_flag_devirtualize_speculatively >> + && opts->x_flag_value_profile_transformations) >> + opts->x_flag_devirtualize_speculatively = false; > Please move x_flag_devirtualize_speculatively and x_flag_profile_reorder_functions > to -fprofile-use only. Done. >> +} >> + >> /* Handle target- and language-independent options. Return zero to >> generate an "unknown option" message. Only options that need >> extra handling need to be listed here; if you simply want >> @@ -1746,50 +1797,23 @@ common_handle_option (struct gcc_options *opts, >> value = true; >> /* No break here - do -fprofile-use processing. */ >> case OPT_fprofile_use: >> - if (!opts_set->x_flag_branch_probabilities) >> - opts->x_flag_branch_probabilities = value; >> - if (!opts_set->x_flag_profile_values) >> - opts->x_flag_profile_values = value; >> - if (!opts_set->x_flag_unroll_loops) >> - opts->x_flag_unroll_loops = value; >> - if (!opts_set->x_flag_peel_loops) >> - opts->x_flag_peel_loops = value; >> - if (!opts_set->x_flag_tracer) >> - opts->x_flag_tracer = value; >> - if (!opts_set->x_flag_value_profile_transformations) >> - opts->x_flag_value_profile_transformations = value; >> - if (!opts_set->x_flag_inline_functions) >> - opts->x_flag_inline_functions = value; >> - if (!opts_set->x_flag_ipa_cp) >> - opts->x_flag_ipa_cp = value; >> - if (!opts_set->x_flag_ipa_cp_clone >> - && value && opts->x_flag_ipa_cp) >> - opts->x_flag_ipa_cp_clone = value; >> - if (!opts_set->x_flag_predictive_commoning) >> - opts->x_flag_predictive_commoning = value; >> - if (!opts_set->x_flag_unswitch_loops) >> - opts->x_flag_unswitch_loops = value; >> - if (!opts_set->x_flag_gcse_after_reload) >> - opts->x_flag_gcse_after_reload = value; >> - if (!opts_set->x_flag_tree_loop_vectorize >> - && !opts_set->x_flag_tree_vectorize) >> - opts->x_flag_tree_loop_vectorize = value; >> - if (!opts_set->x_flag_tree_slp_vectorize >> - && !opts_set->x_flag_tree_vectorize) >> - opts->x_flag_tree_slp_vectorize = value; >> - if (!opts_set->x_flag_vect_cost_model) >> - opts->x_flag_vect_cost_model = VECT_COST_MODEL_DYNAMIC; >> - if (!opts_set->x_flag_tree_loop_distribute_patterns) >> - opts->x_flag_tree_loop_distribute_patterns = value; >> - if (!opts_set->x_flag_profile_reorder_functions) >> - opts->x_flag_profile_reorder_functions = value; >> - /* Indirect call profiling should do all useful transformations >> - speculative devirtualization does. */ >> - if (!opts_set->x_flag_devirtualize_speculatively >> - && opts->x_flag_value_profile_transformations) >> - opts->x_flag_devirtualize_speculatively = false; >> + enable_fdo_optimizations (opts, opts_set, value); >> break; >> >> + case OPT_fauto_profile_: >> + opts->x_auto_profile_file = xstrdup (arg); >> + opts->x_flag_auto_profile = true; >> + value = true; >> + /* No break here - do -fauto-profile processing. */ >> + case OPT_fauto_profile: >> + enable_fdo_optimizations (opts, opts_set, value); >> + if (!opts_set->x_flag_profile_correction) >> + opts->x_flag_profile_correction = value; >> + maybe_set_param_value ( >> + PARAM_EARLY_INLINER_MAX_ITERATIONS, 10, >> + opts->x_param_values, opts_set->x_param_values); >> + break; >> + >> case OPT_fprofile_generate_: >> opts->x_profile_data_prefix = xstrdup (arg); >> value = true; > > Patch is OK with these changes. Please also update news.html and perhaps even > news on the main page? It is quite nice user visible change. Where is news.html? Thanks, Dehao > > Honza [-- Attachment #2: patch.txt --] [-- Type: text/plain, Size: 76252 bytes --] Index: gcc/profile.c =================================================================== --- gcc/profile.c (revision 215826) +++ gcc/profile.c (working copy) @@ -106,6 +106,14 @@ static int total_num_times_called; static int total_hist_br_prob[20]; static int total_num_branches; +/* Helper function to update gcov_working_sets. */ + +void add_working_set (gcov_working_set_t *set) { + int i = 0; + for (; i < NUM_GCOV_WORKING_SETS; i++) + gcov_working_sets[i] = set[i]; +} + /* Forward declarations. */ static void find_spanning_tree (struct edge_list *); Index: gcc/profile.h =================================================================== --- gcc/profile.h (revision 215826) +++ gcc/profile.h (working copy) @@ -47,6 +47,7 @@ extern void init_node_map (bool); extern void del_node_map (void); extern void get_working_sets (void); +extern void add_working_set (gcov_working_set_t *); /* In predict.c. */ extern gcov_type get_hot_bb_threshold (void); Index: gcc/opts.c =================================================================== --- gcc/opts.c (revision 215826) +++ gcc/opts.c (working copy) @@ -1279,6 +1279,50 @@ print_specific_help (unsigned int include_flags, opts->x_help_columns, opts, lang_mask); } +/* Enable FDO-related flags. */ + +static void +enable_fdo_optimizations (struct gcc_options *opts, + struct gcc_options *opts_set, + int value) +{ + if (!opts_set->x_flag_branch_probabilities) + opts->x_flag_branch_probabilities = value; + if (!opts_set->x_flag_profile_values) + opts->x_flag_profile_values = value; + if (!opts_set->x_flag_unroll_loops) + opts->x_flag_unroll_loops = value; + if (!opts_set->x_flag_peel_loops) + opts->x_flag_peel_loops = value; + if (!opts_set->x_flag_tracer) + opts->x_flag_tracer = value; + if (!opts_set->x_flag_value_profile_transformations) + opts->x_flag_value_profile_transformations = value; + if (!opts_set->x_flag_inline_functions) + opts->x_flag_inline_functions = value; + if (!opts_set->x_flag_ipa_cp) + opts->x_flag_ipa_cp = value; + if (!opts_set->x_flag_ipa_cp_clone + && value && opts->x_flag_ipa_cp) + opts->x_flag_ipa_cp_clone = value; + if (!opts_set->x_flag_predictive_commoning) + opts->x_flag_predictive_commoning = value; + if (!opts_set->x_flag_unswitch_loops) + opts->x_flag_unswitch_loops = value; + if (!opts_set->x_flag_gcse_after_reload) + opts->x_flag_gcse_after_reload = value; + if (!opts_set->x_flag_tree_loop_vectorize + && !opts_set->x_flag_tree_vectorize) + opts->x_flag_tree_loop_vectorize = value; + if (!opts_set->x_flag_tree_slp_vectorize + && !opts_set->x_flag_tree_vectorize) + opts->x_flag_tree_slp_vectorize = value; + if (!opts_set->x_flag_vect_cost_model) + opts->x_flag_vect_cost_model = VECT_COST_MODEL_DYNAMIC; + if (!opts_set->x_flag_tree_loop_distribute_patterns) + opts->x_flag_tree_loop_distribute_patterns = value; +} + /* Handle target- and language-independent options. Return zero to generate an "unknown option" message. Only options that need extra handling need to be listed here; if you simply want @@ -1746,50 +1790,30 @@ common_handle_option (struct gcc_options *opts, value = true; /* No break here - do -fprofile-use processing. */ case OPT_fprofile_use: - if (!opts_set->x_flag_branch_probabilities) - opts->x_flag_branch_probabilities = value; - if (!opts_set->x_flag_profile_values) - opts->x_flag_profile_values = value; - if (!opts_set->x_flag_unroll_loops) - opts->x_flag_unroll_loops = value; - if (!opts_set->x_flag_peel_loops) - opts->x_flag_peel_loops = value; - if (!opts_set->x_flag_tracer) - opts->x_flag_tracer = value; - if (!opts_set->x_flag_value_profile_transformations) - opts->x_flag_value_profile_transformations = value; - if (!opts_set->x_flag_inline_functions) - opts->x_flag_inline_functions = value; - if (!opts_set->x_flag_ipa_cp) - opts->x_flag_ipa_cp = value; - if (!opts_set->x_flag_ipa_cp_clone - && value && opts->x_flag_ipa_cp) - opts->x_flag_ipa_cp_clone = value; - if (!opts_set->x_flag_predictive_commoning) - opts->x_flag_predictive_commoning = value; - if (!opts_set->x_flag_unswitch_loops) - opts->x_flag_unswitch_loops = value; - if (!opts_set->x_flag_gcse_after_reload) - opts->x_flag_gcse_after_reload = value; - if (!opts_set->x_flag_tree_loop_vectorize - && !opts_set->x_flag_tree_vectorize) - opts->x_flag_tree_loop_vectorize = value; - if (!opts_set->x_flag_tree_slp_vectorize - && !opts_set->x_flag_tree_vectorize) - opts->x_flag_tree_slp_vectorize = value; - if (!opts_set->x_flag_vect_cost_model) - opts->x_flag_vect_cost_model = VECT_COST_MODEL_DYNAMIC; - if (!opts_set->x_flag_tree_loop_distribute_patterns) - opts->x_flag_tree_loop_distribute_patterns = value; + enable_fdo_optimizations (opts, opts_set, value); if (!opts_set->x_flag_profile_reorder_functions) - opts->x_flag_profile_reorder_functions = value; - /* Indirect call profiling should do all useful transformations - speculative devirtualization does. */ + opts->x_flag_profile_reorder_functions = value; + /* Indirect call profiling should do all useful transformations + speculative devirtualization does. */ if (!opts_set->x_flag_devirtualize_speculatively && opts->x_flag_value_profile_transformations) opts->x_flag_devirtualize_speculatively = false; break; + case OPT_fauto_profile_: + opts->x_auto_profile_file = xstrdup (arg); + opts->x_flag_auto_profile = true; + value = true; + /* No break here - do -fauto-profile processing. */ + case OPT_fauto_profile: + enable_fdo_optimizations (opts, opts_set, value); + if (!opts_set->x_flag_profile_correction) + opts->x_flag_profile_correction = value; + maybe_set_param_value ( + PARAM_EARLY_INLINER_MAX_ITERATIONS, 10, + opts->x_param_values, opts_set->x_param_values); + break; + case OPT_fprofile_generate_: opts->x_profile_data_prefix = xstrdup (arg); value = true; Index: gcc/passes.def =================================================================== --- gcc/passes.def (revision 215826) +++ gcc/passes.def (working copy) @@ -90,6 +90,7 @@ along with GCC; see the file COPYING3. If not see NEXT_PASS (pass_rebuild_cgraph_edges); NEXT_PASS (pass_inline_parameters); POP_INSERT_PASSES () + NEXT_PASS (pass_ipa_auto_profile); NEXT_PASS (pass_ipa_free_inline_summary); NEXT_PASS (pass_ipa_tree_profile); PUSH_INSERT_PASSES_WITHIN (pass_ipa_tree_profile) Index: gcc/debug.h =================================================================== --- gcc/debug.h (revision 215826) +++ gcc/debug.h (working copy) @@ -169,6 +169,7 @@ extern const struct gcc_debug_hooks dbx_debug_hook extern const struct gcc_debug_hooks sdb_debug_hooks; extern const struct gcc_debug_hooks xcoff_debug_hooks; extern const struct gcc_debug_hooks dwarf2_debug_hooks; +extern const struct gcc_debug_hooks auto_profile_debug_hooks; extern const struct gcc_debug_hooks vmsdbg_debug_hooks; /* Dwarf2 frame information. */ Index: gcc/tree-pass.h =================================================================== --- gcc/tree-pass.h (revision 215826) +++ gcc/tree-pass.h (working copy) @@ -449,6 +449,7 @@ extern simple_ipa_opt_pass *make_pass_ipa_lower_em extern simple_ipa_opt_pass *make_pass_ipa_function_and_variable_visibility (gcc::context *ctxt); extern simple_ipa_opt_pass *make_pass_ipa_tree_profile (gcc::context *ctxt); +extern simple_ipa_opt_pass *make_pass_ipa_auto_profile (gcc::context *ctxt); extern simple_ipa_opt_pass *make_pass_early_local_passes (gcc::context *ctxt); Index: gcc/cfgloop.c =================================================================== --- gcc/cfgloop.c (revision 215826) +++ gcc/cfgloop.c (working copy) @@ -1802,7 +1802,7 @@ record_niter_bound (struct loop *loop, const wides } if (realistic && (!loop->any_estimate - || wi::ltu_p (i_bound, loop->nb_iterations_estimate))) + || (!flag_auto_profile && wi::ltu_p (i_bound, loop->nb_iterations_estimate)))) { loop->any_estimate = true; loop->nb_iterations_estimate = i_bound; Index: gcc/coverage.c =================================================================== --- gcc/coverage.c (revision 215826) +++ gcc/coverage.c (working copy) @@ -55,6 +55,7 @@ along with GCC; see the file COPYING3. If not see #include "filenames.h" #include "target.h" #include "params.h" +#include "auto-profile.h" #include "gcov-io.h" #include "gcov-io.c" @@ -1208,7 +1209,9 @@ coverage_init (const char *filename) bbg_file_stamp = local_tick; - if (flag_branch_probabilities) + if (flag_auto_profile) + read_autofdo_file (); + else if (flag_branch_probabilities) read_counts_file (); /* Name of bbg file. */ Index: gcc/auto-profile.c =================================================================== --- gcc/auto-profile.c (revision 0) +++ gcc/auto-profile.c (revision 0) @@ -0,0 +1,1687 @@ +/* Read and annotate call graph profile from the auto profile data file. + Copyright (C) 2014. Free Software Foundation, Inc. + Contributed by Dehao Chen (dehao@google.com) + +This file is part of GCC. + +GCC is free software; you can redistribute it and/or modify it under +the terms of the GNU General Public License as published by the Free +Software Foundation; either version 3, or (at your option) any later +version. + +GCC is distributed in the hope that it will be useful, but WITHOUT ANY +WARRANTY; without even the implied warranty of MERCHANTABILITY or +FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License +for more details. + +You should have received a copy of the GNU General Public License +along with GCC; see the file COPYING3. If not see +<http://www.gnu.org/licenses/>. */ + +#include <string.h> +#include <map> +#include <set> + +#include "config.h" +#include "system.h" +#include "coretypes.h" +#include "tree.h" +#include "tree-pass.h" +#include "flags.h" +#include "basic-block.h" +#include "diagnostic-core.h" +#include "gcov-io.h" +#include "input.h" +#include "profile.h" +#include "langhooks.h" +#include "opts.h" +#include "tree-pass.h" +#include "cfgloop.h" +#include "tree-ssa-alias.h" +#include "tree-cfg.h" +#include "tree-cfgcleanup.h" +#include "tree-ssa-operands.h" +#include "tree-into-ssa.h" +#include "internal-fn.h" +#include "is-a.h" +#include "gimple-expr.h" +#include "gimple.h" +#include "gimple-iterator.h" +#include "gimple-ssa.h" +#include "cgraph.h" +#include "value-prof.h" +#include "coverage.h" +#include "params.h" +#include "ipa-inline.h" +#include "tree-inline.h" +#include "stringpool.h" +#include "auto-profile.h" +#include "vec.h" + +/* The following routines implements AutoFDO optimization. + + This optimization uses sampling profiles to annotate basic block counts + and uses heuristics to estimate branch probabilities. + + There are three phases in AutoFDO: + + Phase 1: Read profile from the profile data file. + The following info is read from the profile datafile: + * string_table: a map between function name and its index. + * autofdo_source_profile: a map from function_instance name to + function_instance. This is represented as a forest of + function_instances. + * WorkingSet: a histogram of how many instructions are covered for a + given percentage of total cycles. This is describing the binary + level information (not source level). This info is used to help + decide if we want aggressive optimizations that could increase + code footprint (e.g. loop unroll etc.) + A function instance is an instance of function that could either be a + standalone symbol, or a clone of a function that is inlined into another + function. + + Phase 2: Early inline + valur profile transformation. + Early inline uses autofdo_source_profile to find if a callsite is: + * inlined in the profiled binary. + * callee body is hot in the profiling run. + If both condition satisfies, early inline will inline the callsite + regardless of the code growth. + Phase 2 is an iterative process. During each iteration, we also check + if an indirect callsite is promoted and inlined in the profiling run. + If yes, vpt will happen to force promote it and in the next iteration, + einline will inline the promoted callsite in the next iteration. + + Phase 3: Annotate control flow graph. + AutoFDO uses a separate pass to: + * Annotate basic block count + * Estimate branch probability + + After the above 3 phases, all profile is readily annotated on the GCC IR. + AutoFDO tries to reuse all FDO infrastructure as much as possible to make + use of the profile. E.g. it uses existing mechanism to calculate the basic + block/edge frequency, as well as the cgraph node/edge count. +*/ + +#define DEFAULT_AUTO_PROFILE_FILE "fbdata.afdo" +#define AUTO_PROFILE_VERSION 1 + +namespace autofdo +{ + +/* Represent a source location: (function_decl, lineno). */ +typedef std::pair<tree, unsigned> decl_lineno; + +/* Represent an inline stack. vector[0] is the leaf node. */ +typedef vec<decl_lineno> inline_stack; + +/* String array that stores function names. */ +typedef vec<char *> string_vector; + +/* Map from function name's index in string_table to target's + execution count. */ +typedef std::map<unsigned, gcov_type> icall_target_map; + +/* Set of gimple stmts. Used to track if the stmt has already been promoted + to direct call. */ +typedef std::set<gimple> stmt_set; + +/* Represent count info of an inline stack. */ +struct count_info +{ + /* Sampled count of the inline stack. */ + gcov_type count; + + /* Map from indirect call target to its sample count. */ + icall_target_map targets; + + /* Whether this inline stack is already used in annotation. + + Each inline stack should only be used to annotate IR once. + This will be enforced when instruction-level discriminator + is supported. */ + bool annotated; +}; + +/* operator< for "const char *". */ +struct string_compare +{ + bool operator()(const char *a, const char *b) const + { + return strcmp (a, b) < 0; + } +}; + +/* Store a string array, indexed by string position in the array. */ +class string_table +{ +public: + string_table () + {} + + ~string_table (); + + /* For a given string, returns its index. */ + int get_index (const char *name) const; + + /* For a given decl, returns the index of the decl name. */ + int get_index_by_decl (tree decl) const; + + /* For a given index, returns the string. */ + const char *get_name (int index) const; + + /* Read profile, return TRUE on success. */ + bool read (); + +private: + typedef std::map<const char *, unsigned, string_compare> string_index_map; + string_vector vector_; + string_index_map map_; +}; + +/* Profile of a function instance: + 1. total_count of the function. + 2. head_count (entry basic block count) of the function (only valid when + function is a top-level function_instance, i.e. it is the original copy + instead of the inlined copy). + 3. map from source location (decl_lineno) to profile (count_info). + 4. map from callsite to callee function_instance. */ +class function_instance +{ +public: + typedef vec<function_instance *> function_instance_stack; + + /* Read the profile and return a function_instance with head count as + HEAD_COUNT. Recursively read callsites to create nested function_instances + too. STACK is used to track the recursive creation process. */ + static function_instance * + read_function_instance (function_instance_stack *stack, + gcov_type head_count); + + /* Recursively deallocate all callsites (nested function_instances). */ + ~function_instance (); + + /* Accessors. */ + int + name () const + { + return name_; + } + gcov_type + total_count () const + { + return total_count_; + } + gcov_type + head_count () const + { + return head_count_; + } + + /* Traverse callsites of the current function_instance to find one at the + location of LINENO and callee name represented in DECL. */ + function_instance *get_function_instance_by_decl (unsigned lineno, + tree decl) const; + + /* Store the profile info for LOC in INFO. Return TRUE if profile info + is found. */ + bool get_count_info (location_t loc, count_info *info) const; + + /* Read the inlined indirect call target profile for STMT and store it in + MAP, return the total count for all inlined indirect calls. */ + gcov_type find_icall_target_map (gimple stmt, icall_target_map *map) const; + + /* Sum of counts that is used during annotation. */ + gcov_type total_annotated_count () const; + + /* Mark LOC as annotated. */ + void mark_annotated (location_t loc); + +private: + /* Callsite, represented as (decl_lineno, callee_function_name_index). */ + typedef std::pair<unsigned, unsigned> callsite; + + /* Map from callsite to callee function_instance. */ + typedef std::map<callsite, function_instance *> callsite_map; + + function_instance (unsigned name, gcov_type head_count) + : name_ (name), total_count_ (0), head_count_ (head_count) + { + } + + /* Map from source location (decl_lineno) to profile (count_info). */ + typedef std::map<unsigned, count_info> position_count_map; + + /* function_instance name index in the string_table. */ + unsigned name_; + + /* Total sample count. */ + gcov_type total_count_; + + /* Entry BB's sample count. */ + gcov_type head_count_; + + /* Map from callsite location to callee function_instance. */ + callsite_map callsites; + + /* Map from source location to count_info. */ + position_count_map pos_counts; +}; + +/* Profile for all functions. */ +class autofdo_source_profile +{ +public: + static autofdo_source_profile * + create () + { + autofdo_source_profile *map = new autofdo_source_profile (); + + if (map->read ()) + return map; + delete map; + return NULL; + } + + ~autofdo_source_profile (); + + /* For a given DECL, returns the top-level function_instance. */ + function_instance *get_function_instance_by_decl (tree decl) const; + + /* Find count_info for a given gimple STMT. If found, store the count_info + in INFO and return true; otherwise return false. */ + bool get_count_info (gimple stmt, count_info *info) const; + + /* Find total count of the callee of EDGE. */ + gcov_type get_callsite_total_count (struct cgraph_edge *edge) const; + + /* Update value profile INFO for STMT from the inlined indirect callsite. + Return true if INFO is updated. */ + bool update_inlined_ind_target (gimple stmt, count_info *info); + + /* Mark LOC as annotated. */ + void mark_annotated (location_t loc); + +private: + /* Map from function_instance name index (in string_table) to + function_instance. */ + typedef std::map<unsigned, function_instance *> name_function_instance_map; + + autofdo_source_profile () {} + + /* Read AutoFDO profile and returns TRUE on success. */ + bool read (); + + /* Return the function_instance in the profile that correspond to the + inline STACK. */ + function_instance * + get_function_instance_by_inline_stack (const inline_stack &stack) const; + + name_function_instance_map map_; +}; + +/* Store the strings read from the profile data file. */ +static string_table *afdo_string_table; + +/* Store the AutoFDO source profile. */ +static autofdo_source_profile *afdo_source_profile; + +/* gcov_ctr_summary structure to store the profile_info. */ +static struct gcov_ctr_summary *afdo_profile_info; + +/* Helper functions. */ + +/* Return the original name of NAME: strip the suffix that starts + with '.' Caller is responsible for freeing RET. */ + +static char * +get_original_name (const char *name) +{ + char *ret = xstrdup (name); + char *find = strchr (ret, '.'); + if (find != NULL) + *find = 0; + return ret; +} + +/* Return the combined location, which is a 32bit integer in which + higher 16 bits stores the line offset of LOC to the start lineno + of DECL, The lower 16 bits stores the discrimnator. */ + +static unsigned +get_combined_location (location_t loc, tree decl) +{ + /* TODO: allow more bits for line and less bits for discriminator. */ + if (LOCATION_LINE (loc) - DECL_SOURCE_LINE (decl) >= (1<<16)) + warning_at (loc, OPT_Woverflow, "Offset exceeds 16 bytes."); + return ((LOCATION_LINE (loc) - DECL_SOURCE_LINE (decl)) << 16); +} + +/* Return the function decl of a given lexical BLOCK. */ + +static tree +get_function_decl_from_block (tree block) +{ + tree decl; + + if (LOCATION_LOCUS (BLOCK_SOURCE_LOCATION (block) == UNKNOWN_LOCATION)) + return NULL_TREE; + + for (decl = BLOCK_ABSTRACT_ORIGIN (block); + decl && (TREE_CODE (decl) == BLOCK); + decl = BLOCK_ABSTRACT_ORIGIN (decl)) + if (TREE_CODE (decl) == FUNCTION_DECL) + break; + return decl; +} + +/* Store inline stack for STMT in STACK. */ + +static void +get_inline_stack (location_t locus, inline_stack *stack) +{ + if (LOCATION_LOCUS (locus) == UNKNOWN_LOCATION) + return; + + tree block = LOCATION_BLOCK (locus); + if (block && TREE_CODE (block) == BLOCK) + { + int level = 0; + for (block = BLOCK_SUPERCONTEXT (block); + block && (TREE_CODE (block) == BLOCK); + block = BLOCK_SUPERCONTEXT (block)) + { + location_t tmp_locus = BLOCK_SOURCE_LOCATION (block); + if (LOCATION_LOCUS (tmp_locus) == UNKNOWN_LOCATION) + continue; + + tree decl = get_function_decl_from_block (block); + stack->safe_push ( + std::make_pair (decl, get_combined_location (locus, decl))); + locus = tmp_locus; + level++; + } + } + stack->safe_push ( + std::make_pair (current_function_decl, + get_combined_location (locus, current_function_decl))); +} + +/* Return STMT's combined location, which is a 32bit integer in which + higher 16 bits stores the line offset of LOC to the start lineno + of DECL, The lower 16 bits stores the discrimnator. */ + +static unsigned +get_relative_location_for_stmt (gimple stmt) +{ + location_t locus = gimple_location (stmt); + if (LOCATION_LOCUS (locus) == UNKNOWN_LOCATION) + return UNKNOWN_LOCATION; + + for (tree block = gimple_block (stmt); block && (TREE_CODE (block) == BLOCK); + block = BLOCK_SUPERCONTEXT (block)) + if (LOCATION_LOCUS (BLOCK_SOURCE_LOCATION (block)) != UNKNOWN_LOCATION) + return get_combined_location (locus, + get_function_decl_from_block (block)); + return get_combined_location (locus, current_function_decl); +} + +/* Return true if BB contains indirect call. */ + +static bool +has_indirect_call (basic_block bb) +{ + gimple_stmt_iterator gsi; + + for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi)) + { + gimple stmt = gsi_stmt (gsi); + if (gimple_code (stmt) == GIMPLE_CALL && !gimple_call_internal_p (stmt) + && (gimple_call_fn (stmt) == NULL + || TREE_CODE (gimple_call_fn (stmt)) != FUNCTION_DECL)) + return true; + } + return false; +} + +/* Member functions for string_table. */ + +/* Deconstructor. */ + +string_table::~string_table () +{ + for (unsigned i = 0; i < vector_.length (); i++) + free (vector_[i]); +} + + +/* Return the index of a given function NAME. Return -1 if NAME is not + found in string table. */ + +int +string_table::get_index (const char *name) const +{ + if (name == NULL) + return -1; + string_index_map::const_iterator iter = map_.find (name); + if (iter == map_.end ()) + return -1; + else + return iter->second; +} + +/* Return the index of a given function DECL. Return -1 if DECL is not + found in string table. */ + +int +string_table::get_index_by_decl (tree decl) const +{ + char *name + = get_original_name (IDENTIFIER_POINTER (DECL_ASSEMBLER_NAME (decl))); + int ret = get_index (name); + free (name); + if (ret != -1) + return ret; + ret = get_index (lang_hooks.dwarf_name (decl, 0)); + if (ret != -1) + return ret; + if (DECL_ABSTRACT_ORIGIN (decl)) + return get_index_by_decl (DECL_ABSTRACT_ORIGIN (decl)); + else + return -1; +} + +/* Return the function name of a given INDEX. */ + +const char * +string_table::get_name (int index) const +{ + gcc_assert (index > 0 && index < (int)vector_.length ()); + return vector_[index]; +} + +/* Read the string table. Return TRUE if reading is successful. */ + +bool +string_table::read () +{ + if (gcov_read_unsigned () != GCOV_TAG_AFDO_FILE_NAMES) + return false; + /* Skip the length of the section. */ + gcov_read_unsigned (); + /* Read in the file name table. */ + unsigned string_num = gcov_read_unsigned (); + for (unsigned i = 0; i < string_num; i++) + { + vector_.safe_push (get_original_name (gcov_read_string ())); + map_[vector_.last ()] = i; + } + return true; +} + +/* Member functions for function_instance. */ + +function_instance::~function_instance () +{ + for (callsite_map::iterator iter = callsites.begin (); + iter != callsites.end (); ++iter) + delete iter->second; +} + +/* Traverse callsites of the current function_instance to find one at the + location of LINENO and callee name represented in DECL. */ + +function_instance * +function_instance::get_function_instance_by_decl (unsigned lineno, + tree decl) const +{ + int func_name_idx = afdo_string_table->get_index_by_decl (decl); + if (func_name_idx != -1) + { + callsite_map::const_iterator ret + = callsites.find (std::make_pair (lineno, func_name_idx)); + if (ret != callsites.end ()) + return ret->second; + } + func_name_idx + = afdo_string_table->get_index (lang_hooks.dwarf_name (decl, 0)); + if (func_name_idx != -1) + { + callsite_map::const_iterator ret + = callsites.find (std::make_pair (lineno, func_name_idx)); + if (ret != callsites.end ()) + return ret->second; + } + if (DECL_ABSTRACT_ORIGIN (decl)) + return get_function_instance_by_decl (lineno, DECL_ABSTRACT_ORIGIN (decl)); + else + return NULL; +} + +/* Store the profile info for LOC in INFO. Return TRUE if profile info + is found. */ + +bool +function_instance::get_count_info (location_t loc, count_info *info) const +{ + position_count_map::const_iterator iter = pos_counts.find (loc); + if (iter == pos_counts.end ()) + return false; + *info = iter->second; + return true; +} + +/* Mark LOC as annotated. */ + +void +function_instance::mark_annotated (location_t loc) +{ + position_count_map::iterator iter = pos_counts.find (loc); + if (iter == pos_counts.end ()) + return; + iter->second.annotated = true; +} + +/* Read the inlinied indirect call target profile for STMT and store it in + MAP, return the total count for all inlined indirect calls. */ + +gcov_type +function_instance::find_icall_target_map (gimple stmt, + icall_target_map *map) const +{ + gcov_type ret = 0; + unsigned stmt_offset = get_relative_location_for_stmt (stmt); + + for (callsite_map::const_iterator iter = callsites.begin (); + iter != callsites.end (); ++iter) + { + unsigned callee = iter->second->name (); + /* Check if callsite location match the stmt. */ + if (iter->first.first != stmt_offset) + continue; + struct cgraph_node *node = cgraph_node::get_for_asmname ( + get_identifier (afdo_string_table->get_name (callee))); + if (node == NULL) + continue; + if (!check_ic_target (stmt, node)) + continue; + (*map)[callee] = iter->second->total_count (); + ret += iter->second->total_count (); + } + return ret; +} + +/* Read the profile and create a function_instance with head count as + HEAD_COUNT. Recursively read callsites to create nested function_instances + too. STACK is used to track the recursive creation process. */ + +/* function instance profile format: + + ENTRY_COUNT: 8 bytes + NAME_INDEX: 4 bytes + NUM_POS_COUNTS: 4 bytes + NUM_CALLSITES: 4 byte + POS_COUNT_1: + POS_1_OFFSET: 4 bytes + NUM_TARGETS: 4 bytes + COUNT: 8 bytes + TARGET_1: + VALUE_PROFILE_TYPE: 4 bytes + TARGET_IDX: 8 bytes + COUNT: 8 bytes + TARGET_2 + ... + TARGET_n + POS_COUNT_2 + ... + POS_COUNT_N + CALLSITE_1: + CALLSITE_1_OFFSET: 4 bytes + FUNCTION_INSTANCE_PROFILE (nested) + CALLSITE_2 + ... + CALLSITE_n. */ + +function_instance * +function_instance::read_function_instance (function_instance_stack *stack, + gcov_type head_count) +{ + unsigned name = gcov_read_unsigned (); + unsigned num_pos_counts = gcov_read_unsigned (); + unsigned num_callsites = gcov_read_unsigned (); + function_instance *s = new function_instance (name, head_count); + stack->safe_push (s); + + for (unsigned i = 0; i < num_pos_counts; i++) + { + unsigned offset = gcov_read_unsigned () & 0xffff0000; + unsigned num_targets = gcov_read_unsigned (); + gcov_type count = gcov_read_counter (); + s->pos_counts[offset].count = count; + for (unsigned j = 0; j < stack->length (); j++) + (*stack)[j]->total_count_ += count; + for (unsigned j = 0; j < num_targets; j++) + { + /* Only indirect call target histogram is supported now. */ + gcov_read_unsigned (); + gcov_type target_idx = gcov_read_counter (); + s->pos_counts[offset].targets[target_idx] = gcov_read_counter (); + } + } + for (unsigned i = 0; i < num_callsites; i++) + { + unsigned offset = gcov_read_unsigned (); + function_instance *callee_function_instance + = read_function_instance (stack, 0); + s->callsites[std::make_pair (offset, callee_function_instance->name ())] + = callee_function_instance; + } + stack->pop (); + return s; +} + +/* Sum of counts that is used during annotation. */ + +gcov_type +function_instance::total_annotated_count () const +{ + gcov_type ret = 0; + for (callsite_map::const_iterator iter = callsites.begin (); + iter != callsites.end (); ++iter) + ret += iter->second->total_annotated_count (); + for (position_count_map::const_iterator iter = pos_counts.begin (); + iter != pos_counts.end (); ++iter) + if (iter->second.annotated) + ret += iter->second.count; + return ret; +} + +/* Member functions for autofdo_source_profile. */ + +autofdo_source_profile::~autofdo_source_profile () +{ + for (name_function_instance_map::const_iterator iter = map_.begin (); + iter != map_.end (); ++iter) + delete iter->second; +} + +/* For a given DECL, returns the top-level function_instance. */ + +function_instance * +autofdo_source_profile::get_function_instance_by_decl (tree decl) const +{ + int index = afdo_string_table->get_index_by_decl (decl); + if (index == -1) + return NULL; + name_function_instance_map::const_iterator ret = map_.find (index); + return ret == map_.end () ? NULL : ret->second; +} + +/* Find count_info for a given gimple STMT. If found, store the count_info + in INFO and return true; otherwise return false. */ + +bool +autofdo_source_profile::get_count_info (gimple stmt, count_info *info) const +{ + if (LOCATION_LOCUS (gimple_location (stmt)) == cfun->function_end_locus) + return false; + + inline_stack stack; + get_inline_stack (gimple_location (stmt), &stack); + if (stack.length () == 0) + return false; + function_instance *s = get_function_instance_by_inline_stack (stack); + if (s == NULL) + return false; + return s->get_count_info (stack[0].second, info); +} + +/* Mark LOC as annotated. */ + +void +autofdo_source_profile::mark_annotated (location_t loc) +{ + inline_stack stack; + get_inline_stack (loc, &stack); + if (stack.length () == 0) + return; + function_instance *s = get_function_instance_by_inline_stack (stack); + if (s == NULL) + return; + s->mark_annotated (stack[0].second); +} + +/* Update value profile INFO for STMT from the inlined indirect callsite. + Return true if INFO is updated. */ + +bool +autofdo_source_profile::update_inlined_ind_target (gimple stmt, + count_info *info) +{ + if (LOCATION_LOCUS (gimple_location (stmt)) == cfun->function_end_locus) + return false; + + count_info old_info; + get_count_info (stmt, &old_info); + gcov_type total = 0; + for (icall_target_map::const_iterator iter = old_info.targets.begin (); + iter != old_info.targets.end (); ++iter) + total += iter->second; + + /* Program behavior changed, original promoted (and inlined) target is not + hot any more. Will avoid promote the original target. + + To check if original promoted target is still hot, we check the total + count of the unpromoted targets (stored in old_info). If it is no less + than half of the callsite count (stored in INFO), the original promoted + target is considered not hot any more. */ + if (total >= info->count / 2) + return false; + + inline_stack stack; + get_inline_stack (gimple_location (stmt), &stack); + if (stack.length () == 0) + return false; + function_instance *s = get_function_instance_by_inline_stack (stack); + if (s == NULL) + return false; + icall_target_map map; + if (s->find_icall_target_map (stmt, &map) == 0) + return false; + for (icall_target_map::const_iterator iter = map.begin (); + iter != map.end (); ++iter) + info->targets[iter->first] = iter->second; + return true; +} + +/* Find total count of the callee of EDGE. */ + +gcov_type +autofdo_source_profile::get_callsite_total_count ( + struct cgraph_edge *edge) const +{ + inline_stack stack; + stack.safe_push (std::make_pair (edge->callee->decl, 0)); + get_inline_stack (gimple_location (edge->call_stmt), &stack); + + function_instance *s = get_function_instance_by_inline_stack (stack); + if (s == NULL + || afdo_string_table->get_index (IDENTIFIER_POINTER ( + DECL_ASSEMBLER_NAME (edge->callee->decl))) != s->name ()) + return 0; + else + return s->total_count (); +} + +/* Read AutoFDO profile and returns TRUE on success. */ + +/* source profile format: + + GCOV_TAG_AFDO_FUNCTION: 4 bytes + LENGTH: 4 bytes + NUM_FUNCTIONS: 4 bytes + FUNCTION_INSTANCE_1 + FUNCTION_INSTANCE_2 + ... + FUNCTION_INSTANCE_N. */ + +bool +autofdo_source_profile::read () +{ + if (gcov_read_unsigned () != GCOV_TAG_AFDO_FUNCTION) + { + inform (0, "Not expected TAG."); + return false; + } + + /* Skip the length of the section. */ + gcov_read_unsigned (); + + /* Read in the function/callsite profile, and store it in local + data structure. */ + unsigned function_num = gcov_read_unsigned (); + for (unsigned i = 0; i < function_num; i++) + { + function_instance::function_instance_stack stack; + function_instance *s = function_instance::read_function_instance ( + &stack, gcov_read_counter ()); + afdo_profile_info->sum_all += s->total_count (); + map_[s->name ()] = s; + } + return true; +} + +/* Return the function_instance in the profile that correspond to the + inline STACK. */ + +function_instance * +autofdo_source_profile::get_function_instance_by_inline_stack ( + const inline_stack &stack) const +{ + name_function_instance_map::const_iterator iter = map_.find ( + afdo_string_table->get_index_by_decl (stack[stack.length () - 1].first)); + if (iter == map_.end()) + return NULL; + function_instance *s = iter->second; + for (unsigned i = stack.length() - 1; i > 0; i--) + { + s = s->get_function_instance_by_decl ( + stack[i].second, stack[i - 1].first); + if (s == NULL) + return NULL; + } + return s; +} + +/* Module profile is only used by LIPO. Here we simply ignore it. */ + +static void +fake_read_autofdo_module_profile () +{ + /* Read in the module info. */ + gcov_read_unsigned (); + + /* Skip the length of the section. */ + gcov_read_unsigned (); + + /* Read in the file name table. */ + unsigned total_module_num = gcov_read_unsigned (); + gcc_assert (total_module_num == 0); +} + +/* Read data from profile data file. */ + +static void +read_profile (void) +{ + if (gcov_open (auto_profile_file, 1) == 0) + error ("Cannot open profile file %s.", auto_profile_file); + + if (gcov_read_unsigned () != GCOV_DATA_MAGIC) + error ("AutoFDO profile magic number does not mathch."); + + /* Skip the version number. */ + unsigned version = gcov_read_unsigned (); + if (version != AUTO_PROFILE_VERSION) + error ("AutoFDO profile version %u does match %u.", + version, AUTO_PROFILE_VERSION); + + /* Skip the empty integer. */ + gcov_read_unsigned (); + + /* string_table. */ + afdo_string_table = new string_table (); + if (!afdo_string_table->read()) + error ("Cannot read string table from %s.", auto_profile_file); + + /* autofdo_source_profile. */ + afdo_source_profile = autofdo_source_profile::create (); + if (afdo_source_profile == NULL) + error ("Cannot read function profile from %s.", auto_profile_file); + + /* autofdo_module_profile. */ + fake_read_autofdo_module_profile (); + + /* Read in the working set. */ + if (gcov_read_unsigned () != GCOV_TAG_AFDO_WORKING_SET) + error ("Cannot read working set from %s.", auto_profile_file); + + /* Skip the length of the section. */ + gcov_read_unsigned (); + gcov_working_set_t set[128]; + for (unsigned i = 0; i < 128; i++) + { + set[i].num_counters = gcov_read_unsigned (); + set[i].min_counter = gcov_read_counter (); + } + add_working_set (set); +} + +/* From AutoFDO profiles, find values inside STMT for that we want to measure + histograms for indirect-call optimization. + + This function is actually served for 2 purposes: + * before annotation, we need to mark histogram, promote and inline + * after annotation, we just need to mark, and let follow-up logic to + decide if it needs to promote and inline. */ + +static void +afdo_indirect_call (gimple_stmt_iterator *gsi, const icall_target_map &map, + bool transform) +{ + gimple stmt = gsi_stmt (*gsi); + tree callee; + + if (map.size () == 0 || gimple_code (stmt) != GIMPLE_CALL + || gimple_call_fndecl (stmt) != NULL_TREE) + return; + + callee = gimple_call_fn (stmt); + + histogram_value hist = gimple_alloc_histogram_value ( + cfun, HIST_TYPE_INDIR_CALL, stmt, callee); + hist->n_counters = 3; + hist->hvalue.counters = XNEWVEC (gcov_type, hist->n_counters); + gimple_add_histogram_value (cfun, stmt, hist); + + gcov_type total = 0; + icall_target_map::const_iterator max_iter = map.end (); + + for (icall_target_map::const_iterator iter = map.begin (); + iter != map.end (); ++iter) + { + total += iter->second; + if (max_iter == map.end () || max_iter->second < iter->second) + max_iter = iter; + } + + hist->hvalue.counters[0] + = (unsigned long long)afdo_string_table->get_name (max_iter->first); + hist->hvalue.counters[1] = max_iter->second; + hist->hvalue.counters[2] = total; + + if (!transform) + return; + + struct cgraph_edge *indirect_edge + = cgraph_node::get (current_function_decl)->get_edge (stmt); + struct cgraph_node *direct_call = cgraph_node::get_for_asmname ( + get_identifier ((const char *) hist->hvalue.counters[0])); + + if (direct_call == NULL || !check_ic_target (stmt, direct_call)) + return; + if (DECL_STRUCT_FUNCTION (direct_call->decl) == NULL) + return; + struct cgraph_edge *new_edge + = indirect_edge->make_speculative (direct_call, 0, 0); + new_edge->redirect_call_stmt_to_callee (); + gimple_remove_histogram_value (cfun, stmt, hist); + inline_call (new_edge, true, NULL, NULL, false); +} + +/* From AutoFDO profiles, find values inside STMT for that we want to measure + histograms and adds them to list VALUES. */ + +static void +afdo_vpt (gimple_stmt_iterator *gsi, const icall_target_map &map, + bool transform) +{ + afdo_indirect_call (gsi, map, transform); +} + +typedef std::set<basic_block> bb_set; +typedef std::set<edge> edge_set; + +static bool +is_bb_annotated (const basic_block bb, const bb_set &annotated) +{ + return annotated.find (bb) != annotated.end (); +} + +static void +set_bb_annotated (basic_block bb, bb_set *annotated) +{ + annotated->insert (bb); +} + +static bool +is_edge_annotated (const edge e, const edge_set &annotated) +{ + return annotated.find (e) != annotated.end (); +} + +static void +set_edge_annotated (edge e, edge_set *annotated) +{ + annotated->insert (e); +} + +/* For a given BB, set its execution count. Attach value profile if a stmt + is not in PROMOTED, because we only want to promot an indirect call once. + Return TRUE if BB is annotated. */ + +static bool +afdo_set_bb_count (basic_block bb, const stmt_set &promoted) +{ + gimple_stmt_iterator gsi; + edge e; + edge_iterator ei; + gcov_type max_count = 0; + bool has_annotated = false; + + for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi)) + { + count_info info; + gimple stmt = gsi_stmt (gsi); + if (gimple_clobber_p (stmt) || is_gimple_debug (stmt)) + continue; + if (afdo_source_profile->get_count_info (stmt, &info)) + { + if (info.count > max_count) + max_count = info.count; + has_annotated = true; + if (info.targets.size () > 0 + && promoted.find (stmt) == promoted.end ()) + afdo_vpt (&gsi, info.targets, false); + } + } + + if (!has_annotated) + return false; + + for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi)) + afdo_source_profile->mark_annotated (gimple_location (gsi_stmt (gsi))); + for (gsi = gsi_start_phis (bb); !gsi_end_p (gsi); gsi_next (&gsi)) + { + gimple phi = gsi_stmt (gsi); + size_t i; + for (i = 0; i < gimple_phi_num_args (phi); i++) + afdo_source_profile->mark_annotated (gimple_phi_arg_location (phi, i)); + } + FOR_EACH_EDGE (e, ei, bb->succs) + afdo_source_profile->mark_annotated (e->goto_locus); + + bb->count = max_count; + return true; +} + +/* BB1 and BB2 are in an equivalent class iff: + 1. BB1 dominates BB2. + 2. BB2 post-dominates BB1. + 3. BB1 and BB2 are in the same loop nest. + This function finds the equivalent class for each basic block, and + stores a pointer to the first BB in its equivalent class. Meanwhile, + set bb counts for the same equivalent class to be idenical. Update + ANNOTATED_BB for the first BB in its equivalent class. */ + +static void +afdo_find_equiv_class (bb_set *annotated_bb) +{ + basic_block bb; + + FOR_ALL_BB_FN (bb, cfun) + bb->aux = NULL; + + FOR_ALL_BB_FN (bb, cfun) + { + vec<basic_block> dom_bbs; + basic_block bb1; + int i; + + if (bb->aux != NULL) + continue; + bb->aux = bb; + dom_bbs = get_dominated_by (CDI_DOMINATORS, bb); + FOR_EACH_VEC_ELT (dom_bbs, i, bb1) + if (bb1->aux == NULL && dominated_by_p (CDI_POST_DOMINATORS, bb, bb1) + && bb1->loop_father == bb->loop_father) + { + bb1->aux = bb; + if (bb1->count > bb->count && is_bb_annotated (bb1, *annotated_bb)) + { + bb->count = MAX (bb->count, bb1->count); + set_bb_annotated (bb, annotated_bb); + } + } + dom_bbs = get_dominated_by (CDI_POST_DOMINATORS, bb); + FOR_EACH_VEC_ELT (dom_bbs, i, bb1) + if (bb1->aux == NULL && dominated_by_p (CDI_DOMINATORS, bb, bb1) + && bb1->loop_father == bb->loop_father) + { + bb1->aux = bb; + if (bb1->count > bb->count && is_bb_annotated (bb1, *annotated_bb)) + { + bb->count = MAX (bb->count, bb1->count); + set_bb_annotated (bb, annotated_bb); + } + } + } +} + +/* If a basic block's count is known, and only one of its in/out edges' count + is unknown, its count can be calculated. Meanwhile, if all of the in/out + edges' counts are known, then the basic block's unknown count can also be + calculated. + IS_SUCC is true if out edges of a basic blocks are examined. + Update ANNOTATED_BB and ANNOTATED_EDGE accordingly. + Return TRUE if any basic block/edge count is changed. */ + +static bool +afdo_propagate_edge (bool is_succ, bb_set *annotated_bb, + edge_set *annotated_edge) +{ + basic_block bb; + bool changed = false; + + FOR_EACH_BB_FN (bb, cfun) + { + edge e, unknown_edge = NULL; + edge_iterator ei; + int num_unknown_edge = 0; + gcov_type total_known_count = 0; + + FOR_EACH_EDGE (e, ei, is_succ ? bb->succs : bb->preds) + if (!is_edge_annotated (e, *annotated_edge)) + num_unknown_edge++, unknown_edge = e; + else + total_known_count += e->count; + + if (num_unknown_edge == 0) + { + if (total_known_count > bb->count) + { + bb->count = total_known_count; + changed = true; + } + if (!is_bb_annotated (bb, *annotated_bb)) + { + set_bb_annotated (bb, annotated_bb); + changed = true; + } + } + else if (num_unknown_edge == 1 && is_bb_annotated (bb, *annotated_bb)) + { + if (bb->count >= total_known_count) + unknown_edge->count = bb->count - total_known_count; + else + unknown_edge->count = 0; + set_edge_annotated (unknown_edge, annotated_edge); + changed = true; + } + } + return changed; +} + +/* Special propagation for circuit expressions. Because GCC translates + control flow into data flow for circuit expressions. E.g. + BB1: + if (a && b) + BB2 + else + BB3 + + will be translated into: + + BB1: + if (a) + goto BB.t1 + else + goto BB.t3 + BB.t1: + if (b) + goto BB.t2 + else + goto BB.t3 + BB.t2: + goto BB.t3 + BB.t3: + tmp = PHI (0 (BB1), 0 (BB.t1), 1 (BB.t2) + if (tmp) + goto BB2 + else + goto BB3 + + In this case, we need to propagate through PHI to determine the edge + count of BB1->BB.t1, BB.t1->BB.t2. + Update ANNOTATED_EDGE accordingly. */ + +static void +afdo_propagate_circuit (const bb_set &annotated_bb, edge_set *annotated_edge) +{ + basic_block bb; + FOR_ALL_BB_FN (bb, cfun) + { + gimple phi_stmt; + tree cmp_rhs, cmp_lhs; + gimple cmp_stmt = last_stmt (bb); + edge e; + edge_iterator ei; + + if (!cmp_stmt || gimple_code (cmp_stmt) != GIMPLE_COND) + continue; + cmp_rhs = gimple_cond_rhs (cmp_stmt); + cmp_lhs = gimple_cond_lhs (cmp_stmt); + if (!TREE_CONSTANT (cmp_rhs) + || !(integer_zerop (cmp_rhs) || integer_onep (cmp_rhs))) + continue; + if (TREE_CODE (cmp_lhs) != SSA_NAME) + continue; + if (!is_bb_annotated (bb, annotated_bb)) + continue; + phi_stmt = SSA_NAME_DEF_STMT (cmp_lhs); + while (phi_stmt && gimple_code (phi_stmt) == GIMPLE_ASSIGN + && gimple_assign_single_p (phi_stmt) + && TREE_CODE (gimple_assign_rhs1 (phi_stmt)) == SSA_NAME) + phi_stmt = SSA_NAME_DEF_STMT (gimple_assign_rhs1 (phi_stmt)); + if (!phi_stmt || gimple_code (phi_stmt) != GIMPLE_PHI) + continue; + FOR_EACH_EDGE (e, ei, bb->succs) + { + unsigned i, total = 0; + edge only_one; + bool check_value_one = (((integer_onep (cmp_rhs)) + ^ (gimple_cond_code (cmp_stmt) == EQ_EXPR)) + ^ ((e->flags & EDGE_TRUE_VALUE) != 0)); + if (!is_edge_annotated (e, *annotated_edge)) + continue; + for (i = 0; i < gimple_phi_num_args (phi_stmt); i++) + { + tree val = gimple_phi_arg_def (phi_stmt, i); + edge ep = gimple_phi_arg_edge (phi_stmt, i); + + if (!TREE_CONSTANT (val) + || !(integer_zerop (val) || integer_onep (val))) + continue; + if (check_value_one ^ integer_onep (val)) + continue; + total++; + only_one = ep; + if (e->probability == 0 && !is_edge_annotated (ep, *annotated_edge)) + { + ep->probability = 0; + ep->count = 0; + set_edge_annotated (ep, annotated_edge); + } + } + if (total == 1 && !is_edge_annotated (only_one, *annotated_edge)) + { + only_one->probability = e->probability; + only_one->count = e->count; + set_edge_annotated (only_one, annotated_edge); + } + } + } +} + +/* Propagate the basic block count and edge count on the control flow + graph. We do the propagation iteratively until stablize. */ + +static void +afdo_propagate (bb_set *annotated_bb, edge_set *annotated_edge) +{ + basic_block bb; + bool changed = true; + int i = 0; + + FOR_ALL_BB_FN (bb, cfun) + { + bb->count = ((basic_block)bb->aux)->count; + if (is_bb_annotated ((const basic_block)bb->aux, *annotated_bb)) + set_bb_annotated (bb, annotated_bb); + } + + while (changed && i++ < 10) + { + changed = false; + + if (afdo_propagate_edge (true, annotated_bb, annotated_edge)) + changed = true; + if (afdo_propagate_edge (false, annotated_bb, annotated_edge)) + changed = true; + afdo_propagate_circuit (*annotated_bb, annotated_edge); + } +} + +/* Propagate counts on control flow graph and calculate branch + probabilities. */ + +static void +afdo_calculate_branch_prob (bb_set *annotated_bb, edge_set *annotated_edge) +{ + basic_block bb; + bool has_sample = false; + + FOR_EACH_BB_FN (bb, cfun) + if (bb->count > 0) + has_sample = true; + + if (!has_sample) + return; + + calculate_dominance_info (CDI_POST_DOMINATORS); + calculate_dominance_info (CDI_DOMINATORS); + loop_optimizer_init (0); + + afdo_find_equiv_class (annotated_bb); + afdo_propagate (annotated_bb, annotated_edge); + + FOR_EACH_BB_FN (bb, cfun) + { + edge e; + edge_iterator ei; + int num_unknown_succ = 0; + gcov_type total_count = 0; + + FOR_EACH_EDGE (e, ei, bb->succs) + { + if (!is_edge_annotated (e, *annotated_edge)) + num_unknown_succ++; + else + total_count += e->count; + } + if (num_unknown_succ == 0 && total_count > 0) + { + FOR_EACH_EDGE (e, ei, bb->succs) + e->probability = (double)e->count * REG_BR_PROB_BASE / total_count; + } + } + FOR_ALL_BB_FN (bb, cfun) + { + edge e; + edge_iterator ei; + + FOR_EACH_EDGE (e, ei, bb->succs) + e->count = (double)bb->count * e->probability / REG_BR_PROB_BASE; + bb->aux = NULL; + } + + loop_optimizer_finalize (); + free_dominance_info (CDI_DOMINATORS); + free_dominance_info (CDI_POST_DOMINATORS); +} + +/* Perform value profile transformation using AutoFDO profile. Add the + promoted stmts to PROMOTED_STMTS. Return TRUE if there is any + indirect call promoted. */ + +static bool +afdo_vpt_for_early_inline (stmt_set *promoted_stmts) +{ + basic_block bb; + if (afdo_source_profile->get_function_instance_by_decl ( + current_function_decl) == NULL) + return false; + + compute_inline_parameters (cgraph_node::get (current_function_decl), true); + + bool has_vpt = false; + FOR_EACH_BB_FN (bb, cfun) + { + if (!has_indirect_call (bb)) + continue; + gimple_stmt_iterator gsi; + + gcov_type bb_count = 0; + for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi)) + { + count_info info; + gimple stmt = gsi_stmt (gsi); + if (afdo_source_profile->get_count_info (stmt, &info)) + bb_count = MAX (bb_count, info.count); + } + + for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi)) + { + gimple stmt = gsi_stmt (gsi); + /* IC_promotion and early_inline_2 is done in multiple iterations. + No need to promoted the stmt if its in promoted_stmts (means + it is already been promoted in the previous iterations). */ + if (gimple_code (stmt) != GIMPLE_CALL || gimple_call_fn (stmt) == NULL + || TREE_CODE (gimple_call_fn (stmt)) == FUNCTION_DECL + || promoted_stmts->find (stmt) != promoted_stmts->end ()) + continue; + + count_info info; + afdo_source_profile->get_count_info (stmt, &info); + info.count = bb_count; + if (afdo_source_profile->update_inlined_ind_target (stmt, &info)) + { + /* Promote the indirect call and update the promoted_stmts. */ + promoted_stmts->insert (stmt); + afdo_vpt (&gsi, info.targets, true); + has_vpt = true; + } + } + } + if (has_vpt) + { + optimize_inline_calls (current_function_decl); + return true; + } + else + return false; +} + +/* Annotate auto profile to the control flow graph. Do not annotate value + profile for stmts in PROMOTED_STMTS. */ + +static void +afdo_annotate_cfg (const stmt_set &promoted_stmts) +{ + basic_block bb; + bb_set annotated_bb; + edge_set annotated_edge; + const function_instance *s + = afdo_source_profile->get_function_instance_by_decl ( + current_function_decl); + + if (s == NULL) + return; + cgraph_node::get (current_function_decl)->count = s->head_count (); + ENTRY_BLOCK_PTR_FOR_FN (cfun)->count = s->head_count (); + gcov_type max_count = ENTRY_BLOCK_PTR_FOR_FN (cfun)->count; + + FOR_EACH_BB_FN (bb, cfun) + { + edge e; + edge_iterator ei; + + bb->count = 0; + FOR_EACH_EDGE (e, ei, bb->succs) + e->count = 0; + + if (afdo_set_bb_count (bb, promoted_stmts)) + set_bb_annotated (bb, &annotated_bb); + if (bb->count > max_count) + max_count = bb->count; + } + if (ENTRY_BLOCK_PTR_FOR_FN (cfun)->count + > ENTRY_BLOCK_PTR_FOR_FN (cfun)->next_bb->count) + { + ENTRY_BLOCK_PTR_FOR_FN (cfun)->next_bb->count + = ENTRY_BLOCK_PTR_FOR_FN (cfun)->count; + set_bb_annotated (ENTRY_BLOCK_PTR_FOR_FN (cfun)->next_bb, &annotated_bb); + } + if (ENTRY_BLOCK_PTR_FOR_FN (cfun)->count + > EXIT_BLOCK_PTR_FOR_FN (cfun)->prev_bb->count) + { + EXIT_BLOCK_PTR_FOR_FN (cfun)->prev_bb->count + = ENTRY_BLOCK_PTR_FOR_FN (cfun)->count; + set_bb_annotated (EXIT_BLOCK_PTR_FOR_FN (cfun)->prev_bb, &annotated_bb); + } + afdo_source_profile->mark_annotated ( + DECL_SOURCE_LOCATION (current_function_decl)); + afdo_source_profile->mark_annotated (cfun->function_start_locus); + afdo_source_profile->mark_annotated (cfun->function_end_locus); + if (max_count > 0) + { + afdo_calculate_branch_prob (&annotated_bb, &annotated_edge); + counts_to_freqs (); + profile_status_for_fn (cfun) = PROFILE_READ; + } + if (flag_value_profile_transformations) + gimple_value_profile_transformations (); +} + +/* Wrapper function to invoke early inliner. */ + +static void +early_inline () +{ + compute_inline_parameters (cgraph_node::get (current_function_decl), true); + unsigned todo = early_inliner (cfun); + if (todo & TODO_update_ssa_any) + update_ssa (TODO_update_ssa); +} + +/* Use AutoFDO profile to annoate the control flow graph. + Return the todo flag. */ + +static unsigned int +auto_profile (void) +{ + struct cgraph_node *node; + + if (symtab->state == FINISHED) + return 0; + + init_node_map (true); + profile_info = autofdo::afdo_profile_info; + + FOR_EACH_FUNCTION (node) + { + if (!gimple_has_body_p (node->decl)) + continue; + + /* Don't profile functions produced for builtin stuff. */ + if (DECL_SOURCE_LOCATION (node->decl) == BUILTINS_LOCATION) + continue; + + push_cfun (DECL_STRUCT_FUNCTION (node->decl)); + + /* First do indirect call promotion and early inline to make the + IR match the profiled binary before actual annotation. + + This is needed because an indirect call might have been promoted + and inlined in the profiled binary. If we do not promote and + inline these indirect calls before annotation, the profile for + these promoted functions will be lost. + + e.g. foo() --indirect_call--> bar() + In profiled binary, the callsite is promoted and inlined, making + the profile look like: + + foo: { + loc_foo_1: count_1 + bar@loc_foo_2: { + loc_bar_1: count_2 + loc_bar_2: count_3 + } + } + + Before AutoFDO pass, loc_foo_2 is not promoted thus not inlined. + If we perform annotation on it, the profile inside bar@loc_foo2 + will be wasted. + + To avoid this, we promote loc_foo_2 and inline the promoted bar + function before annotation, so the profile inside bar@loc_foo2 + will be useful. */ + autofdo::stmt_set promoted_stmts; + for (int i = 0; i < PARAM_VALUE (PARAM_EARLY_INLINER_MAX_ITERATIONS); i++) + { + if (!flag_value_profile_transformations + || !autofdo::afdo_vpt_for_early_inline (&promoted_stmts)) + break; + early_inline (); + } + + early_inline (); + autofdo::afdo_annotate_cfg (promoted_stmts); + compute_function_frequency (); + update_ssa (TODO_update_ssa); + + /* Local pure-const may imply need to fixup the cfg. */ + if (execute_fixup_cfg () & TODO_cleanup_cfg) + cleanup_tree_cfg (); + + free_dominance_info (CDI_DOMINATORS); + free_dominance_info (CDI_POST_DOMINATORS); + cgraph_edge::rebuild_edges (); + pop_cfun (); + } + + return TODO_rebuild_cgraph_edges; +} +} /* namespace autofdo. */ + +/* Read the profile from the profile data file. */ + +void +read_autofdo_file (void) +{ + if (auto_profile_file == NULL) + auto_profile_file = DEFAULT_AUTO_PROFILE_FILE; + + autofdo::afdo_profile_info = (struct gcov_ctr_summary *)xcalloc ( + 1, sizeof (struct gcov_ctr_summary)); + autofdo::afdo_profile_info->runs = 1; + autofdo::afdo_profile_info->sum_max = 0; + autofdo::afdo_profile_info->sum_all = 0; + + /* Read the profile from the profile file. */ + autofdo::read_profile (); +} + +/* Free the resources. */ + +void +end_auto_profile (void) +{ + delete autofdo::afdo_source_profile; + delete autofdo::afdo_string_table; + profile_info = NULL; +} + +/* Returns TRUE if EDGE is hot enough to be inlined early. */ + +bool +afdo_callsite_hot_enough_for_early_inline (struct cgraph_edge *edge) +{ + gcov_type count + = autofdo::afdo_source_profile->get_callsite_total_count (edge); + if (count > 0) + { + bool is_hot; + const struct gcov_ctr_summary *saved_profile_info = profile_info; + /* At earling inline stage, profile_info is not set yet. We need to + temporarily set it to afdo_profile_info to calculate hotness. */ + profile_info = autofdo::afdo_profile_info; + is_hot = maybe_hot_count_p (NULL, count); + profile_info = saved_profile_info; + return is_hot; + } + else + return false; +} + +namespace +{ + +const pass_data pass_data_ipa_auto_profile = { + SIMPLE_IPA_PASS, "afdo", /* name */ + OPTGROUP_NONE, /* optinfo_flags */ + TV_IPA_AUTOFDO, /* tv_id */ + 0, /* properties_required */ + 0, /* properties_provided */ + 0, /* properties_destroyed */ + 0, /* todo_flags_start */ + 0, /* todo_flags_finish */ +}; + +class pass_ipa_auto_profile : public simple_ipa_opt_pass +{ +public: + pass_ipa_auto_profile (gcc::context *ctxt) + : simple_ipa_opt_pass (pass_data_ipa_auto_profile, ctxt) + { + } + + /* opt_pass methods: */ + virtual bool + gate (function *) + { + return flag_auto_profile; + } + virtual unsigned int + execute (function *) + { + return autofdo::auto_profile (); + } +}; // class pass_ipa_auto_profile + +} // anon namespace + +simple_ipa_opt_pass * +make_pass_ipa_auto_profile (gcc::context *ctxt) +{ + return new pass_ipa_auto_profile (ctxt); +} Index: gcc/auto-profile.h =================================================================== --- gcc/auto-profile.h (revision 0) +++ gcc/auto-profile.h (revision 0) @@ -0,0 +1,31 @@ +/* auto-profile.h - Defines data exported from auto-profile.c + Copyright (C) 2014. Free Software Foundation, Inc. + Contributed by Dehao Chen (dehao@google.com) + +This file is part of GCC. + +GCC is free software; you can redistribute it and/or modify it under +the terms of the GNU General Public License as published by the Free +Software Foundation; either version 3, or (at your option) any later +version. + +GCC is distributed in the hope that it will be useful, but WITHOUT ANY +WARRANTY; without even the implied warranty of MERCHANTABILITY or +FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License +for more details. + +You should have received a copy of the GNU General Public License +along with GCC; see the file COPYING3. If not see +<http://www.gnu.org/licenses/>. */ + +#ifndef AUTO_PROFILE_H +#define AUTO_PROFILE_H + +/* Read, process, finalize AutoFDO data structures. */ +extern void read_autofdo_file (void); +extern void end_auto_profile (void); + +/* Returns TRUE if EDGE is hot enough to be inlined early. */ +extern bool afdo_callsite_hot_enough_for_early_inline (struct cgraph_edge *); + +#endif /* AUTO_PROFILE_H */ Index: gcc/toplev.c =================================================================== --- gcc/toplev.c (revision 215826) +++ gcc/toplev.c (working copy) @@ -79,6 +79,7 @@ along with GCC; see the file COPYING3. If not see #include "diagnostic-color.h" #include "context.h" #include "pass_manager.h" +#include "auto-profile.h" #include "optabs.h" #if defined(DBX_DEBUGGING_INFO) || defined(XCOFF_DEBUGGING_INFO) @@ -663,6 +664,10 @@ compile_file (void) targetm.asm_out.output_ident (ident_str); } + /* Auto profile finalization. */ + if (flag_auto_profile) + end_auto_profile (); + /* Invoke registered plugin callbacks. */ invoke_plugin_callbacks (PLUGIN_FINISH_UNIT, NULL); Index: gcc/value-prof.c =================================================================== --- gcc/value-prof.c (revision 215826) +++ gcc/value-prof.c (working copy) @@ -138,7 +138,7 @@ static bool gimple_ic_transform (gimple_stmt_itera /* Allocate histogram value. */ -static histogram_value +histogram_value gimple_alloc_histogram_value (struct function *fun ATTRIBUTE_UNUSED, enum hist_type type, gimple stmt, tree value) { @@ -1320,7 +1320,7 @@ find_func_by_profile_id (int profile_id) may ICE. Here we only do very minimal sanity check just to make compiler happy. Returns true if TARGET is considered ok for call CALL_STMT. */ -static bool +bool check_ic_target (gimple call_stmt, struct cgraph_node *target) { location_t locus; Index: gcc/value-prof.h =================================================================== --- gcc/value-prof.h (revision 215826) +++ gcc/value-prof.h (working copy) @@ -75,6 +75,8 @@ typedef vec<histogram_value> histogram_values; extern void gimple_find_values_to_profile (histogram_values *); extern bool gimple_value_profile_transformations (void); +histogram_value gimple_alloc_histogram_value (struct function *, enum hist_type, + gimple stmt, tree); histogram_value gimple_histogram_value (struct function *, gimple); histogram_value gimple_histogram_value_of_type (struct function *, gimple, enum hist_type); @@ -89,6 +91,7 @@ void verify_histograms (void); void free_histograms (void); void stringop_block_profile (gimple, unsigned int *, HOST_WIDE_INT *); gimple gimple_ic (gimple, struct cgraph_node *, int, gcov_type, gcov_type); +bool check_ic_target (gimple, struct cgraph_node *); /* In tree-profile.c. */ Index: gcc/common.opt =================================================================== --- gcc/common.opt (revision 215826) +++ gcc/common.opt (working copy) @@ -895,6 +895,16 @@ fauto-inc-dec Common Report Var(flag_auto_inc_dec) Init(1) Generate auto-inc/dec instructions +fauto-profile +Common Report Var(flag_auto_profile) Optimization +Use sample profile information for call graph node weights. The default +profile file is fbdata.afdo in 'pwd'. + +fauto-profile= +Common Joined RejectNegative Var(auto_profile_file) +Use sample profile information for call graph node weights. The profile +file is specified in the argument. + ; -fcheck-bounds causes gcc to generate array bounds checks. ; For C, C++ and ObjC: defaults off. ; For Java: defaults to on. Index: gcc/basic-block.h =================================================================== --- gcc/basic-block.h (revision 215826) +++ gcc/basic-block.h (working copy) @@ -724,6 +724,7 @@ extern struct edge_list *pre_edge_rev_lcm (int, sb extern void compute_available (sbitmap *, sbitmap *, sbitmap *, sbitmap *); /* In predict.c */ +extern bool maybe_hot_count_p (struct function *, gcov_type); extern bool maybe_hot_bb_p (struct function *, const_basic_block); extern bool maybe_hot_edge_p (edge); extern bool probably_never_executed_bb_p (struct function *, const_basic_block); Index: gcc/timevar.def =================================================================== --- gcc/timevar.def (revision 215826) +++ gcc/timevar.def (working copy) @@ -89,6 +89,7 @@ DEFTIMEVAR (TV_WHOPR_PARTITIONING , "whopr part DEFTIMEVAR (TV_WHOPR_LTRANS , "whopr ltrans") DEFTIMEVAR (TV_IPA_REFERENCE , "ipa reference") DEFTIMEVAR (TV_IPA_PROFILE , "ipa profile") +DEFTIMEVAR (TV_IPA_AUTOFDO , "auto profile") DEFTIMEVAR (TV_IPA_PURE_CONST , "ipa pure const") DEFTIMEVAR (TV_IPA_PTA , "ipa points-to") DEFTIMEVAR (TV_IPA_SRA , "ipa SRA") Index: gcc/ipa-inline.c =================================================================== --- gcc/ipa-inline.c (revision 215826) +++ gcc/ipa-inline.c (working copy) @@ -121,6 +121,7 @@ along with GCC; see the file COPYING3. If not see #include "ipa-inline.h" #include "ipa-utils.h" #include "sreal.h" +#include "auto-profile.h" #include "cilk.h" #include "builtins.h" @@ -449,6 +450,14 @@ want_early_inline_function_p (struct cgraph_edge * if (DECL_DISREGARD_INLINE_LIMITS (callee->decl)) ; + /* For AutoFDO, we need to make sure that before profile annotation, all + hot paths' IR look exactly the same as profiled binary. As a result, + in einliner, we will disregard size limit and inline those callsites + that are: + * inlined in the profiled binary, and + * the cloned callee has enough samples to be considered "hot". */ + else if (flag_auto_profile && afdo_callsite_hot_enough_for_early_inline (e)) + ; else if (!DECL_DECLARED_INLINE_P (callee->decl) && !flag_inline_small_functions) { @@ -2366,39 +2375,8 @@ early_inline_small_functions (struct cgraph_node * return inlined; } -/* Do inlining of small functions. Doing so early helps profiling and other - passes to be somewhat more effective and avoids some code duplication in - later real inlining pass for testcases with very many function calls. */ - -namespace { - -const pass_data pass_data_early_inline = -{ - GIMPLE_PASS, /* type */ - "einline", /* name */ - OPTGROUP_INLINE, /* optinfo_flags */ - TV_EARLY_INLINING, /* tv_id */ - PROP_ssa, /* properties_required */ - 0, /* properties_provided */ - 0, /* properties_destroyed */ - 0, /* todo_flags_start */ - 0, /* todo_flags_finish */ -}; - -class pass_early_inline : public gimple_opt_pass -{ -public: - pass_early_inline (gcc::context *ctxt) - : gimple_opt_pass (pass_data_early_inline, ctxt) - {} - - /* opt_pass methods: */ - virtual unsigned int execute (function *); - -}; // class pass_early_inline - unsigned int -pass_early_inline::execute (function *fun) +early_inliner (function *fun) { struct cgraph_node *node = cgraph_node::get (current_function_decl); struct cgraph_edge *edge; @@ -2499,6 +2477,43 @@ unsigned int return todo; } +/* Do inlining of small functions. Doing so early helps profiling and other + passes to be somewhat more effective and avoids some code duplication in + later real inlining pass for testcases with very many function calls. */ + +namespace { + +const pass_data pass_data_early_inline = +{ + GIMPLE_PASS, /* type */ + "einline", /* name */ + OPTGROUP_INLINE, /* optinfo_flags */ + TV_EARLY_INLINING, /* tv_id */ + PROP_ssa, /* properties_required */ + 0, /* properties_provided */ + 0, /* properties_destroyed */ + 0, /* todo_flags_start */ + 0, /* todo_flags_finish */ +}; + +class pass_early_inline : public gimple_opt_pass +{ +public: + pass_early_inline (gcc::context *ctxt) + : gimple_opt_pass (pass_data_early_inline, ctxt) + {} + + /* opt_pass methods: */ + virtual unsigned int execute (function *); + +}; // class pass_early_inline + +unsigned int +pass_early_inline::execute (function *fun) +{ + return early_inliner (fun); +} + } // anon namespace gimple_opt_pass * Index: gcc/ipa-inline.h =================================================================== --- gcc/ipa-inline.h (revision 215826) +++ gcc/ipa-inline.h (working copy) @@ -238,6 +238,7 @@ void initialize_growth_caches (void); void free_growth_caches (void); void compute_inline_parameters (struct cgraph_node *, bool); bool speculation_useful_p (struct cgraph_edge *e, bool anticipate_inlining); +unsigned int early_inliner (function *fun); /* In ipa-inline-transform.c */ bool inline_call (struct cgraph_edge *, bool, vec<cgraph_edge *> *, int *, bool, Index: gcc/tree-profile.c =================================================================== --- gcc/tree-profile.c (revision 215826) +++ gcc/tree-profile.c (working copy) @@ -699,8 +699,10 @@ class pass_ipa_tree_profile : public simple_ipa_op bool pass_ipa_tree_profile::gate (function *) { - /* When profile instrumentation, use or test coverage shall be performed. */ - return (!in_lto_p + /* When profile instrumentation, use or test coverage shall be performed. + But for AutoFDO, this there is no instrumentation, thus this pass is + diabled. */ + return (!in_lto_p && !flag_auto_profile && (flag_branch_probabilities || flag_test_coverage || profile_arc_flag)); } Index: gcc/gcov-io.h =================================================================== --- gcc/gcov-io.h (revision 215826) +++ gcc/gcov-io.h (working copy) @@ -244,6 +244,9 @@ typedef uint64_t gcov_type_unsigned; #define GCOV_TAG_PROGRAM_SUMMARY ((gcov_unsigned_t)0xa3000000) #define GCOV_TAG_SUMMARY_LENGTH(NUM) \ (1 + GCOV_COUNTERS_SUMMABLE * (10 + 3 * 2) + (NUM) * 5) +#define GCOV_TAG_AFDO_FILE_NAMES ((gcov_unsigned_t)0xaa000000) +#define GCOV_TAG_AFDO_FUNCTION ((gcov_unsigned_t)0xac000000) +#define GCOV_TAG_AFDO_WORKING_SET ((gcov_unsigned_t)0xaf000000) /* Counters that are collected. */ Index: gcc/doc/invoke.texi =================================================================== --- gcc/doc/invoke.texi (revision 215826) +++ gcc/doc/invoke.texi (working copy) @@ -365,7 +365,8 @@ Objective-C and Objective-C++ Dialects}. @gccoptlist{-faggressive-loop-optimizations -falign-functions[=@var{n}] @gol -falign-jumps[=@var{n}] @gol -falign-labels[=@var{n}] -falign-loops[=@var{n}] @gol --fassociative-math -fauto-inc-dec -fbranch-probabilities @gol +-fassociative-math -fauto-profile -fauto-profile[=@var{path}] @gol +-fauto-inc-dec -fbranch-probabilities @gol -fbranch-target-load-optimize -fbranch-target-load-optimize2 @gol -fbtr-bb-exclusive -fcaller-saves @gol -fcheck-data-deps -fcombine-stack-adjustments -fconserve-stack @gol @@ -9164,6 +9165,41 @@ code. If @var{path} is specified, GCC looks at the @var{path} to find the profile feedback data files. See @option{-fprofile-dir}. + +@item -fauto-profile +@itemx -fauto-profile=@var{path} +@opindex fauto-profile +Enable sampling based feedback directed optimizations, and optimizations +generally profitable only with profile feedback available. + +The following options are enabled: @code{-fbranch-probabilities}, @code{-fvpt}, +@code{-funroll-loops}, @code{-fpeel-loops}, @code{-ftracer}, @code{-ftree-vectorize}, +@code{-finline-functions}, @code{-fipa-cp}, @code{-fipa-cp-clone}, +@code{-fpredictive-commoning}, @code{-funswitch-loops}, +@code{-fgcse-after-reload}, @code{-ftree-loop-distribute-patterns}, + +If @var{path} is specified, GCC looks at the @var{path} to find +the profile feedback data files. + +In order to collect AutoFDO profile, you need to have: + +1. A linux system with linux perf support +2. (optional) An Intel processor with last branch record (LBR) support. This is + to guarantee accurate instruction level profile, which is important for + AutoFDO performance. + +To collect the profile, first use linux perf to collect raw profile +(see @uref{https://perf.wiki.kernel.org/}). + +E.g. +@code{perf record -e br_inst_retired:near_taken -b -o perf.data -- your_program} + +Then use create_gcov tool, which takes raw profile and unstripped binary to +generate AutoFDO profile that can be used by GCC. +(see @uref{https://github.com/google/autofdo}). + +E.g. +@code{create_gcov --binary=your_program.unstripped --profile=perf.data --gcov=profile.afdo} @end table The following options control compiler behavior regarding floating-point Index: gcc/tree-ssa-live.c =================================================================== --- gcc/tree-ssa-live.c (revision 215826) +++ gcc/tree-ssa-live.c (working copy) @@ -605,7 +605,7 @@ remove_unused_scope_block_p (tree scope) ; /* When not generating debug info we can eliminate info on unused variables. */ - else if (debug_info_level == DINFO_LEVEL_NONE) + else if (!flag_auto_profile && debug_info_level == DINFO_LEVEL_NONE) { /* Even for -g0 don't prune outer scopes from artificial functions, otherwise diagnostics using tree_nonartificial_location Index: gcc/Makefile.in =================================================================== --- gcc/Makefile.in (revision 215826) +++ gcc/Makefile.in (working copy) @@ -1153,6 +1153,7 @@ OBJS = \ alias.o \ alloc-pool.o \ auto-inc-dec.o \ + auto-profile.o \ bb-reorder.o \ bitmap.o \ bt-load.o \ Index: gcc/predict.c =================================================================== --- gcc/predict.c (revision 215826) +++ gcc/predict.c (working copy) @@ -161,7 +161,7 @@ set_hot_bb_threshold (gcov_type min) /* Return TRUE if frequency FREQ is considered to be hot. */ -static inline bool +bool maybe_hot_count_p (struct function *fun, gcov_type count) { if (fun && profile_status_for_fn (fun) != PROFILE_READ) @@ -2852,7 +2852,7 @@ counts_to_freqs (void) /* Don't overwrite the estimated frequencies when the profile for the function is missing. We may drop this function PROFILE_GUESSED later in drop_profile (). */ - if (!ENTRY_BLOCK_PTR_FOR_FN (cfun)->count) + if (!flag_auto_profile && !ENTRY_BLOCK_PTR_FOR_FN (cfun)->count) return 0; FOR_BB_BETWEEN (bb, ENTRY_BLOCK_PTR_FOR_FN (cfun), NULL, next_bb) @@ -3223,7 +3223,8 @@ rebuild_frequencies (void) count_max = MAX (bb->count, count_max); if (profile_status_for_fn (cfun) == PROFILE_GUESSED - || (profile_status_for_fn (cfun) == PROFILE_READ && count_max < REG_BR_PROB_BASE/10)) + || (!flag_auto_profile && profile_status_for_fn (cfun) == PROFILE_READ + && count_max < REG_BR_PROB_BASE/10)) { loop_optimizer_init (0); add_noreturn_fake_exit_edges (); ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH] AutoFDO patch for trunk 2014-10-20 21:39 ` Dehao Chen @ 2014-10-21 19:43 ` Markus Trippelsdorf 2014-10-21 21:01 ` Dehao Chen 2014-10-22 8:32 ` [BUILDROBOT] Build failure on AIX (was: [PATCH] AutoFDO patch for trunk) Jan-Benedict Glaw 2014-10-22 15:26 ` [PATCH] AutoFDO patch for trunk Rainer Orth 2 siblings, 1 reply; 25+ messages in thread From: Markus Trippelsdorf @ 2014-10-21 19:43 UTC (permalink / raw) To: Dehao Chen; +Cc: Jan Hubicka, GCC Patches, David Li On 2014.10.20 at 14:21 -0700, Dehao Chen wrote: > >> +If @var{path} is specified, GCC looks at the @var{path} to find > >> +the profile feedback data files. > >> + > >> +In order to collect AutoFDO profile, you need to have: > >> + > >> +1. A linux system with linux perf support > >> +2. An Intel processor with last branch record (LBR) support > > > > Is it really a requirement or will it only make profiles more accurate? > > It's optional, updated the doc. Can you give an example on how this is supposed to work on non Intel CPUs? It is not obvious to me. -- Markus ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH] AutoFDO patch for trunk 2014-10-21 19:43 ` Markus Trippelsdorf @ 2014-10-21 21:01 ` Dehao Chen 2014-10-21 21:30 ` Markus Trippelsdorf 0 siblings, 1 reply; 25+ messages in thread From: Dehao Chen @ 2014-10-21 21:01 UTC (permalink / raw) To: Markus Trippelsdorf; +Cc: Jan Hubicka, GCC Patches, David Li Everything will be the same on non-intel CPUs except for the perf command: perf record -e instructions -- your program. i.e. you need to drop "-b" and use instructions as event. Note that the current algorithm is tuned for accurate instruction level profile, which is not available on non-Intel CPU. But you are more than welcome to tune the propagation algorithm to get most out of inaccurate instruction profile. Cheers, Dehao On Tue, Oct 21, 2014 at 12:30 PM, Markus Trippelsdorf <markus@trippelsdorf.de> wrote: > On 2014.10.20 at 14:21 -0700, Dehao Chen wrote: >> >> +If @var{path} is specified, GCC looks at the @var{path} to find >> >> +the profile feedback data files. >> >> + >> >> +In order to collect AutoFDO profile, you need to have: >> >> + >> >> +1. A linux system with linux perf support >> >> +2. An Intel processor with last branch record (LBR) support >> > >> > Is it really a requirement or will it only make profiles more accurate? >> >> It's optional, updated the doc. > > Can you give an example on how this is supposed to work on non Intel > CPUs? It is not obvious to me. > > -- > Markus ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH] AutoFDO patch for trunk 2014-10-21 21:01 ` Dehao Chen @ 2014-10-21 21:30 ` Markus Trippelsdorf 2014-10-21 22:32 ` Dehao Chen 0 siblings, 1 reply; 25+ messages in thread From: Markus Trippelsdorf @ 2014-10-21 21:30 UTC (permalink / raw) To: Dehao Chen; +Cc: Jan Hubicka, GCC Patches, David Li On 2014.10.21 at 13:53 -0700, Dehao Chen wrote: > Everything will be the same on non-intel CPUs except for the perf command: > > perf record -e instructions -- your program. > > i.e. you need to drop "-b" and use instructions as event. > > Note that the current algorithm is tuned for accurate instruction > level profile, which is not available on non-Intel CPU. But you are > more than welcome to tune the propagation algorithm to get most out of > inaccurate instruction profile. Thanks. But unfortunately it doesn't work: markus@x4 autofdo % g++ -O2 -g /home/markus/bench.cpp markus@x4 autofdo % perf record -e instructions ./a.out ... markus@x4 autofdo % ./create_gcov --binary=a.out --profile=perf.data --gcov=profile.afdo E1021 23:23:30.733412 13689 perf_reader.cc:918] Unsupported event type 10 F1021 23:23:30.733707 13689 perf_parser.cc:114] Check failed: ReadPerfSampleInfo(**parsed_events_[i].raw_event, &sample_info) *** Check failure stack trace: *** @ 0x44469d google::LogMessage::Flush() @ 0x4470b9 google::LogMessageFatal::~LogMessageFatal() @ 0x43c668 quipper::PerfParser::SortParsedEvents() @ 0x43f0cb quipper::PerfParser::ParseRawEvents() @ 0x4221ef autofdo::PerfDataSampleReader::Append() @ 0x421af3 autofdo::SampleReader::ReadAndSetMaxCount() @ 0x417e61 autofdo::ProfileCreator::ReadSample() @ 0x418db2 autofdo::ProfileCreator::CreateProfile() @ 0x4086a5 main @ 0x7fa849ce8fd0 __libc_start_main @ 0x4099cb (unknown) @ (nil) (unknown) [1] 13689 abort ./create_gcov --binary=a.out --profile=perf.data --gcov=profile.afdo -- Markus ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH] AutoFDO patch for trunk 2014-10-21 21:30 ` Markus Trippelsdorf @ 2014-10-21 22:32 ` Dehao Chen 2014-10-22 5:58 ` Markus Trippelsdorf 0 siblings, 1 reply; 25+ messages in thread From: Dehao Chen @ 2014-10-21 22:32 UTC (permalink / raw) To: Markus Trippelsdorf; +Cc: Jan Hubicka, GCC Patches, David Li Looks like the perf data type is incompatible with quipper (perf data parser). Can you send me the perf.data file so that I can take a look. Thanks, Dehao On Tue, Oct 21, 2014 at 2:25 PM, Markus Trippelsdorf <markus@trippelsdorf.de> wrote: > On 2014.10.21 at 13:53 -0700, Dehao Chen wrote: >> Everything will be the same on non-intel CPUs except for the perf command: >> >> perf record -e instructions -- your program. >> >> i.e. you need to drop "-b" and use instructions as event. >> >> Note that the current algorithm is tuned for accurate instruction >> level profile, which is not available on non-Intel CPU. But you are >> more than welcome to tune the propagation algorithm to get most out of >> inaccurate instruction profile. > > Thanks. > But unfortunately it doesn't work: > > markus@x4 autofdo % g++ -O2 -g /home/markus/bench.cpp > markus@x4 autofdo % perf record -e instructions ./a.out > ... > markus@x4 autofdo % ./create_gcov --binary=a.out --profile=perf.data --gcov=profile.afdo > E1021 23:23:30.733412 13689 perf_reader.cc:918] Unsupported event type 10 > F1021 23:23:30.733707 13689 perf_parser.cc:114] Check failed: ReadPerfSampleInfo(**parsed_events_[i].raw_event, &sample_info) > *** Check failure stack trace: *** > @ 0x44469d google::LogMessage::Flush() > @ 0x4470b9 google::LogMessageFatal::~LogMessageFatal() > @ 0x43c668 quipper::PerfParser::SortParsedEvents() > @ 0x43f0cb quipper::PerfParser::ParseRawEvents() > @ 0x4221ef autofdo::PerfDataSampleReader::Append() > @ 0x421af3 autofdo::SampleReader::ReadAndSetMaxCount() > @ 0x417e61 autofdo::ProfileCreator::ReadSample() > @ 0x418db2 autofdo::ProfileCreator::CreateProfile() > @ 0x4086a5 main > @ 0x7fa849ce8fd0 __libc_start_main > @ 0x4099cb (unknown) > @ (nil) (unknown) > [1] 13689 abort ./create_gcov --binary=a.out --profile=perf.data --gcov=profile.afdo > > -- > Markus ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH] AutoFDO patch for trunk 2014-10-21 22:32 ` Dehao Chen @ 2014-10-22 5:58 ` Markus Trippelsdorf 0 siblings, 0 replies; 25+ messages in thread From: Markus Trippelsdorf @ 2014-10-22 5:58 UTC (permalink / raw) To: Dehao Chen; +Cc: Jan Hubicka, GCC Patches, David Li On 2014.10.21 at 15:31 -0700, Dehao Chen wrote: > Looks like the perf data type is incompatible with quipper (perf data > parser). Can you send me the perf.data file so that I can take a look. PERF_RECORD_MMAP2 (aka 10) was added in Linux 3.12 (commit 13d7a2410f). So your autofdo tool simply doesn't work on newer kernels. -- Markus ^ permalink raw reply [flat|nested] 25+ messages in thread
* [BUILDROBOT] Build failure on AIX (was: [PATCH] AutoFDO patch for trunk) 2014-10-20 21:39 ` Dehao Chen 2014-10-21 19:43 ` Markus Trippelsdorf @ 2014-10-22 8:32 ` Jan-Benedict Glaw 2014-10-22 15:26 ` [PATCH] AutoFDO patch for trunk Rainer Orth 2 siblings, 0 replies; 25+ messages in thread From: Jan-Benedict Glaw @ 2014-10-22 8:32 UTC (permalink / raw) To: Dehao Chen; +Cc: Jan Hubicka, GCC Patches, David Li [-- Attachment #1: Type: text/plain, Size: 9983 bytes --] On Mon, 2014-10-20 14:21:19 -0700, Dehao Chen <dehao@google.com> wrote: > The updated patch attached. Will commit the patch in 2~3 hours if no > objection is received. Just noticed that this doesn't build on gcc111, see eg. build #364675 [1], which looks like this: g++ -c -g -O2 -DIN_GCC -DCROSS_DIRECTORY_STRUCTURE -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -DHAVE_CONFIG_H -I. -I. -I/home/jbglaw/repos/gcc/gcc -I/home/jbglaw/repos/gcc/gcc/. -I/home/jbglaw/repos/gcc/gcc/../include -I/home/jbglaw/repos/gcc/gcc/../libcpp/include -I/opt/cfarm/gmp-latest/include -I/opt/cfarm/mpfr-latest/include -I/opt/cfarm/mpc-latest/include -I/home/jbglaw/repos/gcc/gcc/../libdecnumber -I/home/jbglaw/repos/gcc/gcc/../libdecnumber/dpd -I../libdecnumber -I/home/jbglaw/repos/gcc/gcc/../libbacktrace -o auto-profile.o -MT auto-profile.o -MMD -MP -MF ./.deps/auto-profile.TPo /home/jbglaw/repos/gcc/gcc/auto-profile.c In file included from /home/jbglaw/repos/gcc/gcc/system.h:46:0, from /home/jbglaw/repos/gcc/gcc/auto-profile.c:26: /opt/freeware/lib/gcc/powerpc-ibm-aix7.1.0.0/4.8.1/include-fixed/stdio.h:560:40: error: declaration of C function 'int fgetpos64(FILE*, fpos64_t*)' conflicts with extern int fgetpos64(FILE *, fpos64_t *); ^ In file included from /home/jbglaw/repos/gcc/gcc/system.h:46:0, from /home/jbglaw/repos/gcc/gcc/auto-profile.c:26: /opt/freeware/lib/gcc/powerpc-ibm-aix7.1.0.0/4.8.1/include-fixed/stdio.h:355:12: error: previous declaration 'int fgetpos64(FILE*, fpos_t*)' here extern int fgetpos(FILE *__restrict__, fpos_t *__restrict__); ^ In file included from /home/jbglaw/repos/gcc/gcc/system.h:46:0, from /home/jbglaw/repos/gcc/gcc/auto-profile.c:26: /opt/freeware/lib/gcc/powerpc-ibm-aix7.1.0.0/4.8.1/include-fixed/stdio.h:563:41: error: declaration of C function 'int fseeko64(FILE*, off64_t, int)' conflicts with extern int fseeko64(FILE *, off64_t, int); ^ In file included from /home/jbglaw/repos/gcc/gcc/system.h:46:0, from /home/jbglaw/repos/gcc/gcc/auto-profile.c:26: /opt/freeware/lib/gcc/powerpc-ibm-aix7.1.0.0/4.8.1/include-fixed/stdio.h:503:12: error: previous declaration 'int fseeko64(FILE*, off_t, int)' here extern int fseeko(FILE *, off_t, int); ^ In file included from /home/jbglaw/repos/gcc/gcc/system.h:46:0, from /home/jbglaw/repos/gcc/gcc/auto-profile.c:26: /opt/freeware/lib/gcc/powerpc-ibm-aix7.1.0.0/4.8.1/include-fixed/stdio.h:564:46: error: declaration of C function 'int fsetpos64(FILE*, const fpos64_t*)' conflicts with extern int fsetpos64(FILE *, const fpos64_t *); ^ In file included from /home/jbglaw/repos/gcc/gcc/system.h:46:0, from /home/jbglaw/repos/gcc/gcc/auto-profile.c:26: /opt/freeware/lib/gcc/powerpc-ibm-aix7.1.0.0/4.8.1/include-fixed/stdio.h:357:12: error: previous declaration 'int fsetpos64(FILE*, const fpos_t*)' here extern int fsetpos(FILE *, const fpos_t *); ^ In file included from /home/jbglaw/repos/gcc/gcc/system.h:46:0, from /home/jbglaw/repos/gcc/gcc/auto-profile.c:26: /opt/freeware/lib/gcc/powerpc-ibm-aix7.1.0.0/4.8.1/include-fixed/stdio.h:565:31: error: declaration of C function 'off64_t ftello64(FILE*)' conflicts with extern off64_t ftello64(FILE *); ^ In file included from /home/jbglaw/repos/gcc/gcc/system.h:46:0, from /home/jbglaw/repos/gcc/gcc/auto-profile.c:26: /opt/freeware/lib/gcc/powerpc-ibm-aix7.1.0.0/4.8.1/include-fixed/stdio.h:504:14: error: previous declaration 'off_t ftello64(FILE*)' here extern off_t ftello(FILE *); ^ In file included from /home/jbglaw/repos/gcc/gcc/system.h:268:0, from /home/jbglaw/repos/gcc/gcc/auto-profile.c:26: /usr/include/unistd.h:201:41: error: declaration of C function 'off64_t lseek64(int, off64_t, int)' conflicts with extern off64_t lseek64(int, off64_t, int); ^ /usr/include/unistd.h:199:14: error: previous declaration 'off_t lseek64(int, off_t, int)' here extern off_t lseek(int, off_t, int); ^ In file included from /usr/include/unistd.h:844:0, from /home/jbglaw/repos/gcc/gcc/system.h:268, from /home/jbglaw/repos/gcc/gcc/auto-profile.c:26: /usr/include/sys/lockf.h:64:39: error: declaration of C function 'int lockf64(int, int, off64_t)' conflicts with extern int lockf64 (int, int, off64_t); ^ /usr/include/sys/lockf.h:62:13: error: previous declaration 'int lockf64(int, int, off_t)' here extern int lockf (int, int, off_t); ^ In file included from /home/jbglaw/repos/gcc/gcc/system.h:268:0, from /home/jbglaw/repos/gcc/gcc/auto-profile.c:26: /usr/include/unistd.h:917:38: error: declaration of C function 'int ftruncate64(int, off64_t)' conflicts with extern int ftruncate64(int, off64_t); ^ In file included from /home/jbglaw/repos/gcc/gcc/system.h:268:0, from /home/jbglaw/repos/gcc/gcc/auto-profile.c:26: /usr/include/unistd.h:915:14: error: previous declaration 'int ftruncate64(int, off_t)' here extern int ftruncate(int, off_t); ^ In file included from /home/jbglaw/repos/gcc/gcc/system.h:268:0, from /home/jbglaw/repos/gcc/gcc/auto-profile.c:26: /usr/include/unistd.h:974:46: error: declaration of C function 'int truncate64(const char*, off64_t)' conflicts with extern int truncate64(const char *, off64_t); ^ In file included from /home/jbglaw/repos/gcc/gcc/system.h:268:0, from /home/jbglaw/repos/gcc/gcc/auto-profile.c:26: /usr/include/unistd.h:972:14: error: previous declaration 'int truncate64(const char*, off_t)' here extern int truncate(const char *, off_t); ^ In file included from /home/jbglaw/repos/gcc/gcc/system.h:268:0, from /home/jbglaw/repos/gcc/gcc/auto-profile.c:26: /usr/include/unistd.h:993:54: error: declaration of C function 'ssize_t pread64(int, void*, size_t, off64_t)' conflicts with extern ssize_t pread64(int, void *, size_t, off64_t); ^ /usr/include/unistd.h:990:18: error: previous declaration 'ssize_t pread64(int, void*, size_t, off_t)' here extern ssize_t pread(int, void *, size_t, off_t); ^ /usr/include/unistd.h:994:61: error: declaration of C function 'ssize_t pwrite64(int, const void*, size_t, off64_t)' conflicts with extern ssize_t pwrite64(int, const void *, size_t, off64_t); ^ /usr/include/unistd.h:991:18: error: previous declaration 'ssize_t pwrite64(int, const void*, size_t, off_t)' here extern ssize_t pwrite(int, const void *, size_t, off_t); ^ /usr/include/unistd.h:1079:38: error: declaration of C function 'off64_t fclear64(int, off64_t)' conflicts with extern off64_t fclear64(int, off64_t); ^ /usr/include/unistd.h:1076:15: error: previous declaration 'off_t fclear64(int, off_t)' here extern off_t fclear(int, off_t); ^ /usr/include/unistd.h:1080:53: error: declaration of C function 'int fsync_range64(int, int, off64_t, off64_t)' conflicts with extern int fsync_range64(int, int, off64_t, off64_t); ^ /usr/include/unistd.h:1077:13: error: previous declaration 'int fsync_range64(int, int, off_t, off_t)' here extern int fsync_range(int, int, off_t, off_t); ^ In file included from /home/jbglaw/repos/gcc/gcc/system.h:395:0, from /home/jbglaw/repos/gcc/gcc/auto-profile.c:26: /usr/include/sys/mman.h:148:60: error: declaration of C function 'void* mmap64(void*, size_t, int, int, int, off64_t)' conflicts with extern void *mmap64(void *, size_t, int, int, int, off64_t); ^ /usr/include/sys/mman.h:143:15: error: previous declaration 'void* mmap64(void*, size_t, int, int, int, off_t)' here extern void *mmap(void *, size_t, int, int, int, off_t); ^ In file included from /home/jbglaw/repos/gcc/gcc/system.h:1093:0, from /home/jbglaw/repos/gcc/gcc/auto-profile.c:26: /home/jbglaw/repos/gcc/gcc/wide-int.h: In member function 'void generic_wide_int<T>::dump() const': /home/jbglaw/repos/gcc/gcc/hwint.h:110:38: error: expected ')' before 'PRIx64' #define HOST_WIDE_INT_PRINT_HEX "%#" PRIx64 ^ /home/jbglaw/repos/gcc/gcc/wide-int.h:879:22: note: in expansion of macro 'HOST_WIDE_INT_PRINT_HEX' fprintf (stderr, HOST_WIDE_INT_PRINT_HEX ",", val[len - 1 - i]); ^ /home/jbglaw/repos/gcc/gcc/hwint.h:110:38: error: expected ')' before 'PRIx64' #define HOST_WIDE_INT_PRINT_HEX "%#" PRIx64 ^ /home/jbglaw/repos/gcc/gcc/wide-int.h:880:20: note: in expansion of macro 'HOST_WIDE_INT_PRINT_HEX' fprintf (stderr, HOST_WIDE_INT_PRINT_HEX "], precision = %d\n", ^ make[1]: *** [auto-profile.o] Error 1 MfG, JBG [1] http://toolchain.lug-owl.de/buildbot/show_build_details.php?id=364675 -- Jan-Benedict Glaw jbglaw@lug-owl.de +49-172-7608481 Signature of: The course of history shows that as a government grows, liberty the second : decreases." (Thomas Jefferson) [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH] AutoFDO patch for trunk 2014-10-20 21:39 ` Dehao Chen 2014-10-21 19:43 ` Markus Trippelsdorf 2014-10-22 8:32 ` [BUILDROBOT] Build failure on AIX (was: [PATCH] AutoFDO patch for trunk) Jan-Benedict Glaw @ 2014-10-22 15:26 ` Rainer Orth 2 siblings, 0 replies; 25+ messages in thread From: Rainer Orth @ 2014-10-22 15:26 UTC (permalink / raw) To: Dehao Chen; +Cc: Jan Hubicka, GCC Patches, David Li [-- Attachment #1: Type: text/plain, Size: 1053 bytes --] Dehao Chen <dehao@google.com> writes: > The updated patch attached. Will commit the patch in 2~3 hours if no > objection is received. Apart from the AIX bootstrap failure your patch introduced, it also breaks Solaris bootstrap: In file included from ./config.h:6:0, from /vol/gcc/src/hg/trunk/local/gcc/auto-profile.c:25: ./auto-host.h:1976:0: error: "_FILE_OFFSET_BITS" redefined [-Werror] #define _FILE_OFFSET_BITS 64 ^ In file included from /usr/include/iso/string_iso.h:24:0, from /usr/include/string.h:11, from /vol/gcc/src/hg/trunk/local/gcc/auto-profile.c:21: /var/gcc/regression/trunk/11-gcc/build/prev-gcc/include-fixed/sys/feature_tests.h:213:0: note: this is the location of the previous definition #define _FILE_OFFSET_BITS 32 ^ As Joseph is repeating over and over again, *nothing* must be included before config.h, and auto-profile.c violates this. The following patch at least allows the file to compile without errors; no idea if this the best order for the headers involved. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: ap.patch --] [-- Type: text/x-patch, Size: 463 bytes --] diff --git a/gcc/auto-profile.c b/gcc/auto-profile.c --- a/gcc/auto-profile.c +++ b/gcc/auto-profile.c @@ -18,12 +18,13 @@ You should have received a copy of the G along with GCC; see the file COPYING3. If not see <http://www.gnu.org/licenses/>. */ +#include "config.h" +#include "system.h" + #include <string.h> #include <map> #include <set> -#include "config.h" -#include "system.h" #include "coretypes.h" #include "tree.h" #include "tree-pass.h" [-- Attachment #3: Type: text/plain, Size: 152 bytes --] Rainer -- ----------------------------------------------------------------------------- Rainer Orth, Center for Biotechnology, Bielefeld University ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH] AutoFDO patch for trunk 2014-05-07 21:24 [PATCH] AutoFDO patch for trunk Dehao Chen 2014-05-07 21:47 ` Xinliang David Li 2014-05-15 21:04 ` Jan Hubicka @ 2014-05-16 0:46 ` Jan Hubicka 2 siblings, 0 replies; 25+ messages in thread From: Jan Hubicka @ 2014-05-16 0:46 UTC (permalink / raw) To: Dehao Chen; +Cc: GCC Patches, Jan Hubicka, David Li > Index: gcc/auto-profile.c > =================================================================== > --- gcc/auto-profile.c (revision 0) > +++ gcc/auto-profile.c (revision 0) > @@ -0,0 +1,1584 @@ > +/* Calculate branch probabilities, and basic block execution counts. Update the toplevel comment, too. > + > +/* The following routines implements AutoFDO optimization. > + > + This optimization uses sampling profiles to annotate basic block counts > + and uses heuristics to estimate branch probabilities. > + > + There are three phases in AutoFDO: > + > + Phase 1: Read profile from the profile data file. > + The following info is read from the profile datafile: > + * string_table: a map between function name and its index. > + * autofdo_source_profile: a map from function_instance name to > + function_instance. This is represented as a forest of > + function_instances. > + * WorkingSet: a histogram of how many instructions are covered for a > + given percentage of total cycles. > + > + Phase 2: Early inline. > + Early inline uses autofdo_source_profile to find if a callsite is: > + * inlined in the profiled binary. > + * callee body is hot in the profiling run. > + If both condition satisfies, early inline will inline the callsite > + regardless of the code growth. > + > + Phase 3: Annotate control flow graph. > + AutoFDO uses a separate pass to: > + * Annotate basic block count > + * Estimate branch probability > + > + After the above 3 phases, all profile is readily annotated on the GCC IR. > + AutoFDO tries to reuse all FDO infrastructure as much as possible to make > + use of the profile. E.g. it uses existing mechanism to calculate the basic > + block/edge frequency, as well as the cgraph node/edge count. > +*/ I suppose the phases can be made explicit to the PM, so early inliner can stay a pass. > +/* Store a string array, indexed by string position in the array. */ > +class string_table { > +public: > + static string_table *create (); > + > + /* For a given string, returns its index. */ > + int get_index (const char *name) const; > + > + /* For a given decl, returns the index of the decl name. */ > + int get_index_by_decl (tree decl) const; > + > + /* For a given index, returns the string. */ > + const char *get_name (int index) const; I wonder how these should work with LTO. I suppose we can tell user to not LTO for train run, we should teach the table to ignore LTO generated suffixes as well as random seeds (like coverage code already does). What happens when two static functions have same name? > +/* Profile for all functions. */ > +class autofdo_source_profile { > +public: > + static autofdo_source_profile *create () > + { > + autofdo_source_profile *map = new autofdo_source_profile (); I think we still want empty lines here. Otherwise I trust you on following GCC C++ coding standards :) > + if (map->read ()) > + return map; > + delete map; > + return NULL; > + } > +/* Helper functions. */ > + > +/* Return the original name of NAME: strip the suffix that starts > + with '.' */ > + > +static const char *get_original_name (const char *name) > +{ > + char *ret = xstrdup (name); > + char *find = strchr (ret, '.'); > + if (find != NULL) > + *find = 0; > + return ret; > +} Stripping all suffixes will likely make conflicts on static functions, right? Example like this will get you a conflict m() { void t(void) { } t(); } m2() { void t2(void) { } t2(); } that is of course not that important given that nested functions are not top priority but it would be nice to handle it gratefuly. I would definitely merge logic here with logic in gcov that already knows how to skip random seeds and other cases that makes filenames to diverge... > + > +/* Return the combined location, which is a 32bit integer in which > + higher 16 bits stores the line offset of LOC to the start lineno > + of DECL, The lower 16 bits stores the discrimnator. */ > + > +static unsigned > +get_combined_location (location_t loc, tree decl) > +{ > + return ((LOCATION_LINE (loc) - DECL_SOURCE_LINE (decl)) << 16); > +} What happens on overflows here? I guess they are possible - 65536 lines is not infinity these days. The discriminator is added later? > + > +int > +string_table::get_index (const char *name) const > +{ > + if (name == NULL) > + return -1; > + string_index_map::const_iterator iter = map_.find (name); > + if (iter == map_.end()) > + return -1; > + else > + return iter->second; > +} > + > +int > +string_table::get_index_by_decl (tree decl) const Comments missing here. > + > +/* Read the profile and create a function_instance with head count as > + HEAD_COUNT. Recursively read callsites to create nested function_instances > + too. STACK is used to track the recursive creation process. */ > + > +function_instance * > +function_instance::read_function_instance ( > + function_instance_stack *stack, gcov_type head_count) > +{ > + unsigned name = gcov_read_unsigned (); > + unsigned num_pos_counts = gcov_read_unsigned (); > + unsigned num_callsites = gcov_read_unsigned (); > + function_instance *s = new function_instance (name, head_count); > + stack->push_back(s); > + > + for (unsigned i = 0; i < num_pos_counts; i++) > + { > + unsigned offset = gcov_read_unsigned () & 0xffff0000; > + unsigned num_targets = gcov_read_unsigned (); > + gcov_type count = gcov_read_counter (); > + s->pos_counts[offset].count = count; > + for (unsigned j = 0; j < stack->size(); j++) > + (*stack)[j]->total_count_ += count; > + for (unsigned j = 0; j < num_targets; j++) > + { > + /* Only indirect call target histogram is supported now. */ > + gcov_read_unsigned (); > + gcov_type target_idx = gcov_read_counter (); > + s->pos_counts[offset].targets[target_idx] = > + gcov_read_counter (); > + } > + } > + for (unsigned i = 0; i < num_callsites; i++) { > + unsigned offset = gcov_read_unsigned (); > + function_instance *callee_function_instance = > + read_function_instance (stack, 0); > + s->callsites[std::make_pair (offset, callee_function_instance->name ())] = > + callee_function_instance; > + } > + stack->pop_back(); > + return s; Can you please add a summary of the file format and what is read in here? > +/* Find count_info for a given gimple STMT. If found, store the count_info > + in INFO and return true; otherwise return false. */ > + > +bool > +autofdo_source_profile::get_count_info (gimple stmt, count_info *info) const > +{ > + if (LOCATION_LOCUS (gimple_location (stmt)) == cfun->function_end_locus) > + return false; > + > + inline_stack stack; > + get_inline_stack (gimple_location (stmt), &stack); > + if (stack.size () == 0) > + return false; > + function_instance *s = get_function_instance_by_inline_stack (stack); > + if (s == NULL) > + return false; > + return s->get_count_info (stack[0].second, info); > +} > + > +/* Mark LOC as annotated. */ > + > +void > +autofdo_source_profile::mark_annotated (location_t loc) { Please review the file for the coding standards. > + > + /* Program behavior changed, original promoted (and inlined) target is not > + hot any more. Will avoid promote the original target. > + > + To check if original promoted target is still hot, we check the total > + count of the unpromoted targets (stored in old_info). If it is no less > + than half of the callsite count (stored in INFO), the original promoted > + target is considered not hot any more. */ > + if (total >= info->count * 0.5) Probably /2, I am not sure if we can hit roundoff errors with long long->double->long long, but it is better to avoid it anyway. > +/* Module profile is only used by LIPO. Here we simply ignore it. */ > + > +static void > +fake_read_autofdo_module_profile () Probably could get in only after LIPO? > +{ > + /* Read in the module info. */ > + gcov_read_unsigned (); > + > + /* Skip the length of the section. */ > + gcov_read_unsigned (); You probably want to check that length is 0 or so. > + for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi)) > + { > + count_info info; > + gimple stmt = gsi_stmt (gsi); > + if (stmt->code == GIMPLE_DEBUG) > + continue; Probably also clobbers can be skipped. > +/* If a basic block's count is known, and only one of its in/out edges' count > + is unknown, its count can be calculated. > + Meanwhile, if all of the in/out edges' counts are known, then the basic > + block's unknown count can also be calculated. > + IS_SUCC is true if out edges of a basic blocks are examined. > + Return TRUE if any basic block/edge count is changed. */ Interesting profile cleanup. I wonder wy EDGE_ANNOTATED is exported now (i.e. why it is not maintained only withing auto-FDO). Do you have more use for it outside auto-FDO? I was thinking that having a flag about profile being reliable may be useful idea, but never really tried to implement it. > + > +/* Special propagation for circuit expressions. Because GCC translates > + control flow into data flow for circuit expressions. E.g. > + BB1: > + if (a && b) > + BB2 > + else > + BB3 > + > + will be translated into: > + > + BB1: > + if (a) > + goto BB.t1 > + else > + goto BB.t3 > + BB.t1: > + if (b) > + goto BB.t2 > + else > + goto BB.t3 > + BB.t2: > + goto BB.t3 > + BB.t3: > + tmp = PHI (0 (BB1), 0 (BB.t1), 1 (BB.t2) > + if (tmp) > + goto BB2 > + else > + goto BB3 > + > + In this case, we need to propagate through PHI to determine the edge > + count of BB1->BB.t1, BB.t1->BB.t2. */ So here you have known probabilities of BB1, BB2 and BB3 and you need to guess all the rest? Why it need so much of gimple matching? > + > +static void > +afdo_propagate_circuit (void) > +{ > + basic_block bb; > + FOR_ALL_BB_FN (bb, cfun) > + { > + gimple phi_stmt; > + tree cmp_rhs, cmp_lhs; > + gimple cmp_stmt = last_stmt (bb); > + edge e; > + edge_iterator ei; > + > + if (!cmp_stmt || gimple_code (cmp_stmt) != GIMPLE_COND) > + continue; > + cmp_rhs = gimple_cond_rhs (cmp_stmt); > + cmp_lhs = gimple_cond_lhs (cmp_stmt); > + if (!TREE_CONSTANT (cmp_rhs) > + || !(integer_zerop (cmp_rhs) || integer_onep (cmp_rhs))) > + continue; > + if (TREE_CODE (cmp_lhs) != SSA_NAME) > + continue; > + if ((bb->flags & BB_ANNOTATED) == 0) > + continue; > + phi_stmt = SSA_NAME_DEF_STMT (cmp_lhs); > + while (phi_stmt && gimple_code (phi_stmt) == GIMPLE_ASSIGN > + && gimple_assign_single_p (phi_stmt) > + && TREE_CODE (gimple_assign_rhs1 (phi_stmt)) == SSA_NAME) > + phi_stmt = SSA_NAME_DEF_STMT (gimple_assign_rhs1 (phi_stmt)); > + if (!phi_stmt || gimple_code (phi_stmt) != GIMPLE_PHI) > + continue; > + FOR_EACH_EDGE (e, ei, bb->succs) > + { > + unsigned i, total = 0; > + edge only_one; > + bool check_value_one = (((integer_onep (cmp_rhs)) > + ^ (gimple_cond_code (cmp_stmt) == EQ_EXPR)) > + ^ ((e->flags & EDGE_TRUE_VALUE) != 0)); > + if ((e->flags & EDGE_ANNOTATED) == 0) > + continue; > + for (i = 0; i < gimple_phi_num_args (phi_stmt); i++) > + { > + tree val = gimple_phi_arg_def (phi_stmt, i); > + edge ep = gimple_phi_arg_edge (phi_stmt, i); > + > + if (!TREE_CONSTANT (val) || !(integer_zerop (val) > + || integer_onep (val))) > + continue; > + if (check_value_one ^ integer_onep (val)) > + continue; > + total++; > + only_one = ep; > + if (e->probability == 0 && (ep->flags & EDGE_ANNOTATED) == 0) > + { > + ep->probability = 0; > + ep->count = 0; > + ep->flags |= EDGE_ANNOTATED; > + } > + } > + if (total == 1 && (only_one->flags & EDGE_ANNOTATED) == 0) > + { > + only_one->probability = e->probability; > + only_one->count = e->count; > + only_one->flags |= EDGE_ANNOTATED; > + } > + } > + } > +} > + > +/* Propagate the basic block count and edge count on the control flow > + graph. We do the propagation iteratively until stablize. */ > + > +static void > +afdo_propagate (void) > +{ > + basic_block bb; > + bool changed = true; > + int i = 0; > + > + FOR_ALL_BB_FN (bb, cfun) > + { > + bb->count = ((basic_block) bb->aux)->count; > + if ((((basic_block) bb->aux)->flags & BB_ANNOTATED) != 0) > + bb->flags |= BB_ANNOTATED; > + } > + > + while (changed && i++ < 10) > + { > + changed = false; > + > + if (afdo_propagate_edge (true)) > + changed = true; > + if (afdo_propagate_edge (false)) > + changed = true; > + afdo_propagate_circuit (); > + } I wonder if there is any code to share with same (except for the circuit thing) code in profile.c and gcov.c? Profile.c also have the minimal flow algorithm for fixing broken profiles, perhaps it could be useful here? > + afdo_find_equiv_class (); > + afdo_propagate (); > + > + FOR_EACH_BB_FN (bb, cfun) > + { > + edge e; > + edge_iterator ei; > + int num_unknown_succ = 0; > + gcov_type total_count = 0; > + > + FOR_EACH_EDGE (e, ei, bb->succs) > + { > + if ((e->flags & EDGE_ANNOTATED) == 0) > + num_unknown_succ ++; > + else > + total_count += e->count; > + } > + if (num_unknown_succ == 0 && total_count > 0) > + { > + FOR_EACH_EDGE (e, ei, bb->succs) > + e->probability = > + (double) e->count * REG_BR_PROB_BASE / total_count; > + } > + } > + FOR_ALL_BB_FN (bb, cfun) > + { > + edge e; > + edge_iterator ei; > + > + FOR_EACH_EDGE (e, ei, bb->succs) > + e->count = > + (double) bb->count * e->probability / REG_BR_PROB_BASE; > + bb->aux = NULL; > + } > + > + loop_optimizer_finalize (); > + free_dominance_info (CDI_DOMINATORS); > + free_dominance_info (CDI_POST_DOMINATORS); > +} I wonder if using the estimated profile to fill in the missing gaps would make sense? I.e. running the standard branch probabilities->freuqencies propagation code bug this time taking the determined counts as known values? > + > +/* Perform value profile transformation using AutoFDO profile. Add the > + promoted stmts to PROMOTED_STMTS. Return TRUE if there is any > + indirect call promoted. */ > + > +static bool > +afdo_vpt_for_early_inline (stmt_set *promoted_stmts) What exactly this function does? It inlines according to the inline decisions made at train run time and recorded to profile feedback? The patch looks very resonable in general. Lets break it up to smaller chunks and get it merged. It is an interesting stuff! Thanks a lot! Honza ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH] AutoFDO patch for trunk @ 2014-10-22 17:07 David Edelsohn 2014-10-22 17:10 ` Xinliang David Li 0 siblings, 1 reply; 25+ messages in thread From: David Edelsohn @ 2014-10-22 17:07 UTC (permalink / raw) To: Rainer Orth, Dehao Chen; +Cc: Jan Hubicka, GCC Patches, David Li >>>>> Rainer Orth writes: > As Joseph is repeating over and over again, *nothing* must be included > before config.h, and auto-profile.c violates this. > > The following patch at least allows the file to compile without errors; > no idea if this the best order for the headers involved. The patch also fixes the problem on AIX. Thanks, David ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH] AutoFDO patch for trunk 2014-10-22 17:07 David Edelsohn @ 2014-10-22 17:10 ` Xinliang David Li 2014-10-22 17:24 ` Dehao Chen 0 siblings, 1 reply; 25+ messages in thread From: Xinliang David Li @ 2014-10-22 17:10 UTC (permalink / raw) To: David Edelsohn; +Cc: Rainer Orth, Dehao Chen, Jan Hubicka, GCC Patches Can someone pre-approve the patch so that Dehao can check it in after basic testing? David On Wed, Oct 22, 2014 at 10:06 AM, David Edelsohn <dje.gcc@gmail.com> wrote: >>>>>> Rainer Orth writes: > >> As Joseph is repeating over and over again, *nothing* must be included >> before config.h, and auto-profile.c violates this. >> >> The following patch at least allows the file to compile without errors; >> no idea if this the best order for the headers involved. > > > The patch also fixes the problem on AIX. > > > Thanks, David ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH] AutoFDO patch for trunk 2014-10-22 17:10 ` Xinliang David Li @ 2014-10-22 17:24 ` Dehao Chen 0 siblings, 0 replies; 25+ messages in thread From: Dehao Chen @ 2014-10-22 17:24 UTC (permalink / raw) To: Xinliang David Li; +Cc: David Edelsohn, Rainer Orth, Jan Hubicka, GCC Patches The patch tested OK. And I think it's a trivial patch, and already committed it to trunk. About the perf parser. I'm syncing the toolchain to head which should already have newer kernel support. Thanks, Dehao On Wed, Oct 22, 2014 at 10:07 AM, Xinliang David Li <davidxl@google.com> wrote: > Can someone pre-approve the patch so that Dehao can check it in after > basic testing? > > David > > On Wed, Oct 22, 2014 at 10:06 AM, David Edelsohn <dje.gcc@gmail.com> wrote: >>>>>>> Rainer Orth writes: >> >>> As Joseph is repeating over and over again, *nothing* must be included >>> before config.h, and auto-profile.c violates this. >>> >>> The following patch at least allows the file to compile without errors; >>> no idea if this the best order for the headers involved. >> >> >> The patch also fixes the problem on AIX. >> >> >> Thanks, David ^ permalink raw reply [flat|nested] 25+ messages in thread
end of thread, other threads:[~2014-10-22 17:22 UTC | newest] Thread overview: 25+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2014-05-07 21:24 [PATCH] AutoFDO patch for trunk Dehao Chen 2014-05-07 21:47 ` Xinliang David Li 2014-05-15 21:04 ` Jan Hubicka 2014-10-09 0:24 ` Dehao Chen 2014-10-09 22:34 ` Jan Hubicka 2014-10-10 3:15 ` Dehao Chen 2014-10-14 15:08 ` Jan Hubicka 2014-10-14 21:10 ` Dehao Chen 2014-10-14 23:28 ` Dehao Chen 2014-10-15 2:57 ` Andi Kleen 2014-10-15 14:36 ` Jan Hubicka 2014-10-16 18:57 ` Dehao Chen 2014-10-19 10:02 ` Jan Hubicka 2014-10-20 21:39 ` Dehao Chen 2014-10-21 19:43 ` Markus Trippelsdorf 2014-10-21 21:01 ` Dehao Chen 2014-10-21 21:30 ` Markus Trippelsdorf 2014-10-21 22:32 ` Dehao Chen 2014-10-22 5:58 ` Markus Trippelsdorf 2014-10-22 8:32 ` [BUILDROBOT] Build failure on AIX (was: [PATCH] AutoFDO patch for trunk) Jan-Benedict Glaw 2014-10-22 15:26 ` [PATCH] AutoFDO patch for trunk Rainer Orth 2014-05-16 0:46 ` Jan Hubicka 2014-10-22 17:07 David Edelsohn 2014-10-22 17:10 ` Xinliang David Li 2014-10-22 17:24 ` Dehao Chen
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).