This patch improves the main inlining pass so that it is able to inline virtual calls which will have known base type because of previous inlining decisions. Inlining is perhaps the most important part of devirtualization, when I benchmarked a previous version of this patch I got ~3% improvement in 483.xalancbmk execution time. In artificially constructed example (which is dumb but not particularly weird) I got over 60%. I believe many C++ applications where small virtual methods are used extensively will benefit tangibly. In any way, this patch only provides inliner with more choices and so at least in theory should not lead to any regressions. This patch depends on all the previous ones in the series, especially on the new indirect call graph edges and improved OBJ_TYPE_REF folding. It adds a new type of jump function that simply carries over a known base type of the function, adds some information about the O_T_R to the indirect edges and substantially improves the code combining various types of jump functions when inlining. Then the framework we already have for indirect inlining works well with only minor tweaks to extract types from known global constants and the like. The patch also works with LTO and since we now stream indirect inlining node information along the jump functions again (which is the major change since the last submission), it should also work with WHOPR. I have bootstrapped and regression tested all the patches up to this one on x86_64-linux with no problems. I will seek approval to commit them when stage1 opens and I will welcome any comments or suggestions. Thanks, Martin 2010-02-12 Martin Jambor * cgraph.h (cgraph_indirect_call_info): New fields anc_offset, otr_token and polymorphic. * cgraph.c (cgraph_create_indirect_edge): Inilialize the above fields. (cgraph_clone_edge): Copy the above fields. * ipa-prop.h (enum jump_func_type): Added known_type jump function type, reordered items, updated comments. (union jump_func_value): Added base_type field, reordered fields. (enum ipa_lattice_type): Moved down in the file. (struct ipa_param_descriptor): New field polymorphic. (ipa_is_param_polymorphic): New function. * ipa-prop.c (ipa_print_node_jump_functions): Print known type jump functions. (compute_complex_pass_through): Renamed to... (compute_complex_assign_jump_func): this. (compute_complex_ancestor_jump_func): New function. (compute_known_type_jump_func): Likewise. (compute_scalar_jump_functions): Create known type and complex ancestor jump functions. (ipa_note_param_call): New parameter polymorphic, set the corresponding flag in the call note accordingly. (ipa_analyze_call_uses): Renamed to... (ipa_analyze_indirect_call_uses): this. New parameter target, define variable var only in the block where it is used. (ipa_analyze_virtual_call_uses): New function. (ipa_analyze_call_uses): Likewise. (get_binfo_at_offset): Likewise. (combine_known_type_and_ancestor_jfs): Likewise. (update_jump_functions_after_inlining): Implemented handling of a number of new jump function types combination. (print_edge_addition_message): Removed. (make_edge_direct_to_target): New function. (try_make_edge_direct_simple_call): Likewise. (try_make_edge_direct_virtual_call): Likewise. (update_call_notes_after_inlining): Renamed to... (update_indirect_edges_after_inlining): this. Moved edge creation for indirect calls to try_make_edge_direct_simple_call, also calls try_make_edge_direct_virtual_call for virtual calls. (ipa_print_node_params): Changed the header message. (ipa_write_jump_function): Stream also known type jump functions. (ipa_read_jump_function): Likewise. (ipa_write_indirect_edge_info): Stream new fields in cgraph_indirect_call_info. (ipa_read_indirect_edge_info): Likewise. * testsuite/g++.dg/ipa/ivinline-1.C: New test. * testsuite/g++.dg/ipa/ivinline-2.C: New test. * testsuite/g++.dg/ipa/ivinline-3.C: New test. * testsuite/g++.dg/ipa/ivinline-4.C: New test. * testsuite/g++.dg/ipa/ivinline-5.C: New test. * testsuite/g++.dg/ipa/ivinline-6.C: New test. Index: icln/gcc/ipa-prop.c =================================================================== --- icln.orig/gcc/ipa-prop.c +++ icln/gcc/ipa-prop.c @@ -283,6 +283,13 @@ ipa_print_node_jump_functions (FILE *f, fprintf (f, " param %d: ", i); if (type == IPA_JF_UNKNOWN) fprintf (f, "UNKNOWN\n"); + else if (type == IPA_JF_KNOWN_TYPE) + { + tree binfo_type = TREE_TYPE (jump_func->value.base_binfo); + fprintf (f, "KNOWN TYPE, type in binfo is: "); + print_generic_expr (f, binfo_type, 0); + fprintf (f, " (%u)\n", TYPE_UID (binfo_type)); + } else if (type == IPA_JF_CONST) { tree val = jump_func->value.constant; @@ -313,9 +320,11 @@ ipa_print_node_jump_functions (FILE *f, else if (type == IPA_JF_ANCESTOR) { fprintf (f, "ANCESTOR: "); - fprintf (f, "%d, offset "HOST_WIDE_INT_PRINT_DEC"\n", + fprintf (f, "%d, offset "HOST_WIDE_INT_PRINT_DEC", ", jump_func->value.ancestor.formal_id, jump_func->value.ancestor.offset); + print_generic_expr (f, jump_func->value.ancestor.type, 0); + fprintf (dump_file, "\n"); } } } @@ -335,51 +344,60 @@ ipa_print_all_jump_functions (FILE *f) } } -/* Determine whether passing ssa name NAME constitutes a polynomial - pass-through function or getting an address of an acestor and if so, write - such a jump function to JFUNC. INFO describes the caller. */ +/* Given that an actual argument is an SSA_NAME (given in NAME) and is a result + of an assignment statement STMT, try to find out whether NAME can be + described by a (possibly polynomial) pass-through jump-function or an + ancestor jump function and if so, write the appropriate function into + JFUNC */ static void -compute_complex_pass_through (struct ipa_node_params *info, - struct ipa_jump_func *jfunc, - tree name) +compute_complex_assign_jump_func (struct ipa_node_params *info, + struct ipa_jump_func *jfunc, + gimple stmt, tree name) { HOST_WIDE_INT offset, size, max_size; tree op1, op2, type; int index; - gimple stmt = SSA_NAME_DEF_STMT (name); - if (!is_gimple_assign (stmt)) - return; op1 = gimple_assign_rhs1 (stmt); op2 = gimple_assign_rhs2 (stmt); - if (op2) + if (TREE_CODE (op1) == SSA_NAME + && SSA_NAME_IS_DEFAULT_DEF (op1)) { - if (TREE_CODE (op1) != SSA_NAME - || !SSA_NAME_IS_DEFAULT_DEF (op1) - || (TREE_CODE_CLASS (gimple_expr_code (stmt)) != tcc_comparison - && !useless_type_conversion_p (TREE_TYPE (name), - TREE_TYPE (op1))) - || !is_gimple_ip_invariant (op2)) + index = ipa_get_param_decl_index (info, SSA_NAME_VAR (op1)); + if (index < 0) return; - index = ipa_get_param_decl_index (info, SSA_NAME_VAR (op1)); - if (index >= 0) + if (op2) { + if (!is_gimple_ip_invariant (op2) + || (TREE_CODE_CLASS (gimple_expr_code (stmt)) != tcc_comparison + && !useless_type_conversion_p (TREE_TYPE (name), + TREE_TYPE (op1)))) + return; + jfunc->type = IPA_JF_PASS_THROUGH; jfunc->value.pass_through.formal_id = index; jfunc->value.pass_through.operation = gimple_assign_rhs_code (stmt); jfunc->value.pass_through.operand = op2; } + else if (gimple_assign_unary_nop_p (stmt)) + { + jfunc->type = IPA_JF_PASS_THROUGH; + jfunc->value.pass_through.formal_id = index; + jfunc->value.pass_through.operation = NOP_EXPR; + } return; } if (TREE_CODE (op1) != ADDR_EXPR) return; + op1 = TREE_OPERAND (op1, 0); type = TREE_TYPE (op1); - + if (TREE_CODE (type) != RECORD_TYPE) + return; op1 = get_ref_base_and_extent (op1, &offset, &size, &max_size); if (TREE_CODE (op1) != INDIRECT_REF /* If this is a varying address, punt. */ @@ -402,6 +420,120 @@ compute_complex_pass_through (struct ipa } +/* Given that an actual argument is an SSA_NAME that is a result of a phi + statement PHI, try to find out whether NAME is in fact a + multiple-inheritance typecast from a descendant into an ancestor of a formal + parameter and thus can be described by an ancestor jump function and if so, + write the appropriate function into JFUNC. + + Essentially we want to match the following pattern: + + if (obj_2(D) != 0B) + goto ; + else + goto ; + + : + iftmp.1_3 = &obj_2(D)->D.1762; + + : + # iftmp.1_1 = PHI + D.1879_6 = middleman_1 (iftmp.1_1, i_5(D)); + return D.1879_6; */ + +static void +compute_complex_ancestor_jump_func (struct ipa_node_params *info, + struct ipa_jump_func *jfunc, + gimple phi) +{ + HOST_WIDE_INT offset, size, max_size; + gimple assign, cond; + basic_block phi_bb, assign_bb, cond_bb; + tree tmp, parm, expr; + int index, i; + + if (gimple_phi_num_args (phi) != 2 + || !integer_zerop (PHI_ARG_DEF (phi, 1))) + return; + + tmp = PHI_ARG_DEF (phi, 0); + if (TREE_CODE (tmp) != SSA_NAME + || SSA_NAME_IS_DEFAULT_DEF (tmp) + || !POINTER_TYPE_P (TREE_TYPE (tmp)) + || TREE_CODE (TREE_TYPE (TREE_TYPE (tmp))) != RECORD_TYPE) + return; + + assign = SSA_NAME_DEF_STMT (tmp); + assign_bb = gimple_bb (assign); + if (!single_pred_p (assign_bb) + || !gimple_assign_single_p (assign)) + return; + expr = gimple_assign_rhs1 (assign); + + if (TREE_CODE (expr) != ADDR_EXPR) + return; + expr = TREE_OPERAND (expr, 0); + expr = get_ref_base_and_extent (expr, &offset, &size, &max_size); + + if (TREE_CODE (expr) != INDIRECT_REF + /* If this is a varying address, punt. */ + || max_size == -1 + || max_size != size) + return; + parm = TREE_OPERAND (expr, 0); + if (TREE_CODE (parm) != SSA_NAME + || !SSA_NAME_IS_DEFAULT_DEF (parm)) + return; + + index = ipa_get_param_decl_index (info, SSA_NAME_VAR (parm)); + if (index < 0) + return; + + cond_bb = single_pred (assign_bb); + cond = last_stmt (cond_bb); + if (gimple_code (cond) != GIMPLE_COND + || gimple_cond_code (cond) != NE_EXPR + || gimple_cond_lhs (cond) != parm + || !integer_zerop (gimple_cond_rhs (cond))) + return; + + + phi_bb = gimple_bb (phi); + for (i = 0; i < 2; i++) + { + basic_block pred = EDGE_PRED (phi_bb, i)->src; + if (pred != assign_bb && pred != cond_bb) + return; + } + + jfunc->type = IPA_JF_ANCESTOR; + jfunc->value.ancestor.formal_id = index; + jfunc->value.ancestor.offset = offset; + jfunc->value.ancestor.type = TREE_TYPE (TREE_TYPE (tmp)); +} + +/* Given OP whch is passed as an actual argument to a called function, + determine if it is possible to construct a KNOWN_TYPE jump function for it + and if so, create one and store it to JFUNC. */ + +static void +compute_known_type_jump_func (tree op, struct ipa_jump_func *jfunc) +{ + tree binfo; + + if (TREE_CODE (op) != ADDR_EXPR) + return; + + op = TREE_OPERAND (op, 0); + binfo = gimple_get_relevant_ref_binfo (op, NULL_TREE); + if (binfo) + { + jfunc->type = IPA_JF_KNOWN_TYPE; + jfunc->value.base_binfo = binfo; + } +} + + /* Determine the jump functions of scalar arguments. Scalar means SSA names and constants of a number of selected types. INFO is the ipa_node_params structure associated with the caller, FUNCTIONS is a pointer to an array of @@ -439,8 +571,18 @@ compute_scalar_jump_functions (struct ip } } else - compute_complex_pass_through (info, &functions[num], arg); + { + gimple stmt = SSA_NAME_DEF_STMT (arg); + if (is_gimple_assign (stmt)) + compute_complex_assign_jump_func (info, &functions[num], + stmt, arg); + else if (gimple_code (stmt) == GIMPLE_PHI) + compute_complex_ancestor_jump_func (info, &functions[num], + stmt); + } } + else + compute_known_type_jump_func (arg, &functions[num]); } } @@ -740,16 +882,27 @@ ipa_is_ssa_with_stmt_def (tree t) statement. */ static void -ipa_note_param_call (struct cgraph_node *node, int formal_id, gimple stmt) +ipa_note_param_call (struct cgraph_node *node, int param_index, gimple stmt, + bool polymorphic) { struct cgraph_edge *cs; basic_block bb = gimple_bb (stmt); int freq; - freq = compute_call_stmt_bb_frequency (current_function_decl, bb); + freq = compute_call_stmt_bb_frequency (node->decl, bb); cs = cgraph_create_indirect_edge (node, stmt, bb->count, freq, bb->loop_depth); - cs->indirect_info->param_index = formal_id; + cs->indirect_info->param_index = param_index; + cs->indirect_info->anc_offset = 0; + cs->indirect_info->polymorphic = polymorphic; + if (polymorphic) + { + tree otr = gimple_call_fn (stmt); + tree type, token = OBJ_TYPE_REF_TOKEN (otr); + cs->indirect_info->otr_token = tree_low_cst (token, 1); + type = TREE_TYPE (TREE_TYPE (OBJ_TYPE_REF_OBJECT (otr))); + cs->indirect_info->otr_type = type; + } } /* Analyze the CALL and examine uses of formal parameters of the caller NODE @@ -800,12 +953,11 @@ ipa_note_param_call (struct cgraph_node */ static void -ipa_analyze_call_uses (struct cgraph_node *node, struct ipa_node_params *info, - gimple call) +ipa_analyze_indirect_call_uses (struct cgraph_node *node, + struct ipa_node_params *info, + gimple call, tree target) { - tree target = gimple_call_fn (call); gimple def; - tree var; tree n1, n2; gimple d1, d2; tree rec, rec2, cond; @@ -813,16 +965,12 @@ ipa_analyze_call_uses (struct cgraph_nod int index; basic_block bb, virt_bb, join; - if (TREE_CODE (target) != SSA_NAME) - return; - - var = SSA_NAME_VAR (target); if (SSA_NAME_IS_DEFAULT_DEF (target)) { - /* assuming TREE_CODE (var) == PARM_DECL */ + tree var = SSA_NAME_VAR (target); index = ipa_get_param_decl_index (info, var); if (index >= 0) - ipa_note_param_call (node, index, call); + ipa_note_param_call (node, index, call, false); return; } @@ -919,11 +1067,63 @@ ipa_analyze_call_uses (struct cgraph_nod index = ipa_get_param_decl_index (info, rec); if (index >= 0 && !ipa_is_param_modified (info, index)) - ipa_note_param_call (node, index, call); + ipa_note_param_call (node, index, call, false); return; } +/* Analyze a CALL to an OBJ_TYPE_REF which is passed in TARGET and if the + object referenced in the expression is a formal parameter of the caller + (described by INFO), create a call note for the statement. */ + +static void +ipa_analyze_virtual_call_uses (struct cgraph_node *node, + struct ipa_node_params *info, gimple call, + tree target) +{ + tree obj = OBJ_TYPE_REF_OBJECT (target); + tree var; + int index; + + if (TREE_CODE (obj) == ADDR_EXPR) + { + do + { + obj = TREE_OPERAND (obj, 0); + } + while (TREE_CODE (obj) == COMPONENT_REF); + if (TREE_CODE (obj) != INDIRECT_REF) + return; + obj = TREE_OPERAND (obj, 0); + } + + if (TREE_CODE (obj) != SSA_NAME + || !SSA_NAME_IS_DEFAULT_DEF (obj)) + return; + + var = SSA_NAME_VAR (obj); + index = ipa_get_param_decl_index (info, var); + + if (index >= 0) + ipa_note_param_call (node, index, call, true); +} + +/* Analyze a call statement CALL whether and how it utilizes formal parameters + of the caller (described by INFO). */ + +static void +ipa_analyze_call_uses (struct cgraph_node *node, + struct ipa_node_params *info, gimple call) +{ + tree target = gimple_call_fn (call); + + if (TREE_CODE (target) == SSA_NAME) + ipa_analyze_indirect_call_uses (node, info, call, target); + else if (TREE_CODE (target) == OBJ_TYPE_REF) + ipa_analyze_virtual_call_uses (node, info, call, target); +} + + /* Analyze the call statement STMT with respect to formal parameters (described in INFO) of caller given by NODE. Currently it only checks whether formal parameters are called. */ @@ -965,12 +1165,87 @@ ipa_analyze_params_uses (struct cgraph_n info->uses_analysis_done = 1; } +/* Try to find a base info of BINFO that would have its field decl at offset + OFFSET within the BINFO type and which i of EXPECTED_TYPE. If it can be + found, return, otherwise return NULL_TREE. */ + +static tree +get_binfo_at_offset (tree binfo, HOST_WIDE_INT offset, tree expected_type) +{ + tree type; + + if (offset == 0) + return binfo; + + type = TREE_TYPE (binfo); + while (offset > 0) + { + tree base_binfo, found_binfo; + HOST_WIDE_INT pos, size; + tree fld; + int i; + + if (TREE_CODE (type) != RECORD_TYPE) + return NULL_TREE; + + for (fld = TYPE_FIELDS (type); fld; fld = TREE_CHAIN (fld)) + { + if (TREE_CODE (fld) != FIELD_DECL) + continue; + + pos = int_bit_position (fld); + size = tree_low_cst (DECL_SIZE (fld), 1); + if (pos <= offset && (pos + size) > offset) + break; + } + if (!fld) + return NULL_TREE; + + found_binfo = NULL_TREE; + for (i = 0; BINFO_BASE_ITERATE (binfo, i, base_binfo); i++) + if (TREE_TYPE (base_binfo) == TREE_TYPE (fld)) + { + found_binfo = base_binfo; + break; + } + + if (!found_binfo) + return NULL_TREE; + + type = TREE_TYPE (fld); + binfo = found_binfo; + offset -= pos; + } + if (type != expected_type) + return NULL_TREE; + return binfo; +} + +/* Update the jump function DST when the call graph edge correspondng to SRC is + is being inlined, knowing that DST is of type ancestor and src of known + type. */ + +static void +combine_known_type_and_ancestor_jfs (struct ipa_jump_func *src, + struct ipa_jump_func *dst) +{ + tree new_binfo; + + new_binfo = get_binfo_at_offset (src->value.base_binfo, + dst->value.ancestor.offset, + dst->value.ancestor.type); + if (new_binfo) + { + dst->type = IPA_JF_KNOWN_TYPE; + dst->value.base_binfo = new_binfo; + } + else + dst->type = IPA_JF_UNKNOWN; +} + /* Update the jump functions associated with call graph edge E when the call graph edge CS is being inlined, assuming that E->caller is already (possibly - indirectly) inlined into CS->callee and that E has not been inlined. - - We keep pass through functions only if they do not contain any operation. - This is sufficient for inlining and greately simplifies things. */ + indirectly) inlined into CS->callee and that E has not been inlined. */ static void update_jump_functions_after_inlining (struct cgraph_edge *cs, @@ -983,51 +1258,161 @@ update_jump_functions_after_inlining (st for (i = 0; i < count; i++) { - struct ipa_jump_func *src, *dst = ipa_get_ith_jump_func (args, i); + struct ipa_jump_func *dst = ipa_get_ith_jump_func (args, i); if (dst->type == IPA_JF_ANCESTOR) { - dst->type = IPA_JF_UNKNOWN; - continue; - } + struct ipa_jump_func *src; - if (dst->type != IPA_JF_PASS_THROUGH) - continue; + /* Variable number of arguments can cause havoc if we try to access + one that does not exist in the inlined edge. So make sure we + don't. */ + if (dst->value.ancestor.formal_id >= ipa_get_cs_argument_count (top)) + { + dst->type = IPA_JF_UNKNOWN; + continue; + } + + src = ipa_get_ith_jump_func (top, dst->value.ancestor.formal_id); + if (src->type == IPA_JF_KNOWN_TYPE) + combine_known_type_and_ancestor_jfs (src, dst); + else if (src->type == IPA_JF_CONST) + { + struct ipa_jump_func kt_func; - /* We must check range due to calls with variable number of arguments and - we cannot combine jump functions with operations. */ - if (dst->value.pass_through.operation != NOP_EXPR - || (dst->value.pass_through.formal_id - >= ipa_get_cs_argument_count (top))) + kt_func.type = IPA_JF_UNKNOWN; + compute_known_type_jump_func (src->value.constant, &kt_func); + if (kt_func.type == IPA_JF_KNOWN_TYPE) + combine_known_type_and_ancestor_jfs (&kt_func, dst); + else + dst->type = IPA_JF_UNKNOWN; + } + else if (src->type == IPA_JF_PASS_THROUGH + && src->value.pass_through.operation == NOP_EXPR) + dst->value.ancestor.formal_id = src->value.pass_through.formal_id; + else if (src->type == IPA_JF_ANCESTOR) + { + dst->value.ancestor.formal_id = src->value.ancestor.formal_id; + dst->value.ancestor.offset += src->value.ancestor.offset; + } + else + dst->type = IPA_JF_UNKNOWN; + } + else if (dst->type == IPA_JF_PASS_THROUGH) { - dst->type = IPA_JF_UNKNOWN; - continue; + struct ipa_jump_func *src; + /* We must check range due to calls with variable number of arguments + and we cannot combine jump functions with operations. */ + if (dst->value.pass_through.operation == NOP_EXPR + && (dst->value.pass_through.formal_id + < ipa_get_cs_argument_count (top))) + { + src = ipa_get_ith_jump_func (top, + dst->value.pass_through.formal_id); + *dst = *src; + } + else + dst->type = IPA_JF_UNKNOWN; } + } +} + +/* If TARGET is an addr_expr of a function declaration, make it the destination + of an indirect edge IE and return the edge. Otherwise, return NULL. */ + +static struct cgraph_edge * +make_edge_direct_to_target (struct cgraph_edge *ie, tree target) +{ + struct cgraph_node *callee; + + if (TREE_CODE (target) != ADDR_EXPR) + return NULL; + target = TREE_OPERAND (target, 0); + if (TREE_CODE (target) != FUNCTION_DECL) + return NULL; + callee = cgraph_node (target); + if (!callee) + return NULL; + + cgraph_make_edge_direct (ie, callee); + if (dump_file) + { + fprintf (dump_file, "ipa-prop: Discovered %s call to a known target " + "(%s/%i -> %s/%i) for stmt ", + ie->indirect_info->polymorphic ? "a virtual" : "an indirect", + cgraph_node_name (ie->caller), ie->caller->uid, + cgraph_node_name (ie->callee), ie->callee->uid); - src = ipa_get_ith_jump_func (top, dst->value.pass_through.formal_id); - *dst = *src; + if (ie->call_stmt) + print_gimple_stmt (dump_file, ie->call_stmt, 2, TDF_SLIM); + else + fprintf (dump_file, "with uid %i\n", ie->lto_stmt_uid); } + return ie; } -/* Print out a debug message to file F that we have discovered that an indirect - call described by NT is in fact a call of a known constant function described - by JFUNC. NODE is the node where the call is. */ +/* Try to find a destination for indirect edge IE that corresponds to a simple + call or a call of a member function pointer and where the destination is a + pointer formal parameter described by jump function JFUNC. If it can be + determined, return the newly direct edge, otherwise return NULL. */ -static void -print_edge_addition_message (FILE *f, struct cgraph_edge *e, - struct ipa_jump_func *jfunc) +static struct cgraph_edge * +try_make_edge_direct_simple_call (struct cgraph_edge *ie, + struct ipa_jump_func *jfunc) +{ + tree target; + + if (jfunc->type == IPA_JF_CONST) + target = jfunc->value.constant; + else if (jfunc->type == IPA_JF_CONST_MEMBER_PTR) + target = jfunc->value.member_cst.pfn; + else + return NULL; + + return make_edge_direct_to_target (ie, target); +} + +/* Try to find a destination for indirect edge IE that corresponds to a + virtuall call based on a formal parameter which is described by jump + function JFUNC and if it can be determined, make it direct and return the + direct edge. Otherwise, return NULL. */ + +static struct cgraph_edge * +try_make_edge_direct_virtual_call (struct cgraph_edge *ie, + struct ipa_jump_func *jfunc) { - fprintf (f, "ipa-prop: Discovered an indirect call to a known target ("); - if (jfunc->type == IPA_JF_CONST_MEMBER_PTR) + tree binfo, type, target; + HOST_WIDE_INT token; + + if (jfunc->type == IPA_JF_KNOWN_TYPE) + binfo = jfunc->value.base_binfo; + else if (jfunc->type == IPA_JF_CONST) { - print_node_brief (f, "", jfunc->value.member_cst.pfn, 0); - print_node_brief (f, ", ", jfunc->value.member_cst.delta, 0); + tree cst = jfunc->value.constant; + if (TREE_CODE (cst) == ADDR_EXPR) + binfo = gimple_get_relevant_ref_binfo (TREE_OPERAND (cst, 0), + NULL_TREE); + else + return NULL; } else - print_node_brief(f, "", jfunc->value.constant, 0); + return NULL; + + if (!binfo) + return NULL; + + token = ie->indirect_info->otr_token; + type = ie->indirect_info->otr_type; + binfo = get_binfo_at_offset (binfo, ie->indirect_info->anc_offset, type); + if (binfo) + target = gimple_fold_obj_type_ref_known_binfo (token, binfo); + else + return NULL; - fprintf (f, ") in %s: ", cgraph_node_name (e->caller)); - print_gimple_stmt (f, e->call_stmt, 2, TDF_SLIM); + if (target) + return make_edge_direct_to_target (ie, target); + else + return NULL; } /* Update the param called notes associated with NODE when CS is being inlined, @@ -1037,12 +1422,12 @@ print_edge_addition_message (FILE *f, st unless NEW_EDGES is NULL. Return true iff a new edge(s) were created. */ static bool -update_call_notes_after_inlining (struct cgraph_edge *cs, - struct cgraph_node *node, - VEC (cgraph_edge_p, heap) **new_edges) +update_indirect_edges_after_inlining (struct cgraph_edge *cs, + struct cgraph_node *node, + VEC (cgraph_edge_p, heap) **new_edges) { struct ipa_edge_args *top = IPA_EDGE_REF (cs); - struct cgraph_edge *ie, *next_ie; + struct cgraph_edge *ie, *next_ie, *new_direct_edge; bool res = false; ipa_check_create_edge_args (); @@ -1063,7 +1448,7 @@ update_call_notes_after_inlining (struct /* We must check range due to calls with variable number of arguments: */ if (ici->param_index >= ipa_get_cs_argument_count (top)) { - ici->inlining_processed = true; + ici->inlining_processed = 1; continue; } @@ -1071,44 +1456,30 @@ update_call_notes_after_inlining (struct if (jfunc->type == IPA_JF_PASS_THROUGH && jfunc->value.pass_through.operation == NOP_EXPR) ici->param_index = jfunc->value.pass_through.formal_id; - else if (jfunc->type == IPA_JF_CONST - || jfunc->type == IPA_JF_CONST_MEMBER_PTR) + else if (jfunc->type == IPA_JF_ANCESTOR) { - struct cgraph_node *callee; - tree decl; - - ici->inlining_processed = true; - if (jfunc->type == IPA_JF_CONST_MEMBER_PTR) - decl = jfunc->value.member_cst.pfn; - else - decl = jfunc->value.constant; - - if (TREE_CODE (decl) != ADDR_EXPR) - continue; - decl = TREE_OPERAND (decl, 0); - - if (TREE_CODE (decl) != FUNCTION_DECL) - continue; - callee = cgraph_node (decl); - if (!callee || !callee->local.inlinable) - continue; - - res = true; - if (dump_file) - print_edge_addition_message (dump_file, ie, jfunc); - - cgraph_make_edge_direct (ie, callee); - ie->indirect_inlining_edge = 1; - if (new_edges) - VEC_safe_push (cgraph_edge_p, heap, *new_edges, ie); - top = IPA_EDGE_REF (cs); + ici->param_index = jfunc->value.ancestor.formal_id; + ici->anc_offset += jfunc->value.ancestor.offset; } else + /* Either we can find a destination for this edge now or never. */ + ici->inlining_processed = 1; + + if (ici->polymorphic) + new_direct_edge = try_make_edge_direct_virtual_call (ie, jfunc); + else + new_direct_edge = try_make_edge_direct_simple_call (ie, jfunc); + + if (new_direct_edge) { - /* Ancestor jum functions and pass theoughs with operations should - not be used on parameters that then get called. */ - gcc_assert (jfunc->type == IPA_JF_UNKNOWN); - ici->inlining_processed = true; + new_direct_edge->indirect_inlining_edge = 1; + if (new_edges) + { + VEC_safe_push (cgraph_edge_p, heap, *new_edges, + new_direct_edge); + top = IPA_EDGE_REF (cs); + res = true; + } } } @@ -1131,7 +1502,7 @@ propagate_info_to_inlined_callees (struc struct cgraph_edge *e; bool res; - res = update_call_notes_after_inlining (cs, node, new_edges); + res = update_indirect_edges_after_inlining (cs, node, new_edges); for (e = node->callees; e; e = e->next_callee) if (!e->inline_failed) @@ -1389,7 +1760,8 @@ ipa_print_node_params (FILE * f, struct if (!node->analyzed) return; info = IPA_NODE_REF (node); - fprintf (f, " function %s Trees :: \n", cgraph_node_name (node)); + fprintf (f, " function %s parameter descriptors:\n", + cgraph_node_name (node)); count = ipa_get_param_count (info); for (i = 0; i < count; i++) { @@ -1881,6 +2253,9 @@ ipa_write_jump_function (struct output_b { case IPA_JF_UNKNOWN: break; + case IPA_JF_KNOWN_TYPE: + lto_output_tree (ob, jump_func->value.base_binfo, true); + break; case IPA_JF_CONST: lto_output_tree (ob, jump_func->value.constant, true); break; @@ -1918,6 +2293,9 @@ ipa_read_jump_function (struct lto_input { case IPA_JF_UNKNOWN: break; + case IPA_JF_KNOWN_TYPE: + jump_func->value.base_binfo = lto_input_tree (ib, data_in); + break; case IPA_JF_CONST: jump_func->value.constant = lto_input_tree (ib, data_in); break; @@ -1949,10 +2327,19 @@ ipa_write_indirect_edge_info (struct out struct bitpack_d *bp; lto_output_sleb128_stream (ob->main_stream, ii->param_index); + lto_output_sleb128_stream (ob->main_stream, ii->anc_offset); bp = bitpack_create (); + gcc_assert (!ii->inlining_processed); bp_pack_value (bp, ii->inlining_processed, 1); + bp_pack_value (bp, ii->polymorphic, 1); lto_output_bitpack (ob->main_stream, bp); bitpack_delete (bp); + + if (ii->polymorphic) + { + lto_output_sleb128_stream (ob->main_stream, ii->otr_token); + lto_output_tree (ob, ii->otr_type, true); + } } /* Read in parts of cgraph_indirect_call_info corresponding to CS that are @@ -1967,9 +2354,16 @@ ipa_read_indirect_edge_info (struct lto_ struct bitpack_d *bp; ii->param_index = (int) lto_input_sleb128 (ib); + ii->anc_offset = (HOST_WIDE_INT) lto_input_sleb128 (ib); bp = lto_input_bitpack (ib); ii->inlining_processed = bp_unpack_value (bp, 1); + ii->polymorphic = bp_unpack_value (bp, 1); bitpack_delete (bp); + if (ii->polymorphic) + { + ii->otr_token = (HOST_WIDE_INT) lto_input_sleb128 (ib); + ii->otr_type = lto_input_tree (ib, data_in); + } } /* Stream out NODE info to OB. */ Index: icln/gcc/ipa-prop.h =================================================================== --- icln.orig/gcc/ipa-prop.h +++ icln/gcc/ipa-prop.h @@ -38,38 +38,40 @@ along with GCC; see the file COPYING3. argument. Unknown - neither of the above. - IPA_JF_CONST_MEMBER_PTR stands for C++ member pointers, other constants are - represented with IPA_JF_CONST. + IPA_JF_CONST_MEMBER_PTR stands for C++ member pointers, it is a special + constant in this regard. Other constants are represented with IPA_JF_CONST. + + IPA_JF_ANCESTOR is a special pass-through jump function, which means that + the result is an address of a part of the object pointed to by the formal + parameter to which the function refers. It is mainly intended to represent + getting addresses of of ancestor fields in C++ + (e.g. &this_1(D)->D.1766.D.1756). Note that if the original pointer is + NULL, ancestor jump function must behave like a simple pass-through. + + Other pass-through functions can either simply pass on an unchanged formal + parameter or can apply one simple binary operation to it (such jump + functions are called polynomial). + + IPA_JF_KNOWN_TYPE is a special type of an "unknown" function that applies + only to pointer parameters. It means that even though we cannot prove that + the passed value is an interprocedural constant, we still know the exact + type of the containing object which may be valuable for devirtualization. + + Jump functions are computed in ipa-prop.c by function + update_call_notes_after_inlining. Some information can be lost and jump + functions degraded accordingly when inlining, see + update_call_notes_after_inlining in the same file. */ - In addition to "ordinary" pass through functions represented by - IPA_JF_PASS_THROUGH, IPA_JF_ANCESTOR represents getting addresses of of - ancestor fields in C++ (e.g. &this_1(D)->D.1766.D.1756). */ enum jump_func_type { IPA_JF_UNKNOWN = 0, /* newly allocated and zeroed jump functions default */ - IPA_JF_CONST, - IPA_JF_CONST_MEMBER_PTR, - IPA_JF_PASS_THROUGH, - IPA_JF_ANCESTOR -}; - -/* All formal parameters in the program have a lattice associated with it - computed by the interprocedural stage of IPCP. - There are three main values of the lattice: - IPA_TOP - unknown, - IPA_BOTTOM - non constant, - IPA_CONST_VALUE - simple scalar constant, - Cval of formal f will have a constant value if all callsites to this - function have the same constant value passed to f. - Integer and real constants are represented as IPA_CONST_VALUE. */ -enum ipa_lattice_type -{ - IPA_BOTTOM, - IPA_CONST_VALUE, - IPA_TOP + IPA_JF_KNOWN_TYPE, /* represented by field base_binfo */ + IPA_JF_CONST, /* represented by field costant */ + IPA_JF_CONST_MEMBER_PTR, /* represented by field member_cst */ + IPA_JF_PASS_THROUGH, /* represented by field pass_through */ + IPA_JF_ANCESTOR /* represented by field ancestor */ }; - /* Structure holding data required to describe a pass-through jump function. */ struct GTY(()) ipa_pass_through_data @@ -86,8 +88,8 @@ struct GTY(()) ipa_pass_through_data enum tree_code operation; }; -/* Structure holding data required to describe and ancestor pass throu - funkci. */ +/* Structure holding data required to describe an ancestor pass-through + jump function. */ struct GTY(()) ipa_ancestor_jf_data { @@ -118,13 +120,30 @@ struct GTY (()) ipa_jump_func functions and member_cst holds constant c++ member functions. */ union jump_func_value { + tree GTY ((tag ("IPA_JF_KNOWN_TYPE"))) base_binfo; tree GTY ((tag ("IPA_JF_CONST"))) constant; + struct ipa_member_ptr_cst GTY ((tag ("IPA_JF_CONST_MEMBER_PTR"))) member_cst; struct ipa_pass_through_data GTY ((tag ("IPA_JF_PASS_THROUGH"))) pass_through; struct ipa_ancestor_jf_data GTY ((tag ("IPA_JF_ANCESTOR"))) ancestor; - struct ipa_member_ptr_cst GTY ((tag ("IPA_JF_CONST_MEMBER_PTR"))) member_cst; } GTY ((desc ("%1.type"))) value; }; +/* All formal parameters in the program have a lattice associated with it + computed by the interprocedural stage of IPCP. + There are three main values of the lattice: + IPA_TOP - unknown, + IPA_BOTTOM - non constant, + IPA_CONST_VALUE - simple scalar constant, + Cval of formal f will have a constant value if all callsites to this + function have the same constant value passed to f. + Integer and real constants are represented as IPA_CONST_VALUE. */ +enum ipa_lattice_type +{ + IPA_BOTTOM, + IPA_CONST_VALUE, + IPA_TOP +}; + /* All formal parameters in the program have a cval computed by the interprocedural stage of IPCP. See enum ipa_lattice_type for the various types of lattices supported */ Index: icln/gcc/testsuite/g++.dg/ipa/ivinline-1.C =================================================================== --- /dev/null +++ icln/gcc/testsuite/g++.dg/ipa/ivinline-1.C @@ -0,0 +1,61 @@ +/* Verify that simple virtual calls are inlined even without early + inlining. */ +/* { dg-do run } */ +/* { dg-options "-O3 -fdump-ipa-inline -fno-early-inlining -fno-ipa-cp" } */ + +extern "C" void abort (void); + +class A +{ +public: + int data; + virtual int foo (int i); +}; + +class B : public A +{ +public: + virtual int foo (int i); +}; + +class C : public A +{ +public: + virtual int foo (int i); +}; + +int A::foo (int i) +{ + return i + 1; +} + +int B::foo (int i) +{ + return i + 2; +} + +int C::foo (int i) +{ + return i + 3; +} + +int middleman (class A *obj, int i) +{ + return obj->foo (i); +} + +int __attribute__ ((noinline,noclone)) get_input(void) +{ + return 1; +} + +int main (int argc, char *argv[]) +{ + class B b; + if (middleman (&b, get_input ()) != 3) + abort (); + return 0; +} + +/* { dg-final { scan-ipa-dump "B::foo\[^\\n\]*inline copy in int main" "inline" } } */ +/* { dg-final { cleanup-ipa-dump "inline" } } */ Index: icln/gcc/testsuite/g++.dg/ipa/ivinline-2.C =================================================================== --- /dev/null +++ icln/gcc/testsuite/g++.dg/ipa/ivinline-2.C @@ -0,0 +1,60 @@ +/* Verify that simple virtual calls using this pointer are inlined + even without early inlining.. */ +/* { dg-do run } */ +/* { dg-options "-O3 -fdump-ipa-inline -fno-early-inlining -fno-ipa-cp" } */ + +extern "C" void abort (void); + +class A +{ +public: + int data; + virtual int foo (int i); + int middleman (int i) + { + return foo (i); + } +}; + +class B : public A +{ +public: + virtual int foo (int i); +}; + +class C : public A +{ +public: + virtual int foo (int i); +}; + +int A::foo (int i) +{ + return i + 1; +} + +int B::foo (int i) +{ + return i + 2; +} + +int C::foo (int i) +{ + return i + 3; +} + +int __attribute__ ((noinline,noclone)) get_input(void) +{ + return 1; +} + +int main (int argc, char *argv[]) +{ + class B b; + if (b.middleman (get_input ()) != 3) + abort (); + return 0; +} + +/* { dg-final { scan-ipa-dump "B::foo\[^\\n\]*inline copy in int main" "inline" } } */ +/* { dg-final { cleanup-ipa-dump "inline" } } */ Index: icln/gcc/testsuite/g++.dg/ipa/ivinline-3.C =================================================================== --- /dev/null +++ icln/gcc/testsuite/g++.dg/ipa/ivinline-3.C @@ -0,0 +1,61 @@ +/* Verify that simple virtual calls on an object refrence are inlined + even without early inlining. */ +/* { dg-do run } */ +/* { dg-options "-O3 -fdump-ipa-inline -fno-early-inlining -fno-ipa-cp" } */ + +extern "C" void abort (void); + +class A +{ +public: + int data; + virtual int foo (int i); +}; + +class B : public A +{ +public: + virtual int foo (int i); +}; + +class C : public A +{ +public: + virtual int foo (int i); +}; + +int A::foo (int i) +{ + return i + 1; +} + +int B::foo (int i) +{ + return i + 2; +} + +int C::foo (int i) +{ + return i + 3; +} + +int middleman (class A &obj, int i) +{ + return obj.foo (i); +} + +int __attribute__ ((noinline,noclone)) get_input(void) +{ + return 1; +} + +int main (int argc, char *argv[]) +{ + class B b; + if (middleman (b, get_input ()) != 3) + abort (); + return 0; +} + +/* { dg-final { scan-ipa-dump "B::foo\[^\\n\]*inline copy in int main" "inline" } } */ +/* { dg-final { cleanup-ipa-dump "inline" } } */ Index: icln/gcc/testsuite/g++.dg/ipa/ivinline-4.C =================================================================== --- /dev/null +++ icln/gcc/testsuite/g++.dg/ipa/ivinline-4.C @@ -0,0 +1,67 @@ +/* Verify that simple virtual calls are inlined even without early + inlining, even when a typecast to an ancestor is involved along the + way. */ +/* { dg-do run } */ +/* { dg-options "-O3 -fdump-ipa-inline -fno-early-inlining -fno-ipa-cp" } */ + +extern "C" void abort (void); + +class A +{ +public: + int data; + virtual int foo (int i); +}; + +class B : public A +{ +public: + virtual int foo (int i); +}; + +class C : public A +{ +public: + virtual int foo (int i); +}; + +int A::foo (int i) +{ + return i + 1; +} + +int B::foo (int i) +{ + return i + 2; +} + +int C::foo (int i) +{ + return i + 3; +} + +int __attribute__ ((noinline,noclone)) get_input(void) +{ + return 1; +} + +static int middleman_1 (class A *obj, int i) +{ + return obj->foo (i); +} + +static int middleman_2 (class B *obj, int i) +{ + return middleman_1 (obj, i); +} + +int main (int argc, char *argv[]) +{ + class B b; + if (middleman_2 (&b, get_input ()) != 3) + abort (); + return 0; +} + +/* { dg-final { scan-ipa-dump "B::foo\[^\\n\]*inline copy in int main" "inline" } } */ +/* { dg-final { cleanup-ipa-dump "inline" } } */ Index: icln/gcc/testsuite/g++.dg/ipa/ivinline-5.C =================================================================== --- /dev/null +++ icln/gcc/testsuite/g++.dg/ipa/ivinline-5.C @@ -0,0 +1,53 @@ +/* Verify that virtual call inlining does not pick a wrong method when + there is a user defined ancestor in an object. */ +/* { dg-do run } */ +/* { dg-options "-O3 -fdump-ipa-inline -fno-early-inlining -fno-ipa-cp" } */ + +extern "C" void abort (void); + +class A +{ +public: + int data; + virtual int foo (int i); +}; + +class B : public A +{ +public: + class A confusion; + virtual int foo (int i); +}; + +int A::foo (int i) +{ + return i + 1; +} + +int B::foo (int i) +{ + return i + 2; +} + +int middleman (class A *obj, int i) +{ + return obj->foo (i); +} + +int __attribute__ ((noinline,noclone)) get_input(void) +{ + return 1; +} + +int main (int argc, char *argv[]) +{ + class B b; + int i = get_input (); + if ((middleman (&b, i) + 100 * middleman (&b.confusion, i)) != 203) + abort (); + return 0; +} + +/* { dg-final { scan-ipa-dump "A::foo\[^\\n\]*inline copy in int main" "inline" } } */ +/* { dg-final { scan-ipa-dump "B::foo\[^\\n\]*inline copy in int main" "inline" } } */ +/* { dg-final { cleanup-ipa-dump "inline" } } */ Index: icln/gcc/testsuite/g++.dg/ipa/ivinline-6.C =================================================================== --- /dev/null +++ icln/gcc/testsuite/g++.dg/ipa/ivinline-6.C @@ -0,0 +1,56 @@ +/* Verify that virtual call inlining works also when it has to get the + type from an ipa invariant and that even in this case it does not + pick a wrong method when there is a user defined ancestor in an + object. */ +/* { dg-do run } */ +/* { dg-options "-O3 -fdump-ipa-inline -fno-early-inlining -fno-ipa-cp" } */ + +extern "C" void abort (void); + +class A +{ +public: + int data; + virtual int foo (int i); +}; + +class B : public A +{ +public: + class A confusion; + virtual int foo (int i); +}; + +int A::foo (int i) +{ + return i + 1; +} + +int B::foo (int i) +{ + return i + 2; +} + +int middleman (class A *obj, int i) +{ + return obj->foo (i); +} + +int __attribute__ ((noinline,noclone)) get_input(void) +{ + return 1; +} + +class B b; + +int main (int argc, char *argv[]) +{ + int i = get_input (); + if ((middleman (&b, i) + 100 * middleman (&b.confusion, i)) != 203) + abort (); + return 0; +} + +/* { dg-final { scan-ipa-dump "A::foo\[^\\n\]*inline copy in int main" "inline" } } */ +/* { dg-final { scan-ipa-dump "B::foo\[^\\n\]*inline copy in int main" "inline" } } */ +/* { dg-final { cleanup-ipa-dump "inline" } } */ Index: icln/gcc/testsuite/g++.dg/ipa/ivinline-7.C =================================================================== --- /dev/null +++ icln/gcc/testsuite/g++.dg/ipa/ivinline-7.C @@ -0,0 +1,77 @@ +/* Verify that simple virtual calls are inlined even without early + inlining, even when a typecast to an ancestor is involved along the + way and that ancestor is not the first one with virtual functions. */ +/* { dg-do run } */ +/* { dg-options "-O3 -fdump-ipa-inline -fno-early-inlining -fno-ipa-cp" } */ + +extern "C" void abort (void); + +class Distraction +{ +public: + float f; + double d; + Distraction () + { + f = 8.3; + d = 10.2; + } + virtual float bar (float z); +}; + +class A +{ +public: + int data; + virtual int foo (int i); +}; + + +class B : public Distraction, public A +{ +public: + virtual int foo (int i); +}; + +float Distraction::bar (float z) +{ + f += z; + return f/2; +} + +int A::foo (int i) +{ + return i + 1; +} + +int B::foo (int i) +{ + return i + 2; +} + +int __attribute__ ((noinline,noclone)) get_input(void) +{ + return 1; +} + +static int middleman_1 (class A *obj, int i) +{ + return obj->foo (i); +} + +static int middleman_2 (class B *obj, int i) +{ + return middleman_1 (obj, i); +} + +int main (int argc, char *argv[]) +{ + class B b; + + if (middleman_2 (&b, get_input ()) != 3) + abort (); + return 0; +} + +/* { dg-final { scan-ipa-dump "B::foo\[^\\n\]*inline copy in int main" "inline" } } */ +/* { dg-final { cleanup-ipa-dump "inline" } } */ Index: icln/gcc/cgraph.h =================================================================== --- icln.orig/gcc/cgraph.h +++ icln/gcc/cgraph.h @@ -318,11 +318,21 @@ typedef enum { struct GTY(()) cgraph_indirect_call_info { + /* Offset accumulated from ancestor jump functions of inlined call graph + edges. */ + HOST_WIDE_INT anc_offset; + /* OBJ_TYPE_REF_TOKEN of a polymorphic call (if polymorphic is set). */ + HOST_WIDE_INT otr_token; + /* Type of the object from OBJ_TYPE_REF_OBJECT. */ + tree otr_type; /* Index of the parameter that is called. */ int param_index; /* Whether this edge has already been looked at by indirect inlining. */ unsigned int inlining_processed : 1; + /* Set when the call is a virtual call with the parameter being the + associated object pointer rather than a simple direct call. */ + unsigned polymorphic : 1; }; struct GTY((chain_next ("%h.next_caller"), chain_prev ("%h.prev_caller"))) cgraph_edge { Index: icln/gcc/cgraph.c =================================================================== --- icln.orig/gcc/cgraph.c +++ icln/gcc/cgraph.c @@ -1032,8 +1032,8 @@ cgraph_create_indirect_edge (struct cgra initialize_inline_failed (edge); edge->indirect_info = GGC_NEW (struct cgraph_indirect_call_info); + memset (edge->indirect_info, 0, sizeof (*edge->indirect_info)); edge->indirect_info->param_index = -1; - edge->indirect_info->inlining_processed = 0; edge->next_callee = caller->indirect_calls; if (caller->indirect_calls) @@ -1972,9 +1972,16 @@ cgraph_clone_edge (struct cgraph_edge *e } else { + struct cgraph_indirect_call_info *new_ii, *old_ii = e->indirect_info; new_edge = cgraph_create_indirect_edge (n, call_stmt, count, freq, e->loop_nest + loop_nest); - new_edge->indirect_info->param_index = e->indirect_info->param_index; + new_ii = new_edge->indirect_info; + new_ii->param_index = old_ii->param_index; + new_ii->anc_offset = old_ii->anc_offset; + new_ii->otr_token = old_ii->otr_token; + new_ii->otr_type = old_ii->otr_type; + new_ii->inlining_processed = old_ii->inlining_processed; + new_ii->polymorphic = old_ii->polymorphic; } } else