public inbox for gcc-cvs@sourceware.org
help / color / mirror / Atom feed
* [gcc(refs/users/marxin/heads/pgo-reproducibility-test)] ipa: fix handling of multiple speculations (PR93318)
@ 2020-01-30 8:53 Martin Liska
0 siblings, 0 replies; 2+ messages in thread
From: Martin Liska @ 2020-01-30 8:53 UTC (permalink / raw)
To: gcc-cvs
https://gcc.gnu.org/g:40d240315a6e5ff2010769bfa1c19264217b5f1c
commit 40d240315a6e5ff2010769bfa1c19264217b5f1c
Author: Jan Hubicka <jh@suse.cz>
Date: Tue Jan 28 21:30:14 2020 +0100
ipa: fix handling of multiple speculations (PR93318)
This patch started as work to resole Richard's comment on quadratic lookups
in resolve_speculation. While doing it I however noticed multiple problems
in the new speuclative call code which made the patch quite big. In
particular:
1) Before applying speculation we consider only targets with at lest
probability 1/2.
If profile is sane at most two targets can have probability greater or
equal to 1/2. So the new multi-target speculation code got enabled only
in very special scenario when there ae precisely two target with precise
probability 1/2 (which is tested by the single testcase).
As a conseuqence the multiple target logic got minimal test coverage and
this made us to miss several ICEs.
2) Profile updating in profile merging, tree-inline and indirect call
expansion was wrong which led to inconsistent profiles (as already seen
on the testcase).
3) Code responsible to turn speculative call to direct call was broken for
anything with more than one target.
4) There were multiple cases where call_site_hash went out of sync which
eventually leads to an ICE..
5) Some code expects that all speculative call targets forms a sequence in
the callee linked list but there is no code to maintain that invariant
nor a verifier.
Fixing this it became obvious that the current API of speculative_call_info is
not useful because it really builds on fact tht there are precisely three
components (direct call, ref and indirect call) in every speculative call
sequence. I ended up replacing it with iterator API for direct call
(first_speculative_call_target, next_speculative_call_target) and accessors for
the other coponents updating comment in cgraph.h.
Finally I made the work with call site hash more effetive by updating edge
manipulation to keep them in sequence. So first one can be looked up from the
hash and then they can be iterated by callee.
There are other things that can be improved (for example the speculation should
start with most common target first), but I will try to keep that for next
stage1. This patch is mostly about getting rid of ICE and profile corruption
which is a regression from GCC 9.
Honza
gcc/ChangeLog:
2020-01-28 Jan Hubicka <hubicka@ucw.cz>
PR lto/93318
* cgraph.c (cgraph_add_edge_to_call_site_hash): Update call site
hash only when edge is first within the sequence.
(cgraph_edge::set_call_stmt): Update handling of speculative calls.
(symbol_table::create_edge): Do not set target_prob.
(cgraph_edge::remove_caller): Watch for speculative calls when updating
the call site hash.
(cgraph_edge::make_speculative): Drop target_prob parameter.
(cgraph_edge::speculative_call_info): Remove.
(cgraph_edge::first_speculative_call_target): New member function.
(update_call_stmt_hash_for_removing_direct_edge): New function.
(cgraph_edge::resolve_speculation): Rewrite to new API.
(cgraph_edge::speculative_call_for_target): New member function.
(cgraph_edge::make_direct): Rewrite to new API; fix handling of
multiple speculation targets.
(cgraph_edge::redirect_call_stmt_to_callee): Likewise; fix updating
of profile.
(verify_speculative_call): Verify that targets form an interval.
* cgraph.h (cgraph_edge::speculative_call_info): Remove.
(cgraph_edge::first_speculative_call_target): New member function.
(cgraph_edge::next_speculative_call_target): New member function.
(cgraph_edge::speculative_call_target_ref): New member function.
(cgraph_edge;:speculative_call_indirect_edge): New member funtion.
(cgraph_edge): Remove target_prob.
* cgraphclones.c (cgraph_node::set_call_stmt_including_clones):
Fix handling of speculative calls.
* ipa-devirt.c (ipa_devirt): Fix handling of speculative cals.
* ipa-fnsummary.c (analyze_function_body): Likewise.
* ipa-inline.c (speculation_useful_p): Use new speculative call API.
* ipa-profile.c (dump_histogram): Fix formating.
(ipa_profile_generate_summary): Watch for overflows.
(ipa_profile): Do not require probablity to be 1/2; update to new API.
* ipa-prop.c (ipa_make_edge_direct_to_target): Update to new API.
(update_indirect_edges_after_inlining): Update to new API.
* ipa-utils.c (ipa_merge_profiles): Rewrite merging of speculative call
profiles.
* profile-count.h: (profile_probability::adjusted): New.
* tree-inline.c (copy_bb): Update to new speculative call API; fix
updating of profile.
* value-prof.c (gimple_ic_transform): Rename to ...
(dump_ic_profile): ... this one; update dumping.
(stream_in_histogram_value): Fix formating.
(gimple_value_profile_transformations): Update.
gcc/testsuite/ChangeLog:
2020-01-28 Jan Hubicka <hubicka@ucw.cz>
* g++.dg/tree-prof/indir-call-prof.C: Update template.
* gcc.dg/tree-prof/crossmodule-indircall-1.c: Add more targets.
* gcc.dg/tree-prof/crossmodule-indircall-1a.c: Add more targets.
* gcc.dg/tree-prof/indir-call-prof.c: Update template.
Diff:
---
gcc/ChangeLog | 46 ++++++++++++++++++++++++++++++++++++++++++++++
gcc/testsuite/ChangeLog | 7 +++++++
2 files changed, 53 insertions(+)
diff --git a/gcc/ChangeLog b/gcc/ChangeLog
index 05f3b72..d124c4f 100644
--- a/gcc/ChangeLog
+++ b/gcc/ChangeLog
@@ -1,3 +1,49 @@
+2020-01-28 Jan Hubicka <hubicka@ucw.cz>
+
+ PR lto/93318
+ * cgraph.c (cgraph_add_edge_to_call_site_hash): Update call site
+ hash only when edge is first within the sequence.
+ (cgraph_edge::set_call_stmt): Update handling of speculative calls.
+ (symbol_table::create_edge): Do not set target_prob.
+ (cgraph_edge::remove_caller): Watch for speculative calls when updating
+ the call site hash.
+ (cgraph_edge::make_speculative): Drop target_prob parameter.
+ (cgraph_edge::speculative_call_info): Remove.
+ (cgraph_edge::first_speculative_call_target): New member function.
+ (update_call_stmt_hash_for_removing_direct_edge): New function.
+ (cgraph_edge::resolve_speculation): Rewrite to new API.
+ (cgraph_edge::speculative_call_for_target): New member function.
+ (cgraph_edge::make_direct): Rewrite to new API; fix handling of
+ multiple speculation targets.
+ (cgraph_edge::redirect_call_stmt_to_callee): Likewise; fix updating
+ of profile.
+ (verify_speculative_call): Verify that targets form an interval.
+ * cgraph.h (cgraph_edge::speculative_call_info): Remove.
+ (cgraph_edge::first_speculative_call_target): New member function.
+ (cgraph_edge::next_speculative_call_target): New member function.
+ (cgraph_edge::speculative_call_target_ref): New member function.
+ (cgraph_edge;:speculative_call_indirect_edge): New member funtion.
+ (cgraph_edge): Remove target_prob.
+ * cgraphclones.c (cgraph_node::set_call_stmt_including_clones):
+ Fix handling of speculative calls.
+ * ipa-devirt.c (ipa_devirt): Fix handling of speculative cals.
+ * ipa-fnsummary.c (analyze_function_body): Likewise.
+ * ipa-inline.c (speculation_useful_p): Use new speculative call API.
+ * ipa-profile.c (dump_histogram): Fix formating.
+ (ipa_profile_generate_summary): Watch for overflows.
+ (ipa_profile): Do not require probablity to be 1/2; update to new API.
+ * ipa-prop.c (ipa_make_edge_direct_to_target): Update to new API.
+ (update_indirect_edges_after_inlining): Update to new API.
+ * ipa-utils.c (ipa_merge_profiles): Rewrite merging of speculative call
+ profiles.
+ * profile-count.h: (profile_probability::adjusted): New.
+ * tree-inline.c (copy_bb): Update to new speculative call API; fix
+ updating of profile.
+ * value-prof.c (gimple_ic_transform): Rename to ...
+ (dump_ic_profile): ... this one; update dumping.
+ (stream_in_histogram_value): Fix formating.
+ (gimple_value_profile_transformations): Update.
+
2020-01-28 H.J. Lu <hongjiu.lu@intel.com>
PR target/91461
diff --git a/gcc/testsuite/ChangeLog b/gcc/testsuite/ChangeLog
index 37ab4b9..8610164 100644
--- a/gcc/testsuite/ChangeLog
+++ b/gcc/testsuite/ChangeLog
@@ -1,3 +1,10 @@
+2020-01-28 Jan Hubicka <hubicka@ucw.cz>
+
+ * g++.dg/tree-prof/indir-call-prof.C: Update template.
+ * gcc.dg/tree-prof/crossmodule-indircall-1.c: Add more targets.
+ * gcc.dg/tree-prof/crossmodule-indircall-1a.c: Add more targets.
+ * gcc.dg/tree-prof/indir-call-prof.c: Update template.
+
2020-01-28 H.J. Lu <hongjiu.lu@intel.com>
PR target/91461
^ permalink raw reply [flat|nested] 2+ messages in thread
* [gcc(refs/users/marxin/heads/pgo-reproducibility-test)] ipa: fix handling of multiple speculations (PR93318)
@ 2020-01-30 8:52 Martin Liska
0 siblings, 0 replies; 2+ messages in thread
From: Martin Liska @ 2020-01-30 8:52 UTC (permalink / raw)
To: gcc-cvs
https://gcc.gnu.org/g:845bb366adcf702331de3d8022fd0e1c1c918607
commit 845bb366adcf702331de3d8022fd0e1c1c918607
Author: Jan Hubicka <jh@suse.cz>
Date: Tue Jan 28 20:34:56 2020 +0100
ipa: fix handling of multiple speculations (PR93318)
This patch started as work to resole Richard's comment on quadratic lookups
in resolve_speculation. While doing it I however noticed multiple problems
in the new speuclative call code which made the patch quite big. In
particular:
1) Before applying speculation we consider only targets with at lest
probability 1/2.
If profile is sane at most two targets can have probability greater or
equal to 1/2. So the new multi-target speculation code got enabled only
in very special scenario when there ae precisely two target with precise
probability 1/2 (which is tested by the single testcase).
As a conseuqence the multiple target logic got minimal test coverage and
this made us to miss several ICEs.
2) Profile updating in profile merging, tree-inline and indirect call
expansion was wrong which led to inconsistent profiles (as already seen
on the testcase).
3) Code responsible to turn speculative call to direct call was broken for
anything with more than one target.
4) There were multiple cases where call_site_hash went out of sync which
eventually leads to an ICE..
5) Some code expects that all speculative call targets forms a sequence in
the callee linked list but there is no code to maintain that invariant
nor a verifier.
Fixing this it became obvious that the current API of speculative_call_info is
not useful because it really builds on fact tht there are precisely three
components (direct call, ref and indirect call) in every speculative call
sequence. I ended up replacing it with iterator API for direct call
(first_speculative_call_target, next_speculative_call_target) and accessors for
the other coponents updating comment in cgraph.h.
Finally I made the work with call site hash more effetive by updating edge
manipulation to keep them in sequence. So first one can be looked up from the
hash and then they can be iterated by callee.
There are other things that can be improved (for example the speculation should
start with most common target first), but I will try to keep that for next
stage1. This patch is mostly about getting rid of ICE and profile corruption
which is a regression from GCC 9.
gcc/ChangeLog:
PR lto/93318
* cgraph.c (cgraph_add_edge_to_call_site_hash): Update call site
hash only when edge is first within the sequence.
(cgraph_edge::set_call_stmt): Update handling of speculative calls.
(symbol_table::create_edge): Do not set target_prob.
(cgraph_edge::remove_caller): Watch for speculative calls when updating
the call site hash.
(cgraph_edge::make_speculative): Drop target_prob parameter.
(cgraph_edge::speculative_call_info): Remove.
(cgraph_edge::first_speculative_call_target): New member function.
(update_call_stmt_hash_for_removing_direct_edge): New function.
(cgraph_edge::resolve_speculation): Rewrite to new API.
(cgraph_edge::speculative_call_for_target): New member function.
(cgraph_edge::make_direct): Rewrite to new API; fix handling of
multiple speculation targets.
(cgraph_edge::redirect_call_stmt_to_callee): Likewise; fix updating
of profile.
(verify_speculative_call): Verify that targets form an interval.
* cgraph.h (cgraph_edge::speculative_call_info): Remove.
(cgraph_edge::first_speculative_call_target): New member function.
(cgraph_edge::next_speculative_call_target): New member function.
(cgraph_edge::speculative_call_target_ref): New member function.
(cgraph_edge;:speculative_call_indirect_edge): New member funtion.
(cgraph_edge): Remove target_prob.
* cgraphclones.c (cgraph_node::set_call_stmt_including_clones):
Fix handling of speculative calls.
* ipa-devirt.c (ipa_devirt): Fix handling of speculative cals.
* ipa-fnsummary.c (analyze_function_body): Likewise.
* ipa-inline.c (speculation_useful_p): Use new speculative call API.
* ipa-profile.c (dump_histogram): Fix formating.
(ipa_profile_generate_summary): Watch for overflows.
(ipa_profile): Do not require probablity to be 1/2; update to new API.
* ipa-prop.c (ipa_make_edge_direct_to_target): Update to new API.
(update_indirect_edges_after_inlining): Update to new API.
* ipa-utils.c (ipa_merge_profiles): Rewrite merging of speculative call
profiles.
* profile-count.h: (profile_probability::adjusted): New.
* tree-inline.c (copy_bb): Update to new speculative call API; fix
updating of profile.
* value-prof.c (gimple_ic_transform): Rename to ...
(dump_ic_profile): ... this one; update dumping.
(stream_in_histogram_value): Fix formating.
(gimple_value_profile_transformations): Update.
gcc/testsuite/ChangeLog:
* g++.dg/tree-prof/indir-call-prof.C: Update template.
* gcc.dg/tree-prof/crossmodule-indircall-1.c: Add more targets.
* gcc.dg/tree-prof/crossmodule-indircall-1a.c: Add more targets.
* gcc.dg/tree-prof/indir-call-prof.c: Update template.
Diff:
---
gcc/cgraph.c | 347 ++++++++++-----------
gcc/cgraph.h | 87 +++++-
gcc/cgraphclones.c | 18 +-
gcc/ipa-devirt.c | 7 +-
gcc/ipa-fnsummary.c | 36 +--
gcc/ipa-inline.c | 10 +-
gcc/ipa-profile.c | 36 ++-
gcc/ipa-prop.c | 49 ++-
gcc/ipa-utils.c | 241 ++++----------
gcc/profile-count.h | 10 +
gcc/testsuite/g++.dg/tree-prof/indir-call-prof.C | 4 +-
.../gcc.dg/tree-prof/crossmodule-indircall-1.c | 6 +-
.../gcc.dg/tree-prof/crossmodule-indircall-1a.c | 17 +-
gcc/testsuite/gcc.dg/tree-prof/indir-call-prof.c | 4 +-
gcc/tree-inline.c | 91 +++---
gcc/value-prof.c | 58 ++--
16 files changed, 491 insertions(+), 530 deletions(-)
diff --git a/gcc/cgraph.c b/gcc/cgraph.c
index f7ebcc9..3e50b0b 100644
--- a/gcc/cgraph.c
+++ b/gcc/cgraph.c
@@ -713,7 +713,9 @@ cgraph_add_edge_to_call_site_hash (cgraph_edge *e)
if (*slot)
{
gcc_assert (((cgraph_edge *)*slot)->speculative);
- if (e->callee)
+ if (e->callee && (!e->prev_callee
+ || !e->prev_callee->speculative
+ || e->prev_callee->call_stmt != e->call_stmt))
*slot = e;
return;
}
@@ -782,21 +784,40 @@ cgraph_edge::set_call_stmt (cgraph_edge *e, gcall *new_stmt,
when asked to. */
if (update_speculative && e->speculative)
{
- cgraph_edge *direct, *indirect;
+ cgraph_edge *direct, *indirect, *next;
ipa_ref *ref;
bool e_indirect = e->indirect_unknown_callee;
+ int n = 0;
+
+ direct = e->first_speculative_call_target ();
+ indirect = e->speculative_call_indirect_edge ();
+
+ gcall *old_stmt = direct->call_stmt;
+ for (cgraph_edge *d = direct; d; d = next)
+ {
+ next = d->next_speculative_call_target ();
+ cgraph_edge *d2 = set_call_stmt (d, new_stmt, false);
+ gcc_assert (d2 == d);
+ n++;
+ }
+ gcc_checking_assert (indirect->num_speculative_call_targets_p () == n);
+ for (unsigned int i = 0; e->caller->iterate_reference (i, ref); i++)
+ if (ref->speculative && ref->stmt == old_stmt)
+ {
+ ref->stmt = new_stmt;
+ n--;
+ }
- e->speculative_call_info (direct, indirect, ref);
- ref->stmt = new_stmt;
- cgraph_edge *d2 = set_call_stmt (direct, new_stmt, false);
- gcc_assert (direct == d2);
indirect = set_call_stmt (indirect, new_stmt, false);
return e_indirect ? indirect : direct;
}
/* Only direct speculative edges go to call_site_hash. */
if (e->caller->call_site_hash
- && (!e->speculative || !e->indirect_unknown_callee))
+ && (!e->speculative || !e->indirect_unknown_callee)
+ /* It is possible that edge was previously speculative. In this case
+ we have different value in call stmt hash which needs preserving. */
+ && e->caller->get_edge (e->call_stmt) == e)
e->caller->call_site_hash->remove_elt_with_hash
(e->call_stmt, cgraph_edge_hasher::hash (e->call_stmt));
@@ -814,7 +835,14 @@ cgraph_edge::set_call_stmt (cgraph_edge *e, gcall *new_stmt,
function *fun = DECL_STRUCT_FUNCTION (e->caller->decl);
e->can_throw_external = stmt_can_throw_external (fun, new_stmt);
- if (e->caller->call_site_hash)
+ /* Update call stite hash. For speculative calls we only record the first
+ direct edge. */
+ if (e->caller->call_site_hash
+ && (!e->speculative
+ || (e->callee
+ && (!e->prev_callee || !e->prev_callee->speculative
+ || e->prev_callee->call_stmt != e->call_stmt))
+ || (e->speculative && !e->callee)))
cgraph_add_edge_to_call_site_hash (e);
return e;
}
@@ -858,7 +886,6 @@ symbol_table::create_edge (cgraph_node *caller, cgraph_node *callee,
edge->prev_callee = NULL;
edge->next_callee = NULL;
edge->lto_stmt_uid = 0;
- edge->target_prob = 0;
edge->speculative_id = 0;
edge->count = count;
@@ -992,7 +1019,8 @@ cgraph_edge::remove_caller (void)
else
caller->callees = next_callee;
}
- if (caller->call_site_hash)
+ if (caller->call_site_hash
+ && this == caller->get_edge (call_stmt))
caller->call_site_hash->remove_elt_with_hash
(call_stmt, cgraph_edge_hasher::hash (call_stmt));
}
@@ -1049,13 +1077,11 @@ cgraph_edge::remove (cgraph_edge *edge)
speculative_id is used to link direct calls with their corresponding
IPA_REF_ADDR references when representing speculative calls.
- target_prob is the probability of the speculative call.
-
Return direct edge created. */
cgraph_edge *
cgraph_edge::make_speculative (cgraph_node *n2, profile_count direct_count,
- unsigned int speculative_id, int target_prob)
+ unsigned int speculative_id)
{
cgraph_node *n = caller;
ipa_ref *ref = NULL;
@@ -1074,7 +1100,6 @@ cgraph_edge::make_speculative (cgraph_node *n2, profile_count direct_count,
e2->can_throw_external = can_throw_external;
e2->lto_stmt_uid = lto_stmt_uid;
e2->speculative_id = speculative_id;
- e2->target_prob = target_prob;
e2->in_polymorphic_cdtor = in_polymorphic_cdtor;
indirect_info->num_speculative_call_targets++;
count -= e2->count;
@@ -1087,91 +1112,59 @@ cgraph_edge::make_speculative (cgraph_node *n2, profile_count direct_count,
return e2;
}
-/* Speculative calls represent a transformation of indirect calls
- which may be later inserted into gimple in the following form:
-
- if (call_dest == target1)
- target1 ();
- else if (call_dest == target2)
- target2 ();
- else
- call_dest ();
+/* Speculative call consists of an indirect edge and one or more
+ direct edge+ref pairs.
- This is a win in the case when target1 and target2 are common values for
- call_dest as determined by ipa-devirt or indirect call profiling.
- In particular this may enable inlining and other optimizations.
+ Given an edge which is part of speculative call, return the first
+ direct call edge in the speculative call sequence. */
- Speculative call consists of the following main components:
-
- 1) One or more "speculative" direct call (num_speculative_call_targets is
- speculative direct call count belongs to the speculative indirect call)
- 2) One or more IPA_REF_ADDR references (representing the fact that code above
- takes address of target1 and target2)
- 3) The fallback "speculative" indirect call
-
- Direct calls and corresponding references are linked by
- speculative_id.
-
- speculative_call_info returns triple
- (direct_call, indirect call, IPA_REF_ADDR reference)
- when called on one edge participating in the speculative call:
-
- 1) If called on direct call, its corresponding IPA_REF_ADDR and related
- indirect call are returned.
-
- 2) If called on indirect call, it will return one of direct edges and its
- matching IPA_REF_ADDR.
- */
-
-void
-cgraph_edge::speculative_call_info (cgraph_edge *&direct,
- cgraph_edge *&indirect,
- ipa_ref *&reference)
+cgraph_edge *
+cgraph_edge::first_speculative_call_target ()
{
- ipa_ref *ref;
- int i;
- cgraph_edge *e2;
cgraph_edge *e = this;
- if (!e->indirect_unknown_callee)
- for (e2 = e->caller->indirect_calls;
- e2->call_stmt != e->call_stmt || e2->lto_stmt_uid != e->lto_stmt_uid;
- e2 = e2->next_callee)
- ;
- else
+ gcc_checking_assert (e->speculative);
+ if (e->callee)
+ {
+ while (e->prev_callee && e->prev_callee->speculative
+ && e->prev_callee->call_stmt == e->call_stmt
+ && e->prev_callee->lto_stmt_uid == e->lto_stmt_uid)
+ e = e->prev_callee;
+ return e;
+ }
+ /* Call stmt site hash always points to the first target of the
+ speculative call sequence. */
+ if (e->call_stmt)
+ return e->caller->get_edge (e->call_stmt);
+ for (cgraph_edge *e2 = e->caller->callees; true; e2 = e2->next_callee)
+ if (e2->speculative
+ && e->call_stmt == e2->call_stmt
+ && e->lto_stmt_uid == e2->lto_stmt_uid)
+ return e2;
+}
+
+/* We always maintain first direct edge in the call site hash, if one
+ exists. E is going to be removed. See if it is first one and update
+ hash accordingly. INDIRECT is the indirect edge of speculative call.
+ We assume that INDIRECT->num_speculative_call_targets_p () is already
+ updated for removal of E. */
+static void
+update_call_stmt_hash_for_removing_direct_edge (cgraph_edge *e,
+ cgraph_edge *indirect)
+{
+ if (e->caller->call_site_hash)
{
- e2 = e;
- /* We can take advantage of the call stmt hash. */
- if (e2->call_stmt)
+ if (e->caller->get_edge (e->call_stmt) != e)
+ ;
+ else if (!indirect->num_speculative_call_targets_p ())
+ cgraph_update_edge_in_call_site_hash (indirect);
+ else
{
- e = e->caller->get_edge (e2->call_stmt);
- gcc_assert (e->speculative && !e->indirect_unknown_callee);
+ gcc_checking_assert (e->next_callee && e->next_callee->speculative
+ && e->next_callee->call_stmt == e->call_stmt);
+ cgraph_update_edge_in_call_site_hash (e->next_callee);
}
- else
- for (e = e->caller->callees;
- e2->call_stmt != e->call_stmt
- || e2->lto_stmt_uid != e->lto_stmt_uid;
- e = e->next_callee)
- ;
}
- gcc_assert (e->speculative && e2->speculative);
- direct = e;
- indirect = e2;
-
- reference = NULL;
- for (i = 0; e->caller->iterate_reference (i, ref); i++)
- if (ref->speculative && ref->speculative_id == e->speculative_id
- && ((ref->stmt && ref->stmt == e->call_stmt)
- || (!ref->stmt && ref->lto_stmt_uid == e->lto_stmt_uid)))
- {
- reference = ref;
- break;
- }
-
- /* Speculative edge always consist of all three components - direct edge,
- indirect and reference. */
-
- gcc_assert (e && e2 && ref);
}
/* Speculative call EDGE turned out to be direct call to CALLEE_DECL. Remove
@@ -1195,8 +1188,10 @@ cgraph_edge::resolve_speculation (cgraph_edge *edge, tree callee_decl)
cgraph_edge *e2;
ipa_ref *ref;
- gcc_assert (edge->speculative);
- edge->speculative_call_info (e2, edge, ref);
+ gcc_assert (edge->speculative && (!callee_decl || edge->callee));
+ e2 = edge->first_speculative_call_target ();
+ ref = e2->speculative_call_target_ref ();
+ edge = edge->speculative_call_indirect_edge ();
if (!callee_decl
|| !ref->referred->semantically_equivalent_p
(symtab_node::get (callee_decl)))
@@ -1242,31 +1237,30 @@ cgraph_edge::resolve_speculation (cgraph_edge *edge, tree callee_decl)
else
edge->speculative = false;
e2->speculative = false;
+ update_call_stmt_hash_for_removing_direct_edge (e2, edge);
ref->remove_reference ();
if (e2->indirect_unknown_callee || e2->inline_failed)
remove (e2);
else
e2->callee->remove_symbol_and_inline_clones ();
- if (edge->caller->call_site_hash)
- {
- /* We always maintain direct edge in the call site hash, if one
- exists. */
- if (!edge->num_speculative_call_targets_p ())
- cgraph_update_edge_in_call_site_hash (edge);
- else
- {
- cgraph_edge *e;
- for (e = edge->caller->callees;
- e->call_stmt != edge->call_stmt
- || e->lto_stmt_uid != edge->lto_stmt_uid;
- e = e->next_callee)
- ;
- cgraph_update_edge_in_call_site_hash (e);
- }
- }
return edge;
}
+/* Return edge corresponding to speculative call to a given target.
+ NULL if speculative call does not have one. */
+
+cgraph_edge *
+cgraph_edge::speculative_call_for_target (cgraph_node *target)
+{
+ for (cgraph_edge *direct = first_speculative_call_target ();
+ direct;
+ direct = direct->next_speculative_call_target ())
+ if (direct->speculative_call_target_ref ()
+ ->referred->semantically_equivalent_p (target))
+ return direct;
+ return NULL;
+}
+
/* Make an indirect edge with an unknown callee an ordinary edge leading to
CALLEE. Speculations can be resolved in the process and EDGE can be removed
and deallocated. Return the edge that now represents the call. */
@@ -1281,38 +1275,28 @@ cgraph_edge::make_direct (cgraph_edge *edge, cgraph_node *callee)
{
cgraph_edge *found = NULL;
cgraph_edge *direct, *next;
- ipa_ref *ref;
- edge->speculative_call_info (direct, edge, ref);
+ edge = edge->speculative_call_indirect_edge ();
/* Look all speculative targets and remove all but one corresponding
- to callee (if it exists).
- If there is only one target we can save one extra call to
- speculative_call_info. */
- if (edge->num_speculative_call_targets_p () != 1)
- for (direct = edge->caller->callees; direct; direct = next)
- {
- next = direct->next_callee;
- if (direct->call_stmt == edge->call_stmt
- && direct->lto_stmt_uid == edge->lto_stmt_uid)
- {
- direct->speculative_call_info (direct, edge, ref);
-
- /* Compare ref not direct->callee. Direct edge is possibly
- inlined or redirected. */
- if (!ref->referred->semantically_equivalent_p (callee))
- edge = direct->resolve_speculation (direct, NULL);
- else
- {
- gcc_checking_assert (!found);
- found = direct;
- }
- }
- }
- else if (!ref->referred->semantically_equivalent_p (callee))
- edge = direct->resolve_speculation (direct, NULL);
- else
- found = direct;
+ to callee (if it exists). */
+ for (direct = edge->first_speculative_call_target ();
+ direct;
+ direct = next)
+ {
+ next = direct->next_speculative_call_target ();
+
+ /* Compare ref not direct->callee. Direct edge is possibly
+ inlined or redirected. */
+ if (!direct->speculative_call_target_ref ()
+ ->referred->semantically_equivalent_p (callee))
+ edge = direct->resolve_speculation (direct, NULL);
+ else
+ {
+ gcc_checking_assert (!found);
+ found = direct;
+ }
+ }
/* On successful speculation just remove the indirect edge and
return the pre existing direct edge.
@@ -1320,8 +1304,8 @@ cgraph_edge::make_direct (cgraph_edge *edge, cgraph_node *callee)
edge may be inlined or redirected. */
if (found)
{
- resolve_speculation (edge, callee->decl);
- gcc_checking_assert (!found->speculative);
+ cgraph_edge *e2 = resolve_speculation (found, callee->decl);
+ gcc_checking_assert (!found->speculative && e2 == found);
return found;
}
gcc_checking_assert (!edge->speculative);
@@ -1377,17 +1361,19 @@ cgraph_edge::redirect_call_stmt_to_callee (cgraph_edge *e)
if (e->speculative)
{
- cgraph_edge *e2;
- gcall *new_stmt;
- ipa_ref *ref;
-
- e->speculative_call_info (e, e2, ref);
/* If there already is an direct call (i.e. as a result of inliner's
substitution), forget about speculating. */
if (decl)
e = make_direct (e, cgraph_node::get (decl));
else
{
+ /* Be sure we redirect all speculative targets before poking
+ abou tindirect edge. */
+ gcc_checking_assert (e->callee);
+ cgraph_edge *indirect = e->speculative_call_indirect_edge ();
+ gcall *new_stmt;
+ ipa_ref *ref;
+
/* Expand speculation into GIMPLE code. */
if (dump_file)
{
@@ -1398,51 +1384,45 @@ cgraph_edge::redirect_call_stmt_to_callee (cgraph_edge *e)
e->count.dump (dump_file);
fprintf (dump_file, "\n");
}
- gcc_assert (e2->speculative);
push_cfun (DECL_STRUCT_FUNCTION (e->caller->decl));
- profile_probability prob = e->count.probability_in (e->count
- + e2->count);
+ profile_count all = indirect->count;
+ for (cgraph_edge *e2 = e->first_speculative_call_target ();
+ e2;
+ e2 = e2->next_speculative_call_target ())
+ all = all + e2->count;
+ profile_probability prob = e->count.probability_in (all);
if (!prob.initialized_p ())
prob = profile_probability::even ();
+ ref = e->speculative_call_target_ref ();
new_stmt = gimple_ic (e->call_stmt,
dyn_cast<cgraph_node *> (ref->referred),
prob);
e->speculative = false;
- e->caller->set_call_stmt_including_clones (e->call_stmt, new_stmt,
- false);
- e->count = gimple_bb (e->call_stmt)->count;
- if (e2->num_speculative_call_targets_p ())
+ if (indirect->num_speculative_call_targets_p ())
{
/* The indirect edge has multiple speculative targets, don't
remove speculative until all related direct edges are
redirected. */
- e2->indirect_info->num_speculative_call_targets--;
- if (!e2->indirect_info->num_speculative_call_targets)
- e2->speculative = false;
+ indirect->indirect_info->num_speculative_call_targets--;
+ if (!indirect->indirect_info->num_speculative_call_targets)
+ indirect->speculative = false;
}
else
- e2->speculative = false;
- e2->count = gimple_bb (e2->call_stmt)->count;
- ref->speculative = false;
- ref->stmt = NULL;
+ indirect->speculative = false;
/* Indirect edges are not both in the call site hash.
get it updated. */
- if (e->caller->call_site_hash)
- {
- if (!e2->num_speculative_call_targets_p ())
- cgraph_update_edge_in_call_site_hash (e2);
- else
- {
- cgraph_edge *e;
- for (e = e2->caller->callees;
- e->call_stmt != e2->call_stmt
- || e->lto_stmt_uid != e2->lto_stmt_uid;
- e = e->next_callee)
- ;
- cgraph_update_edge_in_call_site_hash (e);
- }
- }
+ update_call_stmt_hash_for_removing_direct_edge (e, indirect);
+ cgraph_edge::set_call_stmt (e, new_stmt, false);
+ e->count = gimple_bb (e->call_stmt)->count;
+
+ /* Once we are done with expanding the sequence, update also indirect
+ call probability. Until then the basic block accounts for the
+ sum of indirect edge and all non-expanded speculations. */
+ if (!indirect->speculative)
+ indirect->count = gimple_bb (indirect->call_stmt)->count;
+ ref->speculative = false;
+ ref->stmt = NULL;
pop_cfun ();
/* Continue redirecting E to proper target. */
}
@@ -3222,10 +3202,21 @@ verify_speculative_call (struct cgraph_node *node, gimple *stmt,
refs[i] = NULL;
}
+ cgraph_edge *first_call = NULL;
+ cgraph_edge *prev_call = NULL;
+
for (cgraph_edge *direct = node->callees; direct;
direct = direct->next_callee)
if (direct->call_stmt == stmt && direct->lto_stmt_uid == lto_stmt_uid)
{
+ if (!first_call)
+ first_call = direct;
+ if (prev_call && direct != prev_call->next_callee)
+ {
+ error ("speculative edges are not adjacent");
+ return true;
+ }
+ prev_call = direct;
if (!direct->speculative)
{
error ("direct call to %s in speculative call sequence has no "
@@ -3249,6 +3240,14 @@ verify_speculative_call (struct cgraph_node *node, gimple *stmt,
direct_calls[direct->speculative_id] = direct;
}
+ if (first_call->call_stmt
+ && first_call != node->get_edge (first_call->call_stmt))
+ {
+ error ("call stmt hash does not point to first direct edge of "
+ "speculative call sequence ");
+ return true;
+ }
+
ipa_ref *ref;
for (int i = 0; node->iterate_reference (i, ref); i++)
if (ref->speculative
diff --git a/gcc/cgraph.h b/gcc/cgraph.h
index 0ace13d..aa4cdc9 100644
--- a/gcc/cgraph.h
+++ b/gcc/cgraph.h
@@ -1765,14 +1765,87 @@ public:
the profile so the direct call is taken COUNT times
with FREQUENCY. speculative_id is used to link direct calls with their
corresponding IPA_REF_ADDR references when representing speculative calls.
- target_prob is the probability of the speculative call. */
+ */
cgraph_edge *make_speculative (cgraph_node *n2, profile_count direct_count,
- unsigned int speculative_id = 0,
- int target_prob = 0);
+ unsigned int speculative_id = 0);
- /* Given speculative call edge, return all three components. */
- void speculative_call_info (cgraph_edge *&direct, cgraph_edge *&indirect,
- ipa_ref *&reference);
+ /* Speculative call consists of an indirect edge and one or more
+ direct edge+ref pairs. Speculative will expand to the following sequence:
+
+ if (call_dest == target1) // reference to target1
+ target1 (); // direct call to target1
+ else if (call_dest == target2) // reference to targt2
+ target2 (); // direct call to target2
+ else
+ call_dest (); // indirect call
+
+ Before the expansion we will have indirect call and the direct call+ref
+ pairs all linked to single statement.
+
+ Note that ref may point to different symbol than the corresponding call
+ becuase the speculated edge may have been optimized (redirected to
+ a clone) or inlined.
+
+ Given an edge which is part of speculative call, return the first
+ direct call edge in the speculative call sequence.
+
+ In the example above called on any cgraph edge in the sequence it will
+ return direct call to target1. */
+ cgraph_edge *first_speculative_call_target ();
+
+ /* Return next speculative call target or NULL if there is none.
+ All targets are required to form an interval in the callee list.
+
+ In example above, if called on call to target1 it will return call to
+ target2. */
+ cgraph_edge *next_speculative_call_target ()
+ {
+ cgraph_edge *e = this;
+ gcc_checking_assert (speculative && callee);
+
+ if (e->next_callee && e->next_callee->speculative
+ && e->next_callee->call_stmt == e->call_stmt
+ && e->next_callee->lto_stmt_uid == e->lto_stmt_uid)
+ return e->next_callee;
+ return NULL;
+ }
+
+ /* When called on any edge in the speculative call return the (unique)
+ indirect call edge in the speculative call sequence. */
+ cgraph_edge *speculative_call_indirect_edge ()
+ {
+ gcc_checking_assert (speculative);
+ if (!callee)
+ return this;
+ for (cgraph_edge *e2 = caller->indirect_calls;
+ true; e2 = e2->next_callee)
+ if (e2->speculative
+ && call_stmt == e2->call_stmt
+ && lto_stmt_uid == e2->lto_stmt_uid)
+ return e2;
+ }
+
+ /* When called on any edge in speculative call and when given any target
+ of ref which is speculated to it returns the corresponding direct call.
+
+ In example above if called on function target2 it will return call to
+ target2. */
+ cgraph_edge *speculative_call_for_target (cgraph_node *);
+
+ /* Return REF corresponding to direct call in the specualtive call
+ sequence. */
+ ipa_ref *speculative_call_target_ref ()
+ {
+ ipa_ref *ref;
+
+ gcc_checking_assert (speculative);
+ for (unsigned int i = 0; caller->iterate_reference (i, ref); i++)
+ if (ref->speculative && ref->speculative_id == speculative_id
+ && ref->stmt == (gimple *)call_stmt
+ && ref->lto_stmt_uid == lto_stmt_uid)
+ return ref;
+ gcc_unreachable ();
+ }
/* Speculative call edge turned out to be direct call to CALLEE_DECL. Remove
the speculative call sequence and return edge representing the call, the
@@ -1874,8 +1947,6 @@ public:
/* The stmt_uid of call_stmt. This is used by LTO to recover the call_stmt
when the function is serialized in. */
unsigned int lto_stmt_uid;
- /* target_prob is the probability of the speculative call. */
- unsigned int target_prob;
/* speculative id is used to link direct calls with their corresponding
IPA_REF_ADDR references when representing speculative calls. */
unsigned int speculative_id : 16;
diff --git a/gcc/cgraphclones.c b/gcc/cgraphclones.c
index 417488b..c73b8f8 100644
--- a/gcc/cgraphclones.c
+++ b/gcc/cgraphclones.c
@@ -765,14 +765,18 @@ cgraph_node::set_call_stmt_including_clones (gimple *old_stmt,
callgraph edges. */
if (edge->speculative && !update_speculative)
{
- cgraph_edge *direct, *indirect;
- ipa_ref *ref;
-
- gcc_assert (!edge->indirect_unknown_callee);
- edge->speculative_call_info (direct, indirect, ref);
- direct->speculative = false;
+ cgraph_edge *indirect = edge->speculative_call_indirect_edge ();
+
+ for (cgraph_edge *next, *direct
+ = edge->first_speculative_call_target ();
+ direct;
+ direct = next)
+ {
+ next = direct->next_speculative_call_target ();
+ direct->speculative_call_target_ref ()->speculative = false;
+ direct->speculative = false;
+ }
indirect->speculative = false;
- ref->speculative = false;
}
}
if (node->clones)
diff --git a/gcc/ipa-devirt.c b/gcc/ipa-devirt.c
index b609a77..b04cae7 100644
--- a/gcc/ipa-devirt.c
+++ b/gcc/ipa-devirt.c
@@ -3757,11 +3757,8 @@ ipa_devirt (void)
with the speculation. */
if (e->speculative)
{
- struct cgraph_edge *e2;
- struct ipa_ref *ref;
- e->speculative_call_info (e2, e, ref);
- if (e2->callee->ultimate_alias_target ()
- == likely_target->ultimate_alias_target ())
+ bool found = e->speculative_call_for_target (likely_target);
+ if (found)
{
fprintf (dump_file, "We agree with speculation\n\n");
nok++;
diff --git a/gcc/ipa-fnsummary.c b/gcc/ipa-fnsummary.c
index dbd53f1..059ea74 100644
--- a/gcc/ipa-fnsummary.c
+++ b/gcc/ipa-fnsummary.c
@@ -2604,34 +2604,26 @@ analyze_function_body (struct cgraph_node *node, bool early)
edge_set_predicate (edge, &bb_predicate);
if (edge->speculative)
{
- cgraph_edge *direct, *indirect, *next_direct;
- ipa_ref *ref;
- edge->speculative_call_info (direct, indirect, ref);
- gcc_assert (direct == edge);
+ cgraph_edge *indirect
+ = edge->speculative_call_indirect_edge ();
ipa_call_summary *es2
= ipa_call_summaries->get_create (indirect);
ipa_call_summaries->duplicate (edge, indirect,
es, es2);
- /* Create and duplicate call summaries for multiple
+ /* Edge is the first direct call.
+ create and duplicate call summaries for multiple
speculative call targets. */
- int num_specs = indirect->num_speculative_call_targets_p ();
- if (num_specs > 1)
- for (next_direct = edge->next_callee;
- next_direct && --num_specs;
- next_direct = next_direct->next_callee)
- {
- next_direct->speculative_call_info (direct, indirect,
- ref);
- if (direct == next_direct && next_direct->speculative
- && edge->call_stmt == stmt)
- {
- ipa_call_summary *es3
- = ipa_call_summaries->get_create (next_direct);
- ipa_call_summaries->duplicate (edge, next_direct,
- es, es3);
- }
- }
+ for (cgraph_edge *direct
+ = edge->next_speculative_call_target ();
+ direct;
+ direct = direct->next_speculative_call_target ())
+ {
+ ipa_call_summary *es3
+ = ipa_call_summaries->get_create (direct);
+ ipa_call_summaries->duplicate (edge, direct,
+ es, es3);
+ }
}
}
diff --git a/gcc/ipa-inline.c b/gcc/ipa-inline.c
index c56cfb0..6b6ba9a 100644
--- a/gcc/ipa-inline.c
+++ b/gcc/ipa-inline.c
@@ -1778,8 +1778,6 @@ speculation_useful_p (struct cgraph_edge *e, bool anticipate_inlining)
enum availability avail;
struct cgraph_node *target = e->callee->ultimate_alias_target (&avail,
e->caller);
- struct cgraph_edge *direct, *indirect;
- struct ipa_ref *ref;
gcc_assert (e->speculative && !e->indirect_unknown_callee);
@@ -1794,14 +1792,14 @@ speculation_useful_p (struct cgraph_edge *e, bool anticipate_inlining)
int ecf_flags = flags_from_decl_or_type (target->decl);
if (ecf_flags & ECF_CONST)
{
- e->speculative_call_info (direct, indirect, ref);
- if (!(indirect->indirect_info->ecf_flags & ECF_CONST))
+ if (!(e->speculative_call_indirect_edge ()->indirect_info
+ ->ecf_flags & ECF_CONST))
return true;
}
else if (ecf_flags & ECF_PURE)
{
- e->speculative_call_info (direct, indirect, ref);
- if (!(indirect->indirect_info->ecf_flags & ECF_PURE))
+ if (!(e->speculative_call_indirect_edge ()->indirect_info
+ ->ecf_flags & ECF_PURE))
return true;
}
}
diff --git a/gcc/ipa-profile.c b/gcc/ipa-profile.c
index 8c5502b..9fbfa90 100644
--- a/gcc/ipa-profile.c
+++ b/gcc/ipa-profile.c
@@ -134,7 +134,8 @@ static void
dump_histogram (FILE *file, vec<histogram_entry *> histogram)
{
unsigned int i;
- gcov_type overall_time = 0, cumulated_time = 0, cumulated_size = 0, overall_size = 0;
+ gcov_type overall_time = 0, cumulated_time = 0, cumulated_size = 0,
+ overall_size = 0;
fprintf (dump_file, "Histogram:\n");
for (i = 0; i < histogram.length (); i++)
@@ -266,7 +267,8 @@ ipa_profile_generate_summary (void)
call_sums = new ipa_profile_call_summaries (symtab);
FOR_EACH_FUNCTION_WITH_GIMPLE_BODY (node)
- if (ENTRY_BLOCK_PTR_FOR_FN (DECL_STRUCT_FUNCTION (node->decl))->count.ipa_p ())
+ if (ENTRY_BLOCK_PTR_FOR_FN
+ (DECL_STRUCT_FUNCTION (node->decl))->count.ipa_p ())
FOR_EACH_BB_FN (bb, DECL_STRUCT_FUNCTION (node->decl))
{
int time = 0;
@@ -300,30 +302,31 @@ ipa_profile_generate_summary (void)
j))
continue;
- if (val == 0)
+ if (val == 0 || count == 0)
continue;
- speculative_call_target item (
- val, GCOV_COMPUTE_SCALE (count, all));
- if (item.target_probability > REG_BR_PROB_BASE)
+ if (count > all)
{
if (dump_file)
fprintf (dump_file,
"Probability capped to 1\n");
- item.target_probability = REG_BR_PROB_BASE;
+ count = all;
}
+ speculative_call_target item (
+ val, GCOV_COMPUTE_SCALE (count, all));
csum->speculative_call_targets.safe_push (item);
}
- gimple_remove_histogram_value (DECL_STRUCT_FUNCTION (node->decl),
- stmt, h);
+ gimple_remove_histogram_value
+ (DECL_STRUCT_FUNCTION (node->decl), stmt, h);
}
}
time += estimate_num_insns (stmt, &eni_time_weights);
size += estimate_num_insns (stmt, &eni_size_weights);
}
if (bb->count.ipa_p () && bb->count.initialized_p ())
- account_time_size (&hashtable, histogram, bb->count.ipa ().to_gcov_type (),
+ account_time_size (&hashtable, histogram,
+ bb->count.ipa ().to_gcov_type (),
time, size);
}
histogram.qsort (cmp_counts);
@@ -864,6 +867,7 @@ ipa_profile (void)
}
unsigned speculative_id = 0;
+ profile_count orig = e->count;
for (unsigned i = 0; i < spec_count; i++)
{
speculative_call_target item
@@ -881,7 +885,8 @@ ipa_profile (void)
item.target_probability
/ (float) REG_BR_PROB_BASE);
}
- if (item.target_probability < REG_BR_PROB_BASE / 2)
+ if (item.target_probability
+ < REG_BR_PROB_BASE / GCOV_TOPN_VALUES / 2)
{
nuseless++;
if (dump_file)
@@ -939,11 +944,12 @@ ipa_profile (void)
n2 = alias;
}
nconverted++;
+ profile_probability prob
+ = profile_probability::from_reg_br_prob_base
+ (item.target_probability).adjusted ();
e->make_speculative (n2,
- e->count.apply_probability (
- item.target_probability),
- speculative_id,
- item.target_probability);
+ orig.apply_probability (prob),
+ speculative_id);
update = true;
speculative_id++;
}
diff --git a/gcc/ipa-prop.c b/gcc/ipa-prop.c
index 12cdb95..7b97154 100644
--- a/gcc/ipa-prop.c
+++ b/gcc/ipa-prop.c
@@ -3237,7 +3237,7 @@ ipa_make_edge_direct_to_target (struct cgraph_edge *ie, tree target,
{
if (dump_file)
fprintf (dump_file, "ipa-prop: Discovered call to a known target "
- "(%s -> %s) but cannot refer to it. Giving up.\n",
+ "(%s -> %s) but cannot refer to it. Giving up.\n",
ie->caller->dump_name (),
ie->callee->dump_name ());
return NULL;
@@ -3248,25 +3248,26 @@ ipa_make_edge_direct_to_target (struct cgraph_edge *ie, tree target,
/* If the edge is already speculated. */
if (speculative && ie->speculative)
{
- struct cgraph_edge *e2;
- struct ipa_ref *ref;
- ie->speculative_call_info (e2, ie, ref);
- if (e2->callee->ultimate_alias_target ()
- != callee->ultimate_alias_target ())
- {
- if (dump_file)
- fprintf (dump_file, "ipa-prop: Discovered call to a speculative "
- "target (%s -> %s) but the call is already "
- "speculated to %s. Giving up.\n",
- ie->caller->dump_name (), callee->dump_name (),
- e2->callee->dump_name ());
- }
- else
+ if (dump_file)
{
- if (dump_file)
- fprintf (dump_file, "ipa-prop: Discovered call to a speculative target "
- "(%s -> %s) this agree with previous speculation.\n",
- ie->caller->dump_name (), callee->dump_name ());
+ cgraph_edge *e2 = ie->speculative_call_for_target (callee);
+ if (!e2)
+ {
+ if (dump_file)
+ fprintf (dump_file, "ipa-prop: Discovered call to a "
+ "speculative target (%s -> %s) but the call is "
+ "already speculated to different target. "
+ "Giving up.\n",
+ ie->caller->dump_name (), callee->dump_name ());
+ }
+ else
+ {
+ if (dump_file)
+ fprintf (dump_file,
+ "ipa-prop: Discovered call to a speculative target "
+ "(%s -> %s) this agree with previous speculation.\n",
+ ie->caller->dump_name (), callee->dump_name ());
+ }
}
return NULL;
}
@@ -3771,7 +3772,6 @@ update_indirect_edges_after_inlining (struct cgraph_edge *cs,
class cgraph_indirect_call_info *ici = ie->indirect_info;
struct ipa_jump_func *jfunc;
int param_index;
- cgraph_node *spec_target = NULL;
next_ie = ie->next_callee;
@@ -3787,14 +3787,11 @@ update_indirect_edges_after_inlining (struct cgraph_edge *cs,
param_index = ici->param_index;
jfunc = ipa_get_ith_jump_func (top, param_index);
+ cgraph_node *spec_target;
+ /* FIXME: This may need updating for multiple calls. */
if (ie->speculative)
- {
- struct cgraph_edge *de;
- struct ipa_ref *ref;
- ie->speculative_call_info (de, ie, ref);
- spec_target = de->callee;
- }
+ spec_target = ie->first_speculative_call_target ()->callee;
if (!opt_for_fn (node->decl, flag_indirect_inlining))
new_direct_edge = NULL;
diff --git a/gcc/ipa-utils.c b/gcc/ipa-utils.c
index 587ad5c..23e7f71 100644
--- a/gcc/ipa-utils.c
+++ b/gcc/ipa-utils.c
@@ -674,201 +674,75 @@ ipa_merge_profiles (struct cgraph_node *dst,
for (e = dst->indirect_calls, e2 = src->indirect_calls; e;
e2 = (e2 ? e2->next_callee : NULL), e = e->next_callee)
{
- profile_count count = gimple_bb (e->call_stmt)->count;
- /* Below code are introduced by r279373 of "Fix merging of common
- traget info.".
-
- ipa-icf runs after ipa-profile, common_target_id with
- common_target_probablity are useless in ipa-icf since they are
- moved from cgraph.h to ipa-profile.c and processed already.
- Need double circulation to find out each mapped direct speculative
- edge and do prob merge. Not easy to construct a case to cover all
- circumstances here. For src and dst both have multiple speculative
- targets, only N:N maps are implemented, 2:0, 2:1, 0:2, 1:2 are not
- implemented yet as too complicated and no test cases to cover. */
- if (copy_counts)
+ if (!e->speculative && !e2->speculative)
{
- /* copy if both e and e2 have same num_speculative_call_targets.
- */
- if (e->num_speculative_call_targets_p ()
- == e2->num_speculative_call_targets_p ())
- {
- int num_specs = e->num_speculative_call_targets_p ();
- cgraph_edge *direct, *indirect, *next_direct;
- cgraph_edge *direct2, *indirect2, *next_direct2;
- ipa_ref *ref;
- for (next_direct = e; next_direct && num_specs--;
- next_direct = direct->next_callee)
- {
- next_direct->speculative_call_info (direct, indirect,
- ref);
-
- int num_specs2 = e2->num_speculative_call_targets_p ();
- for (next_direct2 = e2; next_direct2 && num_specs2--;
- next_direct2 = direct2->next_callee)
- {
- if (e2 && e2->speculative)
- next_direct2->speculative_call_info (direct2,
- indirect2,
- ref);
- if (direct->speculative_id == direct2->speculative_id
- && direct->lto_stmt_uid == direct2->lto_stmt_uid)
- {
- direct->target_prob = direct2->target_prob;
- break;
- }
- }
- }
- }
- else
- gcc_assert (e->num_speculative_call_targets_p ()
- && e->num_speculative_call_targets_p ());
+ /* FIXME: we need to also merge ipa-profile histograms
+ because with LTO merging happens from lto-symtab before
+ these are converted to indirect edges. */
+ e->count = gimple_bb (e->call_stmt)->count;
+ continue;
}
- else if (e->num_speculative_call_targets_p ()
- || e2->num_speculative_call_targets_p ())
+
+ /* When copying just remove all speuclations on dst and then copy
+ one from src. */
+ if (copy_counts)
{
- if (e->num_speculative_call_targets_p ()
- == e2->num_speculative_call_targets_p ())
+ while (e->speculative)
+ cgraph_edge::resolve_speculation (e, NULL);
+ e->count = gimple_bb (e->call_stmt)->count;
+ if (e2->speculative)
{
- int num_specs = e->num_speculative_call_targets_p ();
- cgraph_edge *direct, *indirect, *next_direct;
- cgraph_edge *direct2, *indirect2, *next_direct2;
- ipa_ref *ref;
- for (next_direct = e; next_direct && num_specs--;
- next_direct = direct->next_callee)
+ for (cgraph_edge *e3 = e2->first_speculative_call_target ();
+ e3;
+ e3 = e3->next_speculative_call_target ())
{
- next_direct->speculative_call_info (direct, indirect,
- ref);
-
- int num_specs2 = e2->num_speculative_call_targets_p ();
- for (next_direct2 = e2; next_direct2 && num_specs2--;
- next_direct2 = direct2->next_callee)
- {
- if (e2 && e2->speculative)
- next_direct2->speculative_call_info (direct2,
- indirect2,
- ref);
- if (direct->speculative_id == direct2->speculative_id
- && direct->lto_stmt_uid == direct2->lto_stmt_uid)
- {
- sreal scale1
- = e->count.ipa ().to_sreal_scale (count);
- sreal scale2
- = e2->count.ipa ().to_sreal_scale (count);
-
- if (scale1 == 0 && scale2 == 0)
- scale1 = scale2 = 1;
- sreal sum = scale1 + scale2;
- int scaled_prob1
- = (((sreal)direct->target_prob)
- * scale1 / sum).to_int ();
- int scaled_prob2
- = (((sreal)direct2->target_prob)
- * scale2 / sum).to_int ();
- if (symtab->dump_file)
- {
- fprintf (
- symtab->dump_file,
- "Merging speculative id %i prob %i"
- " and %i prob %i with scales %f %f\n",
- direct->speculative_id, direct->target_prob,
- direct2->speculative_id,
- direct2->target_prob, scale1.to_double (),
- scale2.to_double ());
- fprintf (symtab->dump_file,
- "Combined BB count ");
- count.dump (symtab->dump_file);
- fprintf (symtab->dump_file,
- " dst edge count ");
- e->count.dump (symtab->dump_file);
- fprintf (symtab->dump_file,
- " src edge count ");
- e2->count.dump (symtab->dump_file);
- fprintf (symtab->dump_file, "\n");
- }
- direct->target_prob = scaled_prob1 + scaled_prob2;
- break;
- }
- }
+ cgraph_edge *ns;
+ ns = e->make_speculative
+ (dyn_cast <cgraph_node *>
+ (e3->speculative_call_target_ref ()->referred),
+ e3->count, e3->speculative_id);
+ /* Target may differ from ref (for example it may be
+ redirected to local alias. */
+ ns->redirect_callee (e3->callee);
}
}
- else if (e->num_speculative_call_targets_p ())
- {
- /* Process if only dst is speculative. */
- gcc_assert (!e->num_speculative_call_targets_p ());
- }
- else if (e2->num_speculative_call_targets_p ())
- {
- /* Process if only src is speculative. */
- gcc_assert (!e2->num_speculative_call_targets_p ());
- }
+ continue;
}
- /* When call is speculative, we need to re-distribute probabilities
- the same way as they was. This is not really correct because
- in the other copy the speculation may differ; but probably it
- is not really worth the effort. */
- if (e->speculative)
+ /* Iterate all speculations in SRC, see if corresponding ones exist
+ int DST and if so, sum the counts. Otherwise create new
+ speculation. */
+ int max_spec = 0;
+ for (cgraph_edge *e3 = e->first_speculative_call_target ();
+ e3;
+ e3 = e3->next_speculative_call_target ())
+ if (e3->speculative_id > max_spec)
+ max_spec = e3->speculative_id;
+ for (cgraph_edge *e3 = e2->first_speculative_call_target ();
+ e3;
+ e3 = e3->next_speculative_call_target ())
{
- cgraph_edge *direct, *indirect;
- cgraph_edge *direct2 = NULL, *indirect2 = NULL;
- ipa_ref *ref;
-
- e->speculative_call_info (direct, indirect, ref);
- gcc_assert (e == indirect);
- if (e2 && e2->speculative)
- e2->speculative_call_info (direct2, indirect2, ref);
- if (indirect->count > profile_count::zero ()
- || direct->count > profile_count::zero ())
+ cgraph_edge *te
+ = e->speculative_call_for_target
+ (dyn_cast <cgraph_node *>
+ (e3->speculative_call_target_ref ()->referred));
+ if (te)
+ te->count = te->count + e3->count;
+ else
{
- /* We should mismatch earlier if there is no matching
- indirect edge. */
- if (!e2)
- {
- if (symtab->dump_file)
- fprintf (symtab->dump_file,
- "Mismatch in merging indirect edges\n");
- }
- else if (!e2->speculative)
- indirect->count += e2->count;
- else if (e2->speculative)
- {
- if (DECL_ASSEMBLER_NAME (direct2->callee->decl)
- != DECL_ASSEMBLER_NAME (direct->callee->decl))
- {
- if (direct2->count >= direct->count)
- {
- direct->redirect_callee (direct2->callee);
- indirect->count += indirect2->count
- + direct->count;
- direct->count = direct2->count;
- }
- else
- indirect->count += indirect2->count + direct2->count;
- }
- else
- {
- direct->count += direct2->count;
- indirect->count += indirect2->count;
- }
- }
+ e->count = e->count + e3->count;
+ cgraph_edge *ns;
+ ns = e->make_speculative
+ (dyn_cast <cgraph_node *>
+ (e3->speculative_call_target_ref ()
+ ->referred),
+ e3->count,
+ e3->speculative_id + max_spec + 1);
+ /* Target may differ from ref (for example it may be
+ redirected to local alias. */
+ ns->redirect_callee (e3->callee);
}
- else
- /* At the moment we should have only profile feedback based
- speculations when merging. */
- gcc_unreachable ();
}
- else if (e2 && e2->speculative)
- {
- cgraph_edge *direct, *indirect;
- ipa_ref *ref;
-
- e2->speculative_call_info (direct, indirect, ref);
- e->count = count;
- e->make_speculative (direct->callee, direct->count);
- }
- else
- e->count = count;
}
if (!preserve_body)
src->release_body ();
@@ -882,7 +756,8 @@ ipa_merge_profiles (struct cgraph_node *dst,
src->decl = oldsrcdecl;
}
-/* Return true if call to DEST is known to be self-recusive call withing FUNC. */
+/* Return true if call to DEST is known to be self-recusive
+ call withing FUNC. */
bool
recursive_call_p (tree func, tree dest)
diff --git a/gcc/profile-count.h b/gcc/profile-count.h
index 09217a8..014f597 100644
--- a/gcc/profile-count.h
+++ b/gcc/profile-count.h
@@ -262,6 +262,16 @@ public:
return ret;
}
+ /* Return THIS with quality set to ADJUSTED. */
+ profile_probability adjusted () const
+ {
+ profile_probability ret = *this;
+ if (!initialized_p ())
+ return *this;
+ ret.m_quality = ADJUSTED;
+ return ret;
+ }
+
int to_reg_br_prob_base () const
{
gcc_checking_assert (initialized_p ());
diff --git a/gcc/testsuite/g++.dg/tree-prof/indir-call-prof.C b/gcc/testsuite/g++.dg/tree-prof/indir-call-prof.C
index be896c0..3374744 100644
--- a/gcc/testsuite/g++.dg/tree-prof/indir-call-prof.C
+++ b/gcc/testsuite/g++.dg/tree-prof/indir-call-prof.C
@@ -38,6 +38,6 @@ main (void)
return 0;
}
-/* { dg-final-use-not-autofdo { scan-ipa-dump "Indirect call -> direct call.* AA transformation on insn" "profile" } } */
-/* { dg-final-use-autofdo { scan-ipa-dump "Indirect call -> direct call.* AA transformation on insn" "afdo" } } */
+/* { dg-final-use-not-autofdo { scan-ipa-dump "Indirect call -> direct call.* AA .will resolve by ipa-profile" "profile" } } */
+/* { dg-final-use-autofdo { scan-ipa-dump "Indirect call -> direct call.* AA .will resolve by ipa-profile" "afdo" } } */
/* { dg-final-use-not-autofdo { scan-tree-dump-not "Invalid sum" "optimized" } } */
diff --git a/gcc/testsuite/gcc.dg/tree-prof/crossmodule-indircall-1.c b/gcc/testsuite/gcc.dg/tree-prof/crossmodule-indircall-1.c
index 58109d5..4f65dc6 100644
--- a/gcc/testsuite/gcc.dg/tree-prof/crossmodule-indircall-1.c
+++ b/gcc/testsuite/gcc.dg/tree-prof/crossmodule-indircall-1.c
@@ -3,7 +3,7 @@
/* { dg-options "-O3 -flto -DDOJOB=1" } */
int a;
-extern void (*p[2])(int n);
+extern void (*p[5])(int n);
void abort (void);
int
main()
@@ -14,8 +14,8 @@ main()
p[0](1);
/* This call shall not be converted. */
for (i = 0;i<1000;i++)
- p[i%2](2);
- if (a != 1000)
+ p[i%5](2);
+ if (a != -1000)
abort ();
return 0;
diff --git a/gcc/testsuite/gcc.dg/tree-prof/crossmodule-indircall-1a.c b/gcc/testsuite/gcc.dg/tree-prof/crossmodule-indircall-1a.c
index 568cfa9..9223785 100644
--- a/gcc/testsuite/gcc.dg/tree-prof/crossmodule-indircall-1a.c
+++ b/gcc/testsuite/gcc.dg/tree-prof/crossmodule-indircall-1a.c
@@ -30,8 +30,23 @@ sub(int i)
{
a -= i;
}
+void
+add2(int i)
+{
+ a -= 2*i;
+}
+void
+sub2(int i)
+{
+ a -= 2*i;
+}
+void
+nothing(int i)
+{
+ a -= i;
+}
__attribute__ ((externally_visible))
-void (*p[2])(int)={add, sub};
+void (*p[5])(int)={add, sub, add2, sub2, nothing};
#else
int
main()
diff --git a/gcc/testsuite/gcc.dg/tree-prof/indir-call-prof.c b/gcc/testsuite/gcc.dg/tree-prof/indir-call-prof.c
index 3ca7893..138b85a 100644
--- a/gcc/testsuite/gcc.dg/tree-prof/indir-call-prof.c
+++ b/gcc/testsuite/gcc.dg/tree-prof/indir-call-prof.c
@@ -37,6 +37,6 @@ main (void)
return 0;
}
-/* { dg-final-use-not-autofdo { scan-ipa-dump "Indirect call -> direct call.* a1 transformation on insn" "profile"} } */
-/* { dg-final-use-autofdo { scan-ipa-dump "Indirect call -> direct call.* a1 transformation on insn" "afdo"} } */
+/* { dg-final-use-not-autofdo { scan-ipa-dump "Indirect call -> direct call.* a1 .will resolve by ipa-profile" "profile"} } */
+/* { dg-final-use-autofdo { scan-ipa-dump "Indirect call -> direct call.* a1 .will resolve by ipa-profile" "afdo"} } */
/* { dg-final-use { scan-tree-dump-not "Invalid sum" "optimized"} } */
diff --git a/gcc/tree-inline.c b/gcc/tree-inline.c
index 2197769..5b0050a 100644
--- a/gcc/tree-inline.c
+++ b/gcc/tree-inline.c
@@ -2181,47 +2181,51 @@ copy_bb (copy_body_data *id, basic_block bb,
if (edge)
{
struct cgraph_edge *old_edge = edge;
- profile_count old_cnt = edge->count;
- edge = edge->clone (id->dst_node, call_stmt,
- gimple_uid (stmt),
- num, den,
- true);
-
- /* A speculative call is consist of edges - indirect edge
- and direct edges (one indirect edeg may has multiple
- direct edges). Duplicate the whole thing and
- distribute frequencies accordingly. */
+
+ /* A speculative call is consist of multiple
+ edges - indirect edge and one or more direct edges
+ Duplicate the whole thing and distribute frequencies
+ accordingly. */
if (edge->speculative)
{
- struct cgraph_edge *direct, *indirect;
- struct ipa_ref *ref;
-
- gcc_assert (!edge->indirect_unknown_callee);
- old_edge->speculative_call_info (direct, indirect, ref);
- while (old_edge->next_callee
- && old_edge->next_callee->speculative
- && indirect->num_speculative_call_targets_p ()
- > 1)
+ int n = 0;
+ profile_count direct_cnt
+ = profile_count::zero ();
+
+ /* First figure out the distribution of counts
+ so we can re-scale BB profile accordingly. */
+ for (cgraph_edge *e = old_edge; e;
+ e = e->next_speculative_call_target ())
+ direct_cnt = direct_cnt + e->count;
+
+ cgraph_edge *indirect
+ = old_edge->speculative_call_indirect_edge ();
+ profile_count indir_cnt = indirect->count;
+
+ /* Next iterate all direct edges, clone it and its
+ corresponding reference and update profile. */
+ for (cgraph_edge *e = old_edge;
+ e;
+ e = e->next_speculative_call_target ())
{
- id->dst_node->clone_reference (ref, stmt);
-
- edge = old_edge->next_callee;
- edge = edge->clone (id->dst_node, call_stmt,
- gimple_uid (stmt), num, den,
- true);
- old_edge = old_edge->next_callee;
- gcc_assert (!edge->indirect_unknown_callee);
-
- /* If the indirect edge has multiple speculative
- calls, iterate through all direct calls
- associated to the speculative call and clone
- all related direct edges before cloning the
- related indirect edge. */
- old_edge->speculative_call_info (direct, indirect,
- ref);
+ profile_count cnt = e->count;
+
+ id->dst_node->clone_reference
+ (e->speculative_call_target_ref (), stmt);
+ edge = e->clone (id->dst_node, call_stmt,
+ gimple_uid (stmt), num, den,
+ true);
+ profile_probability prob
+ = cnt.probability_in (direct_cnt
+ + indir_cnt);
+ edge->count
+ = copy_basic_block->count.apply_probability
+ (prob);
+ n++;
}
-
- profile_count indir_cnt = indirect->count;
+ gcc_checking_assert
+ (indirect->num_speculative_call_targets_p ()
+ == n);
/* Duplicate the indirect edge after all direct edges
cloned. */
@@ -2231,14 +2235,19 @@ copy_bb (copy_body_data *id, basic_block bb,
true);
profile_probability prob
- = indir_cnt.probability_in (old_cnt + indir_cnt);
+ = indir_cnt.probability_in (direct_cnt
+ + indir_cnt);
indirect->count
= copy_basic_block->count.apply_probability (prob);
- edge->count = copy_basic_block->count - indirect->count;
- id->dst_node->clone_reference (ref, stmt);
}
else
- edge->count = copy_basic_block->count;
+ {
+ edge = edge->clone (id->dst_node, call_stmt,
+ gimple_uid (stmt),
+ num, den,
+ true);
+ edge->count = copy_basic_block->count;
+ }
}
break;
diff --git a/gcc/value-prof.c b/gcc/value-prof.c
index b7c7d7e..f0456c8 100644
--- a/gcc/value-prof.c
+++ b/gcc/value-prof.c
@@ -106,7 +106,7 @@ static bool gimple_divmod_fixed_value_transform (gimple_stmt_iterator *);
static bool gimple_mod_pow2_value_transform (gimple_stmt_iterator *);
static bool gimple_mod_subtract_transform (gimple_stmt_iterator *);
static bool gimple_stringops_transform (gimple_stmt_iterator *);
-static void gimple_ic_transform (gimple_stmt_iterator *);
+static void dump_ic_profile (gimple_stmt_iterator *gsi);
/* Allocate histogram value. */
@@ -386,7 +386,9 @@ stream_in_histogram_value (class lto_input_block *ib, gimple *stmt)
default:
gcc_unreachable ();
}
- new_val->hvalue.counters = XNEWVAR (gcov_type, sizeof (*new_val->hvalue.counters) * ncounters);
+ new_val->hvalue.counters = XNEWVAR (gcov_type,
+ sizeof (*new_val->hvalue.counters)
+ * ncounters);
new_val->n_counters = ncounters;
for (i = 0; i < ncounters; i++)
new_val->hvalue.counters[i] = streamer_read_gcov_count (ib);
@@ -629,7 +631,8 @@ gimple_value_profile_transformations (void)
}
/* The function never thansforms a GIMPLE statement. */
- gimple_ic_transform (&gsi);
+ if (dump_enabled_p ())
+ dump_ic_profile (&gsi);
}
}
@@ -1388,13 +1391,10 @@ gimple_ic (gcall *icall_stmt, struct cgraph_node *direct_call,
return dcall_stmt;
}
-/* There maybe multiple indirect targets in histogram. Check every
- indirect/virtual call if callee function exists, if not exist, leave it to
- LTO stage for later process. Modify code of this indirect call to an if-else
- structure in ipa-profile finally. */
+/* Dump info about indirect call profile. */
static void
-gimple_ic_transform (gimple_stmt_iterator *gsi)
+dump_ic_profile (gimple_stmt_iterator *gsi)
{
gcall *stmt;
histogram_value histogram;
@@ -1423,37 +1423,25 @@ gimple_ic_transform (gimple_stmt_iterator *gsi)
if (!get_nth_most_common_value (NULL, "indirect call", histogram, &val,
&count, &all, j))
return;
-
- /* Minimum probability. should be higher than 25%. */
- if (4 * count <= all)
- return;
+ if (!count)
+ continue;
direct_call = find_func_by_profile_id ((int) val);
if (direct_call == NULL)
- {
- if (val)
- {
- if (dump_enabled_p ())
- dump_printf_loc (
- MSG_MISSED_OPTIMIZATION, stmt,
- "Indirect call -> direct call from other "
- "module %T=> %i (will resolve only with LTO)\n",
- gimple_call_fn (stmt), (int) val);
- }
- return;
- }
-
- if (dump_enabled_p ())
- {
- dump_printf_loc (MSG_OPTIMIZED_LOCATIONS, stmt,
- "Indirect call -> direct call "
- "%T => %T transformation on insn postponed\n",
- gimple_call_fn (stmt), direct_call->decl);
- dump_printf_loc (MSG_NOTE, stmt,
- "hist->count %" PRId64 " hist->all %" PRId64 "\n",
- count, all);
- }
+ dump_printf_loc (
+ MSG_MISSED_OPTIMIZATION, stmt,
+ "Indirect call -> direct call from other "
+ "module %T=> %i (will resolve by ipa-profile only with LTO)\n",
+ gimple_call_fn (stmt), (int) val);
+ else
+ dump_printf_loc (MSG_OPTIMIZED_LOCATIONS, stmt,
+ "Indirect call -> direct call "
+ "%T => %T (will resolve by ipa-profile)\n",
+ gimple_call_fn (stmt), direct_call->decl);
+ dump_printf_loc (MSG_NOTE, stmt,
+ "hist->count %" PRId64 " hist->all %" PRId64 "\n",
+ count, all);
}
}
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2020-01-30 8:52 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-01-30 8:53 [gcc(refs/users/marxin/heads/pgo-reproducibility-test)] ipa: fix handling of multiple speculations (PR93318) Martin Liska
-- strict thread matches above, loose matches on Subject: below --
2020-01-30 8:52 Martin Liska
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).