public inbox for gcc-cvs@sourceware.org
help / color / mirror / Atom feed
* [gcc(refs/users/aoliva/heads/testme)] hardcfr: add optional checkpoints
@ 2022-09-07 23:58 Alexandre Oliva
  0 siblings, 0 replies; 13+ messages in thread
From: Alexandre Oliva @ 2022-09-07 23:58 UTC (permalink / raw)
  To: gcc-cvs

https://gcc.gnu.org/g:0d4a420c2382596f9df22db84893a985e07226ac

commit 0d4a420c2382596f9df22db84893a985e07226ac
Author: Alexandre Oliva <oliva@adacore.com>
Date:   Fri Sep 2 20:33:12 2022 -0300

    hardcfr: add optional checkpoints
    
    Previously, control flow redundancy only checked the visited bitmap
    against the control flow graph at return points and before mandatory
    tail calls, missing various other possibilities of exiting a
    subprogram, such as by raising or propagating exceptions, and calling
    noreturn functions.  The checks inserted before returns also prevented
    potential tail-call optimizations.
    
    This incremental change introduces options to control checking at each
    of these previously-missed checkpoints.  Unless disabled, a cleanup is
    introduced to check when an exceptions escapes a subprogram.  To avoid
    disrupting sibcall optimizations, when they are enabled, checks are
    introduced before calls whose results are immediately returned,
    whether or not they are ultimately optimized.  If enabled, checks are
    introduced before noreturn calls and exception raises, or only before
    nothrow noreturn calls.
    
    Add examples of code transformations to the GNAT RM.
    
    
    for  gcc/ada/ChangeLog
    
            * doc/gnat_rm/security_hardening_features.rst: Document
            optional hardcfr checkpoints.
    
    for  gcc/ChangeLog
    
            * common.opt (-fhardcfr-check-returning-calls): New.
            (-fhardcfr-check-exceptions): New.
            (-fhardcfr-check-noreturn-calls=*): New.  (Enum
            hardcfr_check_noreturn_calls): New.  * doc/invoke.texi:
            Document them.  * flag-types.h (enum hardcfr_noret): New.  *
            gimple-harden-control-flow.cc: Include more headers.
            (pass_data_harden_control_flow_redundancy): Move
            properties_finish to pass return.
            (pass_harden_control_flow_redundancy::gate): Move
            no-edge-to-exit checking to execute, after exception
            transformations.  Use symbolic NUM_FIXED_BLOCKS.  Test for CFG
            before testing block count.  (check_returning_calls_p): New.
            (hardcfr_scan_block): New.  (returnign_call_p): New.
            (chk_edges_t): New.  (hardcfr_sibcall_search_block): New.
            (hardcfr_sibcall_search_preds): New.
            (rt_bb_visited::num2idx): Use symbolic NUM_FIXED_BLOCKS.
            (rt_bb_visited::rt_bb_visited): Likewise.  Take checkpoints
            count, test it to select out-of-line checks.
            (rt_bb_visited::insert_exit_check): Removed.
            (rt_bb_visited::insert_exit_check_in_blocks): New.
            (rt_bb_visited::insert_exit_check_on_edge): New.
            (rt_bb_visited::check): Take checkpoint edges and blocks.
            Move initialization insertion to the end.  Insert checks on
            edges, and before calls in select blocks.  Clobber the visited
            array to the check sequence.  Output actions to dump_file.
            (rt_bb_visited::visit): Take checkpoint and postcheck.  Assume
            an edge to exit if checkpoint, and skip testing if postcheck.
            (always_throwing_noreturn_call_p): New, dummy.
            (pass_harden_control_flow_redundancy::execute): Introduce
            cleanup block if requested and needed.  Scan blocks for
            noreturn and returning calls if checkpoints before them were
            requested.  Log actions to dump_file.  Visit blocks in index
            order.
    
    for  libgcc/ChangeLog
    
            * hardcfr.c (block2mask): Split out of...
            (visited_p): ... this.
            (check_seq): Move trapping out, return result instead.
            (__hardcfr_debug_cfg): New.
            (__hardcfr_check_fail): New, with optional verbose error
            printing before trapping.
            (__hardcfr_check): Call __hardcfr_check_fail when check_seq
            fails.
    
    for  gcc/testsuite/ChangeLog
    
            * c-c++-common/harden-cfr-noret-never-O0.c: New.
            * c-c++-common/torture/harden-cfr-noret-never.c: New.
            * c-c++-common/torture/harden-cfr-noret-noexcept.c: New.
            * c-c++-common/torture/harden-cfr-noret-nothrow.c: New.
            * c-c++-common/torture/harden-cfr-noret.c: New.
            * c-c++-common/torture/harden-cfr-notail.c: New.
            * c-c++-common/torture/harden-cfr-tail.c: Extend.
            * g++.dg/harden-cfr-throw-always-O0.C: New.
            * g++.dg/harden-cfr-throw-returning-O0.C: New.
            * g++.dg/torture/harden-cfr-noret-always-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-noret-never-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-noret-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-throw-always.C: New.
            * g++.dg/torture/harden-cfr-throw-nocleanup.C: New.
            * g++.dg/torture/harden-cfr-throw-returning.C: New.
            * g++.dg/torture/harden-cfr-throw.C: New.
            * gcc.dg/torture/harden-cfr-noret-no-nothrow.c: New.
            * gcc.dg/torture/harden-cfr-tail-ub.c: New.
            * gnat.dg/hardcfr.adb: Disable checking at exceptions.

Diff:
---
 .../doc/gnat_rm/security_hardening_features.rst    | 126 ++-
 gcc/common.opt                                     |  33 +
 gcc/doc/invoke.texi                                |  74 +-
 gcc/flag-types.h                                   |  10 +
 gcc/gimple-harden-control-flow.cc                  | 916 ++++++++++++++++++---
 .../c-c++-common/harden-cfr-noret-never-O0.c       |  12 +
 .../c-c++-common/torture/harden-cfr-noret-never.c  |  18 +
 .../torture/harden-cfr-noret-noexcept.c            |  16 +
 .../torture/harden-cfr-noret-nothrow.c             |  13 +
 .../c-c++-common/torture/harden-cfr-noret.c        |  38 +
 .../c-c++-common/torture/harden-cfr-notail.c       |   8 +
 .../c-c++-common/torture/harden-cfr-returning.c    |  35 +
 .../c-c++-common/torture/harden-cfr-tail.c         |  55 +-
 gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C  |  11 +
 .../g++.dg/harden-cfr-throw-returning-O0.C         |  10 +
 .../torture/harden-cfr-noret-always-no-nothrow.C   |  16 +
 .../torture/harden-cfr-noret-never-no-nothrow.C    |  18 +
 .../g++.dg/torture/harden-cfr-noret-no-nothrow.C   |  23 +
 .../g++.dg/torture/harden-cfr-throw-always.C       |  20 +
 .../g++.dg/torture/harden-cfr-throw-nocleanup.C    |  11 +
 .../g++.dg/torture/harden-cfr-throw-returning.C    |  31 +
 gcc/testsuite/g++.dg/torture/harden-cfr-throw.C    |  65 ++
 .../gcc.dg/torture/harden-cfr-noret-no-nothrow.c   |  15 +
 gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c  |  40 +
 gcc/testsuite/gnat.dg/hardcfr.adb                  |   2 +-
 libgcc/hardcfr.c                                   | 119 ++-
 26 files changed, 1598 insertions(+), 137 deletions(-)

diff --git a/gcc/ada/doc/gnat_rm/security_hardening_features.rst b/gcc/ada/doc/gnat_rm/security_hardening_features.rst
index f5fdc8e46b4..4dfda486795 100644
--- a/gcc/ada/doc/gnat_rm/security_hardening_features.rst
+++ b/gcc/ada/doc/gnat_rm/security_hardening_features.rst
@@ -263,11 +263,127 @@ For each block that is marked as visited, the mechanism checks that at
 least one of its predecessors, and at least one of its successors, are
 also marked as visited.
 
-Verification is performed just before returning.  Subprogram
-executions that complete by raising or propagating an exception bypass
-verification-and-return points.  A subprogram that can only complete
-by raising or propagating an exception may have instrumentation
-disabled altogether.
+Verification is performed just before a subprogram returns.  The
+following fragment:
+
+.. code-block:: ada
+
+   if X then
+     Y := F (Z);
+     return;
+   end if;
+
+
+gets turned into:
+
+.. code-block:: ada
+
+   type Visited_Bitmap is array (1..N) of Boolean with Pack;
+   Visited : aliased Visited_Bitmap := (others => False);
+   --  Bitmap of visited blocks.  N is the basic block count.
+   [...]
+   --  Basic block #I
+   Visited(I) := True;
+   if X then
+     --  Basic block #J
+     Visited(J) := True;
+     Y := F (Z);
+     CFR.Check (N, Visited'Access, CFG'Access);
+     --  CFR is a hypothetical package whose Check procedure calls
+     --  libgcc's __hardcfr_check, that traps if the Visited bitmap
+     --  does not hold a valid path in CFG, the run-time
+     --  representation of the control flow graph in the enclosing
+     --  subprogram.
+     return;
+   end if;
+   --  Basic block #K
+   Visited(K) := True;
+
+
+Verification would also be performed before must-tail calls, and
+before early-marked potential tail calls, but these do not occur in
+practice, as potential tail-calls are only detected in late
+optimization phases, too late for this transformation to act on it.
+
+In order to avoid adding verification after potential tail calls,
+which would prevent tail-call optimization, we recognize returning
+calls, i.e., calls whose result, if any, is returned by the calling
+subprogram to its caller immediately after the call returns.
+Verification is performed before such calls, whether or not they are
+ultimately optimized to tail calls.  This behavior is enabled by
+default whenever sibcall optimization is enabled (see
+:switch:`-foptimize-sibling-calls`); it may be disabled with
+:switch:`-fno-hardcfr-check-returning-calls`, or enabled with
+:switch:`-fhardcfr-check-returning-calls`, regardless of the
+optimization, but the lack of other optimizations may prevent calls
+from being recognized as returning calls:
+
+.. code-block:: ada
+
+     --  CFR.Check here, with -fhardcfr-check-returning-calls.
+     P (X);
+     --  CFR.Check here, with -fno-hardcfr-check-returning-calls.
+     return;
+
+or:
+
+.. code-block:: ada
+
+     --  CFR.Check here, with -fhardcfr-check-returning-calls.
+     R := F (X);
+     --  CFR.Check here, with -fno-hardcfr-check-returning-calls.
+     return R;
+
+
+Any subprogram from which an exception may escape, i.e., that may
+raise or propagate an exception that isn't handled internally, is
+conceptually enclosed by a cleanup handler that performs verification,
+unless this is disabled with :switch:`-fno-hardcfr-check-exceptions`.
+With this feature enabled, a subprogram body containing:
+
+.. code-block:: ada
+
+     --  ...
+       Y := F (X);  -- May raise exceptions.
+     --  ...
+       raise E;  -- Not handled internally.
+     --  ...
+
+
+gets modified as follows:
+
+.. code-block:: ada
+
+   begin
+     --  ...
+       Y := F (X);  -- May raise exceptions.
+     --  ...
+       raise E;  -- Not handled internally.
+     --  ...
+   exception
+     when others =>
+       CFR.Check (N, Visited'Access, CFG'Access);
+       raise;
+   end;
+
+
+Verification may also be performed before No_Return calls, whether
+only nothrow ones, with
+:switch:`-fhardcfr-check-noreturn-calls=nothrow`, or all of them, with
+:switch:`-fhardcfr-check-noreturn-calls=always`.  The default is
+:switch:`-fhardcfr-check-noreturn-calls=never` for this feature, that
+disables checking before No_Return calls.
+
+When a No_Return call returns control to its caller through an
+exception, verification may have already been performed before the
+call, if :switch:`-fhardcfr-check-noreturn-calls=always` is in effect.
+The compiler arranges for already-checked No_Return calls without a
+preexisting handler to bypass the implicitly-added cleanup handler and
+thus the redundant check, but a local exception or cleanup handler, if
+present, will modify the set of visited blocks, and checking will take
+place again when the caller reaches the next verification point,
+whether it is a return or reraise statement after the exception is
+otherwise handled, or even another No_Return call.
 
 The instrumentation for hardening with control flow redundancy can be
 observed in dump files generated by the command-line option
diff --git a/gcc/common.opt b/gcc/common.opt
index 661dcf2f485..983cb4db7c6 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -1795,6 +1795,39 @@ fharden-control-flow-redundancy
 Common Var(flag_harden_control_flow_redundancy) Optimization
 Harden control flow by recording and checking execution paths.
 
+fhardcfr-check-returning-calls
+Common Var(flag_harden_control_flow_redundancy_check_returning_calls) Init(-1) Optimization
+Check CFR execution paths also before calls followed by returns of their results.
+
+fhardcfr-check-exceptions
+Common Var(flag_harden_control_flow_redundancy_check_exceptions) Init(-1) Optimization
+Check CFR execution paths also when exiting a function through an exception.
+
+fhardcfr-check-noreturn-calls=
+Common Joined RejectNegative Enum(hardcfr_check_noreturn_calls) Var(flag_harden_control_flow_redundancy_check_noreturn) Init(HCFRNR_UNSPECIFIED) Optimization
+-fhardcfr-check-noreturn-calls=[always|nothrow|never]	Check CFR execution paths also before calling noreturn functions.
+
+Enum
+Name(hardcfr_check_noreturn_calls) Type(enum hardcfr_noret) UnknownError(unknown hardcfr noreturn checking level %qs)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(never) Value(HCFRNR_NEVER)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(nothrow) Value(HCFRNR_NOTHROW)
+
+; ??? There could be yet another option here, that checked before
+; noreturn calls, except for those known to always throw, if we had
+; means to distinguish noreturn functions known to always throw, such
+; as those used to (re)raise exceptions, from those that merely might
+; throw.  "not always" stands for "not always-throwing", but it also
+; contrasts with "always" below.
+; EnumValue
+; Enum(hardcfr_check_noreturn_calls) String(not-always) Value(HCFRNR_NOT_ALWAYS)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(always) Value(HCFRNR_ALWAYS)
+
 ; Nonzero means ignore `#ident' directives.  0 means handle them.
 ; Generate position-independent code for executables if possible
 ; On SVR4 targets, it also controls whether or not to emit a
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index 5c91a2e4fdb..f7ecc90e587 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -625,7 +625,9 @@ Objective-C and Objective-C++ Dialects}.
 -fsanitize-undefined-trap-on-error  -fbounds-check @gol
 -fcf-protection=@r{[}full@r{|}branch@r{|}return@r{|}none@r{|}check@r{]} @gol
 -fharden-compares -fharden-conditional-branches @gol
--fharden-control-flow-redundancy @gol
+-fharden-control-flow-redundancy  -fhardcfr-check-exceptions  @gol
+-fhardcfr-check-returning-calls  @gol
+-fhardcfr-check-noreturn-calls=@r{[}always@r{|}nothrow@r{|}never@r{]}  @gol
 -fstack-protector  -fstack-protector-all  -fstack-protector-strong @gol
 -fstack-protector-explicit  -fstack-check @gol
 -fstack-limit-register=@var{reg}  -fstack-limit-symbol=@var{sym} @gol
@@ -16572,11 +16574,75 @@ conditionals.
 @item -fharden-control-flow-redundancy
 @opindex fharden-control-flow-redundancy
 Emit extra code to set booleans when entering basic blocks, and to
-verify, at function exits, that they amount to an execution path that is
-consistent with the control flow graph, trapping otherwise.  Tuning
-options @option{--param hardcfr-max-blocks} and @option{--param
+verify and trap, at function exits, when the booleans do not form an
+execution path that is compatible with the control flow graph.
+
+Verification takes place before returns, before mandatory tail calls
+(see below) and, optionally, before escaping exceptions with
+@option{-fhardcfr-check-exceptions}, before returning calls with
+@option{-fhardcfr-check-returning-calls}, and before noreturn calls with
+@option{-fhardcfr-check-noreturn-calls}).  Tuning options
+@option{--param hardcfr-max-blocks} and @option{--param
 hardcfr-max-inline-blocks} are available.
 
+Tail call optimization takes place too late to affect control flow
+redundancy, but calls annotated as mandatory tail calls by language
+front-ends, and any calls marked early enough as potential tail calls
+would also have verification issued before the call, but these
+possibilities are merely theoretical, as these conditions can only be
+met when using custom compiler plugins.
+
+@item -fhardcfr-check-exceptions
+@opindex fhardcfr-check-exceptions
+@opindex fno-hardcfr-check-exceptions
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph at exception
+escape points, as if the function body was wrapped with a cleanup
+handler that performed the check and reraised.  This option is enabled
+by default; use @option{-fno-hardcfr-check-exceptions} to disable it.
+
+@item -fhardcfr-check-returning-calls
+@opindex fhardcfr-check-returning-calls
+@opindex fno-hardcfr-check-returning-calls
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph before any
+function call immediately followed by a return of its result, if any, so
+as to not prevent tail-call optimization, whether or not it is
+ultimately optimized to a tail call.
+
+This option is enabled by default whenever sibling call optimizations
+are enabled (see @option{-foptimize-sibling-calls}), but it can be
+enabled (or disabled, using its negated form) explicitly, regardless of
+the optimizations.
+
+@item -fhardcfr-check-noreturn-calls=@r{[}always@r{|}nothrow@r{|}never@r{]}
+@opindex fhardcfr-check-noreturn-calls
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph before
+@code{noreturn} calls, either all of them (@option{always}), those that
+may not return control to the caller through an exception either
+(@option{nothrow}), or none of them (@option{never}, the default).
+
+Checking before a @code{noreturn} function that may return control to
+the caller through an exception may cause checking to be performed more
+than once, if the exception is caught in the caller, whether by a
+handler or a cleanup.  When @option{-fhardcfr-check-exceptions} is also
+enabled, the compiler will avoid associating a @code{noreturn} call with
+the implicitly-added cleanup handler, since it would be redundant with
+the check performed before the call, but other handlers or cleanups in
+the function, if activated, will modify the recorded execution path and
+check it again when another checkpoint is hit.  The checkpoint may even
+be another @code{noreturn} call, so checking may end up performed
+multiple times.
+
+Various optimizers may cause calls to be marked as @code{noreturn}
+and/or @code{nothrow}, even in the absence of the corresponding
+attributes, which may affect the placement of checks before calls, as
+well as the addition of implicit cleanup handlers for them.  This
+unpredictability, and the fact that raising and reraising exceptions
+frequently amounts to implicitly calling @code{noreturn} functions, have
+made @option{never} the default setting for this option.
+
 @item -fstack-protector
 @opindex fstack-protector
 Emit extra code to check for buffer overflows, such as stack smashing
diff --git a/gcc/flag-types.h b/gcc/flag-types.h
index d2e751060ff..3fae7548cab 100644
--- a/gcc/flag-types.h
+++ b/gcc/flag-types.h
@@ -157,6 +157,16 @@ enum stack_reuse_level
   SR_ALL
 };
 
+/* Control Flow Redundancy hardening options for noreturn calls.  */
+enum hardcfr_noret
+{
+  HCFRNR_NEVER,
+  HCFRNR_NOTHROW,
+  HCFRNR_NOT_ALWAYS, /* Reserved for future use.  */
+  HCFRNR_ALWAYS,
+  HCFRNR_UNSPECIFIED = -1
+};
+
 /* The live patching level.  */
 enum live_patching_level
 {
diff --git a/gcc/gimple-harden-control-flow.cc b/gcc/gimple-harden-control-flow.cc
index 6b08846dbb1..1c93bf622e8 100644
--- a/gcc/gimple-harden-control-flow.cc
+++ b/gcc/gimple-harden-control-flow.cc
@@ -29,7 +29,12 @@ along with GCC; see the file COPYING3.  If not see
 #include "tree-pass.h"
 #include "ssa.h"
 #include "gimple-iterator.h"
+#include "gimple-pretty-print.h"
 #include "tree-cfg.h"
+#include "tree-cfgcleanup.h"
+#include "tree-eh.h"
+#include "except.h"
+#include "sbitmap.h"
 #include "basic-block.h"
 #include "cfghooks.h"
 #include "cfgloop.h"
@@ -60,9 +65,7 @@ const pass_data pass_data_harden_control_flow_redundancy = {
   0,	    // properties_provided
   0,	    // properties_destroyed
   TODO_cleanup_cfg, // properties_start
-  TODO_update_ssa
-  | TODO_cleanup_cfg
-  | TODO_verify_il, // properties_finish
+  0,        // properties_finish
 };
 
 class pass_harden_control_flow_redundancy : public gimple_opt_pass
@@ -79,16 +82,6 @@ public:
     if (!flag_harden_control_flow_redundancy)
       return false;
 
-    /* We don't verify when an exception escapes, propagated or raised
-       by the function itself, so we're only concerned with edges to
-       the exit block.  If there aren't any, the function doesn't
-       return normally, so there won't be any checking point, so
-       there's no point in running the pass.  Should we add
-       verification at exception escapes, we should at least look at
-       !flag_exceptions here.  */
-    if (EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (fun)->preds) == 0)
-      return false;
-
     /* Functions that return more than once, like setjmp and vfork
        (that also gets this flag set), will start recording a path
        after the first return, and then may take another path when
@@ -117,8 +110,9 @@ public:
 	return false;
       }
 
-    if (param_hardcfr_max_blocks > 0
-	&& n_basic_blocks_for_fn (fun) - 2 > param_hardcfr_max_blocks)
+    if (fun->cfg && param_hardcfr_max_blocks > 0
+	&& (n_basic_blocks_for_fn (fun) - NUM_FIXED_BLOCKS
+	    > param_hardcfr_max_blocks))
       {
 	warning_at (DECL_SOURCE_LOCATION (fun->decl), 0,
 		    "%qD has more than %u blocks, the requested"
@@ -134,6 +128,293 @@ public:
 
 }
 
+/* Return TRUE iff CFR checks should be inserted before returning
+   calls.  */
+
+static bool
+check_returning_calls_p ()
+{
+  return
+    flag_harden_control_flow_redundancy_check_returning_calls > 0
+    || (flag_harden_control_flow_redundancy_check_returning_calls < 0
+	/* Gates pass_tail_calls.  */
+	&& flag_optimize_sibling_calls
+	/* Gates pass_all_ptimizations.  */
+	&& optimize >= 1 && !optimize_debug);
+}
+
+/* Scan BB from the end, updating *RETPTR if given as return stmts and
+   copies are found.  Return a call or a stmt that cannot appear after
+   a tail call, or NULL if the top of the block is reached without
+   finding any.  */
+
+static gimple *
+hardcfr_scan_block (basic_block bb, tree **retptr)
+{
+  gimple_stmt_iterator gsi;
+  for (gsi = gsi_last_bb (bb); !gsi_end_p (gsi); gsi_prev (&gsi))
+    {
+      gimple *stmt = gsi_stmt (gsi);
+
+      /* Ignore labels, returns, nops, clobbers and debug stmts.  */
+      if (gimple_code (stmt) == GIMPLE_LABEL
+	  || gimple_code (stmt) == GIMPLE_NOP
+	  || gimple_code (stmt) == GIMPLE_PREDICT
+	  || gimple_clobber_p (stmt)
+	  || is_gimple_debug (stmt))
+	continue;
+
+      if (gimple_code (stmt) == GIMPLE_RETURN)
+	{
+	  greturn *gret = as_a <greturn *> (stmt);
+	  if (retptr)
+	    {
+	      gcc_checking_assert (!*retptr);
+	      *retptr = gimple_return_retval_ptr (gret);
+	    }
+	  continue;
+	}
+
+      /* Check for a call.  */
+      if (is_gimple_call (stmt))
+	return stmt;
+
+      /* Allow simple copies to the return value, updating the return
+	 value to be found in earlier assignments.  */
+      if (retptr && *retptr && gimple_assign_single_p (stmt)
+	  && **retptr == gimple_assign_lhs (stmt))
+	{
+	  *retptr = gimple_assign_rhs1_ptr (stmt);
+	  continue;
+	}
+
+      return stmt;
+    }
+
+  /* Any other kind of stmt will prevent a tail call.  */
+  return NULL;
+}
+
+/* Return TRUE iff CALL is to be preceded by a CFR checkpoint, i.e.,
+   if it's a returning call (one whose result is ultimately returned
+   without intervening non-copy statements) and we're checking
+   returning calls, a __builtin_return call (noreturn with a path to
+   the exit block), a must-tail call, or a tail call.  */
+
+static bool
+returning_call_p (gcall *call)
+{
+  if (!(gimple_call_noreturn_p (call)
+	|| gimple_call_must_tail_p (call)
+	|| gimple_call_tail_p (call)
+	|| check_returning_calls_p ()))
+    return false;
+
+  /* Quickly check that there's a path to exit compatible with a
+     returning call.  Detect infinite loops through the counter.  */
+  basic_block bb = gimple_bb (call);
+  auto_vec<basic_block, 10> path;
+  for (int i = n_basic_blocks_for_fn (cfun);
+       bb != EXIT_BLOCK_PTR_FOR_FN (cfun) && i--;
+       bb = single_succ (bb))
+    if (!single_succ_p (bb)
+	|| (single_succ_edge (bb)->flags & EDGE_EH) != 0)
+      return false;
+    else
+      path.safe_push (bb);
+
+  /* Check the stmts in the blocks and trace the return value.  */
+  tree *retptr = NULL;
+  for (;;)
+    {
+      gcc_checking_assert (!path.is_empty ());
+      basic_block bb = path.pop ();
+      gimple *stop = hardcfr_scan_block (bb, &retptr);
+      if (stop)
+	{
+	  if (stop != call)
+	    return false;
+	  gcc_checking_assert (path.is_empty ());
+	  break;
+	}
+
+      gphi *retphi = NULL;
+      if (retptr && *retptr && TREE_CODE (*retptr) == SSA_NAME
+	  && !SSA_NAME_IS_DEFAULT_DEF (*retptr)
+	  && SSA_NAME_DEF_STMT (*retptr)
+	  && is_a <gphi *> (SSA_NAME_DEF_STMT (*retptr))
+	  && gimple_bb (SSA_NAME_DEF_STMT (*retptr)) == bb)
+	{
+	  retphi = as_a <gphi *> (SSA_NAME_DEF_STMT (*retptr));
+	  gcc_checking_assert (gimple_phi_result (retphi) == *retptr);
+	}
+      else
+	continue;
+
+      gcc_checking_assert (!path.is_empty ());
+      edge e = single_succ_edge (path.last ());
+      int i = EDGE_COUNT (bb->preds);
+      while (i--)
+	if (EDGE_PRED (bb, i) == e)
+	  break;
+      gcc_checking_assert (i >= 0);
+      retptr = gimple_phi_arg_def_ptr (retphi, i);
+    }
+
+  return (gimple_call_noreturn_p (call)
+	  || gimple_call_must_tail_p (call)
+	  || gimple_call_tail_p (call)
+	  || (gimple_call_lhs (call) == (retptr ? *retptr : NULL)
+	      && check_returning_calls_p ()));
+}
+
+typedef auto_vec<edge, 10> chk_edges_t;
+
+/* Declare for mutual recursion.  */
+static bool hardcfr_sibcall_search_preds (basic_block bb,
+					  chk_edges_t &chk_edges,
+					  int &count_chkcall,
+					  auto_sbitmap &chkcall_blocks,
+					  int &count_postchk,
+					  auto_sbitmap &postchk_blocks,
+					  tree *retptr);
+
+/* Search backwards from the end of BB for a mandatory or potential
+   sibcall.  Schedule the block to be handled sort-of like noreturn if
+   so.  Recurse to preds, with updated RETPTR, if the block only
+   contains stmts that may follow such a call, scheduling checking at
+   edges and marking blocks as post-check as needed.  Return true iff,
+   at the end of the block, a check will have already been
+   performed.  */
+
+static bool
+hardcfr_sibcall_search_block (basic_block bb,
+			      chk_edges_t &chk_edges,
+			      int &count_chkcall,
+			      auto_sbitmap &chkcall_blocks,
+			      int &count_postchk,
+			      auto_sbitmap &postchk_blocks,
+			      tree *retptr)
+{
+  /* Conditionals and internal exceptions rule out tail calls.  */
+  if (!single_succ_p (bb)
+      || (single_succ_edge (bb)->flags & EDGE_EH) != 0)
+    return false;
+
+  gimple *stmt = hardcfr_scan_block (bb, &retptr);
+  if (!stmt)
+    return hardcfr_sibcall_search_preds (bb, chk_edges,
+					 count_chkcall, chkcall_blocks,
+					 count_postchk, postchk_blocks,
+					 retptr);
+
+  if (!is_a <gcall *> (stmt))
+    return false;
+
+  /* Avoid disrupting mandatory or early-marked tail calls,
+     inserting the check before them.  This works for
+     must-tail calls, but tail calling as an optimization is
+     detected too late for us.
+
+     Also check for noreturn calls here.  Noreturn calls won't
+     normally have edges to exit, so they won't be found here,
+     but __builtin_return does, and we must check before
+     it, so handle it like a tail call.  */
+  gcall *call = as_a <gcall *> (stmt);
+  if (!(gimple_call_noreturn_p (call)
+	|| gimple_call_must_tail_p (call)
+	|| gimple_call_tail_p (call)
+	|| (gimple_call_lhs (call) == (retptr ? *retptr : NULL)
+	    && check_returning_calls_p ())))
+    return false;
+
+  gcc_checking_assert (returning_call_p (call));
+
+  /* We found a call that is to be preceded by checking.  */
+  if (bitmap_set_bit (chkcall_blocks, bb->index))
+    ++count_chkcall;
+  else
+    gcc_unreachable ();
+  return true;
+}
+
+
+/* Search preds of BB for a mandatory or potential sibcall or
+   returning call, and arrange for the blocks containing them to have
+   a check inserted before the call, like noreturn calls.  If any
+   preds are found to perform checking, schedule checks at the edges
+   of those that don't, and mark BB as postcheck..  */
+
+static bool
+hardcfr_sibcall_search_preds (basic_block bb,
+			      chk_edges_t &chk_edges,
+			      int &count_chkcall,
+			      auto_sbitmap &chkcall_blocks,
+			      int &count_postchk,
+			      auto_sbitmap &postchk_blocks,
+			      tree *retptr)
+{
+  /* For the exit block, we wish to force a check at every
+     predecessor, so pretend we've already found a pred that had
+     checking, so that we schedule checking at every one of its pred
+     edges.  */
+  bool first = bb->index >= NUM_FIXED_BLOCKS;
+  bool postchecked = true;
+
+  gphi *retphi = NULL;
+  if (retptr && *retptr && TREE_CODE (*retptr) == SSA_NAME
+      && !SSA_NAME_IS_DEFAULT_DEF (*retptr)
+      && SSA_NAME_DEF_STMT (*retptr)
+      && is_a <gphi *> (SSA_NAME_DEF_STMT (*retptr))
+      && gimple_bb (SSA_NAME_DEF_STMT (*retptr)) == bb)
+    {
+      retphi = as_a <gphi *> (SSA_NAME_DEF_STMT (*retptr));
+      gcc_checking_assert (gimple_phi_result (retphi) == *retptr);
+    }
+
+  for (int i = EDGE_COUNT (bb->preds); i--; first = false)
+    {
+      edge e = EDGE_PRED (bb, i);
+
+      bool checked
+	= hardcfr_sibcall_search_block (e->src, chk_edges,
+					count_chkcall, chkcall_blocks,
+					count_postchk, postchk_blocks,
+					!retphi ? retptr
+					: gimple_phi_arg_def_ptr (retphi, i));
+
+      if (first)
+	{
+	  postchecked = checked;
+	  continue;
+	}
+
+      /* When we first find a checked block, force a check at every
+	 other incoming edge we've already visited, and those we
+	 visit afterwards that don't have their own check, so that
+	 when we reach BB, the check has already been performed.  */
+      if (!postchecked && checked)
+	{
+	  for (int j = EDGE_COUNT (bb->preds); --j > i; )
+	    chk_edges.safe_push (EDGE_PRED (bb, j));
+	  postchecked = true;
+	}
+      if (postchecked && !checked)
+	chk_edges.safe_push (EDGE_PRED (bb, i));
+    }
+
+  if (postchecked && bb->index >= NUM_FIXED_BLOCKS)
+    {
+      if (bitmap_set_bit (postchk_blocks, bb->index))
+	count_postchk++;
+      else
+	gcc_unreachable ();
+    }
+
+  return postchecked;
+}
+
+
 class rt_bb_visited
 {
   /* Use a sufficiently wide unsigned type to hold basic block numbers.  */
@@ -176,8 +457,8 @@ class rt_bb_visited
      neither ENTRY nor EXIT, but maybe one-past-the-end, to compute
      the visited array length.  */
   blknum num2idx (blknum n) {
-    gcc_checking_assert (n >= 2 && n <= nblocks);
-    return (n - 2);
+    gcc_checking_assert (n >= NUM_FIXED_BLOCKS && n <= nblocks);
+    return (n - NUM_FIXED_BLOCKS);
   }
   /* Return the block vindex for BB, that must not be ENTRY or
      EXIT.  */
@@ -249,8 +530,7 @@ class rt_bb_visited
   }
 
   /* Set the bit corresponding to BB in VISITED.  Add to SEQ any
-     required gimple statements, and return SEQ, possibly
-     modified.  */
+     required gimple stmts, and return SEQ, possibly modified.  */
   gimple_seq vset (basic_block bb, gimple_seq seq = NULL)
   {
     tree bit, setme = vword (bb, &bit);
@@ -270,7 +550,7 @@ class rt_bb_visited
 
 public:
   /* Prepare to add control flow redundancy testing to CFUN.  */
-  rt_bb_visited ()
+  rt_bb_visited (int checkpoints)
     : nblocks (n_basic_blocks_for_fn (cfun)),
       vword_type (NULL), ckseq (NULL), rtcfg (NULL)
   {
@@ -353,8 +633,8 @@ public:
 					 NULL, NULL);
     gimple_seq_add_stmt (&ckseq, detach);
 
-    if (nblocks - 2 > blknum (param_hardcfr_max_inline_blocks)
-	|| !single_pred_p (EXIT_BLOCK_PTR_FOR_FN (cfun)))
+    if (nblocks - NUM_FIXED_BLOCKS > blknum (param_hardcfr_max_inline_blocks)
+	|| checkpoints > 1)
       {
 	/* Make sure vword_bits is wide enough for the representation
 	   of nblocks in rtcfg.  Compare with vword_bits << vword_bits,
@@ -379,65 +659,33 @@ public:
     gimple_seq_add_stmt (&ckseq, ckfail_init);
   }
 
-  /* Insert SEQ on E, or close enough (e.g., before a noreturn or tail
-     call at the end of E->src).  */
-  void insert_exit_check (gimple_seq seq, edge e)
+  /* Insert SEQ before a resx or a call in INSBB.  */
+  void insert_exit_check_in_block (gimple_seq seq, basic_block insbb)
   {
-    basic_block insbb = e->src;
-
-    /* If the returning block ends with a noreturn call, insert
-       checking before it.  This is particularly important for
-       __builtin_return.  Other noreturn calls won't have an edge to
-       the exit block, so they won't even be considered as exit paths.
-
-       Insert-on-edge inserts before other return stmts, but not
-       before calls, and if a single-block function had the check
-       sequence inserted after a noreturn call, it would be useless,
-       but checking would still flag it as malformed if block 2 has a
-       fallthru edge to the exit block.
-
-       Also avoid disrupting tail calls, inserting the check before
-       them.  This works for must-tail calls, but tail calling as an
-       optimization is detected too late for us.  */
     gimple_stmt_iterator gsi = gsi_last_bb (insbb);
-    gimple *ret = gsi_stmt (gsi);
-    if (ret && is_a <greturn *> (ret))
-      {
+
+    while (!gsi_end_p (gsi))
+      if (is_a <gresx *> (gsi_stmt (gsi))
+	  || is_a <gcall *> (gsi_stmt (gsi)))
+	break;
+      else
 	gsi_prev (&gsi);
-	if (!gsi_end_p (gsi))
-	  ret = gsi_stmt (gsi);
-      }
-    if (ret && is_a <gcall *> (ret)
-	&& (gimple_call_noreturn_p (ret)
-	    || gimple_call_must_tail_p (as_a <gcall *> (ret))
-	    || gimple_call_tail_p (as_a <gcall *> (ret))))
-      gsi_insert_seq_before (&gsi, seq, GSI_SAME_STMT);
-    else
-      gsi_insert_seq_on_edge_immediate (e, seq);
+
+    gsi_insert_seq_before (&gsi, seq, GSI_SAME_STMT);
+  }
+
+  /* Insert SEQ on E.  */
+  void insert_exit_check_on_edge (gimple_seq seq, edge e)
+  {
+    gsi_insert_seq_on_edge_immediate (e, seq);
   }
 
-  /* Add checking code on every exit edge, and initialization code on
+  /* Add checking code to CHK_EDGES, and initialization code on
      the entry edge.  Before this point, the CFG has been undisturbed,
      and all the needed data has been collected and safely stowed.  */
-  void check ()
+  void check (chk_edges_t &chk_edges,
+	      int count_chkcall, auto_sbitmap const &chkcall_blocks)
   {
-    /* Insert initializers for visited at the entry.  */
-    gimple_seq iseq = NULL;
-
-    gcall *vinit = gimple_build_call (builtin_decl_explicit
-				      (BUILT_IN_MEMSET), 3,
-				      build1 (ADDR_EXPR,
-					      build_pointer_type
-					      (TREE_TYPE (visited)),
-					      visited),
-				      integer_zero_node,
-				      TYPE_SIZE_UNIT (TREE_TYPE (visited)));
-    gimple_seq_add_stmt (&iseq, vinit);
-
-    gsi_insert_seq_on_edge_immediate (single_succ_edge
-				      (ENTRY_BLOCK_PTR_FOR_FN (cfun)),
-				      iseq);
-
     /* If we're using out-of-line checking, create and statically
        initialize the CFG checking representation, generate the
        checker call for the checking sequence, and insert it in all
@@ -498,28 +746,116 @@ public:
 						     rtcfg));
 	gimple_seq_add_stmt (&ckseq, call_chk);
 
+	gimple *clobber = gimple_build_assign (visited,
+					       build_clobber
+					       (TREE_TYPE (visited)));
+	gimple_seq_add_stmt (&ckseq, clobber);
+
 	/* If we have multiple exit edges, insert (copies of)
 	   ckseq in all of them.  */
-	for (int i = EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (cfun)->preds);
-	     i--; )
+	for (int i = chk_edges.length (); i--; )
 	  {
 	    gimple_seq seq = ckseq;
 	    /* Copy the sequence, unless we're dealing with the
 	       last edge (we're counting down to zero).  */
-	    if (i)
+	    if (i || count_chkcall)
+	      seq = gimple_seq_copy (seq);
+
+	    edge e = chk_edges[i];
+
+	    if (dump_file)
+	      {
+		if (e->dest == EXIT_BLOCK_PTR_FOR_FN (cfun))
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to exit.\n",
+			   e->src->index);
+		else
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to postcheck block %i.\n",
+			   e->src->index, e->dest->index);
+	      }
+
+	    insert_exit_check_on_edge (seq, e);
+
+	    gcc_checking_assert (!bitmap_bit_p (chkcall_blocks, e->src->index));
+	  }
+
+	sbitmap_iterator it;
+	unsigned i;
+	EXECUTE_IF_SET_IN_BITMAP (chkcall_blocks, 0, i, it)
+	  {
+	    basic_block bb = BASIC_BLOCK_FOR_FN (cfun, i);
+
+	    gimple_seq seq = ckseq;
+	    gcc_checking_assert (count_chkcall > 0);
+	    if (--count_chkcall)
 	      seq = gimple_seq_copy (seq);
 
-	    insert_exit_check (seq,
-			       EDGE_PRED (EXIT_BLOCK_PTR_FOR_FN (cfun), i));
+	    if (dump_file)
+	      fprintf (dump_file,
+		       "Inserting out-of-line check before stmt in block %i.\n",
+		       bb->index);
+
+	    insert_exit_check_in_block (seq, bb);
 	  }
+
+	gcc_checking_assert (count_chkcall == 0);
       }
     else
       {
 	/* Inline checking requires a single exit edge.  */
-	gimple *last = gsi_stmt (gsi_last (ckseq));
+	gimple *last = gimple_build_assign (visited,
+					       build_clobber
+					       (TREE_TYPE (visited)));
+	gimple_seq_add_stmt (&ckseq, last);
 
-	insert_exit_check (ckseq,
-			   single_pred_edge (EXIT_BLOCK_PTR_FOR_FN (cfun)));
+	if (!count_chkcall)
+	  {
+	    edge e = single_pred_edge (EXIT_BLOCK_PTR_FOR_FN (cfun));
+
+	    if (dump_file)
+	      {
+		if (e->dest == EXIT_BLOCK_PTR_FOR_FN (cfun))
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to postcheck block %i.\n",
+			   e->src->index, e->dest->index);
+		else
+		  fprintf (dump_file,
+			   "Inserting inline check in"
+			   " block %i's edge to exit.\n",
+			   e->src->index);
+	      }
+
+	    insert_exit_check_on_edge (ckseq, e);
+	  }
+	else
+	  {
+	    gcc_checking_assert (count_chkcall == 1);
+
+	    sbitmap_iterator it;
+	    unsigned i;
+	    EXECUTE_IF_SET_IN_BITMAP (chkcall_blocks, 0, i, it)
+	      {
+		basic_block bb = BASIC_BLOCK_FOR_FN (cfun, i);
+
+		gimple_seq seq = ckseq;
+		gcc_checking_assert (count_chkcall > 0);
+		if (--count_chkcall)
+		  seq = gimple_seq_copy (seq);
+
+		if (dump_file)
+		  fprintf (dump_file,
+			   "Inserting inline check before stmt in block %i.\n",
+			   bb->index);
+
+		insert_exit_check_in_block (seq, bb);
+	      }
+
+	    gcc_checking_assert (count_chkcall == 0);
+	  }
 
 	/* The inserted ckseq computes CKFAIL at LAST.  Now we have to
 	   conditionally trap on it.  */
@@ -540,8 +876,7 @@ public:
 	  add_bb_to_loop (trp, current_loops->tree_root);
 
 	/* Insert a conditional branch to the trap block.  If the
-	   conditional wouldn't be the last statement, split the
-	   block.  */
+	   conditional wouldn't be the last stmt, split the block.  */
 	gimple_stmt_iterator gsi = gsi_for_stmt (last);
 	if (!gsi_one_before_end_p (gsi))
 	  split_block (gsi_bb (gsi), gsi_stmt (gsi));
@@ -564,6 +899,24 @@ public:
 	if (dom_info_available_p (CDI_DOMINATORS))
 	  set_immediate_dominator (CDI_DOMINATORS, trp, gimple_bb (last));
       }
+
+    /* Insert initializers for visited at the entry.  Do this after
+       other insertions, to avoid messing with block numbers.  */
+    gimple_seq iseq = NULL;
+
+    gcall *vinit = gimple_build_call (builtin_decl_explicit
+				      (BUILT_IN_MEMSET), 3,
+				      build1 (ADDR_EXPR,
+					      build_pointer_type
+					      (TREE_TYPE (visited)),
+					      visited),
+				      integer_zero_node,
+				      TYPE_SIZE_UNIT (TREE_TYPE (visited)));
+    gimple_seq_add_stmt (&iseq, vinit);
+
+    gsi_insert_seq_on_edge_immediate (single_succ_edge
+				      (ENTRY_BLOCK_PTR_FOR_FN (cfun)),
+				      iseq);
   }
 
   /* Push onto RTCFG a (mask, index) pair to test for IBB when BB is
@@ -607,7 +960,7 @@ public:
     return false;
   }
 
-  /* Add to CKSEQ statements to clear CKPART if OBB is visited.  */
+  /* Add to CKSEQ stmts to clear CKPART if OBB is visited.  */
   void
   build_block_check (basic_block obb)
   {
@@ -632,34 +985,48 @@ public:
 
   /* Add to BB code to set its bit in VISITED, and add to RTCFG or
      CKSEQ the data or code needed to check BB's predecessors and
-     successors.  Do NOT change the CFG.  */
-  void visit (basic_block bb)
+     successors.  If CHECKPOINT, assume the block is a checkpoint,
+     whether or not it has an edge to EXIT.  If POSTCHECK, assume the
+     block post-dominates checkpoints and therefore no bitmap setting
+     or checks are to be performed in or for it.  Do NOT change the
+     CFG.  */
+  void visit (basic_block bb, bool checkpoint, bool postcheck)
   {
     /* Set the bit in VISITED when entering the block.  */
     gimple_stmt_iterator gsi = gsi_after_labels (bb);
-    gsi_insert_seq_before (&gsi, vset (bb), GSI_SAME_STMT);
+    if (!postcheck)
+      gsi_insert_seq_before (&gsi, vset (bb), GSI_SAME_STMT);
 
     if (rtcfg)
       {
-	/* Build a list of (index, mask) terminated by (NULL, 0).
-	   Consolidate masks with the same index when they're
-	   adjacent.  First, predecessors.  Count backwards, because
-	   we're going to reverse the list.  The order shouldn't
-	   matter, but let's not make it surprising.  */
-	for (int i = EDGE_COUNT (bb->preds); i--; )
-	  if (push_rtcfg_pair (EDGE_PRED (bb, i)->src, bb,
-			       ENTRY_BLOCK_PTR_FOR_FN (cfun)))
-	    break;
+	if (!postcheck)
+	  {
+	    /* Build a list of (index, mask) terminated by (NULL, 0).
+	       Consolidate masks with the same index when they're
+	       adjacent.  First, predecessors.  Count backwards, because
+	       we're going to reverse the list.  The order shouldn't
+	       matter, but let's not make it surprising.  */
+	    for (int i = EDGE_COUNT (bb->preds); i--; )
+	      if (push_rtcfg_pair (EDGE_PRED (bb, i)->src, bb,
+				   ENTRY_BLOCK_PTR_FOR_FN (cfun)))
+		break;
+	  }
 	rtcfg = tree_cons (NULL_TREE, build_int_cst (vword_type, 0), rtcfg);
 
-	/* Then, successors.  */
-	for (int i = EDGE_COUNT (bb->succs); i--; )
-	  if (push_rtcfg_pair (EDGE_SUCC (bb, i)->dest, bb,
-			       EXIT_BLOCK_PTR_FOR_FN (cfun)))
-	    break;
+	if (!postcheck)
+	  {
+	    /* Then, successors.  */
+	    if (!checkpoint
+		|| !push_rtcfg_pair (EXIT_BLOCK_PTR_FOR_FN (cfun),
+				     bb, EXIT_BLOCK_PTR_FOR_FN (cfun)))
+	      for (int i = EDGE_COUNT (bb->succs); i--; )
+		if (push_rtcfg_pair (EDGE_SUCC (bb, i)->dest, bb,
+				     EXIT_BLOCK_PTR_FOR_FN (cfun)))
+		  break;
+	  }
 	rtcfg = tree_cons (NULL_TREE, build_int_cst (vword_type, 0), rtcfg);
       }
-    else
+    else if (!postcheck)
       {
 	/* Schedule test to fail if the block was reached but somehow none
 	   of its predecessors were.  */
@@ -677,6 +1044,8 @@ public:
 	gassign *blkruns = gimple_build_assign (ckpart, unshare_expr (bit));
 	gimple_seq_add_stmt (&ckseq, blkruns);
 
+	if (checkpoint)
+	  build_block_check (EXIT_BLOCK_PTR_FOR_FN (cfun));
 	for (int i = 0, e = EDGE_COUNT (bb->succs); i < e; i++)
 	  build_block_check (EDGE_SUCC (bb, i)->dest);
 
@@ -687,21 +1056,350 @@ public:
   }
 };
 
+/* It might be useful to avoid checking before noreturn calls that are
+   known to always finish by throwing an exception, rather than by
+   ending the program or looping forever.  Such functions would have
+   to be annotated somehow, with an attribute or flag.
+   Exception-raising functions, such as C++'s __cxa_throw,
+   __cxa_rethrow, and Ada's */
+static bool
+always_throwing_noreturn_call_p (gimple *)
+{
+  return false;
+}
+
 /* Control flow redundancy hardening: record the execution path, and
    verify at exit that an expect path was taken.  */
 
 unsigned int
-pass_harden_control_flow_redundancy::execute (function *)
+pass_harden_control_flow_redundancy::execute (function *fun)
 {
-  rt_bb_visited vstd;
-
+  bool const check_at_escaping_exceptions
+    = (flag_exceptions
+       && flag_harden_control_flow_redundancy_check_exceptions);
+  bool const check_before_noreturn_calls
+    = flag_harden_control_flow_redundancy_check_noreturn > HCFRNR_NEVER;
+  bool const check_before_nothrow_noreturn_calls
+    = (check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn >= HCFRNR_NOTHROW);
+  bool const check_before_throwing_noreturn_calls
+    = (flag_exceptions
+       && check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn > HCFRNR_NOTHROW);
+  bool const check_before_always_throwing_noreturn_calls
+    = (flag_exceptions
+       && check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn >= HCFRNR_ALWAYS);
   basic_block bb;
-  FOR_EACH_BB_FN (bb, cfun)
-    vstd.visit (bb);
+  basic_block bb_eh_cleanup = NULL;
+
+  if (check_at_escaping_exceptions)
+    {
+      int lp_eh_cleanup = -1;
+
+      /* Record the preexisting blocks, to avoid visiting newly-created
+	 blocks.  */
+      auto_sbitmap to_visit (last_basic_block_for_fn (fun));
+      bitmap_clear (to_visit);
+
+      FOR_EACH_BB_FN (bb, fun)
+	bitmap_set_bit (to_visit, bb->index);
+
+      /* Scan the blocks for stmts with escaping exceptions, that
+	 wouldn't be denoted in the CFG, and associate them with an
+	 empty cleanup handler around the whole function.  Walk
+	 backwards, so that even when we split the block, */
+      sbitmap_iterator it;
+      unsigned i;
+      EXECUTE_IF_SET_IN_BITMAP (to_visit, 0, i, it)
+	{
+	  bb = BASIC_BLOCK_FOR_FN (fun, i);
+
+	  for (gimple_stmt_iterator gsi = gsi_last_bb (bb);
+	       !gsi_end_p (gsi); gsi_prev (&gsi))
+	    {
+	      gimple *stmt = gsi_stmt (gsi);
+	      if (!stmt_could_throw_p (fun, stmt))
+		continue;
+
+	      /* If it must not throw, or if it already has a handler,
+		 we need not worry about it.  */
+	      if (lookup_stmt_eh_lp (stmt) != 0)
+		continue;
+
+	      /* Don't split blocks at, nor add EH edges to, tail
+		 calls, we will add verification before the call
+		 anyway.  */
+	      if (is_a <gcall *> (stmt)
+		  && (gimple_call_must_tail_p (as_a <gcall *> (stmt))
+		      || gimple_call_tail_p (as_a <gcall *> (stmt))
+		      || returning_call_p (as_a <gcall *> (stmt))))
+		continue;
+
+	      if (!gsi_one_before_end_p (gsi))
+		split_block (bb, stmt);
+	      /* A resx or noreturn call needs not be associated with
+		 the cleanup handler if we're going to add checking
+		 before it.  We only test cases that didn't require
+		 block splitting because noreturn calls would always
+		 be at the end of blocks, and we test for zero
+		 successors because if there is an edge, it's not
+		 noreturn, as any EH edges would have already been
+		 caught by the lookup_stmt_eh_lp test above.  */
+	      else if (check_before_noreturn_calls
+		       && EDGE_COUNT (bb->succs) == 0
+		       && (is_a <gresx *> (stmt)
+			   ? check_before_always_throwing_noreturn_calls
+			   : (!is_a <gcall *> (stmt)
+			      || !gimple_call_noreturn_p (stmt))
+			   ? (gcc_unreachable (), false)
+			   : (!flag_exceptions
+			      || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+			   ? check_before_nothrow_noreturn_calls
+			   : always_throwing_noreturn_call_p (stmt)
+			   ? check_before_always_throwing_noreturn_calls
+			   : check_before_throwing_noreturn_calls))
+		{
+		  if (dump_file)
+		    {
+		      fprintf (dump_file,
+			       "Bypassing cleanup for noreturn stmt"
+			       " in block %i:\n",
+			       bb->index);
+		      print_gimple_stmt (dump_file, stmt, 0);
+		    }
+		  continue;
+		}
+
+	      if (!bb_eh_cleanup)
+		{
+		  bb_eh_cleanup = create_empty_bb (bb);
+		  if (dom_info_available_p (CDI_DOMINATORS))
+		    set_immediate_dominator (CDI_DOMINATORS, bb_eh_cleanup, bb);
+		  if (current_loops)
+		    add_bb_to_loop (bb_eh_cleanup, current_loops->tree_root);
+
+		  /* Make the new block an EH cleanup for the call.  */
+		  eh_region new_r = gen_eh_region_cleanup (NULL);
+		  eh_landing_pad lp = gen_eh_landing_pad (new_r);
+		  tree label = gimple_block_label (bb_eh_cleanup);
+		  lp->post_landing_pad = label;
+		  EH_LANDING_PAD_NR (label) = lp_eh_cleanup = lp->index;
+
+		  /* Just propagate the exception.
+		     We will later insert the verifier call.  */
+		  gimple_stmt_iterator ehgsi;
+		  ehgsi = gsi_after_labels (bb_eh_cleanup);
+		  gresx *resx = gimple_build_resx (new_r->index);
+		  gsi_insert_before (&ehgsi, resx, GSI_SAME_STMT);
+
+		  if (dump_file)
+		    fprintf (dump_file,
+			     "Created cleanup block %i:\n",
+			     bb_eh_cleanup->index);
+		}
+	      else if (dom_info_available_p (CDI_DOMINATORS))
+		{
+		  basic_block immdom;
+		  immdom = get_immediate_dominator (CDI_DOMINATORS,
+						    bb_eh_cleanup);
+		  if (!dominated_by_p (CDI_DOMINATORS, bb, immdom))
+		    {
+		      immdom = nearest_common_dominator (CDI_DOMINATORS,
+							 immdom, bb);
+		      set_immediate_dominator (CDI_DOMINATORS,
+					       bb_eh_cleanup, immdom);
+		    }
+		}
+
+	      if (dump_file)
+		{
+		  fprintf (dump_file,
+			   "Associated cleanup block with stmt in block %i:\n",
+			   bb->index);
+		  print_gimple_stmt (dump_file, stmt, 0);
+		}
+
+	      add_stmt_to_eh_lp (stmt, lp_eh_cleanup);
+	      /* Finally, wire the EH cleanup block into the CFG.  */
+	      make_eh_edges (stmt);
+	    }
+	}
+
+      if (bb_eh_cleanup)
+	{
+	  /* A cfg_cleanup after bb_eh_cleanup makes for a more compact
+	     rtcfg, and it avoids bb numbering differences when we split
+	     blocks because of trailing debug insns only.  */
+	  cleanup_tree_cfg ();
+	  gcc_checking_assert (EDGE_COUNT (bb_eh_cleanup->succs) == 0);
+	}
+    }
+
+  /* These record blocks with calls that are to be preceded by
+     checkpoints, such as noreturn calls (if so chosen), must-tail
+     calls, potential early-marked tail calls, and returning calls (if
+     so chosen).  */
+  int count_chkcall = 0;
+  auto_sbitmap chkcall_blocks (last_basic_block_for_fn (fun));
+  bitmap_clear (chkcall_blocks);
+
+  /* We wish to add verification at blocks without successors, such as
+     noreturn calls (raising or not) and the reraise at the cleanup
+     block, but not other reraises: they will go through the cleanup
+     block.  */
+  if (check_before_noreturn_calls)
+    FOR_EACH_BB_FN (bb, fun)
+      {
+	gimple_stmt_iterator gsi = gsi_last_bb (bb);
+	if (gsi_end_p (gsi))
+	  continue;
+	gimple *stmt = gsi_stmt (gsi);
 
-  vstd.check ();
+	if (EDGE_COUNT (bb->succs) == 0)
+	  {
+	    /* A stmt at the end of a block without any successors is
+	       either a resx or a noreturn call without a local
+	       handler.  Check that it's one of the desired
+	       checkpoints.  */
+	    if (flag_exceptions && is_a <gresx *> (stmt)
+		? (check_before_always_throwing_noreturn_calls
+		   || bb == bb_eh_cleanup)
+		: (!is_a <gcall *> (stmt)
+		   || !gimple_call_noreturn_p (stmt))
+		? (/* Catch cases in which successors would be
+		      expected.  */
+		   gcc_unreachable (), false)
+		: (!flag_exceptions
+		   || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+		? check_before_nothrow_noreturn_calls
+		: always_throwing_noreturn_call_p (stmt)
+		? check_before_always_throwing_noreturn_calls
+		: check_before_throwing_noreturn_calls)
+	      {
+		if (dump_file)
+		  {
+		    fprintf (dump_file,
+			     "Scheduling check before stmt"
+			     " in succ-less block %i:\n",
+			     bb->index);
+		    print_gimple_stmt (dump_file, stmt, 0);
+		  }
+
+		if (bitmap_set_bit (chkcall_blocks, bb->index))
+		  count_chkcall++;
+		else
+		  gcc_unreachable ();
+	      }
+	    continue;
+	  }
 
-  return 0;
+	/* If there are no exceptions, then any noreturn call must have
+	   zero successor edges.  Otherwise, check for blocks without
+	   non-EH successors, but skip those with resx stmts and edges
+	   (i.e., those other than that in bb_eh_cleanup), since those
+	   will go through bb_eh_cleanup, that will have been counted as
+	   noreturn above because it has no successors.  */
+	gcc_checking_assert (bb != bb_eh_cleanup
+			     || !check_at_escaping_exceptions);
+	if (flag_exceptions && is_a <gresx *> (stmt)
+	    ? check_before_always_throwing_noreturn_calls
+	    : (!is_a <gcall *> (stmt)
+	       || !gimple_call_noreturn_p (stmt))
+	    ? false
+	    : (!flag_exceptions
+	       || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+	    ? (/* Catch cases that should not have successors.  */
+	       gcc_unreachable (), check_before_nothrow_noreturn_calls)
+	    : always_throwing_noreturn_call_p (stmt)
+	    ? check_before_always_throwing_noreturn_calls
+	    : check_before_throwing_noreturn_calls)
+	  {
+	    gcc_checking_assert (single_succ_p (bb)
+				 && (single_succ_edge (bb)->flags & EDGE_EH));
+
+	    if (dump_file)
+	      {
+		fprintf (dump_file,
+			 "Scheduling check before stmt"
+			 " in EH-succ block %i:\n",
+			 bb->index);
+		print_gimple_stmt (dump_file, stmt, 0);
+	      }
+
+	    if (bitmap_set_bit (chkcall_blocks, bb->index))
+	      count_chkcall++;
+	    else
+	      gcc_unreachable ();
+	  }
+      }
+  else if (bb_eh_cleanup)
+    {
+      if (bitmap_set_bit (chkcall_blocks, bb_eh_cleanup->index))
+	count_chkcall++;
+      else
+	gcc_unreachable ();
+    }
+
+  gcc_checking_assert (!bb_eh_cleanup
+		       || bitmap_bit_p (chkcall_blocks, bb_eh_cleanup->index));
+
+  /* If we don't have edges to exit nor noreturn calls (including the
+     cleanup reraise), then we may skip instrumentation: that would
+     amount to a function that ends with an infinite loop.  */
+  if (!count_chkcall
+      && EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (fun)->preds) == 0)
+    {
+      if (dump_file)
+	fprintf (dump_file,
+		 "Disabling CFR, no exit paths to check\n");
+
+      return 0;
+    }
+
+  /* Search for must-tail calls, early-marked potential tail calls,
+     and, if requested, returning calls.  As we introduce early
+     checks, */
+  int count_postchk = 0;
+  auto_sbitmap postchk_blocks (last_basic_block_for_fn (fun));
+  bitmap_clear (postchk_blocks);
+  chk_edges_t chk_edges;
+  hardcfr_sibcall_search_preds (EXIT_BLOCK_PTR_FOR_FN (fun), chk_edges,
+				count_chkcall, chkcall_blocks,
+				count_postchk, postchk_blocks,
+				NULL);
+
+  rt_bb_visited vstd (chk_edges.length () + count_chkcall);
+
+  auto_sbitmap combined_blocks (last_basic_block_for_fn (fun));
+  bitmap_copy (combined_blocks, chkcall_blocks);
+  int i;
+  edge *e;
+  FOR_EACH_VEC_ELT (chk_edges, i, e)
+    if (!bitmap_set_bit (combined_blocks, (*e)->src->index))
+      gcc_unreachable ();
+
+  /* Visit blocks in index order, because building rtcfg depends on
+     that.  Blocks must be compact, which the cleanup_cfg requirement
+     ensures.  This would also enable FOR_EACH_BB_FN to be used to
+     iterate in index order, but bb_eh_cleanup block splits and
+     insertions changes that.  */
+  gcc_checking_assert (n_basic_blocks_for_fn (fun)
+		       == last_basic_block_for_fn (fun));
+  for (int i = NUM_FIXED_BLOCKS; i < n_basic_blocks_for_fn (fun); i++)
+    {
+      bb = BASIC_BLOCK_FOR_FN (fun, i);
+      gcc_checking_assert (bb->index == i);
+      vstd.visit (bb, bitmap_bit_p (combined_blocks, i),
+		  bitmap_bit_p (postchk_blocks, i));
+    }
+
+  vstd.check (chk_edges, count_chkcall, chkcall_blocks);
+
+  return
+    TODO_update_ssa
+    | TODO_cleanup_cfg
+    | TODO_verify_il;
 }
 
 /* Instantiate a hardcfr pass.  */
diff --git a/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c b/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c
new file mode 100644
index 00000000000..a6992eb9f8e
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -O0 -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we don't insert checking before noreturn calls.  -O0 is tested
+   separately because h is not found to be noreturn without optimization.  */
+
+#include "torture/harden-cfr-noret.c"
+
+/* No out-of-line checks.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* Only one inline check at the end of f and of h2.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c
new file mode 100644
index 00000000000..8bd2d13ac18
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c
@@ -0,0 +1,18 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we don't insert checking before noreturn calls.  -O0 is tested
+   separately because h is not found to be noreturn without optimization, which
+   affects codegen for h2, so h2 is omitted here at -O0.  */
+
+#if !__OPTIMIZE__
+# define OMIT_H2
+#endif
+
+#include "harden-cfr-noret.c"
+
+
+/* No out-of-line checks.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* Only one inline check at the end of f.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c
new file mode 100644
index 00000000000..a804a6cfe59
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fno-exceptions -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that -fno-exceptions makes for implicit nothrow in noreturn
+   handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#include "harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c
new file mode 100644
index 00000000000..f390cfdbc59
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checking before nothrow noreturn calls.  */
+
+#include "harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c
new file mode 100644
index 00000000000..fdd803109a4
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c
@@ -0,0 +1,38 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checking before all noreturn calls.  */
+
+#ifndef ATTR_NOTHROW_OPT /* Overridden in harden-cfr-noret-noexcept.  */
+#define ATTR_NOTHROW_OPT __attribute__ ((__nothrow__))
+#endif
+
+extern void __attribute__ ((__noreturn__)) ATTR_NOTHROW_OPT g (void);
+
+void f(int i) {
+  if (i)
+    /* Out-of-line checks here...  */
+    g ();
+  /* ... and here.  */
+}
+
+void __attribute__ ((__noinline__, __noclone__))
+h(void) {
+  /* Inline check here.  */
+  g ();
+}
+
+#ifndef OMIT_H2 /* from harden-cfr-noret-never.  */
+void h2(void) {
+  /* Inline check either here, whether because of noreturn or tail call...  */
+  h ();
+  /* ... or here, if not optimizing.  */
+}
+#endif
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c
new file mode 100644
index 00000000000..6d11487bbba
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c
@@ -0,0 +1,8 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+#include "harden-cfr-tail.c"
+
+/* Inline checking after the calls, disabling tail calling.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 5 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 0 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c
new file mode 100644
index 00000000000..550b02ca088
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c
@@ -0,0 +1,35 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-exceptions -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checks before returning calls and alternate paths, even
+   at -O0, because of the explicit command-line flag.  */
+
+void g (void);
+void g2 (void);
+void g3 (void);
+
+void f (int i) {
+  if (!i)
+    /* Out-of-line checks here...  */
+    g ();
+  else if (i > 0)
+    /* here...  */
+    g2 ();
+  /* else */
+    /* and in the implicit else here.  */
+}
+
+void f2 (int i) {
+  if (!i)
+    /* Out-of-line check here...  */
+    g ();
+  else if (i > 0)
+    /* here...  */
+    g2 ();
+  else
+    /* and here.  */
+    g3 ();
+}
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 6 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 0 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
index 40d76c5c163..d5467eafa9f 100644
--- a/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
@@ -1,17 +1,52 @@
 /* { dg-do compile } */
-/* { dg-options "-fharden-control-flow-redundancy -fdump-tree-hardcfr -ffat-lto-objects" } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-hardcfr-check-exceptions -fdump-tree-hardcfr -ffat-lto-objects -Wno-return-type" } */
 
-/* We'd like to check that we insert checking so as to not disrupt tail calls,
-   but unfortunately mandatory tail calls are not available in C, and optimizing
-   calls as tail calls only takes place after hardcfr.  */
+/* Check that we insert CFR checking so as to not disrupt tail calls.
+   Mandatory tail calls are not available in C, and optimizing calls as tail
+   calls only takes place after hardcfr, so we insert checking before calls
+   followed by copies and return stmts with the same return value, that might
+   (or might not) end up optimized to tail calls.  */
 
-extern int g(int i);
+extern int g (int i);
 
-int f(int i) {
+int f1(int i) {
+  /* Inline check before the returning call.  */
   return g (i);
 }
 
-/* Inline checking before the tail call.  */
-/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
-/* Inline checking before the tail call.  */
-/* { dg-final { scan-tree-dump-times "\\\[tail\]" 1 "hardcfr" { xfail *-*-* } } } */
+extern void g2 (int i);
+
+void f2(int i) {
+  /* Inline check before the returning call, that ignores the returned value,
+     matching the value-less return.  */
+  g2 (i);
+  return;
+}
+
+void f3(int i) {
+  /* Inline check before the returning call.  */
+  g (i);
+}
+
+void f4(int i) {
+  if (i)
+    /* Out-of-line check before the returning call.  */
+    return g2 (i);
+  /* Out-of-line check before implicit return.  */
+}
+
+int f5(int i) {
+  /* Not regarded as a returning call, returning value other than callee's
+     returned value.  */
+  g (i);
+  /* Inline check after the non-returning call.  */
+  return i;
+}
+
+/* Out-of-line checks in f4, before returning calls and before return.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking in all other functions.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 4 "hardcfr" } } */
+/* Check before tail-call in all but f5, but f4 is out-of-line.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 3 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C b/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C
new file mode 100644
index 00000000000..17ea79f7cfb
--- /dev/null
+++ b/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C
@@ -0,0 +1,11 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects -O0" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions, and also checking before noreturn
+   calls.  h2 and h2b get an extra resx without ehcleanup.  */
+
+#include "torture/harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 16 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C b/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C
new file mode 100644
index 00000000000..f0338ccc361
--- /dev/null
+++ b/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C
@@ -0,0 +1,10 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -foptimize-sibling-calls -fdump-tree-hardcfr -O0" } */
+
+/* -fhardcfr-check-returning-calls gets implicitly disabled because,
+   -at O0, -foptimize-sibling-calls has no effect.  */
+
+#include "torture/harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 12 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C
new file mode 100644
index 00000000000..0d35920c7ee
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  */
+
+#include "harden-cfr-noret-no-nothrow.C"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Bypassing cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C
new file mode 100644
index 00000000000..b7d247ff43c
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C
@@ -0,0 +1,18 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  Expected results for =never and =nothrow are the same,
+   since the functions are not nothrow.  */
+
+#include "harden-cfr-noret-no-nothrow.C"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Associated cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C
new file mode 100644
index 00000000000..62c58cfd406
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C
@@ -0,0 +1,23 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#if ! __OPTIMIZE__
+void __attribute__ ((__noreturn__)) h (void);
+#endif
+
+#include "../../c-c++-common/torture/harden-cfr-noret.c"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Associated cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C
new file mode 100644
index 00000000000..0286f6e6d3f
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C
@@ -0,0 +1,20 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions, and also checking before noreturn
+   calls.  */
+
+#if ! __OPTIMIZE__
+/* Without optimization, functions with cleanups end up with an extra
+   resx that is not optimized out, so arrange to optimize them.  */
+void __attribute__ ((__optimize__ (1))) h2(void);
+void __attribute__ ((__optimize__ (1))) h2b(void);
+#endif
+
+#include "harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 14 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
+/* h, h2, h2b, and h4.  */
+/* { dg-final { scan-tree-dump-times "Bypassing" 4 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C
new file mode 100644
index 00000000000..885b0b236af
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C
@@ -0,0 +1,11 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions  -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we do not insert cleanups for checking around the bodies
+   of maybe-throwing functions.  h4 doesn't get any checks, because we
+   don't have noreturn checking enabled.  */
+
+#include "harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 6 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C
new file mode 100644
index 00000000000..32def637255
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C
@@ -0,0 +1,31 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -foptimize-sibling-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions.  These results depend on checking before
+   returning calls, which is only enabled when sibcall optimizations
+   are enabled, so change the optimization mode to -O1 for f and f2,
+   so that -foptimize-sibling-calls can take effect and enable
+   -fhardcfr-check-returning-calls, so that we get the same results.
+   There is a separate test for -O0.  */
+
+#if ! __OPTIMIZE__
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) f(int i);
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) f2(int i);
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) h3(void);
+#endif
+
+#include "harden-cfr-throw.C"
+
+/* f gets out-of-line checks before the unwrapped tail call and in the
+   else edge.  */
+/* f2 gets out-of-line checks before both unwrapped tail calls.  */
+/* h gets out-of-line checks before the implicit return and in the
+   cleanup block.  */
+/* h2 and h2b get out-of-line checks before the cleanup returning
+   call, and in the cleanup block.  */
+/* h3 gets an inline check before the __cxa_end_catch returning call.  */
+/* h4 gets an inline check in the cleanup block.  */
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 10 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C
new file mode 100644
index 00000000000..992fbdad381
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C
@@ -0,0 +1,65 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions.  */
+
+extern void g (void);
+extern void g2 (void);
+
+void f(int i) {
+  if (i)
+    g ();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void f2(int i) {
+  if (i)
+    g ();
+  else
+    g2 ();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h(void) {
+  try {
+    g ();
+  } catch (...) {
+    throw;
+  }
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+struct needs_cleanup {
+  ~needs_cleanup();
+};
+
+void h2(void) {
+  needs_cleanup y; /* No check in the cleanup handler.  */
+  g();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+extern void __attribute__ ((__nothrow__)) another_cleanup (void*);
+
+void h2b(void) {
+  int x __attribute__ ((cleanup (another_cleanup)));
+  g();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h3(void) {
+  try {
+    throw 1;
+  } catch (...) {
+  }
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h4(void) {
+  throw 1;
+  /* Inline check in the cleanup around the __cxa_throw noreturn call.  */
+}
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 12 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c b/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c
new file mode 100644
index 00000000000..8e4ee1fab08
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c
@@ -0,0 +1,15 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C makes for implicit nothrow in noreturn handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#include "../../c-c++-common/torture/harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c b/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c
new file mode 100644
index 00000000000..634d98f1ffc
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c
@@ -0,0 +1,40 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-hardcfr-check-exceptions -fdump-tree-hardcfr -ffat-lto-objects -Wno-return-type" } */
+
+/* In C only, check some additional cases (comparing with
+   c-c++-common/torture/harden-cfr-tail.c) of falling off the end of non-void
+   function.  C++ would issue an unreachable call in these cases.  */
+
+extern int g (int i);
+
+int f1(int i) {
+  /* Inline check before the returning call, that doesn't return anything.  */
+  g (i);
+  /* Implicit return without value, despite the return type; this combination
+     enables tail-calling of g, and is recognized as a returning call.  */
+}
+
+extern void g2 (int i);
+
+int f2(int i) {
+  /* Inline check before the returning call, that disregards its return
+     value.  */
+  g2 (i);
+  /* Implicit return without value, despite the return type; this combination
+     enables tail-calling of g2, and is recognized as a returning call.  */
+}
+
+int f3(int i) {
+  if (i)
+    /* Out-of-line check before the returning call.  */
+    return g (i);
+  /* Out-of-line check before implicit return.  */
+}
+
+/* Out-of-line checks in f3, before returning calls and before return.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking in all other functions.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
+/* Check before tail-call in all functions, but f3 is out-of-line.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/gnat.dg/hardcfr.adb b/gcc/testsuite/gnat.dg/hardcfr.adb
index b578a913d75..abe1605c029 100644
--- a/gcc/testsuite/gnat.dg/hardcfr.adb
+++ b/gcc/testsuite/gnat.dg/hardcfr.adb
@@ -1,5 +1,5 @@
 --  { dg-do run }
---  { dg-options "-fharden-control-flow-redundancy -fdump-tree-hardcfr --param=hardcfr-max-blocks=22 --param=hardcfr-max-inline-blocks=12 -O0" }
+--  { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions -fdump-tree-hardcfr --param=hardcfr-max-blocks=22 --param=hardcfr-max-inline-blocks=12 -O0" }
 
 procedure HardCFR is
    function F (I, J : Integer) return Integer is
diff --git a/libgcc/hardcfr.c b/libgcc/hardcfr.c
index 8ef29428111..55f6b995f2f 100644
--- a/libgcc/hardcfr.c
+++ b/libgcc/hardcfr.c
@@ -60,14 +60,25 @@ extern void __hardcfr_check (size_t blocks,
 			     vword const *visited,
 			     vword const *cfg);
 
+/* Compute the MASK for the bit representing BLOCK in WORDIDX's vword in a
+   visited blocks bit array.  */
+static inline void
+block2mask (size_t const block, vword *const mask, size_t *const wordidx)
+{
+  size_t wbits = __CHAR_BIT__ * sizeof (vword);
+  *wordidx = block / wbits;
+  *mask = (vword)1 << (block % wbits);
+}
 
 /* Check whether the bit corresponding to BLOCK is set in VISITED.  */
 static inline bool
 visited_p (size_t const block, vword const *const visited)
 {
-  size_t wbits = __CHAR_BIT__ * sizeof (vword);
-  vword w = visited[block / wbits];
-  return (w & ((vword)1 << (block % wbits))) != 0;
+  vword mask;
+  size_t wordidx;
+  block2mask (block, &mask, &wordidx);
+  vword w = visited[wordidx];
+  return (w & mask) != 0;
 }
 
 /* Read and consume a mask from **CFG_IT.  (Consume meaning advancing the
@@ -118,19 +129,19 @@ consume_seq (vword const **const cfg_it)
    we reach the terminator without finding any.  Consume the entire sequence
    otherwise, so that *CFG_IT points just past the terminator, which may be the
    beginning of the next sequence.  */
-static inline void
+static inline bool
 check_seq (vword const *const visited, vword const **const cfg_it)
 {
   vword mask;
   size_t wordidx;
 
   /* If the block was visited, check that at least one of the
-     preds was also visited.  */
+     preds/succs was also visited.  */
   do
     /* If we get to the end of the sequence without finding any
        match, something is amiss.  */
     if (!next_pair (cfg_it, &mask, &wordidx))
-      __builtin_trap ();
+      return false;
   /* Keep searching until we find a match, at which point the
      condition is satisfied.  */
   while (!test_mask (visited, mask, wordidx));
@@ -139,6 +150,94 @@ check_seq (vword const *const visited, vword const **const cfg_it)
      skipped the block, so as to position the iterator at the beginning of the
      next .  */
   consume_seq (cfg_it);
+
+  return true;
+}
+
+/* Print out the CFG with BLOCKS blocks, presumed to be associated with CALLER.
+   This is expected to be optimized out entirely, unless the verbose part of
+   __hardcfr_check_fail is enabled.  */
+static inline void
+__hardcfr_debug_cfg (size_t const blocks,
+		     void const *const caller,
+		     vword const *const cfg)
+{
+  __builtin_printf ("CFG at %p, for %p", cfg, caller);
+  vword const *cfg_it = cfg;
+  for (size_t i = 0; i < blocks; i++)
+    {
+      vword mask; size_t wordidx;
+      block2mask (i, &mask, &wordidx);
+      __builtin_printf ("\nblock %lu (%lu/0x%lx)\npreds: ",
+			(unsigned long)i,
+			(unsigned long)wordidx, (unsigned long)mask);
+      while (next_pair (&cfg_it, &mask, &wordidx))
+	__builtin_printf (" (%lu/0x%lx)",
+			  (unsigned long)wordidx, (unsigned long)mask);
+      __builtin_printf ("\nsuccs: ");
+      while (next_pair (&cfg_it, &mask, &wordidx))
+	__builtin_printf (" (%lu/0x%lx)",
+			  (unsigned long)wordidx, (unsigned long)mask);
+    }
+  __builtin_printf ("\n");
+}
+
+#ifndef ATTRIBUTE_UNUSED
+# define ATTRIBUTE_UNUSED __attribute__ ((__unused__))
+#endif
+
+/* This is called when an out-of-line hardcfr check fails.  All the arguments
+   are ignored, and it just traps, unless HARDCFR_VERBOSE_FAIL is enabled.  IF
+   it is, it prints the PART of the CFG, expected to have BLOCKS blocks, that
+   failed at CALLER's BLOCK, and the VISITED bitmap.  When the verbose mode is
+   enabled, it also forces __hardcfr_debug_cfg (above) to be compiled into an
+   out-of-line function, that could be called from a debugger.
+   */
+static inline void
+__hardcfr_check_fail (size_t const blocks ATTRIBUTE_UNUSED,
+		      vword const *const visited,
+		      vword const *const cfg ATTRIBUTE_UNUSED,
+		      size_t const block ATTRIBUTE_UNUSED,
+		      int const part ATTRIBUTE_UNUSED,
+		      void const *const caller ATTRIBUTE_UNUSED)
+{
+#if HARDCFR_VERBOSE_FAIL
+  static const char *parts[] = { "preds", "succs" };
+
+  vword mask; size_t wordidx;
+  block2mask (block, &mask, &wordidx);
+  __builtin_printf ("hardcfr fail at %p block %lu (%lu/0x%lx), expected %s:",
+		    caller, (unsigned long)block,
+		    (unsigned long)wordidx, (unsigned long)mask,
+		    parts[part]);
+
+  /* Skip data for previous blocks.  */
+  vword const *cfg_it = cfg;
+  for (size_t i = block; i--; )
+    {
+      consume_seq (&cfg_it);
+      consume_seq (&cfg_it);
+    }
+  for (size_t i = part; i--; )
+    consume_seq (&cfg_it);
+
+  while (next_pair (&cfg_it, &mask, &wordidx))
+    __builtin_printf (" (%lu/0x%lx)",
+		      (unsigned long)wordidx, (unsigned long)mask);
+
+  __builtin_printf ("\nvisited:");
+  block2mask (blocks, &mask, &wordidx);
+  for (size_t i = 0; i <= wordidx; i++)
+    __builtin_printf (" (%lu/0x%lx)",
+		      (unsigned long)i, (unsigned long)visited[i]);
+  __builtin_printf ("\n");
+
+  /* Reference __hardcfr_debug_cfg so that it's output out-of-line, so that it
+     can be called from a debugger.  */
+  if (!caller || caller == __hardcfr_debug_cfg)
+    return;
+#endif
+  __builtin_trap ();
 }
 
 /* Check that, for each of the BLOCKS basic blocks, if its bit is set in
@@ -168,9 +267,13 @@ __hardcfr_check (size_t const blocks,
       else
 	{
 	  /* Check predecessors.  */
-	  check_seq (visited, &cfg_it);
+	  if (!check_seq (visited, &cfg_it))
+	    __hardcfr_check_fail (blocks, visited, cfg, i, 0,
+				  __builtin_return_address (0));
 	  /* Check successors.  */
-	  check_seq (visited, &cfg_it);
+	  if (!check_seq (visited, &cfg_it))
+	    __hardcfr_check_fail (blocks, visited, cfg, i, 1,
+				  __builtin_return_address (0));
 	}
     }
 }

^ permalink raw reply	[flat|nested] 13+ messages in thread
* [gcc(refs/users/aoliva/heads/testme)] hardcfr: add optional checkpoints
@ 2023-06-09  6:16 Alexandre Oliva
  0 siblings, 0 replies; 13+ messages in thread
From: Alexandre Oliva @ 2023-06-09  6:16 UTC (permalink / raw)
  To: gcc-cvs

https://gcc.gnu.org/g:a35dea054642fee54092002156693f88fdb7813f

commit a35dea054642fee54092002156693f88fdb7813f
Author: Alexandre Oliva <oliva@adacore.com>
Date:   Thu Jun 8 01:34:57 2023 -0300

    hardcfr: add optional checkpoints
    
    Previously, control flow redundancy only checked the visited bitmap
    against the control flow graph at return points and before mandatory
    tail calls, missing various other possibilities of exiting a
    subprogram, such as by raising or propagating exceptions, and calling
    noreturn functions.  The checks inserted before returns also prevented
    potential tail-call optimizations.
    
    This incremental change introduces options to control checking at each
    of these previously-missed checkpoints.  Unless disabled, a cleanup is
    introduced to check when an exceptions escapes a subprogram.  To avoid
    disrupting sibcall optimizations, when they are enabled, checks are
    introduced before calls whose results are immediately returned,
    whether or not they are ultimately optimized.  If enabled, checks are
    introduced before noreturn calls and exception raises, or only before
    nothrow noreturn calls.
    
    
    for  gcc/ChangeLog
    
            * common.opt (-fhardcfr-check-returning-calls): New.
            (-fhardcfr-check-exceptions): New.
            (-fhardcfr-check-noreturn-calls=*): New.
            (Enum hardcfr_check_noreturn_calls): New.
            * doc/invoke.texi: Document them.
            * flag-types.h (enum hardcfr_noret): New.
            * gimple-harden-control-flow.cc: Include more headers.
            (pass_data_harden_control_flow_redundancy): Move
            properties_finish to pass return.
            (pass_harden_control_flow_redundancy::gate): Move
            no-edge-to-exit checking to execute, after exception
            transformations.  Use symbolic NUM_FIXED_BLOCKS.  Test for CFG
            before testing block count.
            (check_returning_calls_p): New.
            (hardcfr_scan_block): New.
            (returnign_call_p): New.
            (chk_edges_t): New.
            (hardcfr_sibcall_search_block): New.
            (hardcfr_sibcall_search_preds): New.
            (rt_bb_visited::num2idx): Use symbolic NUM_FIXED_BLOCKS.
            (rt_bb_visited::rt_bb_visited): Likewise.  Take checkpoints
            count, test it to select out-of-line checks.
            (rt_bb_visited::insert_exit_check): Removed.
            (rt_bb_visited::insert_exit_check_in_blocks): New.
            (rt_bb_visited::insert_exit_check_on_edge): New.
            (rt_bb_visited::check): Take checkpoint edges and blocks.
            Move initialization insertion to the end.  Insert checks on
            edges, and before calls in select blocks.  Clobber the visited
            array to the check sequence.  Output actions to dump_file.
            (rt_bb_visited::visit): Take checkpoint and postcheck.  Assume
            an edge to exit if checkpoint, and skip testing if postcheck.
            (always_throwing_noreturn_call_p): New, dummy.
            (pass_harden_control_flow_redundancy::execute): Introduce
            cleanup block if requested and needed.  Scan blocks for
            noreturn and returning calls if checkpoints before them were
            requested.  Log actions to dump_file.  Visit blocks in index
            order.
    
    for  gcc/testsuite/ChangeLog
    
            * c-c++-common/harden-cfr-noret-never-O0.c: New.
            * c-c++-common/torture/harden-cfr-noret-never.c: New.
            * c-c++-common/torture/harden-cfr-noret-noexcept.c: New.
            * c-c++-common/torture/harden-cfr-noret-nothrow.c: New.
            * c-c++-common/torture/harden-cfr-noret.c: New.
            * c-c++-common/torture/harden-cfr-notail.c: New.
            * c-c++-common/torture/harden-cfr-returning.c: New.
            * c-c++-common/torture/harden-cfr-tail.c: Extend.
            * c-c++-common/torture/harden-cfr-bret-always.c: New.
            * c-c++-common/torture/harden-cfr-bret-never.c: New.
            * c-c++-common/torture/harden-cfr-bret-noopt.c: New.
            * c-c++-common/torture/harden-cfr-bret-noret.c: New.
            * c-c++-common/torture/harden-cfr-bret-nothrow.c: New.
            * c-c++-common/torture/harden-cfr-bret-retcl.c: New.
            * c-c++-common/torture/harden-cfr-bret.c (g): New.
            * g++.dg/harden-cfr-throw-always-O0.C: New.
            * g++.dg/harden-cfr-throw-returning-O0.C: New.
            * g++.dg/torture/harden-cfr-noret-always-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-noret-never-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-noret-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-throw-always.C: New.
            * g++.dg/torture/harden-cfr-throw-nocleanup.C: New.
            * g++.dg/torture/harden-cfr-throw-returning.C: New.
            * g++.dg/torture/harden-cfr-throw.C: New.
            * gcc.dg/torture/harden-cfr-noret-no-nothrow.c: New.
            * gcc.dg/torture/harden-cfr-tail-ub.c: New.
            * gnat.dg/hardcfr.adb: Disable checking at exceptions.
    
    for  libgcc/ChangeLog
    
            * hardcfr.c (block2mask): Split out of...
            (visited_p): ... this.
            (check_seq): Move trapping out, return result instead.
            (__hardcfr_debug_cfg): New.
            (__hardcfr_check_fail): New, with optional verbose error
            printing before trapping.
            (__hardcfr_check): Call __hardcfr_check_fail when check_seq
            fails.

Diff:
---
 gcc/common.opt                                     |  33 +
 gcc/doc/invoke.texi                                |  74 +-
 gcc/flag-types.h                                   |  10 +
 gcc/gimple-harden-control-flow.cc                  | 926 ++++++++++++++++++---
 .../c-c++-common/harden-cfr-noret-never-O0.c       |  12 +
 .../c-c++-common/torture/harden-cfr-bret-always.c  |  13 +
 .../c-c++-common/torture/harden-cfr-bret-never.c   |  13 +
 .../c-c++-common/torture/harden-cfr-bret-noopt.c   |  12 +
 .../c-c++-common/torture/harden-cfr-bret-noret.c   |  12 +
 .../c-c++-common/torture/harden-cfr-bret-nothrow.c |  13 +
 .../c-c++-common/torture/harden-cfr-bret-retcl.c   |  12 +
 .../c-c++-common/torture/harden-cfr-bret.c         |   8 +-
 .../c-c++-common/torture/harden-cfr-noret-never.c  |  18 +
 .../torture/harden-cfr-noret-noexcept.c            |  16 +
 .../torture/harden-cfr-noret-nothrow.c             |  13 +
 .../c-c++-common/torture/harden-cfr-noret.c        |  38 +
 .../c-c++-common/torture/harden-cfr-notail.c       |   8 +
 .../c-c++-common/torture/harden-cfr-returning.c    |  35 +
 .../c-c++-common/torture/harden-cfr-tail.c         |  55 +-
 gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C  |  11 +
 .../g++.dg/harden-cfr-throw-returning-O0.C         |  10 +
 .../torture/harden-cfr-noret-always-no-nothrow.C   |  16 +
 .../torture/harden-cfr-noret-never-no-nothrow.C    |  18 +
 .../g++.dg/torture/harden-cfr-noret-no-nothrow.C   |  23 +
 .../g++.dg/torture/harden-cfr-throw-always.C       |  20 +
 .../g++.dg/torture/harden-cfr-throw-nocleanup.C    |  11 +
 .../g++.dg/torture/harden-cfr-throw-returning.C    |  31 +
 gcc/testsuite/g++.dg/torture/harden-cfr-throw.C    |  65 ++
 .../gcc.dg/torture/harden-cfr-noret-no-nothrow.c   |  15 +
 gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c  |  40 +
 gcc/testsuite/gnat.dg/hardcfr.adb                  |   2 +-
 libgcc/hardcfr.c                                   | 119 ++-
 32 files changed, 1568 insertions(+), 134 deletions(-)

diff --git a/gcc/common.opt b/gcc/common.opt
index cf2c22dd7e0..27e4bcb692b 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -1805,6 +1805,39 @@ fharden-control-flow-redundancy
 Common Var(flag_harden_control_flow_redundancy) Optimization
 Harden control flow by recording and checking execution paths.
 
+fhardcfr-check-returning-calls
+Common Var(flag_harden_control_flow_redundancy_check_returning_calls) Init(-1) Optimization
+Check CFR execution paths also before calls followed by returns of their results.
+
+fhardcfr-check-exceptions
+Common Var(flag_harden_control_flow_redundancy_check_exceptions) Init(-1) Optimization
+Check CFR execution paths also when exiting a function through an exception.
+
+fhardcfr-check-noreturn-calls=
+Common Joined RejectNegative Enum(hardcfr_check_noreturn_calls) Var(flag_harden_control_flow_redundancy_check_noreturn) Init(HCFRNR_UNSPECIFIED) Optimization
+-fhardcfr-check-noreturn-calls=[always|nothrow|never]	Check CFR execution paths also before calling noreturn functions.
+
+Enum
+Name(hardcfr_check_noreturn_calls) Type(enum hardcfr_noret) UnknownError(unknown hardcfr noreturn checking level %qs)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(never) Value(HCFRNR_NEVER)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(nothrow) Value(HCFRNR_NOTHROW)
+
+; ??? There could be yet another option here, that checked before
+; noreturn calls, except for those known to always throw, if we had
+; means to distinguish noreturn functions known to always throw, such
+; as those used to (re)raise exceptions, from those that merely might
+; throw.  "not always" stands for "not always-throwing", but it also
+; contrasts with "always" below.
+; EnumValue
+; Enum(hardcfr_check_noreturn_calls) String(not-always) Value(HCFRNR_NOT_ALWAYS)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(always) Value(HCFRNR_ALWAYS)
+
 ; Nonzero means ignore `#ident' directives.  0 means handle them.
 ; Generate position-independent code for executables if possible
 ; On SVR4 targets, it also controls whether or not to emit a
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index 265fb65340e..386ef9c4ad3 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -634,7 +634,9 @@ Objective-C and Objective-C++ Dialects}.
 -fsanitize-undefined-trap-on-error  -fbounds-check
 -fcf-protection=@r{[}full@r{|}branch@r{|}return@r{|}none@r{|}check@r{]}
 -fharden-compares -fharden-conditional-branches
--fharden-control-flow-redundancy
+-fharden-control-flow-redundancy  -fhardcfr-check-exceptions
+-fhardcfr-check-returning-calls
+-fhardcfr-check-noreturn-calls=@r{[}always@r{|}nothrow@r{|}never@r{]}
 -fstack-protector  -fstack-protector-all  -fstack-protector-strong
 -fstack-protector-explicit  -fstack-check
 -fstack-limit-register=@var{reg}  -fstack-limit-symbol=@var{sym}
@@ -17150,11 +17152,75 @@ conditionals.
 @opindex fharden-control-flow-redundancy
 @item -fharden-control-flow-redundancy
 Emit extra code to set booleans when entering basic blocks, and to
-verify, at function exits, that they amount to an execution path that is
-consistent with the control flow graph, trapping otherwise.  Tuning
-options @option{--param hardcfr-max-blocks} and @option{--param
+verify and trap, at function exits, when the booleans do not form an
+execution path that is compatible with the control flow graph.
+
+Verification takes place before returns, before mandatory tail calls
+(see below) and, optionally, before escaping exceptions with
+@option{-fhardcfr-check-exceptions}, before returning calls with
+@option{-fhardcfr-check-returning-calls}, and before noreturn calls with
+@option{-fhardcfr-check-noreturn-calls}).  Tuning options
+@option{--param hardcfr-max-blocks} and @option{--param
 hardcfr-max-inline-blocks} are available.
 
+Tail call optimization takes place too late to affect control flow
+redundancy, but calls annotated as mandatory tail calls by language
+front-ends, and any calls marked early enough as potential tail calls
+would also have verification issued before the call, but these
+possibilities are merely theoretical, as these conditions can only be
+met when using custom compiler plugins.
+
+@opindex fhardcfr-check-exceptions
+@opindex fno-hardcfr-check-exceptions
+@item -fhardcfr-check-exceptions
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph at exception
+escape points, as if the function body was wrapped with a cleanup
+handler that performed the check and reraised.  This option is enabled
+by default; use @option{-fno-hardcfr-check-exceptions} to disable it.
+
+@opindex fhardcfr-check-returning-calls
+@opindex fno-hardcfr-check-returning-calls
+@item -fhardcfr-check-returning-calls
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph before any
+function call immediately followed by a return of its result, if any, so
+as to not prevent tail-call optimization, whether or not it is
+ultimately optimized to a tail call.
+
+This option is enabled by default whenever sibling call optimizations
+are enabled (see @option{-foptimize-sibling-calls}), but it can be
+enabled (or disabled, using its negated form) explicitly, regardless of
+the optimizations.
+
+@opindex fhardcfr-check-noreturn-calls
+@item -fhardcfr-check-noreturn-calls=@r{[}always@r{|}nothrow@r{|}never@r{]}
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph before
+@code{noreturn} calls, either all of them (@option{always}), those that
+may not return control to the caller through an exception either
+(@option{nothrow}), or none of them (@option{never}, the default).
+
+Checking before a @code{noreturn} function that may return control to
+the caller through an exception may cause checking to be performed more
+than once, if the exception is caught in the caller, whether by a
+handler or a cleanup.  When @option{-fhardcfr-check-exceptions} is also
+enabled, the compiler will avoid associating a @code{noreturn} call with
+the implicitly-added cleanup handler, since it would be redundant with
+the check performed before the call, but other handlers or cleanups in
+the function, if activated, will modify the recorded execution path and
+check it again when another checkpoint is hit.  The checkpoint may even
+be another @code{noreturn} call, so checking may end up performed
+multiple times.
+
+Various optimizers may cause calls to be marked as @code{noreturn}
+and/or @code{nothrow}, even in the absence of the corresponding
+attributes, which may affect the placement of checks before calls, as
+well as the addition of implicit cleanup handlers for them.  This
+unpredictability, and the fact that raising and reraising exceptions
+frequently amounts to implicitly calling @code{noreturn} functions, have
+made @option{never} the default setting for this option.
+
 @opindex fstack-protector
 @item -fstack-protector
 Emit extra code to check for buffer overflows, such as stack smashing
diff --git a/gcc/flag-types.h b/gcc/flag-types.h
index 78dbdbe90dd..6d6fd19fcd6 100644
--- a/gcc/flag-types.h
+++ b/gcc/flag-types.h
@@ -157,6 +157,16 @@ enum stack_reuse_level
   SR_ALL
 };
 
+/* Control Flow Redundancy hardening options for noreturn calls.  */
+enum hardcfr_noret
+{
+  HCFRNR_NEVER,
+  HCFRNR_NOTHROW,
+  HCFRNR_NOT_ALWAYS, /* Reserved for future use.  */
+  HCFRNR_ALWAYS,
+  HCFRNR_UNSPECIFIED = -1
+};
+
 /* The live patching level.  */
 enum live_patching_level
 {
diff --git a/gcc/gimple-harden-control-flow.cc b/gcc/gimple-harden-control-flow.cc
index 6b08846dbb1..5c66762d448 100644
--- a/gcc/gimple-harden-control-flow.cc
+++ b/gcc/gimple-harden-control-flow.cc
@@ -29,7 +29,12 @@ along with GCC; see the file COPYING3.  If not see
 #include "tree-pass.h"
 #include "ssa.h"
 #include "gimple-iterator.h"
+#include "gimple-pretty-print.h"
 #include "tree-cfg.h"
+#include "tree-cfgcleanup.h"
+#include "tree-eh.h"
+#include "except.h"
+#include "sbitmap.h"
 #include "basic-block.h"
 #include "cfghooks.h"
 #include "cfgloop.h"
@@ -60,9 +65,7 @@ const pass_data pass_data_harden_control_flow_redundancy = {
   0,	    // properties_provided
   0,	    // properties_destroyed
   TODO_cleanup_cfg, // properties_start
-  TODO_update_ssa
-  | TODO_cleanup_cfg
-  | TODO_verify_il, // properties_finish
+  0,        // properties_finish
 };
 
 class pass_harden_control_flow_redundancy : public gimple_opt_pass
@@ -79,16 +82,6 @@ public:
     if (!flag_harden_control_flow_redundancy)
       return false;
 
-    /* We don't verify when an exception escapes, propagated or raised
-       by the function itself, so we're only concerned with edges to
-       the exit block.  If there aren't any, the function doesn't
-       return normally, so there won't be any checking point, so
-       there's no point in running the pass.  Should we add
-       verification at exception escapes, we should at least look at
-       !flag_exceptions here.  */
-    if (EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (fun)->preds) == 0)
-      return false;
-
     /* Functions that return more than once, like setjmp and vfork
        (that also gets this flag set), will start recording a path
        after the first return, and then may take another path when
@@ -117,8 +110,9 @@ public:
 	return false;
       }
 
-    if (param_hardcfr_max_blocks > 0
-	&& n_basic_blocks_for_fn (fun) - 2 > param_hardcfr_max_blocks)
+    if (fun->cfg && param_hardcfr_max_blocks > 0
+	&& (n_basic_blocks_for_fn (fun) - NUM_FIXED_BLOCKS
+	    > param_hardcfr_max_blocks))
       {
 	warning_at (DECL_SOURCE_LOCATION (fun->decl), 0,
 		    "%qD has more than %u blocks, the requested"
@@ -134,6 +128,293 @@ public:
 
 }
 
+/* Return TRUE iff CFR checks should be inserted before returning
+   calls.  */
+
+static bool
+check_returning_calls_p ()
+{
+  return
+    flag_harden_control_flow_redundancy_check_returning_calls > 0
+    || (flag_harden_control_flow_redundancy_check_returning_calls < 0
+	/* Gates pass_tail_calls.  */
+	&& flag_optimize_sibling_calls
+	/* Gates pass_all_optimizations.  */
+	&& optimize >= 1 && !optimize_debug);
+}
+
+/* Scan BB from the end, updating *RETPTR if given as return stmts and
+   copies are found.  Return a call or a stmt that cannot appear after
+   a tail call, or NULL if the top of the block is reached without
+   finding any.  */
+
+static gimple *
+hardcfr_scan_block (basic_block bb, tree **retptr)
+{
+  gimple_stmt_iterator gsi;
+  for (gsi = gsi_last_bb (bb); !gsi_end_p (gsi); gsi_prev (&gsi))
+    {
+      gimple *stmt = gsi_stmt (gsi);
+
+      /* Ignore labels, returns, nops, clobbers and debug stmts.  */
+      if (gimple_code (stmt) == GIMPLE_LABEL
+	  || gimple_code (stmt) == GIMPLE_NOP
+	  || gimple_code (stmt) == GIMPLE_PREDICT
+	  || gimple_clobber_p (stmt)
+	  || is_gimple_debug (stmt))
+	continue;
+
+      if (gimple_code (stmt) == GIMPLE_RETURN)
+	{
+	  greturn *gret = as_a <greturn *> (stmt);
+	  if (retptr)
+	    {
+	      gcc_checking_assert (!*retptr);
+	      *retptr = gimple_return_retval_ptr (gret);
+	    }
+	  continue;
+	}
+
+      /* Check for a call.  */
+      if (is_gimple_call (stmt))
+	return stmt;
+
+      /* Allow simple copies to the return value, updating the return
+	 value to be found in earlier assignments.  */
+      if (retptr && *retptr && gimple_assign_single_p (stmt)
+	  && **retptr == gimple_assign_lhs (stmt))
+	{
+	  *retptr = gimple_assign_rhs1_ptr (stmt);
+	  continue;
+	}
+
+      return stmt;
+    }
+
+  /* Any other kind of stmt will prevent a tail call.  */
+  return NULL;
+}
+
+/* Return TRUE iff CALL is to be preceded by a CFR checkpoint, i.e.,
+   if it's a returning call (one whose result is ultimately returned
+   without intervening non-copy statements) and we're checking
+   returning calls, a __builtin_return call (noreturn with a path to
+   the exit block), a must-tail call, or a tail call.  */
+
+static bool
+returning_call_p (gcall *call)
+{
+  if (!(gimple_call_noreturn_p (call)
+	|| gimple_call_must_tail_p (call)
+	|| gimple_call_tail_p (call)
+	|| check_returning_calls_p ()))
+    return false;
+
+  /* Quickly check that there's a path to exit compatible with a
+     returning call.  Detect infinite loops through the counter.  */
+  basic_block bb = gimple_bb (call);
+  auto_vec<basic_block, 10> path;
+  for (int i = n_basic_blocks_for_fn (cfun);
+       bb != EXIT_BLOCK_PTR_FOR_FN (cfun) && i--;
+       bb = single_succ (bb))
+    if (!single_succ_p (bb)
+	|| (single_succ_edge (bb)->flags & EDGE_EH) != 0)
+      return false;
+    else
+      path.safe_push (bb);
+
+  /* Check the stmts in the blocks and trace the return value.  */
+  tree *retptr = NULL;
+  for (;;)
+    {
+      gcc_checking_assert (!path.is_empty ());
+      basic_block bb = path.pop ();
+      gimple *stop = hardcfr_scan_block (bb, &retptr);
+      if (stop)
+	{
+	  if (stop != call)
+	    return false;
+	  gcc_checking_assert (path.is_empty ());
+	  break;
+	}
+
+      gphi *retphi = NULL;
+      if (retptr && *retptr && TREE_CODE (*retptr) == SSA_NAME
+	  && !SSA_NAME_IS_DEFAULT_DEF (*retptr)
+	  && SSA_NAME_DEF_STMT (*retptr)
+	  && is_a <gphi *> (SSA_NAME_DEF_STMT (*retptr))
+	  && gimple_bb (SSA_NAME_DEF_STMT (*retptr)) == bb)
+	{
+	  retphi = as_a <gphi *> (SSA_NAME_DEF_STMT (*retptr));
+	  gcc_checking_assert (gimple_phi_result (retphi) == *retptr);
+	}
+      else
+	continue;
+
+      gcc_checking_assert (!path.is_empty ());
+      edge e = single_succ_edge (path.last ());
+      int i = EDGE_COUNT (bb->preds);
+      while (i--)
+	if (EDGE_PRED (bb, i) == e)
+	  break;
+      gcc_checking_assert (i >= 0);
+      retptr = gimple_phi_arg_def_ptr (retphi, i);
+    }
+
+  return (gimple_call_noreturn_p (call)
+	  || gimple_call_must_tail_p (call)
+	  || gimple_call_tail_p (call)
+	  || (gimple_call_lhs (call) == (retptr ? *retptr : NULL)
+	      && check_returning_calls_p ()));
+}
+
+typedef auto_vec<edge, 10> chk_edges_t;
+
+/* Declare for mutual recursion.  */
+static bool hardcfr_sibcall_search_preds (basic_block bb,
+					  chk_edges_t &chk_edges,
+					  int &count_chkcall,
+					  auto_sbitmap &chkcall_blocks,
+					  int &count_postchk,
+					  auto_sbitmap &postchk_blocks,
+					  tree *retptr);
+
+/* Search backwards from the end of BB for a mandatory or potential
+   sibcall.  Schedule the block to be handled sort-of like noreturn if
+   so.  Recurse to preds, with updated RETPTR, if the block only
+   contains stmts that may follow such a call, scheduling checking at
+   edges and marking blocks as post-check as needed.  Return true iff,
+   at the end of the block, a check will have already been
+   performed.  */
+
+static bool
+hardcfr_sibcall_search_block (basic_block bb,
+			      chk_edges_t &chk_edges,
+			      int &count_chkcall,
+			      auto_sbitmap &chkcall_blocks,
+			      int &count_postchk,
+			      auto_sbitmap &postchk_blocks,
+			      tree *retptr)
+{
+  /* Conditionals and internal exceptions rule out tail calls.  */
+  if (!single_succ_p (bb)
+      || (single_succ_edge (bb)->flags & EDGE_EH) != 0)
+    return false;
+
+  gimple *stmt = hardcfr_scan_block (bb, &retptr);
+  if (!stmt)
+    return hardcfr_sibcall_search_preds (bb, chk_edges,
+					 count_chkcall, chkcall_blocks,
+					 count_postchk, postchk_blocks,
+					 retptr);
+
+  if (!is_a <gcall *> (stmt))
+    return false;
+
+  /* Avoid disrupting mandatory or early-marked tail calls,
+     inserting the check before them.  This works for
+     must-tail calls, but tail calling as an optimization is
+     detected too late for us.
+
+     Also check for noreturn calls here.  Noreturn calls won't
+     normally have edges to exit, so they won't be found here,
+     but __builtin_return does, and we must check before
+     it, so handle it like a tail call.  */
+  gcall *call = as_a <gcall *> (stmt);
+  if (!(gimple_call_noreturn_p (call)
+	|| gimple_call_must_tail_p (call)
+	|| gimple_call_tail_p (call)
+	|| (gimple_call_lhs (call) == (retptr ? *retptr : NULL)
+	    && check_returning_calls_p ())))
+    return false;
+
+  gcc_checking_assert (returning_call_p (call));
+
+  /* We found a call that is to be preceded by checking.  */
+  if (bitmap_set_bit (chkcall_blocks, bb->index))
+    ++count_chkcall;
+  else
+    gcc_unreachable ();
+  return true;
+}
+
+
+/* Search preds of BB for a mandatory or potential sibcall or
+   returning call, and arrange for the blocks containing them to have
+   a check inserted before the call, like noreturn calls.  If any
+   preds are found to perform checking, schedule checks at the edges
+   of those that don't, and mark BB as postcheck..  */
+
+static bool
+hardcfr_sibcall_search_preds (basic_block bb,
+			      chk_edges_t &chk_edges,
+			      int &count_chkcall,
+			      auto_sbitmap &chkcall_blocks,
+			      int &count_postchk,
+			      auto_sbitmap &postchk_blocks,
+			      tree *retptr)
+{
+  /* For the exit block, we wish to force a check at every
+     predecessor, so pretend we've already found a pred that had
+     checking, so that we schedule checking at every one of its pred
+     edges.  */
+  bool first = bb->index >= NUM_FIXED_BLOCKS;
+  bool postchecked = true;
+
+  gphi *retphi = NULL;
+  if (retptr && *retptr && TREE_CODE (*retptr) == SSA_NAME
+      && !SSA_NAME_IS_DEFAULT_DEF (*retptr)
+      && SSA_NAME_DEF_STMT (*retptr)
+      && is_a <gphi *> (SSA_NAME_DEF_STMT (*retptr))
+      && gimple_bb (SSA_NAME_DEF_STMT (*retptr)) == bb)
+    {
+      retphi = as_a <gphi *> (SSA_NAME_DEF_STMT (*retptr));
+      gcc_checking_assert (gimple_phi_result (retphi) == *retptr);
+    }
+
+  for (int i = EDGE_COUNT (bb->preds); i--; first = false)
+    {
+      edge e = EDGE_PRED (bb, i);
+
+      bool checked
+	= hardcfr_sibcall_search_block (e->src, chk_edges,
+					count_chkcall, chkcall_blocks,
+					count_postchk, postchk_blocks,
+					!retphi ? retptr
+					: gimple_phi_arg_def_ptr (retphi, i));
+
+      if (first)
+	{
+	  postchecked = checked;
+	  continue;
+	}
+
+      /* When we first find a checked block, force a check at every
+	 other incoming edge we've already visited, and those we
+	 visit afterwards that don't have their own check, so that
+	 when we reach BB, the check has already been performed.  */
+      if (!postchecked && checked)
+	{
+	  for (int j = EDGE_COUNT (bb->preds); --j > i; )
+	    chk_edges.safe_push (EDGE_PRED (bb, j));
+	  postchecked = true;
+	}
+      if (postchecked && !checked)
+	chk_edges.safe_push (EDGE_PRED (bb, i));
+    }
+
+  if (postchecked && bb->index >= NUM_FIXED_BLOCKS)
+    {
+      if (bitmap_set_bit (postchk_blocks, bb->index))
+	count_postchk++;
+      else
+	gcc_unreachable ();
+    }
+
+  return postchecked;
+}
+
+
 class rt_bb_visited
 {
   /* Use a sufficiently wide unsigned type to hold basic block numbers.  */
@@ -176,8 +457,8 @@ class rt_bb_visited
      neither ENTRY nor EXIT, but maybe one-past-the-end, to compute
      the visited array length.  */
   blknum num2idx (blknum n) {
-    gcc_checking_assert (n >= 2 && n <= nblocks);
-    return (n - 2);
+    gcc_checking_assert (n >= NUM_FIXED_BLOCKS && n <= nblocks);
+    return (n - NUM_FIXED_BLOCKS);
   }
   /* Return the block vindex for BB, that must not be ENTRY or
      EXIT.  */
@@ -249,8 +530,7 @@ class rt_bb_visited
   }
 
   /* Set the bit corresponding to BB in VISITED.  Add to SEQ any
-     required gimple statements, and return SEQ, possibly
-     modified.  */
+     required gimple stmts, and return SEQ, possibly modified.  */
   gimple_seq vset (basic_block bb, gimple_seq seq = NULL)
   {
     tree bit, setme = vword (bb, &bit);
@@ -270,7 +550,7 @@ class rt_bb_visited
 
 public:
   /* Prepare to add control flow redundancy testing to CFUN.  */
-  rt_bb_visited ()
+  rt_bb_visited (int checkpoints)
     : nblocks (n_basic_blocks_for_fn (cfun)),
       vword_type (NULL), ckseq (NULL), rtcfg (NULL)
   {
@@ -353,8 +633,8 @@ public:
 					 NULL, NULL);
     gimple_seq_add_stmt (&ckseq, detach);
 
-    if (nblocks - 2 > blknum (param_hardcfr_max_inline_blocks)
-	|| !single_pred_p (EXIT_BLOCK_PTR_FOR_FN (cfun)))
+    if (nblocks - NUM_FIXED_BLOCKS > blknum (param_hardcfr_max_inline_blocks)
+	|| checkpoints > 1)
       {
 	/* Make sure vword_bits is wide enough for the representation
 	   of nblocks in rtcfg.  Compare with vword_bits << vword_bits,
@@ -379,65 +659,34 @@ public:
     gimple_seq_add_stmt (&ckseq, ckfail_init);
   }
 
-  /* Insert SEQ on E, or close enough (e.g., before a noreturn or tail
-     call at the end of E->src).  */
-  void insert_exit_check (gimple_seq seq, edge e)
+  /* Insert SEQ before a resx or a call in INSBB.  */
+  void insert_exit_check_in_block (gimple_seq seq, basic_block insbb)
   {
-    basic_block insbb = e->src;
-
-    /* If the returning block ends with a noreturn call, insert
-       checking before it.  This is particularly important for
-       __builtin_return.  Other noreturn calls won't have an edge to
-       the exit block, so they won't even be considered as exit paths.
-
-       Insert-on-edge inserts before other return stmts, but not
-       before calls, and if a single-block function had the check
-       sequence inserted after a noreturn call, it would be useless,
-       but checking would still flag it as malformed if block 2 has a
-       fallthru edge to the exit block.
-
-       Also avoid disrupting tail calls, inserting the check before
-       them.  This works for must-tail calls, but tail calling as an
-       optimization is detected too late for us.  */
     gimple_stmt_iterator gsi = gsi_last_bb (insbb);
-    gimple *ret = gsi_stmt (gsi);
-    if (ret && is_a <greturn *> (ret))
-      {
+
+    while (!gsi_end_p (gsi))
+      if (is_a <gresx *> (gsi_stmt (gsi))
+	  || is_a <gcall *> (gsi_stmt (gsi)))
+	break;
+      else
 	gsi_prev (&gsi);
-	if (!gsi_end_p (gsi))
-	  ret = gsi_stmt (gsi);
-      }
-    if (ret && is_a <gcall *> (ret)
-	&& (gimple_call_noreturn_p (ret)
-	    || gimple_call_must_tail_p (as_a <gcall *> (ret))
-	    || gimple_call_tail_p (as_a <gcall *> (ret))))
-      gsi_insert_seq_before (&gsi, seq, GSI_SAME_STMT);
-    else
-      gsi_insert_seq_on_edge_immediate (e, seq);
+
+    gsi_insert_seq_before (&gsi, seq, GSI_SAME_STMT);
   }
 
-  /* Add checking code on every exit edge, and initialization code on
-     the entry edge.  Before this point, the CFG has been undisturbed,
-     and all the needed data has been collected and safely stowed.  */
-  void check ()
+  /* Insert SEQ on E.  */
+  void insert_exit_check_on_edge (gimple_seq seq, edge e)
   {
-    /* Insert initializers for visited at the entry.  */
-    gimple_seq iseq = NULL;
-
-    gcall *vinit = gimple_build_call (builtin_decl_explicit
-				      (BUILT_IN_MEMSET), 3,
-				      build1 (ADDR_EXPR,
-					      build_pointer_type
-					      (TREE_TYPE (visited)),
-					      visited),
-				      integer_zero_node,
-				      TYPE_SIZE_UNIT (TREE_TYPE (visited)));
-    gimple_seq_add_stmt (&iseq, vinit);
-
-    gsi_insert_seq_on_edge_immediate (single_succ_edge
-				      (ENTRY_BLOCK_PTR_FOR_FN (cfun)),
-				      iseq);
+    gsi_insert_seq_on_edge_immediate (e, seq);
+  }
 
+  /* Add checking code to CHK_EDGES and CHKCALL_BLOCKS, and
+     initialization code on the entry edge.  Before this point, the
+     CFG has been undisturbed, and all the needed data has been
+     collected and safely stowed.  */
+  void check (chk_edges_t &chk_edges,
+	      int count_chkcall, auto_sbitmap const &chkcall_blocks)
+  {
     /* If we're using out-of-line checking, create and statically
        initialize the CFG checking representation, generate the
        checker call for the checking sequence, and insert it in all
@@ -498,28 +747,116 @@ public:
 						     rtcfg));
 	gimple_seq_add_stmt (&ckseq, call_chk);
 
+	gimple *clobber = gimple_build_assign (visited,
+					       build_clobber
+					       (TREE_TYPE (visited)));
+	gimple_seq_add_stmt (&ckseq, clobber);
+
 	/* If we have multiple exit edges, insert (copies of)
 	   ckseq in all of them.  */
-	for (int i = EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (cfun)->preds);
-	     i--; )
+	for (int i = chk_edges.length (); i--; )
 	  {
 	    gimple_seq seq = ckseq;
 	    /* Copy the sequence, unless we're dealing with the
 	       last edge (we're counting down to zero).  */
-	    if (i)
+	    if (i || count_chkcall)
+	      seq = gimple_seq_copy (seq);
+
+	    edge e = chk_edges[i];
+
+	    if (dump_file)
+	      {
+		if (e->dest == EXIT_BLOCK_PTR_FOR_FN (cfun))
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to exit.\n",
+			   e->src->index);
+		else
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to postcheck block %i.\n",
+			   e->src->index, e->dest->index);
+	      }
+
+	    insert_exit_check_on_edge (seq, e);
+
+	    gcc_checking_assert (!bitmap_bit_p (chkcall_blocks, e->src->index));
+	  }
+
+	sbitmap_iterator it;
+	unsigned i;
+	EXECUTE_IF_SET_IN_BITMAP (chkcall_blocks, 0, i, it)
+	  {
+	    basic_block bb = BASIC_BLOCK_FOR_FN (cfun, i);
+
+	    gimple_seq seq = ckseq;
+	    gcc_checking_assert (count_chkcall > 0);
+	    if (--count_chkcall)
 	      seq = gimple_seq_copy (seq);
 
-	    insert_exit_check (seq,
-			       EDGE_PRED (EXIT_BLOCK_PTR_FOR_FN (cfun), i));
+	    if (dump_file)
+	      fprintf (dump_file,
+		       "Inserting out-of-line check before stmt in block %i.\n",
+		       bb->index);
+
+	    insert_exit_check_in_block (seq, bb);
 	  }
+
+	gcc_checking_assert (count_chkcall == 0);
       }
     else
       {
 	/* Inline checking requires a single exit edge.  */
-	gimple *last = gsi_stmt (gsi_last (ckseq));
+	gimple *last = gimple_build_assign (visited,
+					    build_clobber
+					    (TREE_TYPE (visited)));
+	gimple_seq_add_stmt (&ckseq, last);
 
-	insert_exit_check (ckseq,
-			   single_pred_edge (EXIT_BLOCK_PTR_FOR_FN (cfun)));
+	if (!count_chkcall)
+	  {
+	    edge e = single_pred_edge (EXIT_BLOCK_PTR_FOR_FN (cfun));
+
+	    if (dump_file)
+	      {
+		if (e->dest == EXIT_BLOCK_PTR_FOR_FN (cfun))
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to postcheck block %i.\n",
+			   e->src->index, e->dest->index);
+		else
+		  fprintf (dump_file,
+			   "Inserting inline check in"
+			   " block %i's edge to exit.\n",
+			   e->src->index);
+	      }
+
+	    insert_exit_check_on_edge (ckseq, e);
+	  }
+	else
+	  {
+	    gcc_checking_assert (count_chkcall == 1);
+
+	    sbitmap_iterator it;
+	    unsigned i;
+	    EXECUTE_IF_SET_IN_BITMAP (chkcall_blocks, 0, i, it)
+	      {
+		basic_block bb = BASIC_BLOCK_FOR_FN (cfun, i);
+
+		gimple_seq seq = ckseq;
+		gcc_checking_assert (count_chkcall > 0);
+		if (--count_chkcall)
+		  seq = gimple_seq_copy (seq);
+
+		if (dump_file)
+		  fprintf (dump_file,
+			   "Inserting inline check before stmt in block %i.\n",
+			   bb->index);
+
+		insert_exit_check_in_block (seq, bb);
+	      }
+
+	    gcc_checking_assert (count_chkcall == 0);
+	  }
 
 	/* The inserted ckseq computes CKFAIL at LAST.  Now we have to
 	   conditionally trap on it.  */
@@ -540,8 +877,7 @@ public:
 	  add_bb_to_loop (trp, current_loops->tree_root);
 
 	/* Insert a conditional branch to the trap block.  If the
-	   conditional wouldn't be the last statement, split the
-	   block.  */
+	   conditional wouldn't be the last stmt, split the block.  */
 	gimple_stmt_iterator gsi = gsi_for_stmt (last);
 	if (!gsi_one_before_end_p (gsi))
 	  split_block (gsi_bb (gsi), gsi_stmt (gsi));
@@ -564,6 +900,24 @@ public:
 	if (dom_info_available_p (CDI_DOMINATORS))
 	  set_immediate_dominator (CDI_DOMINATORS, trp, gimple_bb (last));
       }
+
+    /* Insert initializers for visited at the entry.  Do this after
+       other insertions, to avoid messing with block numbers.  */
+    gimple_seq iseq = NULL;
+
+    gcall *vinit = gimple_build_call (builtin_decl_explicit
+				      (BUILT_IN_MEMSET), 3,
+				      build1 (ADDR_EXPR,
+					      build_pointer_type
+					      (TREE_TYPE (visited)),
+					      visited),
+				      integer_zero_node,
+				      TYPE_SIZE_UNIT (TREE_TYPE (visited)));
+    gimple_seq_add_stmt (&iseq, vinit);
+
+    gsi_insert_seq_on_edge_immediate (single_succ_edge
+				      (ENTRY_BLOCK_PTR_FOR_FN (cfun)),
+				      iseq);
   }
 
   /* Push onto RTCFG a (mask, index) pair to test for IBB when BB is
@@ -607,7 +961,7 @@ public:
     return false;
   }
 
-  /* Add to CKSEQ statements to clear CKPART if OBB is visited.  */
+  /* Add to CKSEQ stmts to clear CKPART if OBB is visited.  */
   void
   build_block_check (basic_block obb)
   {
@@ -632,34 +986,48 @@ public:
 
   /* Add to BB code to set its bit in VISITED, and add to RTCFG or
      CKSEQ the data or code needed to check BB's predecessors and
-     successors.  Do NOT change the CFG.  */
-  void visit (basic_block bb)
+     successors.  If CHECKPOINT, assume the block is a checkpoint,
+     whether or not it has an edge to EXIT.  If POSTCHECK, assume the
+     block post-dominates checkpoints and therefore no bitmap setting
+     or checks are to be performed in or for it.  Do NOT change the
+     CFG.  */
+  void visit (basic_block bb, bool checkpoint, bool postcheck)
   {
     /* Set the bit in VISITED when entering the block.  */
     gimple_stmt_iterator gsi = gsi_after_labels (bb);
-    gsi_insert_seq_before (&gsi, vset (bb), GSI_SAME_STMT);
+    if (!postcheck)
+      gsi_insert_seq_before (&gsi, vset (bb), GSI_SAME_STMT);
 
     if (rtcfg)
       {
-	/* Build a list of (index, mask) terminated by (NULL, 0).
-	   Consolidate masks with the same index when they're
-	   adjacent.  First, predecessors.  Count backwards, because
-	   we're going to reverse the list.  The order shouldn't
-	   matter, but let's not make it surprising.  */
-	for (int i = EDGE_COUNT (bb->preds); i--; )
-	  if (push_rtcfg_pair (EDGE_PRED (bb, i)->src, bb,
-			       ENTRY_BLOCK_PTR_FOR_FN (cfun)))
-	    break;
+	if (!postcheck)
+	  {
+	    /* Build a list of (index, mask) terminated by (NULL, 0).
+	       Consolidate masks with the same index when they're
+	       adjacent.  First, predecessors.  Count backwards, because
+	       we're going to reverse the list.  The order shouldn't
+	       matter, but let's not make it surprising.  */
+	    for (int i = EDGE_COUNT (bb->preds); i--; )
+	      if (push_rtcfg_pair (EDGE_PRED (bb, i)->src, bb,
+				   ENTRY_BLOCK_PTR_FOR_FN (cfun)))
+		break;
+	  }
 	rtcfg = tree_cons (NULL_TREE, build_int_cst (vword_type, 0), rtcfg);
 
-	/* Then, successors.  */
-	for (int i = EDGE_COUNT (bb->succs); i--; )
-	  if (push_rtcfg_pair (EDGE_SUCC (bb, i)->dest, bb,
-			       EXIT_BLOCK_PTR_FOR_FN (cfun)))
-	    break;
+	if (!postcheck)
+	  {
+	    /* Then, successors.  */
+	    if (!checkpoint
+		|| !push_rtcfg_pair (EXIT_BLOCK_PTR_FOR_FN (cfun),
+				     bb, EXIT_BLOCK_PTR_FOR_FN (cfun)))
+	      for (int i = EDGE_COUNT (bb->succs); i--; )
+		if (push_rtcfg_pair (EDGE_SUCC (bb, i)->dest, bb,
+				     EXIT_BLOCK_PTR_FOR_FN (cfun)))
+		  break;
+	  }
 	rtcfg = tree_cons (NULL_TREE, build_int_cst (vword_type, 0), rtcfg);
       }
-    else
+    else if (!postcheck)
       {
 	/* Schedule test to fail if the block was reached but somehow none
 	   of its predecessors were.  */
@@ -677,6 +1045,8 @@ public:
 	gassign *blkruns = gimple_build_assign (ckpart, unshare_expr (bit));
 	gimple_seq_add_stmt (&ckseq, blkruns);
 
+	if (checkpoint)
+	  build_block_check (EXIT_BLOCK_PTR_FOR_FN (cfun));
 	for (int i = 0, e = EDGE_COUNT (bb->succs); i < e; i++)
 	  build_block_check (EDGE_SUCC (bb, i)->dest);
 
@@ -687,21 +1057,357 @@ public:
   }
 };
 
+/* It might be useful to avoid checking before noreturn calls that are
+   known to always finish by throwing an exception, rather than by
+   ending the program or looping forever.  Such functions would have
+   to be annotated somehow, with an attribute or flag, so that
+   exception-raising functions, such as C++'s __cxa_throw,
+   __cxa_rethrow, and Ada's gnat_rcheck_*, gnat_reraise*,
+   ada.exception.raise_exception*, and the language-independent
+   unwinders could be detected here and handled differently from other
+   noreturn functions.  */
+static bool
+always_throwing_noreturn_call_p (gimple *)
+{
+  return false;
+}
+
 /* Control flow redundancy hardening: record the execution path, and
    verify at exit that an expect path was taken.  */
 
 unsigned int
-pass_harden_control_flow_redundancy::execute (function *)
+pass_harden_control_flow_redundancy::execute (function *fun)
 {
-  rt_bb_visited vstd;
-
+  bool const check_at_escaping_exceptions
+    = (flag_exceptions
+       && flag_harden_control_flow_redundancy_check_exceptions);
+  bool const check_before_noreturn_calls
+    = flag_harden_control_flow_redundancy_check_noreturn > HCFRNR_NEVER;
+  bool const check_before_nothrow_noreturn_calls
+    = (check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn >= HCFRNR_NOTHROW);
+  bool const check_before_throwing_noreturn_calls
+    = (flag_exceptions
+       && check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn > HCFRNR_NOTHROW);
+  bool const check_before_always_throwing_noreturn_calls
+    = (flag_exceptions
+       && check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn >= HCFRNR_ALWAYS);
   basic_block bb;
-  FOR_EACH_BB_FN (bb, cfun)
-    vstd.visit (bb);
+  basic_block bb_eh_cleanup = NULL;
+
+  if (check_at_escaping_exceptions)
+    {
+      int lp_eh_cleanup = -1;
+
+      /* Record the preexisting blocks, to avoid visiting newly-created
+	 blocks.  */
+      auto_sbitmap to_visit (last_basic_block_for_fn (fun));
+      bitmap_clear (to_visit);
+
+      FOR_EACH_BB_FN (bb, fun)
+	bitmap_set_bit (to_visit, bb->index);
+
+      /* Scan the blocks for stmts with escaping exceptions, that
+	 wouldn't be denoted in the CFG, and associate them with an
+	 empty cleanup handler around the whole function.  Walk
+	 backwards, so that even when we split the block, */
+      sbitmap_iterator it;
+      unsigned i;
+      EXECUTE_IF_SET_IN_BITMAP (to_visit, 0, i, it)
+	{
+	  bb = BASIC_BLOCK_FOR_FN (fun, i);
+
+	  for (gimple_stmt_iterator gsi = gsi_last_bb (bb);
+	       !gsi_end_p (gsi); gsi_prev (&gsi))
+	    {
+	      gimple *stmt = gsi_stmt (gsi);
+	      if (!stmt_could_throw_p (fun, stmt))
+		continue;
+
+	      /* If it must not throw, or if it already has a handler,
+		 we need not worry about it.  */
+	      if (lookup_stmt_eh_lp (stmt) != 0)
+		continue;
+
+	      /* Don't split blocks at, nor add EH edges to, tail
+		 calls, we will add verification before the call
+		 anyway.  */
+	      if (is_a <gcall *> (stmt)
+		  && (gimple_call_must_tail_p (as_a <gcall *> (stmt))
+		      || gimple_call_tail_p (as_a <gcall *> (stmt))
+		      || returning_call_p (as_a <gcall *> (stmt))))
+		continue;
+
+	      if (!gsi_one_before_end_p (gsi))
+		split_block (bb, stmt);
+	      /* A resx or noreturn call needs not be associated with
+		 the cleanup handler if we're going to add checking
+		 before it.  We only test cases that didn't require
+		 block splitting because noreturn calls would always
+		 be at the end of blocks, and we test for zero
+		 successors because if there is an edge, it's not
+		 noreturn, as any EH edges would have already been
+		 caught by the lookup_stmt_eh_lp test above.  */
+	      else if (check_before_noreturn_calls
+		       && EDGE_COUNT (bb->succs) == 0
+		       && (is_a <gresx *> (stmt)
+			   ? check_before_always_throwing_noreturn_calls
+			   : (!is_a <gcall *> (stmt)
+			      || !gimple_call_noreturn_p (stmt))
+			   ? (gcc_unreachable (), false)
+			   : (!flag_exceptions
+			      || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+			   ? check_before_nothrow_noreturn_calls
+			   : always_throwing_noreturn_call_p (stmt)
+			   ? check_before_always_throwing_noreturn_calls
+			   : check_before_throwing_noreturn_calls))
+		{
+		  if (dump_file)
+		    {
+		      fprintf (dump_file,
+			       "Bypassing cleanup for noreturn stmt"
+			       " in block %i:\n",
+			       bb->index);
+		      print_gimple_stmt (dump_file, stmt, 0);
+		    }
+		  continue;
+		}
+
+	      if (!bb_eh_cleanup)
+		{
+		  bb_eh_cleanup = create_empty_bb (bb);
+		  if (dom_info_available_p (CDI_DOMINATORS))
+		    set_immediate_dominator (CDI_DOMINATORS, bb_eh_cleanup, bb);
+		  if (current_loops)
+		    add_bb_to_loop (bb_eh_cleanup, current_loops->tree_root);
+
+		  /* Make the new block an EH cleanup for the call.  */
+		  eh_region new_r = gen_eh_region_cleanup (NULL);
+		  eh_landing_pad lp = gen_eh_landing_pad (new_r);
+		  tree label = gimple_block_label (bb_eh_cleanup);
+		  lp->post_landing_pad = label;
+		  EH_LANDING_PAD_NR (label) = lp_eh_cleanup = lp->index;
+
+		  /* Just propagate the exception.
+		     We will later insert the verifier call.  */
+		  gimple_stmt_iterator ehgsi;
+		  ehgsi = gsi_after_labels (bb_eh_cleanup);
+		  gresx *resx = gimple_build_resx (new_r->index);
+		  gsi_insert_before (&ehgsi, resx, GSI_SAME_STMT);
+
+		  if (dump_file)
+		    fprintf (dump_file,
+			     "Created cleanup block %i:\n",
+			     bb_eh_cleanup->index);
+		}
+	      else if (dom_info_available_p (CDI_DOMINATORS))
+		{
+		  basic_block immdom;
+		  immdom = get_immediate_dominator (CDI_DOMINATORS,
+						    bb_eh_cleanup);
+		  if (!dominated_by_p (CDI_DOMINATORS, bb, immdom))
+		    {
+		      immdom = nearest_common_dominator (CDI_DOMINATORS,
+							 immdom, bb);
+		      set_immediate_dominator (CDI_DOMINATORS,
+					       bb_eh_cleanup, immdom);
+		    }
+		}
+
+	      if (dump_file)
+		{
+		  fprintf (dump_file,
+			   "Associated cleanup block with stmt in block %i:\n",
+			   bb->index);
+		  print_gimple_stmt (dump_file, stmt, 0);
+		}
+
+	      add_stmt_to_eh_lp (stmt, lp_eh_cleanup);
+	      /* Finally, wire the EH cleanup block into the CFG.  */
+	      make_eh_edges (stmt);
+	    }
+	}
+
+      if (bb_eh_cleanup)
+	{
+	  /* A cfg_cleanup after bb_eh_cleanup makes for a more compact
+	     rtcfg, and it avoids bb numbering differences when we split
+	     blocks because of trailing debug insns only.  */
+	  cleanup_tree_cfg ();
+	  gcc_checking_assert (EDGE_COUNT (bb_eh_cleanup->succs) == 0);
+	}
+    }
+
+  /* These record blocks with calls that are to be preceded by
+     checkpoints, such as noreturn calls (if so chosen), must-tail
+     calls, potential early-marked tail calls, and returning calls (if
+     so chosen).  */
+  int count_chkcall = 0;
+  auto_sbitmap chkcall_blocks (last_basic_block_for_fn (fun));
+  bitmap_clear (chkcall_blocks);
+
+  /* We wish to add verification at blocks without successors, such as
+     noreturn calls (raising or not) and the reraise at the cleanup
+     block, but not other reraises: they will go through the cleanup
+     block.  */
+  if (check_before_noreturn_calls)
+    FOR_EACH_BB_FN (bb, fun)
+      {
+	gimple_stmt_iterator gsi = gsi_last_bb (bb);
+	if (gsi_end_p (gsi))
+	  continue;
+	gimple *stmt = gsi_stmt (gsi);
 
-  vstd.check ();
+	if (EDGE_COUNT (bb->succs) == 0)
+	  {
+	    /* A stmt at the end of a block without any successors is
+	       either a resx or a noreturn call without a local
+	       handler.  Check that it's one of the desired
+	       checkpoints.  */
+	    if (flag_exceptions && is_a <gresx *> (stmt)
+		? (check_before_always_throwing_noreturn_calls
+		   || bb == bb_eh_cleanup)
+		: (!is_a <gcall *> (stmt)
+		   || !gimple_call_noreturn_p (stmt))
+		? (/* Catch cases in which successors would be
+		      expected.  */
+		   gcc_unreachable (), false)
+		: (!flag_exceptions
+		   || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+		? check_before_nothrow_noreturn_calls
+		: always_throwing_noreturn_call_p (stmt)
+		? check_before_always_throwing_noreturn_calls
+		: check_before_throwing_noreturn_calls)
+	      {
+		if (dump_file)
+		  {
+		    fprintf (dump_file,
+			     "Scheduling check before stmt"
+			     " in succ-less block %i:\n",
+			     bb->index);
+		    print_gimple_stmt (dump_file, stmt, 0);
+		  }
+
+		if (bitmap_set_bit (chkcall_blocks, bb->index))
+		  count_chkcall++;
+		else
+		  gcc_unreachable ();
+	      }
+	    continue;
+	  }
 
-  return 0;
+	/* If there are no exceptions, it would seem like any noreturn
+	   call must have zero successor edges, but __builtin_return
+	   gets successor edges.  We don't want to handle it here, it
+	   will be dealt with in sibcall_search_preds.  Otherwise,
+	   check for blocks without non-EH successors, but skip those
+	   with resx stmts and edges (i.e., those other than that in
+	   bb_eh_cleanup), since those will go through bb_eh_cleanup,
+	   that will have been counted as noreturn above because it
+	   has no successors.  */
+	gcc_checking_assert (bb != bb_eh_cleanup
+			     || !check_at_escaping_exceptions);
+	if (flag_exceptions && is_a <gresx *> (stmt)
+	    ? check_before_always_throwing_noreturn_calls
+	    : (!is_a <gcall *> (stmt)
+	       || !gimple_call_noreturn_p (stmt))
+	    ? false
+	    : (!flag_exceptions
+	       || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+	    ? false /* rather than check_before_nothrow_noreturn_calls */
+	    : always_throwing_noreturn_call_p (stmt)
+	    ? check_before_always_throwing_noreturn_calls
+	    : check_before_throwing_noreturn_calls)
+	  {
+	    gcc_checking_assert (single_succ_p (bb)
+				 && (single_succ_edge (bb)->flags & EDGE_EH));
+
+	    if (dump_file)
+	      {
+		fprintf (dump_file,
+			 "Scheduling check before stmt"
+			 " in EH-succ block %i:\n",
+			 bb->index);
+		print_gimple_stmt (dump_file, stmt, 0);
+	      }
+
+	    if (bitmap_set_bit (chkcall_blocks, bb->index))
+	      count_chkcall++;
+	    else
+	      gcc_unreachable ();
+	  }
+      }
+  else if (bb_eh_cleanup)
+    {
+      if (bitmap_set_bit (chkcall_blocks, bb_eh_cleanup->index))
+	count_chkcall++;
+      else
+	gcc_unreachable ();
+    }
+
+  gcc_checking_assert (!bb_eh_cleanup
+		       || bitmap_bit_p (chkcall_blocks, bb_eh_cleanup->index));
+
+  /* If we don't have edges to exit nor noreturn calls (including the
+     cleanup reraise), then we may skip instrumentation: that would
+     amount to a function that ends with an infinite loop.  */
+  if (!count_chkcall
+      && EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (fun)->preds) == 0)
+    {
+      if (dump_file)
+	fprintf (dump_file,
+		 "Disabling CFR, no exit paths to check\n");
+
+      return 0;
+    }
+
+  /* Search for must-tail calls, early-marked potential tail calls,
+     and, if requested, returning calls.  As we introduce early
+     checks, */
+  int count_postchk = 0;
+  auto_sbitmap postchk_blocks (last_basic_block_for_fn (fun));
+  bitmap_clear (postchk_blocks);
+  chk_edges_t chk_edges;
+  hardcfr_sibcall_search_preds (EXIT_BLOCK_PTR_FOR_FN (fun), chk_edges,
+				count_chkcall, chkcall_blocks,
+				count_postchk, postchk_blocks,
+				NULL);
+
+  rt_bb_visited vstd (chk_edges.length () + count_chkcall);
+
+  auto_sbitmap combined_blocks (last_basic_block_for_fn (fun));
+  bitmap_copy (combined_blocks, chkcall_blocks);
+  int i;
+  edge *e;
+  FOR_EACH_VEC_ELT (chk_edges, i, e)
+    if (!bitmap_set_bit (combined_blocks, (*e)->src->index))
+      /* There may be multiple chk_edges with the same src block;
+	 guard againt overlaps with chkcall_blocks only.  */
+      gcc_assert (!bitmap_bit_p (chkcall_blocks, (*e)->src->index));
+
+  /* Visit blocks in index order, because building rtcfg depends on
+     that.  Blocks must be compact, which the cleanup_cfg requirement
+     ensures.  This would also enable FOR_EACH_BB_FN to be used to
+     iterate in index order, but bb_eh_cleanup block splits and
+     insertions changes that.  */
+  gcc_checking_assert (n_basic_blocks_for_fn (fun)
+		       == last_basic_block_for_fn (fun));
+  for (int i = NUM_FIXED_BLOCKS; i < n_basic_blocks_for_fn (fun); i++)
+    {
+      bb = BASIC_BLOCK_FOR_FN (fun, i);
+      gcc_checking_assert (bb->index == i);
+      vstd.visit (bb, bitmap_bit_p (combined_blocks, i),
+		  bitmap_bit_p (postchk_blocks, i));
+    }
+
+  vstd.check (chk_edges, count_chkcall, chkcall_blocks);
+
+  return
+    TODO_update_ssa
+    | TODO_cleanup_cfg
+    | TODO_verify_il;
 }
 
 /* Instantiate a hardcfr pass.  */
diff --git a/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c b/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c
new file mode 100644
index 00000000000..a6992eb9f8e
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -O0 -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we don't insert checking before noreturn calls.  -O0 is tested
+   separately because h is not found to be noreturn without optimization.  */
+
+#include "torture/harden-cfr-noret.c"
+
+/* No out-of-line checks.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* Only one inline check at the end of f and of h2.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-always.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-always.c
new file mode 100644
index 00000000000..779896c60e8
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-always.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even enabling all checks before noreturn calls (leaving
+   returning calls enabled), we get checks before __builtin_return without
+   duplication (__builtin_return is both noreturn and a returning call).  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-never.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-never.c
new file mode 100644
index 00000000000..49ce17f5b93
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-never.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even enabling checks before never noreturn calls (leaving
+   returning calls enabled), we get checks before __builtin_return without
+   duplication (__builtin_return is both noreturn and a returning call).  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noopt.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noopt.c
new file mode 100644
index 00000000000..1512614791f
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noopt.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even disabling checks before both noreturn and returning
+   calls, we still get checks before __builtin_return.  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noret.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noret.c
new file mode 100644
index 00000000000..fd95bb7e3e3
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noret.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even disabling checks before returning calls (leaving noreturn
+   calls enabled), we still get checks before __builtin_return.  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-nothrow.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-nothrow.c
new file mode 100644
index 00000000000..c5c361234c4
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-nothrow.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even enabling checks before nothrow noreturn calls (leaving
+   returning calls enabled), we get checks before __builtin_return without
+   duplication (__builtin_return is both noreturn and a returning call).  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-retcl.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-retcl.c
new file mode 100644
index 00000000000..137dfbb95d6
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-retcl.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even disabling checks before noreturn calls (leaving returning
+   calls enabled), we still get checks before __builtin_return.  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c
index 70acdc95f25..b459ff6b864 100644
--- a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c
@@ -7,5 +7,11 @@ int f(int i) {
   return i;
 }
 
-/* Out-of-line checking, before both builtin_return and return.  */
+int g(int i) {
+  __builtin_return (&i);
+}
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
 /* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c
new file mode 100644
index 00000000000..8bd2d13ac18
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c
@@ -0,0 +1,18 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we don't insert checking before noreturn calls.  -O0 is tested
+   separately because h is not found to be noreturn without optimization, which
+   affects codegen for h2, so h2 is omitted here at -O0.  */
+
+#if !__OPTIMIZE__
+# define OMIT_H2
+#endif
+
+#include "harden-cfr-noret.c"
+
+
+/* No out-of-line checks.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* Only one inline check at the end of f.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c
new file mode 100644
index 00000000000..a804a6cfe59
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fno-exceptions -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that -fno-exceptions makes for implicit nothrow in noreturn
+   handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#include "harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c
new file mode 100644
index 00000000000..f390cfdbc59
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checking before nothrow noreturn calls.  */
+
+#include "harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c
new file mode 100644
index 00000000000..fdd803109a4
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c
@@ -0,0 +1,38 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checking before all noreturn calls.  */
+
+#ifndef ATTR_NOTHROW_OPT /* Overridden in harden-cfr-noret-noexcept.  */
+#define ATTR_NOTHROW_OPT __attribute__ ((__nothrow__))
+#endif
+
+extern void __attribute__ ((__noreturn__)) ATTR_NOTHROW_OPT g (void);
+
+void f(int i) {
+  if (i)
+    /* Out-of-line checks here...  */
+    g ();
+  /* ... and here.  */
+}
+
+void __attribute__ ((__noinline__, __noclone__))
+h(void) {
+  /* Inline check here.  */
+  g ();
+}
+
+#ifndef OMIT_H2 /* from harden-cfr-noret-never.  */
+void h2(void) {
+  /* Inline check either here, whether because of noreturn or tail call...  */
+  h ();
+  /* ... or here, if not optimizing.  */
+}
+#endif
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c
new file mode 100644
index 00000000000..6d11487bbba
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c
@@ -0,0 +1,8 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+#include "harden-cfr-tail.c"
+
+/* Inline checking after the calls, disabling tail calling.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 5 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 0 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c
new file mode 100644
index 00000000000..550b02ca088
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c
@@ -0,0 +1,35 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-exceptions -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checks before returning calls and alternate paths, even
+   at -O0, because of the explicit command-line flag.  */
+
+void g (void);
+void g2 (void);
+void g3 (void);
+
+void f (int i) {
+  if (!i)
+    /* Out-of-line checks here...  */
+    g ();
+  else if (i > 0)
+    /* here...  */
+    g2 ();
+  /* else */
+    /* and in the implicit else here.  */
+}
+
+void f2 (int i) {
+  if (!i)
+    /* Out-of-line check here...  */
+    g ();
+  else if (i > 0)
+    /* here...  */
+    g2 ();
+  else
+    /* and here.  */
+    g3 ();
+}
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 6 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 0 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
index 40d76c5c163..d5467eafa9f 100644
--- a/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
@@ -1,17 +1,52 @@
 /* { dg-do compile } */
-/* { dg-options "-fharden-control-flow-redundancy -fdump-tree-hardcfr -ffat-lto-objects" } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-hardcfr-check-exceptions -fdump-tree-hardcfr -ffat-lto-objects -Wno-return-type" } */
 
-/* We'd like to check that we insert checking so as to not disrupt tail calls,
-   but unfortunately mandatory tail calls are not available in C, and optimizing
-   calls as tail calls only takes place after hardcfr.  */
+/* Check that we insert CFR checking so as to not disrupt tail calls.
+   Mandatory tail calls are not available in C, and optimizing calls as tail
+   calls only takes place after hardcfr, so we insert checking before calls
+   followed by copies and return stmts with the same return value, that might
+   (or might not) end up optimized to tail calls.  */
 
-extern int g(int i);
+extern int g (int i);
 
-int f(int i) {
+int f1(int i) {
+  /* Inline check before the returning call.  */
   return g (i);
 }
 
-/* Inline checking before the tail call.  */
-/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
-/* Inline checking before the tail call.  */
-/* { dg-final { scan-tree-dump-times "\\\[tail\]" 1 "hardcfr" { xfail *-*-* } } } */
+extern void g2 (int i);
+
+void f2(int i) {
+  /* Inline check before the returning call, that ignores the returned value,
+     matching the value-less return.  */
+  g2 (i);
+  return;
+}
+
+void f3(int i) {
+  /* Inline check before the returning call.  */
+  g (i);
+}
+
+void f4(int i) {
+  if (i)
+    /* Out-of-line check before the returning call.  */
+    return g2 (i);
+  /* Out-of-line check before implicit return.  */
+}
+
+int f5(int i) {
+  /* Not regarded as a returning call, returning value other than callee's
+     returned value.  */
+  g (i);
+  /* Inline check after the non-returning call.  */
+  return i;
+}
+
+/* Out-of-line checks in f4, before returning calls and before return.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking in all other functions.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 4 "hardcfr" } } */
+/* Check before tail-call in all but f5, but f4 is out-of-line.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 3 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C b/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C
new file mode 100644
index 00000000000..17ea79f7cfb
--- /dev/null
+++ b/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C
@@ -0,0 +1,11 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects -O0" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions, and also checking before noreturn
+   calls.  h2 and h2b get an extra resx without ehcleanup.  */
+
+#include "torture/harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 16 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C b/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C
new file mode 100644
index 00000000000..f0338ccc361
--- /dev/null
+++ b/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C
@@ -0,0 +1,10 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -foptimize-sibling-calls -fdump-tree-hardcfr -O0" } */
+
+/* -fhardcfr-check-returning-calls gets implicitly disabled because,
+   -at O0, -foptimize-sibling-calls has no effect.  */
+
+#include "torture/harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 12 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C
new file mode 100644
index 00000000000..0d35920c7ee
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  */
+
+#include "harden-cfr-noret-no-nothrow.C"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Bypassing cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C
new file mode 100644
index 00000000000..b7d247ff43c
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C
@@ -0,0 +1,18 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  Expected results for =never and =nothrow are the same,
+   since the functions are not nothrow.  */
+
+#include "harden-cfr-noret-no-nothrow.C"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Associated cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C
new file mode 100644
index 00000000000..62c58cfd406
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C
@@ -0,0 +1,23 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#if ! __OPTIMIZE__
+void __attribute__ ((__noreturn__)) h (void);
+#endif
+
+#include "../../c-c++-common/torture/harden-cfr-noret.c"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Associated cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C
new file mode 100644
index 00000000000..0286f6e6d3f
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C
@@ -0,0 +1,20 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions, and also checking before noreturn
+   calls.  */
+
+#if ! __OPTIMIZE__
+/* Without optimization, functions with cleanups end up with an extra
+   resx that is not optimized out, so arrange to optimize them.  */
+void __attribute__ ((__optimize__ (1))) h2(void);
+void __attribute__ ((__optimize__ (1))) h2b(void);
+#endif
+
+#include "harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 14 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
+/* h, h2, h2b, and h4.  */
+/* { dg-final { scan-tree-dump-times "Bypassing" 4 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C
new file mode 100644
index 00000000000..885b0b236af
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C
@@ -0,0 +1,11 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions  -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we do not insert cleanups for checking around the bodies
+   of maybe-throwing functions.  h4 doesn't get any checks, because we
+   don't have noreturn checking enabled.  */
+
+#include "harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 6 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C
new file mode 100644
index 00000000000..32def637255
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C
@@ -0,0 +1,31 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -foptimize-sibling-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions.  These results depend on checking before
+   returning calls, which is only enabled when sibcall optimizations
+   are enabled, so change the optimization mode to -O1 for f and f2,
+   so that -foptimize-sibling-calls can take effect and enable
+   -fhardcfr-check-returning-calls, so that we get the same results.
+   There is a separate test for -O0.  */
+
+#if ! __OPTIMIZE__
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) f(int i);
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) f2(int i);
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) h3(void);
+#endif
+
+#include "harden-cfr-throw.C"
+
+/* f gets out-of-line checks before the unwrapped tail call and in the
+   else edge.  */
+/* f2 gets out-of-line checks before both unwrapped tail calls.  */
+/* h gets out-of-line checks before the implicit return and in the
+   cleanup block.  */
+/* h2 and h2b get out-of-line checks before the cleanup returning
+   call, and in the cleanup block.  */
+/* h3 gets an inline check before the __cxa_end_catch returning call.  */
+/* h4 gets an inline check in the cleanup block.  */
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 10 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C
new file mode 100644
index 00000000000..992fbdad381
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C
@@ -0,0 +1,65 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions.  */
+
+extern void g (void);
+extern void g2 (void);
+
+void f(int i) {
+  if (i)
+    g ();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void f2(int i) {
+  if (i)
+    g ();
+  else
+    g2 ();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h(void) {
+  try {
+    g ();
+  } catch (...) {
+    throw;
+  }
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+struct needs_cleanup {
+  ~needs_cleanup();
+};
+
+void h2(void) {
+  needs_cleanup y; /* No check in the cleanup handler.  */
+  g();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+extern void __attribute__ ((__nothrow__)) another_cleanup (void*);
+
+void h2b(void) {
+  int x __attribute__ ((cleanup (another_cleanup)));
+  g();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h3(void) {
+  try {
+    throw 1;
+  } catch (...) {
+  }
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h4(void) {
+  throw 1;
+  /* Inline check in the cleanup around the __cxa_throw noreturn call.  */
+}
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 12 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c b/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c
new file mode 100644
index 00000000000..8e4ee1fab08
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c
@@ -0,0 +1,15 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C makes for implicit nothrow in noreturn handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#include "../../c-c++-common/torture/harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c b/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c
new file mode 100644
index 00000000000..634d98f1ffc
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c
@@ -0,0 +1,40 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-hardcfr-check-exceptions -fdump-tree-hardcfr -ffat-lto-objects -Wno-return-type" } */
+
+/* In C only, check some additional cases (comparing with
+   c-c++-common/torture/harden-cfr-tail.c) of falling off the end of non-void
+   function.  C++ would issue an unreachable call in these cases.  */
+
+extern int g (int i);
+
+int f1(int i) {
+  /* Inline check before the returning call, that doesn't return anything.  */
+  g (i);
+  /* Implicit return without value, despite the return type; this combination
+     enables tail-calling of g, and is recognized as a returning call.  */
+}
+
+extern void g2 (int i);
+
+int f2(int i) {
+  /* Inline check before the returning call, that disregards its return
+     value.  */
+  g2 (i);
+  /* Implicit return without value, despite the return type; this combination
+     enables tail-calling of g2, and is recognized as a returning call.  */
+}
+
+int f3(int i) {
+  if (i)
+    /* Out-of-line check before the returning call.  */
+    return g (i);
+  /* Out-of-line check before implicit return.  */
+}
+
+/* Out-of-line checks in f3, before returning calls and before return.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking in all other functions.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
+/* Check before tail-call in all functions, but f3 is out-of-line.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/gnat.dg/hardcfr.adb b/gcc/testsuite/gnat.dg/hardcfr.adb
index b578a913d75..abe1605c029 100644
--- a/gcc/testsuite/gnat.dg/hardcfr.adb
+++ b/gcc/testsuite/gnat.dg/hardcfr.adb
@@ -1,5 +1,5 @@
 --  { dg-do run }
---  { dg-options "-fharden-control-flow-redundancy -fdump-tree-hardcfr --param=hardcfr-max-blocks=22 --param=hardcfr-max-inline-blocks=12 -O0" }
+--  { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions -fdump-tree-hardcfr --param=hardcfr-max-blocks=22 --param=hardcfr-max-inline-blocks=12 -O0" }
 
 procedure HardCFR is
    function F (I, J : Integer) return Integer is
diff --git a/libgcc/hardcfr.c b/libgcc/hardcfr.c
index 8ef29428111..55f6b995f2f 100644
--- a/libgcc/hardcfr.c
+++ b/libgcc/hardcfr.c
@@ -60,14 +60,25 @@ extern void __hardcfr_check (size_t blocks,
 			     vword const *visited,
 			     vword const *cfg);
 
+/* Compute the MASK for the bit representing BLOCK in WORDIDX's vword in a
+   visited blocks bit array.  */
+static inline void
+block2mask (size_t const block, vword *const mask, size_t *const wordidx)
+{
+  size_t wbits = __CHAR_BIT__ * sizeof (vword);
+  *wordidx = block / wbits;
+  *mask = (vword)1 << (block % wbits);
+}
 
 /* Check whether the bit corresponding to BLOCK is set in VISITED.  */
 static inline bool
 visited_p (size_t const block, vword const *const visited)
 {
-  size_t wbits = __CHAR_BIT__ * sizeof (vword);
-  vword w = visited[block / wbits];
-  return (w & ((vword)1 << (block % wbits))) != 0;
+  vword mask;
+  size_t wordidx;
+  block2mask (block, &mask, &wordidx);
+  vword w = visited[wordidx];
+  return (w & mask) != 0;
 }
 
 /* Read and consume a mask from **CFG_IT.  (Consume meaning advancing the
@@ -118,19 +129,19 @@ consume_seq (vword const **const cfg_it)
    we reach the terminator without finding any.  Consume the entire sequence
    otherwise, so that *CFG_IT points just past the terminator, which may be the
    beginning of the next sequence.  */
-static inline void
+static inline bool
 check_seq (vword const *const visited, vword const **const cfg_it)
 {
   vword mask;
   size_t wordidx;
 
   /* If the block was visited, check that at least one of the
-     preds was also visited.  */
+     preds/succs was also visited.  */
   do
     /* If we get to the end of the sequence without finding any
        match, something is amiss.  */
     if (!next_pair (cfg_it, &mask, &wordidx))
-      __builtin_trap ();
+      return false;
   /* Keep searching until we find a match, at which point the
      condition is satisfied.  */
   while (!test_mask (visited, mask, wordidx));
@@ -139,6 +150,94 @@ check_seq (vword const *const visited, vword const **const cfg_it)
      skipped the block, so as to position the iterator at the beginning of the
      next .  */
   consume_seq (cfg_it);
+
+  return true;
+}
+
+/* Print out the CFG with BLOCKS blocks, presumed to be associated with CALLER.
+   This is expected to be optimized out entirely, unless the verbose part of
+   __hardcfr_check_fail is enabled.  */
+static inline void
+__hardcfr_debug_cfg (size_t const blocks,
+		     void const *const caller,
+		     vword const *const cfg)
+{
+  __builtin_printf ("CFG at %p, for %p", cfg, caller);
+  vword const *cfg_it = cfg;
+  for (size_t i = 0; i < blocks; i++)
+    {
+      vword mask; size_t wordidx;
+      block2mask (i, &mask, &wordidx);
+      __builtin_printf ("\nblock %lu (%lu/0x%lx)\npreds: ",
+			(unsigned long)i,
+			(unsigned long)wordidx, (unsigned long)mask);
+      while (next_pair (&cfg_it, &mask, &wordidx))
+	__builtin_printf (" (%lu/0x%lx)",
+			  (unsigned long)wordidx, (unsigned long)mask);
+      __builtin_printf ("\nsuccs: ");
+      while (next_pair (&cfg_it, &mask, &wordidx))
+	__builtin_printf (" (%lu/0x%lx)",
+			  (unsigned long)wordidx, (unsigned long)mask);
+    }
+  __builtin_printf ("\n");
+}
+
+#ifndef ATTRIBUTE_UNUSED
+# define ATTRIBUTE_UNUSED __attribute__ ((__unused__))
+#endif
+
+/* This is called when an out-of-line hardcfr check fails.  All the arguments
+   are ignored, and it just traps, unless HARDCFR_VERBOSE_FAIL is enabled.  IF
+   it is, it prints the PART of the CFG, expected to have BLOCKS blocks, that
+   failed at CALLER's BLOCK, and the VISITED bitmap.  When the verbose mode is
+   enabled, it also forces __hardcfr_debug_cfg (above) to be compiled into an
+   out-of-line function, that could be called from a debugger.
+   */
+static inline void
+__hardcfr_check_fail (size_t const blocks ATTRIBUTE_UNUSED,
+		      vword const *const visited,
+		      vword const *const cfg ATTRIBUTE_UNUSED,
+		      size_t const block ATTRIBUTE_UNUSED,
+		      int const part ATTRIBUTE_UNUSED,
+		      void const *const caller ATTRIBUTE_UNUSED)
+{
+#if HARDCFR_VERBOSE_FAIL
+  static const char *parts[] = { "preds", "succs" };
+
+  vword mask; size_t wordidx;
+  block2mask (block, &mask, &wordidx);
+  __builtin_printf ("hardcfr fail at %p block %lu (%lu/0x%lx), expected %s:",
+		    caller, (unsigned long)block,
+		    (unsigned long)wordidx, (unsigned long)mask,
+		    parts[part]);
+
+  /* Skip data for previous blocks.  */
+  vword const *cfg_it = cfg;
+  for (size_t i = block; i--; )
+    {
+      consume_seq (&cfg_it);
+      consume_seq (&cfg_it);
+    }
+  for (size_t i = part; i--; )
+    consume_seq (&cfg_it);
+
+  while (next_pair (&cfg_it, &mask, &wordidx))
+    __builtin_printf (" (%lu/0x%lx)",
+		      (unsigned long)wordidx, (unsigned long)mask);
+
+  __builtin_printf ("\nvisited:");
+  block2mask (blocks, &mask, &wordidx);
+  for (size_t i = 0; i <= wordidx; i++)
+    __builtin_printf (" (%lu/0x%lx)",
+		      (unsigned long)i, (unsigned long)visited[i]);
+  __builtin_printf ("\n");
+
+  /* Reference __hardcfr_debug_cfg so that it's output out-of-line, so that it
+     can be called from a debugger.  */
+  if (!caller || caller == __hardcfr_debug_cfg)
+    return;
+#endif
+  __builtin_trap ();
 }
 
 /* Check that, for each of the BLOCKS basic blocks, if its bit is set in
@@ -168,9 +267,13 @@ __hardcfr_check (size_t const blocks,
       else
 	{
 	  /* Check predecessors.  */
-	  check_seq (visited, &cfg_it);
+	  if (!check_seq (visited, &cfg_it))
+	    __hardcfr_check_fail (blocks, visited, cfg, i, 0,
+				  __builtin_return_address (0));
 	  /* Check successors.  */
-	  check_seq (visited, &cfg_it);
+	  if (!check_seq (visited, &cfg_it))
+	    __hardcfr_check_fail (blocks, visited, cfg, i, 1,
+				  __builtin_return_address (0));
 	}
     }
 }

^ permalink raw reply	[flat|nested] 13+ messages in thread
* [gcc(refs/users/aoliva/heads/testme)] hardcfr: add optional checkpoints
@ 2023-06-08 10:58 Alexandre Oliva
  0 siblings, 0 replies; 13+ messages in thread
From: Alexandre Oliva @ 2023-06-08 10:58 UTC (permalink / raw)
  To: gcc-cvs

https://gcc.gnu.org/g:f2bb0e3aa7cdf9c7adb34224aeda2947ce04df7e

commit f2bb0e3aa7cdf9c7adb34224aeda2947ce04df7e
Author: Alexandre Oliva <oliva@adacore.com>
Date:   Wed Oct 19 20:36:18 2022 -0300

    hardcfr: add optional checkpoints
    
    Previously, control flow redundancy only checked the visited bitmap
    against the control flow graph at return points and before mandatory
    tail calls, missing various other possibilities of exiting a
    subprogram, such as by raising or propagating exceptions, and calling
    noreturn functions.  The checks inserted before returns also prevented
    potential tail-call optimizations.
    
    This incremental change introduces options to control checking at each
    of these previously-missed checkpoints.  Unless disabled, a cleanup is
    introduced to check when an exceptions escapes a subprogram.  To avoid
    disrupting sibcall optimizations, when they are enabled, checks are
    introduced before calls whose results are immediately returned,
    whether or not they are ultimately optimized.  If enabled, checks are
    introduced before noreturn calls and exception raises, or only before
    nothrow noreturn calls.
    
    
    for  gcc/ChangeLog
    
            * common.opt (-fhardcfr-check-returning-calls): New.
            (-fhardcfr-check-exceptions): New.
            (-fhardcfr-check-noreturn-calls=*): New.
            (Enum hardcfr_check_noreturn_calls): New.
            * doc/invoke.texi: Document them.
            * flag-types.h (enum hardcfr_noret): New.
            * gimple-harden-control-flow.cc: Include more headers.
            (pass_data_harden_control_flow_redundancy): Move
            properties_finish to pass return.
            (pass_harden_control_flow_redundancy::gate): Move
            no-edge-to-exit checking to execute, after exception
            transformations.  Use symbolic NUM_FIXED_BLOCKS.  Test for CFG
            before testing block count.
            (check_returning_calls_p): New.
            (hardcfr_scan_block): New.
            (returnign_call_p): New.
            (chk_edges_t): New.
            (hardcfr_sibcall_search_block): New.
            (hardcfr_sibcall_search_preds): New.
            (rt_bb_visited::num2idx): Use symbolic NUM_FIXED_BLOCKS.
            (rt_bb_visited::rt_bb_visited): Likewise.  Take checkpoints
            count, test it to select out-of-line checks.
            (rt_bb_visited::insert_exit_check): Removed.
            (rt_bb_visited::insert_exit_check_in_blocks): New.
            (rt_bb_visited::insert_exit_check_on_edge): New.
            (rt_bb_visited::check): Take checkpoint edges and blocks.
            Move initialization insertion to the end.  Insert checks on
            edges, and before calls in select blocks.  Clobber the visited
            array to the check sequence.  Output actions to dump_file.
            (rt_bb_visited::visit): Take checkpoint and postcheck.  Assume
            an edge to exit if checkpoint, and skip testing if postcheck.
            (always_throwing_noreturn_call_p): New, dummy.
            (pass_harden_control_flow_redundancy::execute): Introduce
            cleanup block if requested and needed.  Scan blocks for
            noreturn and returning calls if checkpoints before them were
            requested.  Log actions to dump_file.  Visit blocks in index
            order.
    
    for  gcc/testsuite/ChangeLog
    
            * c-c++-common/harden-cfr-noret-never-O0.c: New.
            * c-c++-common/torture/harden-cfr-noret-never.c: New.
            * c-c++-common/torture/harden-cfr-noret-noexcept.c: New.
            * c-c++-common/torture/harden-cfr-noret-nothrow.c: New.
            * c-c++-common/torture/harden-cfr-noret.c: New.
            * c-c++-common/torture/harden-cfr-notail.c: New.
            * c-c++-common/torture/harden-cfr-returning.c: New.
            * c-c++-common/torture/harden-cfr-tail.c: Extend.
            * c-c++-common/torture/harden-cfr-bret-always.c: New.
            * c-c++-common/torture/harden-cfr-bret-never.c: New.
            * c-c++-common/torture/harden-cfr-bret-noopt.c: New.
            * c-c++-common/torture/harden-cfr-bret-noret.c: New.
            * c-c++-common/torture/harden-cfr-bret-nothrow.c: New.
            * c-c++-common/torture/harden-cfr-bret-retcl.c: New.
            * c-c++-common/torture/harden-cfr-bret.c (g): New.
            * g++.dg/harden-cfr-throw-always-O0.C: New.
            * g++.dg/harden-cfr-throw-returning-O0.C: New.
            * g++.dg/torture/harden-cfr-noret-always-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-noret-never-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-noret-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-throw-always.C: New.
            * g++.dg/torture/harden-cfr-throw-nocleanup.C: New.
            * g++.dg/torture/harden-cfr-throw-returning.C: New.
            * g++.dg/torture/harden-cfr-throw.C: New.
            * gcc.dg/torture/harden-cfr-noret-no-nothrow.c: New.
            * gcc.dg/torture/harden-cfr-tail-ub.c: New.
            * gnat.dg/hardcfr.adb: Disable checking at exceptions.
    
    for  libgcc/ChangeLog
    
            * hardcfr.c (block2mask): Split out of...
            (visited_p): ... this.
            (check_seq): Move trapping out, return result instead.
            (__hardcfr_debug_cfg): New.
            (__hardcfr_check_fail): New, with optional verbose error
            printing before trapping.
            (__hardcfr_check): Call __hardcfr_check_fail when check_seq
            fails.

Diff:
---
 gcc/common.opt                                     |  33 +
 gcc/doc/invoke.texi                                |  74 +-
 gcc/flag-types.h                                   |  10 +
 gcc/gimple-harden-control-flow.cc                  | 926 ++++++++++++++++++---
 .../c-c++-common/harden-cfr-noret-never-O0.c       |  12 +
 .../c-c++-common/torture/harden-cfr-bret-always.c  |  13 +
 .../c-c++-common/torture/harden-cfr-bret-never.c   |  13 +
 .../c-c++-common/torture/harden-cfr-bret-noopt.c   |  12 +
 .../c-c++-common/torture/harden-cfr-bret-noret.c   |  12 +
 .../c-c++-common/torture/harden-cfr-bret-nothrow.c |  13 +
 .../c-c++-common/torture/harden-cfr-bret-retcl.c   |  12 +
 .../c-c++-common/torture/harden-cfr-bret.c         |   8 +-
 .../c-c++-common/torture/harden-cfr-noret-never.c  |  18 +
 .../torture/harden-cfr-noret-noexcept.c            |  16 +
 .../torture/harden-cfr-noret-nothrow.c             |  13 +
 .../c-c++-common/torture/harden-cfr-noret.c        |  38 +
 .../c-c++-common/torture/harden-cfr-notail.c       |   8 +
 .../c-c++-common/torture/harden-cfr-returning.c    |  35 +
 .../c-c++-common/torture/harden-cfr-tail.c         |  55 +-
 gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C  |  11 +
 .../g++.dg/harden-cfr-throw-returning-O0.C         |  10 +
 .../torture/harden-cfr-noret-always-no-nothrow.C   |  16 +
 .../torture/harden-cfr-noret-never-no-nothrow.C    |  18 +
 .../g++.dg/torture/harden-cfr-noret-no-nothrow.C   |  23 +
 .../g++.dg/torture/harden-cfr-throw-always.C       |  20 +
 .../g++.dg/torture/harden-cfr-throw-nocleanup.C    |  11 +
 .../g++.dg/torture/harden-cfr-throw-returning.C    |  31 +
 gcc/testsuite/g++.dg/torture/harden-cfr-throw.C    |  65 ++
 .../gcc.dg/torture/harden-cfr-noret-no-nothrow.c   |  15 +
 gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c  |  40 +
 gcc/testsuite/gnat.dg/hardcfr.adb                  |   2 +-
 libgcc/hardcfr.c                                   | 119 ++-
 32 files changed, 1568 insertions(+), 134 deletions(-)

diff --git a/gcc/common.opt b/gcc/common.opt
index ff89b2d1140..c2855cc19a9 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -1805,6 +1805,39 @@ fharden-control-flow-redundancy
 Common Var(flag_harden_control_flow_redundancy) Optimization
 Harden control flow by recording and checking execution paths.
 
+fhardcfr-check-returning-calls
+Common Var(flag_harden_control_flow_redundancy_check_returning_calls) Init(-1) Optimization
+Check CFR execution paths also before calls followed by returns of their results.
+
+fhardcfr-check-exceptions
+Common Var(flag_harden_control_flow_redundancy_check_exceptions) Init(-1) Optimization
+Check CFR execution paths also when exiting a function through an exception.
+
+fhardcfr-check-noreturn-calls=
+Common Joined RejectNegative Enum(hardcfr_check_noreturn_calls) Var(flag_harden_control_flow_redundancy_check_noreturn) Init(HCFRNR_UNSPECIFIED) Optimization
+-fhardcfr-check-noreturn-calls=[always|nothrow|never]	Check CFR execution paths also before calling noreturn functions.
+
+Enum
+Name(hardcfr_check_noreturn_calls) Type(enum hardcfr_noret) UnknownError(unknown hardcfr noreturn checking level %qs)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(never) Value(HCFRNR_NEVER)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(nothrow) Value(HCFRNR_NOTHROW)
+
+; ??? There could be yet another option here, that checked before
+; noreturn calls, except for those known to always throw, if we had
+; means to distinguish noreturn functions known to always throw, such
+; as those used to (re)raise exceptions, from those that merely might
+; throw.  "not always" stands for "not always-throwing", but it also
+; contrasts with "always" below.
+; EnumValue
+; Enum(hardcfr_check_noreturn_calls) String(not-always) Value(HCFRNR_NOT_ALWAYS)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(always) Value(HCFRNR_ALWAYS)
+
 ; Nonzero means ignore `#ident' directives.  0 means handle them.
 ; Generate position-independent code for executables if possible
 ; On SVR4 targets, it also controls whether or not to emit a
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index 87a33b037c1..7742d21f142 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -635,7 +635,9 @@ Objective-C and Objective-C++ Dialects}.
 -fsanitize-undefined-trap-on-error  -fbounds-check
 -fcf-protection=@r{[}full@r{|}branch@r{|}return@r{|}none@r{|}check@r{]}
 -fharden-compares -fharden-conditional-branches
--fharden-control-flow-redundancy
+-fharden-control-flow-redundancy  -fhardcfr-check-exceptions
+-fhardcfr-check-returning-calls
+-fhardcfr-check-noreturn-calls=@r{[}always@r{|}nothrow@r{|}never@r{]}
 -fstack-protector  -fstack-protector-all  -fstack-protector-strong
 -fstack-protector-explicit  -fstack-check
 -fstack-limit-register=@var{reg}  -fstack-limit-symbol=@var{sym}
@@ -17243,11 +17245,75 @@ conditionals.
 @opindex fharden-control-flow-redundancy
 @item -fharden-control-flow-redundancy
 Emit extra code to set booleans when entering basic blocks, and to
-verify, at function exits, that they amount to an execution path that is
-consistent with the control flow graph, trapping otherwise.  Tuning
-options @option{--param hardcfr-max-blocks} and @option{--param
+verify and trap, at function exits, when the booleans do not form an
+execution path that is compatible with the control flow graph.
+
+Verification takes place before returns, before mandatory tail calls
+(see below) and, optionally, before escaping exceptions with
+@option{-fhardcfr-check-exceptions}, before returning calls with
+@option{-fhardcfr-check-returning-calls}, and before noreturn calls with
+@option{-fhardcfr-check-noreturn-calls}).  Tuning options
+@option{--param hardcfr-max-blocks} and @option{--param
 hardcfr-max-inline-blocks} are available.
 
+Tail call optimization takes place too late to affect control flow
+redundancy, but calls annotated as mandatory tail calls by language
+front-ends, and any calls marked early enough as potential tail calls
+would also have verification issued before the call, but these
+possibilities are merely theoretical, as these conditions can only be
+met when using custom compiler plugins.
+
+@opindex fhardcfr-check-exceptions
+@opindex fno-hardcfr-check-exceptions
+@item -fhardcfr-check-exceptions
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph at exception
+escape points, as if the function body was wrapped with a cleanup
+handler that performed the check and reraised.  This option is enabled
+by default; use @option{-fno-hardcfr-check-exceptions} to disable it.
+
+@opindex fhardcfr-check-returning-calls
+@opindex fno-hardcfr-check-returning-calls
+@item -fhardcfr-check-returning-calls
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph before any
+function call immediately followed by a return of its result, if any, so
+as to not prevent tail-call optimization, whether or not it is
+ultimately optimized to a tail call.
+
+This option is enabled by default whenever sibling call optimizations
+are enabled (see @option{-foptimize-sibling-calls}), but it can be
+enabled (or disabled, using its negated form) explicitly, regardless of
+the optimizations.
+
+@opindex fhardcfr-check-noreturn-calls
+@item -fhardcfr-check-noreturn-calls=@r{[}always@r{|}nothrow@r{|}never@r{]}
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph before
+@code{noreturn} calls, either all of them (@option{always}), those that
+may not return control to the caller through an exception either
+(@option{nothrow}), or none of them (@option{never}, the default).
+
+Checking before a @code{noreturn} function that may return control to
+the caller through an exception may cause checking to be performed more
+than once, if the exception is caught in the caller, whether by a
+handler or a cleanup.  When @option{-fhardcfr-check-exceptions} is also
+enabled, the compiler will avoid associating a @code{noreturn} call with
+the implicitly-added cleanup handler, since it would be redundant with
+the check performed before the call, but other handlers or cleanups in
+the function, if activated, will modify the recorded execution path and
+check it again when another checkpoint is hit.  The checkpoint may even
+be another @code{noreturn} call, so checking may end up performed
+multiple times.
+
+Various optimizers may cause calls to be marked as @code{noreturn}
+and/or @code{nothrow}, even in the absence of the corresponding
+attributes, which may affect the placement of checks before calls, as
+well as the addition of implicit cleanup handlers for them.  This
+unpredictability, and the fact that raising and reraising exceptions
+frequently amounts to implicitly calling @code{noreturn} functions, have
+made @option{never} the default setting for this option.
+
 @opindex fstack-protector
 @item -fstack-protector
 Emit extra code to check for buffer overflows, such as stack smashing
diff --git a/gcc/flag-types.h b/gcc/flag-types.h
index 78dbdbe90dd..6d6fd19fcd6 100644
--- a/gcc/flag-types.h
+++ b/gcc/flag-types.h
@@ -157,6 +157,16 @@ enum stack_reuse_level
   SR_ALL
 };
 
+/* Control Flow Redundancy hardening options for noreturn calls.  */
+enum hardcfr_noret
+{
+  HCFRNR_NEVER,
+  HCFRNR_NOTHROW,
+  HCFRNR_NOT_ALWAYS, /* Reserved for future use.  */
+  HCFRNR_ALWAYS,
+  HCFRNR_UNSPECIFIED = -1
+};
+
 /* The live patching level.  */
 enum live_patching_level
 {
diff --git a/gcc/gimple-harden-control-flow.cc b/gcc/gimple-harden-control-flow.cc
index 6b08846dbb1..5c66762d448 100644
--- a/gcc/gimple-harden-control-flow.cc
+++ b/gcc/gimple-harden-control-flow.cc
@@ -29,7 +29,12 @@ along with GCC; see the file COPYING3.  If not see
 #include "tree-pass.h"
 #include "ssa.h"
 #include "gimple-iterator.h"
+#include "gimple-pretty-print.h"
 #include "tree-cfg.h"
+#include "tree-cfgcleanup.h"
+#include "tree-eh.h"
+#include "except.h"
+#include "sbitmap.h"
 #include "basic-block.h"
 #include "cfghooks.h"
 #include "cfgloop.h"
@@ -60,9 +65,7 @@ const pass_data pass_data_harden_control_flow_redundancy = {
   0,	    // properties_provided
   0,	    // properties_destroyed
   TODO_cleanup_cfg, // properties_start
-  TODO_update_ssa
-  | TODO_cleanup_cfg
-  | TODO_verify_il, // properties_finish
+  0,        // properties_finish
 };
 
 class pass_harden_control_flow_redundancy : public gimple_opt_pass
@@ -79,16 +82,6 @@ public:
     if (!flag_harden_control_flow_redundancy)
       return false;
 
-    /* We don't verify when an exception escapes, propagated or raised
-       by the function itself, so we're only concerned with edges to
-       the exit block.  If there aren't any, the function doesn't
-       return normally, so there won't be any checking point, so
-       there's no point in running the pass.  Should we add
-       verification at exception escapes, we should at least look at
-       !flag_exceptions here.  */
-    if (EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (fun)->preds) == 0)
-      return false;
-
     /* Functions that return more than once, like setjmp and vfork
        (that also gets this flag set), will start recording a path
        after the first return, and then may take another path when
@@ -117,8 +110,9 @@ public:
 	return false;
       }
 
-    if (param_hardcfr_max_blocks > 0
-	&& n_basic_blocks_for_fn (fun) - 2 > param_hardcfr_max_blocks)
+    if (fun->cfg && param_hardcfr_max_blocks > 0
+	&& (n_basic_blocks_for_fn (fun) - NUM_FIXED_BLOCKS
+	    > param_hardcfr_max_blocks))
       {
 	warning_at (DECL_SOURCE_LOCATION (fun->decl), 0,
 		    "%qD has more than %u blocks, the requested"
@@ -134,6 +128,293 @@ public:
 
 }
 
+/* Return TRUE iff CFR checks should be inserted before returning
+   calls.  */
+
+static bool
+check_returning_calls_p ()
+{
+  return
+    flag_harden_control_flow_redundancy_check_returning_calls > 0
+    || (flag_harden_control_flow_redundancy_check_returning_calls < 0
+	/* Gates pass_tail_calls.  */
+	&& flag_optimize_sibling_calls
+	/* Gates pass_all_optimizations.  */
+	&& optimize >= 1 && !optimize_debug);
+}
+
+/* Scan BB from the end, updating *RETPTR if given as return stmts and
+   copies are found.  Return a call or a stmt that cannot appear after
+   a tail call, or NULL if the top of the block is reached without
+   finding any.  */
+
+static gimple *
+hardcfr_scan_block (basic_block bb, tree **retptr)
+{
+  gimple_stmt_iterator gsi;
+  for (gsi = gsi_last_bb (bb); !gsi_end_p (gsi); gsi_prev (&gsi))
+    {
+      gimple *stmt = gsi_stmt (gsi);
+
+      /* Ignore labels, returns, nops, clobbers and debug stmts.  */
+      if (gimple_code (stmt) == GIMPLE_LABEL
+	  || gimple_code (stmt) == GIMPLE_NOP
+	  || gimple_code (stmt) == GIMPLE_PREDICT
+	  || gimple_clobber_p (stmt)
+	  || is_gimple_debug (stmt))
+	continue;
+
+      if (gimple_code (stmt) == GIMPLE_RETURN)
+	{
+	  greturn *gret = as_a <greturn *> (stmt);
+	  if (retptr)
+	    {
+	      gcc_checking_assert (!*retptr);
+	      *retptr = gimple_return_retval_ptr (gret);
+	    }
+	  continue;
+	}
+
+      /* Check for a call.  */
+      if (is_gimple_call (stmt))
+	return stmt;
+
+      /* Allow simple copies to the return value, updating the return
+	 value to be found in earlier assignments.  */
+      if (retptr && *retptr && gimple_assign_single_p (stmt)
+	  && **retptr == gimple_assign_lhs (stmt))
+	{
+	  *retptr = gimple_assign_rhs1_ptr (stmt);
+	  continue;
+	}
+
+      return stmt;
+    }
+
+  /* Any other kind of stmt will prevent a tail call.  */
+  return NULL;
+}
+
+/* Return TRUE iff CALL is to be preceded by a CFR checkpoint, i.e.,
+   if it's a returning call (one whose result is ultimately returned
+   without intervening non-copy statements) and we're checking
+   returning calls, a __builtin_return call (noreturn with a path to
+   the exit block), a must-tail call, or a tail call.  */
+
+static bool
+returning_call_p (gcall *call)
+{
+  if (!(gimple_call_noreturn_p (call)
+	|| gimple_call_must_tail_p (call)
+	|| gimple_call_tail_p (call)
+	|| check_returning_calls_p ()))
+    return false;
+
+  /* Quickly check that there's a path to exit compatible with a
+     returning call.  Detect infinite loops through the counter.  */
+  basic_block bb = gimple_bb (call);
+  auto_vec<basic_block, 10> path;
+  for (int i = n_basic_blocks_for_fn (cfun);
+       bb != EXIT_BLOCK_PTR_FOR_FN (cfun) && i--;
+       bb = single_succ (bb))
+    if (!single_succ_p (bb)
+	|| (single_succ_edge (bb)->flags & EDGE_EH) != 0)
+      return false;
+    else
+      path.safe_push (bb);
+
+  /* Check the stmts in the blocks and trace the return value.  */
+  tree *retptr = NULL;
+  for (;;)
+    {
+      gcc_checking_assert (!path.is_empty ());
+      basic_block bb = path.pop ();
+      gimple *stop = hardcfr_scan_block (bb, &retptr);
+      if (stop)
+	{
+	  if (stop != call)
+	    return false;
+	  gcc_checking_assert (path.is_empty ());
+	  break;
+	}
+
+      gphi *retphi = NULL;
+      if (retptr && *retptr && TREE_CODE (*retptr) == SSA_NAME
+	  && !SSA_NAME_IS_DEFAULT_DEF (*retptr)
+	  && SSA_NAME_DEF_STMT (*retptr)
+	  && is_a <gphi *> (SSA_NAME_DEF_STMT (*retptr))
+	  && gimple_bb (SSA_NAME_DEF_STMT (*retptr)) == bb)
+	{
+	  retphi = as_a <gphi *> (SSA_NAME_DEF_STMT (*retptr));
+	  gcc_checking_assert (gimple_phi_result (retphi) == *retptr);
+	}
+      else
+	continue;
+
+      gcc_checking_assert (!path.is_empty ());
+      edge e = single_succ_edge (path.last ());
+      int i = EDGE_COUNT (bb->preds);
+      while (i--)
+	if (EDGE_PRED (bb, i) == e)
+	  break;
+      gcc_checking_assert (i >= 0);
+      retptr = gimple_phi_arg_def_ptr (retphi, i);
+    }
+
+  return (gimple_call_noreturn_p (call)
+	  || gimple_call_must_tail_p (call)
+	  || gimple_call_tail_p (call)
+	  || (gimple_call_lhs (call) == (retptr ? *retptr : NULL)
+	      && check_returning_calls_p ()));
+}
+
+typedef auto_vec<edge, 10> chk_edges_t;
+
+/* Declare for mutual recursion.  */
+static bool hardcfr_sibcall_search_preds (basic_block bb,
+					  chk_edges_t &chk_edges,
+					  int &count_chkcall,
+					  auto_sbitmap &chkcall_blocks,
+					  int &count_postchk,
+					  auto_sbitmap &postchk_blocks,
+					  tree *retptr);
+
+/* Search backwards from the end of BB for a mandatory or potential
+   sibcall.  Schedule the block to be handled sort-of like noreturn if
+   so.  Recurse to preds, with updated RETPTR, if the block only
+   contains stmts that may follow such a call, scheduling checking at
+   edges and marking blocks as post-check as needed.  Return true iff,
+   at the end of the block, a check will have already been
+   performed.  */
+
+static bool
+hardcfr_sibcall_search_block (basic_block bb,
+			      chk_edges_t &chk_edges,
+			      int &count_chkcall,
+			      auto_sbitmap &chkcall_blocks,
+			      int &count_postchk,
+			      auto_sbitmap &postchk_blocks,
+			      tree *retptr)
+{
+  /* Conditionals and internal exceptions rule out tail calls.  */
+  if (!single_succ_p (bb)
+      || (single_succ_edge (bb)->flags & EDGE_EH) != 0)
+    return false;
+
+  gimple *stmt = hardcfr_scan_block (bb, &retptr);
+  if (!stmt)
+    return hardcfr_sibcall_search_preds (bb, chk_edges,
+					 count_chkcall, chkcall_blocks,
+					 count_postchk, postchk_blocks,
+					 retptr);
+
+  if (!is_a <gcall *> (stmt))
+    return false;
+
+  /* Avoid disrupting mandatory or early-marked tail calls,
+     inserting the check before them.  This works for
+     must-tail calls, but tail calling as an optimization is
+     detected too late for us.
+
+     Also check for noreturn calls here.  Noreturn calls won't
+     normally have edges to exit, so they won't be found here,
+     but __builtin_return does, and we must check before
+     it, so handle it like a tail call.  */
+  gcall *call = as_a <gcall *> (stmt);
+  if (!(gimple_call_noreturn_p (call)
+	|| gimple_call_must_tail_p (call)
+	|| gimple_call_tail_p (call)
+	|| (gimple_call_lhs (call) == (retptr ? *retptr : NULL)
+	    && check_returning_calls_p ())))
+    return false;
+
+  gcc_checking_assert (returning_call_p (call));
+
+  /* We found a call that is to be preceded by checking.  */
+  if (bitmap_set_bit (chkcall_blocks, bb->index))
+    ++count_chkcall;
+  else
+    gcc_unreachable ();
+  return true;
+}
+
+
+/* Search preds of BB for a mandatory or potential sibcall or
+   returning call, and arrange for the blocks containing them to have
+   a check inserted before the call, like noreturn calls.  If any
+   preds are found to perform checking, schedule checks at the edges
+   of those that don't, and mark BB as postcheck..  */
+
+static bool
+hardcfr_sibcall_search_preds (basic_block bb,
+			      chk_edges_t &chk_edges,
+			      int &count_chkcall,
+			      auto_sbitmap &chkcall_blocks,
+			      int &count_postchk,
+			      auto_sbitmap &postchk_blocks,
+			      tree *retptr)
+{
+  /* For the exit block, we wish to force a check at every
+     predecessor, so pretend we've already found a pred that had
+     checking, so that we schedule checking at every one of its pred
+     edges.  */
+  bool first = bb->index >= NUM_FIXED_BLOCKS;
+  bool postchecked = true;
+
+  gphi *retphi = NULL;
+  if (retptr && *retptr && TREE_CODE (*retptr) == SSA_NAME
+      && !SSA_NAME_IS_DEFAULT_DEF (*retptr)
+      && SSA_NAME_DEF_STMT (*retptr)
+      && is_a <gphi *> (SSA_NAME_DEF_STMT (*retptr))
+      && gimple_bb (SSA_NAME_DEF_STMT (*retptr)) == bb)
+    {
+      retphi = as_a <gphi *> (SSA_NAME_DEF_STMT (*retptr));
+      gcc_checking_assert (gimple_phi_result (retphi) == *retptr);
+    }
+
+  for (int i = EDGE_COUNT (bb->preds); i--; first = false)
+    {
+      edge e = EDGE_PRED (bb, i);
+
+      bool checked
+	= hardcfr_sibcall_search_block (e->src, chk_edges,
+					count_chkcall, chkcall_blocks,
+					count_postchk, postchk_blocks,
+					!retphi ? retptr
+					: gimple_phi_arg_def_ptr (retphi, i));
+
+      if (first)
+	{
+	  postchecked = checked;
+	  continue;
+	}
+
+      /* When we first find a checked block, force a check at every
+	 other incoming edge we've already visited, and those we
+	 visit afterwards that don't have their own check, so that
+	 when we reach BB, the check has already been performed.  */
+      if (!postchecked && checked)
+	{
+	  for (int j = EDGE_COUNT (bb->preds); --j > i; )
+	    chk_edges.safe_push (EDGE_PRED (bb, j));
+	  postchecked = true;
+	}
+      if (postchecked && !checked)
+	chk_edges.safe_push (EDGE_PRED (bb, i));
+    }
+
+  if (postchecked && bb->index >= NUM_FIXED_BLOCKS)
+    {
+      if (bitmap_set_bit (postchk_blocks, bb->index))
+	count_postchk++;
+      else
+	gcc_unreachable ();
+    }
+
+  return postchecked;
+}
+
+
 class rt_bb_visited
 {
   /* Use a sufficiently wide unsigned type to hold basic block numbers.  */
@@ -176,8 +457,8 @@ class rt_bb_visited
      neither ENTRY nor EXIT, but maybe one-past-the-end, to compute
      the visited array length.  */
   blknum num2idx (blknum n) {
-    gcc_checking_assert (n >= 2 && n <= nblocks);
-    return (n - 2);
+    gcc_checking_assert (n >= NUM_FIXED_BLOCKS && n <= nblocks);
+    return (n - NUM_FIXED_BLOCKS);
   }
   /* Return the block vindex for BB, that must not be ENTRY or
      EXIT.  */
@@ -249,8 +530,7 @@ class rt_bb_visited
   }
 
   /* Set the bit corresponding to BB in VISITED.  Add to SEQ any
-     required gimple statements, and return SEQ, possibly
-     modified.  */
+     required gimple stmts, and return SEQ, possibly modified.  */
   gimple_seq vset (basic_block bb, gimple_seq seq = NULL)
   {
     tree bit, setme = vword (bb, &bit);
@@ -270,7 +550,7 @@ class rt_bb_visited
 
 public:
   /* Prepare to add control flow redundancy testing to CFUN.  */
-  rt_bb_visited ()
+  rt_bb_visited (int checkpoints)
     : nblocks (n_basic_blocks_for_fn (cfun)),
       vword_type (NULL), ckseq (NULL), rtcfg (NULL)
   {
@@ -353,8 +633,8 @@ public:
 					 NULL, NULL);
     gimple_seq_add_stmt (&ckseq, detach);
 
-    if (nblocks - 2 > blknum (param_hardcfr_max_inline_blocks)
-	|| !single_pred_p (EXIT_BLOCK_PTR_FOR_FN (cfun)))
+    if (nblocks - NUM_FIXED_BLOCKS > blknum (param_hardcfr_max_inline_blocks)
+	|| checkpoints > 1)
       {
 	/* Make sure vword_bits is wide enough for the representation
 	   of nblocks in rtcfg.  Compare with vword_bits << vword_bits,
@@ -379,65 +659,34 @@ public:
     gimple_seq_add_stmt (&ckseq, ckfail_init);
   }
 
-  /* Insert SEQ on E, or close enough (e.g., before a noreturn or tail
-     call at the end of E->src).  */
-  void insert_exit_check (gimple_seq seq, edge e)
+  /* Insert SEQ before a resx or a call in INSBB.  */
+  void insert_exit_check_in_block (gimple_seq seq, basic_block insbb)
   {
-    basic_block insbb = e->src;
-
-    /* If the returning block ends with a noreturn call, insert
-       checking before it.  This is particularly important for
-       __builtin_return.  Other noreturn calls won't have an edge to
-       the exit block, so they won't even be considered as exit paths.
-
-       Insert-on-edge inserts before other return stmts, but not
-       before calls, and if a single-block function had the check
-       sequence inserted after a noreturn call, it would be useless,
-       but checking would still flag it as malformed if block 2 has a
-       fallthru edge to the exit block.
-
-       Also avoid disrupting tail calls, inserting the check before
-       them.  This works for must-tail calls, but tail calling as an
-       optimization is detected too late for us.  */
     gimple_stmt_iterator gsi = gsi_last_bb (insbb);
-    gimple *ret = gsi_stmt (gsi);
-    if (ret && is_a <greturn *> (ret))
-      {
+
+    while (!gsi_end_p (gsi))
+      if (is_a <gresx *> (gsi_stmt (gsi))
+	  || is_a <gcall *> (gsi_stmt (gsi)))
+	break;
+      else
 	gsi_prev (&gsi);
-	if (!gsi_end_p (gsi))
-	  ret = gsi_stmt (gsi);
-      }
-    if (ret && is_a <gcall *> (ret)
-	&& (gimple_call_noreturn_p (ret)
-	    || gimple_call_must_tail_p (as_a <gcall *> (ret))
-	    || gimple_call_tail_p (as_a <gcall *> (ret))))
-      gsi_insert_seq_before (&gsi, seq, GSI_SAME_STMT);
-    else
-      gsi_insert_seq_on_edge_immediate (e, seq);
+
+    gsi_insert_seq_before (&gsi, seq, GSI_SAME_STMT);
   }
 
-  /* Add checking code on every exit edge, and initialization code on
-     the entry edge.  Before this point, the CFG has been undisturbed,
-     and all the needed data has been collected and safely stowed.  */
-  void check ()
+  /* Insert SEQ on E.  */
+  void insert_exit_check_on_edge (gimple_seq seq, edge e)
   {
-    /* Insert initializers for visited at the entry.  */
-    gimple_seq iseq = NULL;
-
-    gcall *vinit = gimple_build_call (builtin_decl_explicit
-				      (BUILT_IN_MEMSET), 3,
-				      build1 (ADDR_EXPR,
-					      build_pointer_type
-					      (TREE_TYPE (visited)),
-					      visited),
-				      integer_zero_node,
-				      TYPE_SIZE_UNIT (TREE_TYPE (visited)));
-    gimple_seq_add_stmt (&iseq, vinit);
-
-    gsi_insert_seq_on_edge_immediate (single_succ_edge
-				      (ENTRY_BLOCK_PTR_FOR_FN (cfun)),
-				      iseq);
+    gsi_insert_seq_on_edge_immediate (e, seq);
+  }
 
+  /* Add checking code to CHK_EDGES and CHKCALL_BLOCKS, and
+     initialization code on the entry edge.  Before this point, the
+     CFG has been undisturbed, and all the needed data has been
+     collected and safely stowed.  */
+  void check (chk_edges_t &chk_edges,
+	      int count_chkcall, auto_sbitmap const &chkcall_blocks)
+  {
     /* If we're using out-of-line checking, create and statically
        initialize the CFG checking representation, generate the
        checker call for the checking sequence, and insert it in all
@@ -498,28 +747,116 @@ public:
 						     rtcfg));
 	gimple_seq_add_stmt (&ckseq, call_chk);
 
+	gimple *clobber = gimple_build_assign (visited,
+					       build_clobber
+					       (TREE_TYPE (visited)));
+	gimple_seq_add_stmt (&ckseq, clobber);
+
 	/* If we have multiple exit edges, insert (copies of)
 	   ckseq in all of them.  */
-	for (int i = EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (cfun)->preds);
-	     i--; )
+	for (int i = chk_edges.length (); i--; )
 	  {
 	    gimple_seq seq = ckseq;
 	    /* Copy the sequence, unless we're dealing with the
 	       last edge (we're counting down to zero).  */
-	    if (i)
+	    if (i || count_chkcall)
+	      seq = gimple_seq_copy (seq);
+
+	    edge e = chk_edges[i];
+
+	    if (dump_file)
+	      {
+		if (e->dest == EXIT_BLOCK_PTR_FOR_FN (cfun))
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to exit.\n",
+			   e->src->index);
+		else
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to postcheck block %i.\n",
+			   e->src->index, e->dest->index);
+	      }
+
+	    insert_exit_check_on_edge (seq, e);
+
+	    gcc_checking_assert (!bitmap_bit_p (chkcall_blocks, e->src->index));
+	  }
+
+	sbitmap_iterator it;
+	unsigned i;
+	EXECUTE_IF_SET_IN_BITMAP (chkcall_blocks, 0, i, it)
+	  {
+	    basic_block bb = BASIC_BLOCK_FOR_FN (cfun, i);
+
+	    gimple_seq seq = ckseq;
+	    gcc_checking_assert (count_chkcall > 0);
+	    if (--count_chkcall)
 	      seq = gimple_seq_copy (seq);
 
-	    insert_exit_check (seq,
-			       EDGE_PRED (EXIT_BLOCK_PTR_FOR_FN (cfun), i));
+	    if (dump_file)
+	      fprintf (dump_file,
+		       "Inserting out-of-line check before stmt in block %i.\n",
+		       bb->index);
+
+	    insert_exit_check_in_block (seq, bb);
 	  }
+
+	gcc_checking_assert (count_chkcall == 0);
       }
     else
       {
 	/* Inline checking requires a single exit edge.  */
-	gimple *last = gsi_stmt (gsi_last (ckseq));
+	gimple *last = gimple_build_assign (visited,
+					    build_clobber
+					    (TREE_TYPE (visited)));
+	gimple_seq_add_stmt (&ckseq, last);
 
-	insert_exit_check (ckseq,
-			   single_pred_edge (EXIT_BLOCK_PTR_FOR_FN (cfun)));
+	if (!count_chkcall)
+	  {
+	    edge e = single_pred_edge (EXIT_BLOCK_PTR_FOR_FN (cfun));
+
+	    if (dump_file)
+	      {
+		if (e->dest == EXIT_BLOCK_PTR_FOR_FN (cfun))
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to postcheck block %i.\n",
+			   e->src->index, e->dest->index);
+		else
+		  fprintf (dump_file,
+			   "Inserting inline check in"
+			   " block %i's edge to exit.\n",
+			   e->src->index);
+	      }
+
+	    insert_exit_check_on_edge (ckseq, e);
+	  }
+	else
+	  {
+	    gcc_checking_assert (count_chkcall == 1);
+
+	    sbitmap_iterator it;
+	    unsigned i;
+	    EXECUTE_IF_SET_IN_BITMAP (chkcall_blocks, 0, i, it)
+	      {
+		basic_block bb = BASIC_BLOCK_FOR_FN (cfun, i);
+
+		gimple_seq seq = ckseq;
+		gcc_checking_assert (count_chkcall > 0);
+		if (--count_chkcall)
+		  seq = gimple_seq_copy (seq);
+
+		if (dump_file)
+		  fprintf (dump_file,
+			   "Inserting inline check before stmt in block %i.\n",
+			   bb->index);
+
+		insert_exit_check_in_block (seq, bb);
+	      }
+
+	    gcc_checking_assert (count_chkcall == 0);
+	  }
 
 	/* The inserted ckseq computes CKFAIL at LAST.  Now we have to
 	   conditionally trap on it.  */
@@ -540,8 +877,7 @@ public:
 	  add_bb_to_loop (trp, current_loops->tree_root);
 
 	/* Insert a conditional branch to the trap block.  If the
-	   conditional wouldn't be the last statement, split the
-	   block.  */
+	   conditional wouldn't be the last stmt, split the block.  */
 	gimple_stmt_iterator gsi = gsi_for_stmt (last);
 	if (!gsi_one_before_end_p (gsi))
 	  split_block (gsi_bb (gsi), gsi_stmt (gsi));
@@ -564,6 +900,24 @@ public:
 	if (dom_info_available_p (CDI_DOMINATORS))
 	  set_immediate_dominator (CDI_DOMINATORS, trp, gimple_bb (last));
       }
+
+    /* Insert initializers for visited at the entry.  Do this after
+       other insertions, to avoid messing with block numbers.  */
+    gimple_seq iseq = NULL;
+
+    gcall *vinit = gimple_build_call (builtin_decl_explicit
+				      (BUILT_IN_MEMSET), 3,
+				      build1 (ADDR_EXPR,
+					      build_pointer_type
+					      (TREE_TYPE (visited)),
+					      visited),
+				      integer_zero_node,
+				      TYPE_SIZE_UNIT (TREE_TYPE (visited)));
+    gimple_seq_add_stmt (&iseq, vinit);
+
+    gsi_insert_seq_on_edge_immediate (single_succ_edge
+				      (ENTRY_BLOCK_PTR_FOR_FN (cfun)),
+				      iseq);
   }
 
   /* Push onto RTCFG a (mask, index) pair to test for IBB when BB is
@@ -607,7 +961,7 @@ public:
     return false;
   }
 
-  /* Add to CKSEQ statements to clear CKPART if OBB is visited.  */
+  /* Add to CKSEQ stmts to clear CKPART if OBB is visited.  */
   void
   build_block_check (basic_block obb)
   {
@@ -632,34 +986,48 @@ public:
 
   /* Add to BB code to set its bit in VISITED, and add to RTCFG or
      CKSEQ the data or code needed to check BB's predecessors and
-     successors.  Do NOT change the CFG.  */
-  void visit (basic_block bb)
+     successors.  If CHECKPOINT, assume the block is a checkpoint,
+     whether or not it has an edge to EXIT.  If POSTCHECK, assume the
+     block post-dominates checkpoints and therefore no bitmap setting
+     or checks are to be performed in or for it.  Do NOT change the
+     CFG.  */
+  void visit (basic_block bb, bool checkpoint, bool postcheck)
   {
     /* Set the bit in VISITED when entering the block.  */
     gimple_stmt_iterator gsi = gsi_after_labels (bb);
-    gsi_insert_seq_before (&gsi, vset (bb), GSI_SAME_STMT);
+    if (!postcheck)
+      gsi_insert_seq_before (&gsi, vset (bb), GSI_SAME_STMT);
 
     if (rtcfg)
       {
-	/* Build a list of (index, mask) terminated by (NULL, 0).
-	   Consolidate masks with the same index when they're
-	   adjacent.  First, predecessors.  Count backwards, because
-	   we're going to reverse the list.  The order shouldn't
-	   matter, but let's not make it surprising.  */
-	for (int i = EDGE_COUNT (bb->preds); i--; )
-	  if (push_rtcfg_pair (EDGE_PRED (bb, i)->src, bb,
-			       ENTRY_BLOCK_PTR_FOR_FN (cfun)))
-	    break;
+	if (!postcheck)
+	  {
+	    /* Build a list of (index, mask) terminated by (NULL, 0).
+	       Consolidate masks with the same index when they're
+	       adjacent.  First, predecessors.  Count backwards, because
+	       we're going to reverse the list.  The order shouldn't
+	       matter, but let's not make it surprising.  */
+	    for (int i = EDGE_COUNT (bb->preds); i--; )
+	      if (push_rtcfg_pair (EDGE_PRED (bb, i)->src, bb,
+				   ENTRY_BLOCK_PTR_FOR_FN (cfun)))
+		break;
+	  }
 	rtcfg = tree_cons (NULL_TREE, build_int_cst (vword_type, 0), rtcfg);
 
-	/* Then, successors.  */
-	for (int i = EDGE_COUNT (bb->succs); i--; )
-	  if (push_rtcfg_pair (EDGE_SUCC (bb, i)->dest, bb,
-			       EXIT_BLOCK_PTR_FOR_FN (cfun)))
-	    break;
+	if (!postcheck)
+	  {
+	    /* Then, successors.  */
+	    if (!checkpoint
+		|| !push_rtcfg_pair (EXIT_BLOCK_PTR_FOR_FN (cfun),
+				     bb, EXIT_BLOCK_PTR_FOR_FN (cfun)))
+	      for (int i = EDGE_COUNT (bb->succs); i--; )
+		if (push_rtcfg_pair (EDGE_SUCC (bb, i)->dest, bb,
+				     EXIT_BLOCK_PTR_FOR_FN (cfun)))
+		  break;
+	  }
 	rtcfg = tree_cons (NULL_TREE, build_int_cst (vword_type, 0), rtcfg);
       }
-    else
+    else if (!postcheck)
       {
 	/* Schedule test to fail if the block was reached but somehow none
 	   of its predecessors were.  */
@@ -677,6 +1045,8 @@ public:
 	gassign *blkruns = gimple_build_assign (ckpart, unshare_expr (bit));
 	gimple_seq_add_stmt (&ckseq, blkruns);
 
+	if (checkpoint)
+	  build_block_check (EXIT_BLOCK_PTR_FOR_FN (cfun));
 	for (int i = 0, e = EDGE_COUNT (bb->succs); i < e; i++)
 	  build_block_check (EDGE_SUCC (bb, i)->dest);
 
@@ -687,21 +1057,357 @@ public:
   }
 };
 
+/* It might be useful to avoid checking before noreturn calls that are
+   known to always finish by throwing an exception, rather than by
+   ending the program or looping forever.  Such functions would have
+   to be annotated somehow, with an attribute or flag, so that
+   exception-raising functions, such as C++'s __cxa_throw,
+   __cxa_rethrow, and Ada's gnat_rcheck_*, gnat_reraise*,
+   ada.exception.raise_exception*, and the language-independent
+   unwinders could be detected here and handled differently from other
+   noreturn functions.  */
+static bool
+always_throwing_noreturn_call_p (gimple *)
+{
+  return false;
+}
+
 /* Control flow redundancy hardening: record the execution path, and
    verify at exit that an expect path was taken.  */
 
 unsigned int
-pass_harden_control_flow_redundancy::execute (function *)
+pass_harden_control_flow_redundancy::execute (function *fun)
 {
-  rt_bb_visited vstd;
-
+  bool const check_at_escaping_exceptions
+    = (flag_exceptions
+       && flag_harden_control_flow_redundancy_check_exceptions);
+  bool const check_before_noreturn_calls
+    = flag_harden_control_flow_redundancy_check_noreturn > HCFRNR_NEVER;
+  bool const check_before_nothrow_noreturn_calls
+    = (check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn >= HCFRNR_NOTHROW);
+  bool const check_before_throwing_noreturn_calls
+    = (flag_exceptions
+       && check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn > HCFRNR_NOTHROW);
+  bool const check_before_always_throwing_noreturn_calls
+    = (flag_exceptions
+       && check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn >= HCFRNR_ALWAYS);
   basic_block bb;
-  FOR_EACH_BB_FN (bb, cfun)
-    vstd.visit (bb);
+  basic_block bb_eh_cleanup = NULL;
+
+  if (check_at_escaping_exceptions)
+    {
+      int lp_eh_cleanup = -1;
+
+      /* Record the preexisting blocks, to avoid visiting newly-created
+	 blocks.  */
+      auto_sbitmap to_visit (last_basic_block_for_fn (fun));
+      bitmap_clear (to_visit);
+
+      FOR_EACH_BB_FN (bb, fun)
+	bitmap_set_bit (to_visit, bb->index);
+
+      /* Scan the blocks for stmts with escaping exceptions, that
+	 wouldn't be denoted in the CFG, and associate them with an
+	 empty cleanup handler around the whole function.  Walk
+	 backwards, so that even when we split the block, */
+      sbitmap_iterator it;
+      unsigned i;
+      EXECUTE_IF_SET_IN_BITMAP (to_visit, 0, i, it)
+	{
+	  bb = BASIC_BLOCK_FOR_FN (fun, i);
+
+	  for (gimple_stmt_iterator gsi = gsi_last_bb (bb);
+	       !gsi_end_p (gsi); gsi_prev (&gsi))
+	    {
+	      gimple *stmt = gsi_stmt (gsi);
+	      if (!stmt_could_throw_p (fun, stmt))
+		continue;
+
+	      /* If it must not throw, or if it already has a handler,
+		 we need not worry about it.  */
+	      if (lookup_stmt_eh_lp (stmt) != 0)
+		continue;
+
+	      /* Don't split blocks at, nor add EH edges to, tail
+		 calls, we will add verification before the call
+		 anyway.  */
+	      if (is_a <gcall *> (stmt)
+		  && (gimple_call_must_tail_p (as_a <gcall *> (stmt))
+		      || gimple_call_tail_p (as_a <gcall *> (stmt))
+		      || returning_call_p (as_a <gcall *> (stmt))))
+		continue;
+
+	      if (!gsi_one_before_end_p (gsi))
+		split_block (bb, stmt);
+	      /* A resx or noreturn call needs not be associated with
+		 the cleanup handler if we're going to add checking
+		 before it.  We only test cases that didn't require
+		 block splitting because noreturn calls would always
+		 be at the end of blocks, and we test for zero
+		 successors because if there is an edge, it's not
+		 noreturn, as any EH edges would have already been
+		 caught by the lookup_stmt_eh_lp test above.  */
+	      else if (check_before_noreturn_calls
+		       && EDGE_COUNT (bb->succs) == 0
+		       && (is_a <gresx *> (stmt)
+			   ? check_before_always_throwing_noreturn_calls
+			   : (!is_a <gcall *> (stmt)
+			      || !gimple_call_noreturn_p (stmt))
+			   ? (gcc_unreachable (), false)
+			   : (!flag_exceptions
+			      || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+			   ? check_before_nothrow_noreturn_calls
+			   : always_throwing_noreturn_call_p (stmt)
+			   ? check_before_always_throwing_noreturn_calls
+			   : check_before_throwing_noreturn_calls))
+		{
+		  if (dump_file)
+		    {
+		      fprintf (dump_file,
+			       "Bypassing cleanup for noreturn stmt"
+			       " in block %i:\n",
+			       bb->index);
+		      print_gimple_stmt (dump_file, stmt, 0);
+		    }
+		  continue;
+		}
+
+	      if (!bb_eh_cleanup)
+		{
+		  bb_eh_cleanup = create_empty_bb (bb);
+		  if (dom_info_available_p (CDI_DOMINATORS))
+		    set_immediate_dominator (CDI_DOMINATORS, bb_eh_cleanup, bb);
+		  if (current_loops)
+		    add_bb_to_loop (bb_eh_cleanup, current_loops->tree_root);
+
+		  /* Make the new block an EH cleanup for the call.  */
+		  eh_region new_r = gen_eh_region_cleanup (NULL);
+		  eh_landing_pad lp = gen_eh_landing_pad (new_r);
+		  tree label = gimple_block_label (bb_eh_cleanup);
+		  lp->post_landing_pad = label;
+		  EH_LANDING_PAD_NR (label) = lp_eh_cleanup = lp->index;
+
+		  /* Just propagate the exception.
+		     We will later insert the verifier call.  */
+		  gimple_stmt_iterator ehgsi;
+		  ehgsi = gsi_after_labels (bb_eh_cleanup);
+		  gresx *resx = gimple_build_resx (new_r->index);
+		  gsi_insert_before (&ehgsi, resx, GSI_SAME_STMT);
+
+		  if (dump_file)
+		    fprintf (dump_file,
+			     "Created cleanup block %i:\n",
+			     bb_eh_cleanup->index);
+		}
+	      else if (dom_info_available_p (CDI_DOMINATORS))
+		{
+		  basic_block immdom;
+		  immdom = get_immediate_dominator (CDI_DOMINATORS,
+						    bb_eh_cleanup);
+		  if (!dominated_by_p (CDI_DOMINATORS, bb, immdom))
+		    {
+		      immdom = nearest_common_dominator (CDI_DOMINATORS,
+							 immdom, bb);
+		      set_immediate_dominator (CDI_DOMINATORS,
+					       bb_eh_cleanup, immdom);
+		    }
+		}
+
+	      if (dump_file)
+		{
+		  fprintf (dump_file,
+			   "Associated cleanup block with stmt in block %i:\n",
+			   bb->index);
+		  print_gimple_stmt (dump_file, stmt, 0);
+		}
+
+	      add_stmt_to_eh_lp (stmt, lp_eh_cleanup);
+	      /* Finally, wire the EH cleanup block into the CFG.  */
+	      make_eh_edges (stmt);
+	    }
+	}
+
+      if (bb_eh_cleanup)
+	{
+	  /* A cfg_cleanup after bb_eh_cleanup makes for a more compact
+	     rtcfg, and it avoids bb numbering differences when we split
+	     blocks because of trailing debug insns only.  */
+	  cleanup_tree_cfg ();
+	  gcc_checking_assert (EDGE_COUNT (bb_eh_cleanup->succs) == 0);
+	}
+    }
+
+  /* These record blocks with calls that are to be preceded by
+     checkpoints, such as noreturn calls (if so chosen), must-tail
+     calls, potential early-marked tail calls, and returning calls (if
+     so chosen).  */
+  int count_chkcall = 0;
+  auto_sbitmap chkcall_blocks (last_basic_block_for_fn (fun));
+  bitmap_clear (chkcall_blocks);
+
+  /* We wish to add verification at blocks without successors, such as
+     noreturn calls (raising or not) and the reraise at the cleanup
+     block, but not other reraises: they will go through the cleanup
+     block.  */
+  if (check_before_noreturn_calls)
+    FOR_EACH_BB_FN (bb, fun)
+      {
+	gimple_stmt_iterator gsi = gsi_last_bb (bb);
+	if (gsi_end_p (gsi))
+	  continue;
+	gimple *stmt = gsi_stmt (gsi);
 
-  vstd.check ();
+	if (EDGE_COUNT (bb->succs) == 0)
+	  {
+	    /* A stmt at the end of a block without any successors is
+	       either a resx or a noreturn call without a local
+	       handler.  Check that it's one of the desired
+	       checkpoints.  */
+	    if (flag_exceptions && is_a <gresx *> (stmt)
+		? (check_before_always_throwing_noreturn_calls
+		   || bb == bb_eh_cleanup)
+		: (!is_a <gcall *> (stmt)
+		   || !gimple_call_noreturn_p (stmt))
+		? (/* Catch cases in which successors would be
+		      expected.  */
+		   gcc_unreachable (), false)
+		: (!flag_exceptions
+		   || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+		? check_before_nothrow_noreturn_calls
+		: always_throwing_noreturn_call_p (stmt)
+		? check_before_always_throwing_noreturn_calls
+		: check_before_throwing_noreturn_calls)
+	      {
+		if (dump_file)
+		  {
+		    fprintf (dump_file,
+			     "Scheduling check before stmt"
+			     " in succ-less block %i:\n",
+			     bb->index);
+		    print_gimple_stmt (dump_file, stmt, 0);
+		  }
+
+		if (bitmap_set_bit (chkcall_blocks, bb->index))
+		  count_chkcall++;
+		else
+		  gcc_unreachable ();
+	      }
+	    continue;
+	  }
 
-  return 0;
+	/* If there are no exceptions, it would seem like any noreturn
+	   call must have zero successor edges, but __builtin_return
+	   gets successor edges.  We don't want to handle it here, it
+	   will be dealt with in sibcall_search_preds.  Otherwise,
+	   check for blocks without non-EH successors, but skip those
+	   with resx stmts and edges (i.e., those other than that in
+	   bb_eh_cleanup), since those will go through bb_eh_cleanup,
+	   that will have been counted as noreturn above because it
+	   has no successors.  */
+	gcc_checking_assert (bb != bb_eh_cleanup
+			     || !check_at_escaping_exceptions);
+	if (flag_exceptions && is_a <gresx *> (stmt)
+	    ? check_before_always_throwing_noreturn_calls
+	    : (!is_a <gcall *> (stmt)
+	       || !gimple_call_noreturn_p (stmt))
+	    ? false
+	    : (!flag_exceptions
+	       || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+	    ? false /* rather than check_before_nothrow_noreturn_calls */
+	    : always_throwing_noreturn_call_p (stmt)
+	    ? check_before_always_throwing_noreturn_calls
+	    : check_before_throwing_noreturn_calls)
+	  {
+	    gcc_checking_assert (single_succ_p (bb)
+				 && (single_succ_edge (bb)->flags & EDGE_EH));
+
+	    if (dump_file)
+	      {
+		fprintf (dump_file,
+			 "Scheduling check before stmt"
+			 " in EH-succ block %i:\n",
+			 bb->index);
+		print_gimple_stmt (dump_file, stmt, 0);
+	      }
+
+	    if (bitmap_set_bit (chkcall_blocks, bb->index))
+	      count_chkcall++;
+	    else
+	      gcc_unreachable ();
+	  }
+      }
+  else if (bb_eh_cleanup)
+    {
+      if (bitmap_set_bit (chkcall_blocks, bb_eh_cleanup->index))
+	count_chkcall++;
+      else
+	gcc_unreachable ();
+    }
+
+  gcc_checking_assert (!bb_eh_cleanup
+		       || bitmap_bit_p (chkcall_blocks, bb_eh_cleanup->index));
+
+  /* If we don't have edges to exit nor noreturn calls (including the
+     cleanup reraise), then we may skip instrumentation: that would
+     amount to a function that ends with an infinite loop.  */
+  if (!count_chkcall
+      && EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (fun)->preds) == 0)
+    {
+      if (dump_file)
+	fprintf (dump_file,
+		 "Disabling CFR, no exit paths to check\n");
+
+      return 0;
+    }
+
+  /* Search for must-tail calls, early-marked potential tail calls,
+     and, if requested, returning calls.  As we introduce early
+     checks, */
+  int count_postchk = 0;
+  auto_sbitmap postchk_blocks (last_basic_block_for_fn (fun));
+  bitmap_clear (postchk_blocks);
+  chk_edges_t chk_edges;
+  hardcfr_sibcall_search_preds (EXIT_BLOCK_PTR_FOR_FN (fun), chk_edges,
+				count_chkcall, chkcall_blocks,
+				count_postchk, postchk_blocks,
+				NULL);
+
+  rt_bb_visited vstd (chk_edges.length () + count_chkcall);
+
+  auto_sbitmap combined_blocks (last_basic_block_for_fn (fun));
+  bitmap_copy (combined_blocks, chkcall_blocks);
+  int i;
+  edge *e;
+  FOR_EACH_VEC_ELT (chk_edges, i, e)
+    if (!bitmap_set_bit (combined_blocks, (*e)->src->index))
+      /* There may be multiple chk_edges with the same src block;
+	 guard againt overlaps with chkcall_blocks only.  */
+      gcc_assert (!bitmap_bit_p (chkcall_blocks, (*e)->src->index));
+
+  /* Visit blocks in index order, because building rtcfg depends on
+     that.  Blocks must be compact, which the cleanup_cfg requirement
+     ensures.  This would also enable FOR_EACH_BB_FN to be used to
+     iterate in index order, but bb_eh_cleanup block splits and
+     insertions changes that.  */
+  gcc_checking_assert (n_basic_blocks_for_fn (fun)
+		       == last_basic_block_for_fn (fun));
+  for (int i = NUM_FIXED_BLOCKS; i < n_basic_blocks_for_fn (fun); i++)
+    {
+      bb = BASIC_BLOCK_FOR_FN (fun, i);
+      gcc_checking_assert (bb->index == i);
+      vstd.visit (bb, bitmap_bit_p (combined_blocks, i),
+		  bitmap_bit_p (postchk_blocks, i));
+    }
+
+  vstd.check (chk_edges, count_chkcall, chkcall_blocks);
+
+  return
+    TODO_update_ssa
+    | TODO_cleanup_cfg
+    | TODO_verify_il;
 }
 
 /* Instantiate a hardcfr pass.  */
diff --git a/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c b/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c
new file mode 100644
index 00000000000..a6992eb9f8e
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -O0 -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we don't insert checking before noreturn calls.  -O0 is tested
+   separately because h is not found to be noreturn without optimization.  */
+
+#include "torture/harden-cfr-noret.c"
+
+/* No out-of-line checks.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* Only one inline check at the end of f and of h2.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-always.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-always.c
new file mode 100644
index 00000000000..779896c60e8
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-always.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even enabling all checks before noreturn calls (leaving
+   returning calls enabled), we get checks before __builtin_return without
+   duplication (__builtin_return is both noreturn and a returning call).  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-never.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-never.c
new file mode 100644
index 00000000000..49ce17f5b93
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-never.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even enabling checks before never noreturn calls (leaving
+   returning calls enabled), we get checks before __builtin_return without
+   duplication (__builtin_return is both noreturn and a returning call).  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noopt.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noopt.c
new file mode 100644
index 00000000000..1512614791f
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noopt.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even disabling checks before both noreturn and returning
+   calls, we still get checks before __builtin_return.  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noret.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noret.c
new file mode 100644
index 00000000000..fd95bb7e3e3
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noret.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even disabling checks before returning calls (leaving noreturn
+   calls enabled), we still get checks before __builtin_return.  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-nothrow.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-nothrow.c
new file mode 100644
index 00000000000..c5c361234c4
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-nothrow.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even enabling checks before nothrow noreturn calls (leaving
+   returning calls enabled), we get checks before __builtin_return without
+   duplication (__builtin_return is both noreturn and a returning call).  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-retcl.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-retcl.c
new file mode 100644
index 00000000000..137dfbb95d6
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-retcl.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even disabling checks before noreturn calls (leaving returning
+   calls enabled), we still get checks before __builtin_return.  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c
index 70acdc95f25..b459ff6b864 100644
--- a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c
@@ -7,5 +7,11 @@ int f(int i) {
   return i;
 }
 
-/* Out-of-line checking, before both builtin_return and return.  */
+int g(int i) {
+  __builtin_return (&i);
+}
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
 /* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c
new file mode 100644
index 00000000000..8bd2d13ac18
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c
@@ -0,0 +1,18 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we don't insert checking before noreturn calls.  -O0 is tested
+   separately because h is not found to be noreturn without optimization, which
+   affects codegen for h2, so h2 is omitted here at -O0.  */
+
+#if !__OPTIMIZE__
+# define OMIT_H2
+#endif
+
+#include "harden-cfr-noret.c"
+
+
+/* No out-of-line checks.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* Only one inline check at the end of f.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c
new file mode 100644
index 00000000000..a804a6cfe59
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fno-exceptions -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that -fno-exceptions makes for implicit nothrow in noreturn
+   handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#include "harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c
new file mode 100644
index 00000000000..f390cfdbc59
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checking before nothrow noreturn calls.  */
+
+#include "harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c
new file mode 100644
index 00000000000..fdd803109a4
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c
@@ -0,0 +1,38 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checking before all noreturn calls.  */
+
+#ifndef ATTR_NOTHROW_OPT /* Overridden in harden-cfr-noret-noexcept.  */
+#define ATTR_NOTHROW_OPT __attribute__ ((__nothrow__))
+#endif
+
+extern void __attribute__ ((__noreturn__)) ATTR_NOTHROW_OPT g (void);
+
+void f(int i) {
+  if (i)
+    /* Out-of-line checks here...  */
+    g ();
+  /* ... and here.  */
+}
+
+void __attribute__ ((__noinline__, __noclone__))
+h(void) {
+  /* Inline check here.  */
+  g ();
+}
+
+#ifndef OMIT_H2 /* from harden-cfr-noret-never.  */
+void h2(void) {
+  /* Inline check either here, whether because of noreturn or tail call...  */
+  h ();
+  /* ... or here, if not optimizing.  */
+}
+#endif
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c
new file mode 100644
index 00000000000..6d11487bbba
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c
@@ -0,0 +1,8 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+#include "harden-cfr-tail.c"
+
+/* Inline checking after the calls, disabling tail calling.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 5 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 0 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c
new file mode 100644
index 00000000000..550b02ca088
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c
@@ -0,0 +1,35 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-exceptions -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checks before returning calls and alternate paths, even
+   at -O0, because of the explicit command-line flag.  */
+
+void g (void);
+void g2 (void);
+void g3 (void);
+
+void f (int i) {
+  if (!i)
+    /* Out-of-line checks here...  */
+    g ();
+  else if (i > 0)
+    /* here...  */
+    g2 ();
+  /* else */
+    /* and in the implicit else here.  */
+}
+
+void f2 (int i) {
+  if (!i)
+    /* Out-of-line check here...  */
+    g ();
+  else if (i > 0)
+    /* here...  */
+    g2 ();
+  else
+    /* and here.  */
+    g3 ();
+}
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 6 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 0 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
index 40d76c5c163..d5467eafa9f 100644
--- a/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
@@ -1,17 +1,52 @@
 /* { dg-do compile } */
-/* { dg-options "-fharden-control-flow-redundancy -fdump-tree-hardcfr -ffat-lto-objects" } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-hardcfr-check-exceptions -fdump-tree-hardcfr -ffat-lto-objects -Wno-return-type" } */
 
-/* We'd like to check that we insert checking so as to not disrupt tail calls,
-   but unfortunately mandatory tail calls are not available in C, and optimizing
-   calls as tail calls only takes place after hardcfr.  */
+/* Check that we insert CFR checking so as to not disrupt tail calls.
+   Mandatory tail calls are not available in C, and optimizing calls as tail
+   calls only takes place after hardcfr, so we insert checking before calls
+   followed by copies and return stmts with the same return value, that might
+   (or might not) end up optimized to tail calls.  */
 
-extern int g(int i);
+extern int g (int i);
 
-int f(int i) {
+int f1(int i) {
+  /* Inline check before the returning call.  */
   return g (i);
 }
 
-/* Inline checking before the tail call.  */
-/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
-/* Inline checking before the tail call.  */
-/* { dg-final { scan-tree-dump-times "\\\[tail\]" 1 "hardcfr" { xfail *-*-* } } } */
+extern void g2 (int i);
+
+void f2(int i) {
+  /* Inline check before the returning call, that ignores the returned value,
+     matching the value-less return.  */
+  g2 (i);
+  return;
+}
+
+void f3(int i) {
+  /* Inline check before the returning call.  */
+  g (i);
+}
+
+void f4(int i) {
+  if (i)
+    /* Out-of-line check before the returning call.  */
+    return g2 (i);
+  /* Out-of-line check before implicit return.  */
+}
+
+int f5(int i) {
+  /* Not regarded as a returning call, returning value other than callee's
+     returned value.  */
+  g (i);
+  /* Inline check after the non-returning call.  */
+  return i;
+}
+
+/* Out-of-line checks in f4, before returning calls and before return.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking in all other functions.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 4 "hardcfr" } } */
+/* Check before tail-call in all but f5, but f4 is out-of-line.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 3 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C b/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C
new file mode 100644
index 00000000000..17ea79f7cfb
--- /dev/null
+++ b/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C
@@ -0,0 +1,11 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects -O0" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions, and also checking before noreturn
+   calls.  h2 and h2b get an extra resx without ehcleanup.  */
+
+#include "torture/harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 16 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C b/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C
new file mode 100644
index 00000000000..f0338ccc361
--- /dev/null
+++ b/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C
@@ -0,0 +1,10 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -foptimize-sibling-calls -fdump-tree-hardcfr -O0" } */
+
+/* -fhardcfr-check-returning-calls gets implicitly disabled because,
+   -at O0, -foptimize-sibling-calls has no effect.  */
+
+#include "torture/harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 12 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C
new file mode 100644
index 00000000000..0d35920c7ee
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  */
+
+#include "harden-cfr-noret-no-nothrow.C"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Bypassing cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C
new file mode 100644
index 00000000000..b7d247ff43c
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C
@@ -0,0 +1,18 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  Expected results for =never and =nothrow are the same,
+   since the functions are not nothrow.  */
+
+#include "harden-cfr-noret-no-nothrow.C"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Associated cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C
new file mode 100644
index 00000000000..62c58cfd406
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C
@@ -0,0 +1,23 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#if ! __OPTIMIZE__
+void __attribute__ ((__noreturn__)) h (void);
+#endif
+
+#include "../../c-c++-common/torture/harden-cfr-noret.c"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Associated cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C
new file mode 100644
index 00000000000..0286f6e6d3f
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C
@@ -0,0 +1,20 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions, and also checking before noreturn
+   calls.  */
+
+#if ! __OPTIMIZE__
+/* Without optimization, functions with cleanups end up with an extra
+   resx that is not optimized out, so arrange to optimize them.  */
+void __attribute__ ((__optimize__ (1))) h2(void);
+void __attribute__ ((__optimize__ (1))) h2b(void);
+#endif
+
+#include "harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 14 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
+/* h, h2, h2b, and h4.  */
+/* { dg-final { scan-tree-dump-times "Bypassing" 4 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C
new file mode 100644
index 00000000000..885b0b236af
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C
@@ -0,0 +1,11 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions  -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we do not insert cleanups for checking around the bodies
+   of maybe-throwing functions.  h4 doesn't get any checks, because we
+   don't have noreturn checking enabled.  */
+
+#include "harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 6 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C
new file mode 100644
index 00000000000..32def637255
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C
@@ -0,0 +1,31 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -foptimize-sibling-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions.  These results depend on checking before
+   returning calls, which is only enabled when sibcall optimizations
+   are enabled, so change the optimization mode to -O1 for f and f2,
+   so that -foptimize-sibling-calls can take effect and enable
+   -fhardcfr-check-returning-calls, so that we get the same results.
+   There is a separate test for -O0.  */
+
+#if ! __OPTIMIZE__
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) f(int i);
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) f2(int i);
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) h3(void);
+#endif
+
+#include "harden-cfr-throw.C"
+
+/* f gets out-of-line checks before the unwrapped tail call and in the
+   else edge.  */
+/* f2 gets out-of-line checks before both unwrapped tail calls.  */
+/* h gets out-of-line checks before the implicit return and in the
+   cleanup block.  */
+/* h2 and h2b get out-of-line checks before the cleanup returning
+   call, and in the cleanup block.  */
+/* h3 gets an inline check before the __cxa_end_catch returning call.  */
+/* h4 gets an inline check in the cleanup block.  */
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 10 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C
new file mode 100644
index 00000000000..992fbdad381
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C
@@ -0,0 +1,65 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions.  */
+
+extern void g (void);
+extern void g2 (void);
+
+void f(int i) {
+  if (i)
+    g ();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void f2(int i) {
+  if (i)
+    g ();
+  else
+    g2 ();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h(void) {
+  try {
+    g ();
+  } catch (...) {
+    throw;
+  }
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+struct needs_cleanup {
+  ~needs_cleanup();
+};
+
+void h2(void) {
+  needs_cleanup y; /* No check in the cleanup handler.  */
+  g();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+extern void __attribute__ ((__nothrow__)) another_cleanup (void*);
+
+void h2b(void) {
+  int x __attribute__ ((cleanup (another_cleanup)));
+  g();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h3(void) {
+  try {
+    throw 1;
+  } catch (...) {
+  }
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h4(void) {
+  throw 1;
+  /* Inline check in the cleanup around the __cxa_throw noreturn call.  */
+}
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 12 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c b/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c
new file mode 100644
index 00000000000..8e4ee1fab08
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c
@@ -0,0 +1,15 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C makes for implicit nothrow in noreturn handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#include "../../c-c++-common/torture/harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c b/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c
new file mode 100644
index 00000000000..634d98f1ffc
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c
@@ -0,0 +1,40 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-hardcfr-check-exceptions -fdump-tree-hardcfr -ffat-lto-objects -Wno-return-type" } */
+
+/* In C only, check some additional cases (comparing with
+   c-c++-common/torture/harden-cfr-tail.c) of falling off the end of non-void
+   function.  C++ would issue an unreachable call in these cases.  */
+
+extern int g (int i);
+
+int f1(int i) {
+  /* Inline check before the returning call, that doesn't return anything.  */
+  g (i);
+  /* Implicit return without value, despite the return type; this combination
+     enables tail-calling of g, and is recognized as a returning call.  */
+}
+
+extern void g2 (int i);
+
+int f2(int i) {
+  /* Inline check before the returning call, that disregards its return
+     value.  */
+  g2 (i);
+  /* Implicit return without value, despite the return type; this combination
+     enables tail-calling of g2, and is recognized as a returning call.  */
+}
+
+int f3(int i) {
+  if (i)
+    /* Out-of-line check before the returning call.  */
+    return g (i);
+  /* Out-of-line check before implicit return.  */
+}
+
+/* Out-of-line checks in f3, before returning calls and before return.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking in all other functions.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
+/* Check before tail-call in all functions, but f3 is out-of-line.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/gnat.dg/hardcfr.adb b/gcc/testsuite/gnat.dg/hardcfr.adb
index b578a913d75..abe1605c029 100644
--- a/gcc/testsuite/gnat.dg/hardcfr.adb
+++ b/gcc/testsuite/gnat.dg/hardcfr.adb
@@ -1,5 +1,5 @@
 --  { dg-do run }
---  { dg-options "-fharden-control-flow-redundancy -fdump-tree-hardcfr --param=hardcfr-max-blocks=22 --param=hardcfr-max-inline-blocks=12 -O0" }
+--  { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions -fdump-tree-hardcfr --param=hardcfr-max-blocks=22 --param=hardcfr-max-inline-blocks=12 -O0" }
 
 procedure HardCFR is
    function F (I, J : Integer) return Integer is
diff --git a/libgcc/hardcfr.c b/libgcc/hardcfr.c
index 8ef29428111..55f6b995f2f 100644
--- a/libgcc/hardcfr.c
+++ b/libgcc/hardcfr.c
@@ -60,14 +60,25 @@ extern void __hardcfr_check (size_t blocks,
 			     vword const *visited,
 			     vword const *cfg);
 
+/* Compute the MASK for the bit representing BLOCK in WORDIDX's vword in a
+   visited blocks bit array.  */
+static inline void
+block2mask (size_t const block, vword *const mask, size_t *const wordidx)
+{
+  size_t wbits = __CHAR_BIT__ * sizeof (vword);
+  *wordidx = block / wbits;
+  *mask = (vword)1 << (block % wbits);
+}
 
 /* Check whether the bit corresponding to BLOCK is set in VISITED.  */
 static inline bool
 visited_p (size_t const block, vword const *const visited)
 {
-  size_t wbits = __CHAR_BIT__ * sizeof (vword);
-  vword w = visited[block / wbits];
-  return (w & ((vword)1 << (block % wbits))) != 0;
+  vword mask;
+  size_t wordidx;
+  block2mask (block, &mask, &wordidx);
+  vword w = visited[wordidx];
+  return (w & mask) != 0;
 }
 
 /* Read and consume a mask from **CFG_IT.  (Consume meaning advancing the
@@ -118,19 +129,19 @@ consume_seq (vword const **const cfg_it)
    we reach the terminator without finding any.  Consume the entire sequence
    otherwise, so that *CFG_IT points just past the terminator, which may be the
    beginning of the next sequence.  */
-static inline void
+static inline bool
 check_seq (vword const *const visited, vword const **const cfg_it)
 {
   vword mask;
   size_t wordidx;
 
   /* If the block was visited, check that at least one of the
-     preds was also visited.  */
+     preds/succs was also visited.  */
   do
     /* If we get to the end of the sequence without finding any
        match, something is amiss.  */
     if (!next_pair (cfg_it, &mask, &wordidx))
-      __builtin_trap ();
+      return false;
   /* Keep searching until we find a match, at which point the
      condition is satisfied.  */
   while (!test_mask (visited, mask, wordidx));
@@ -139,6 +150,94 @@ check_seq (vword const *const visited, vword const **const cfg_it)
      skipped the block, so as to position the iterator at the beginning of the
      next .  */
   consume_seq (cfg_it);
+
+  return true;
+}
+
+/* Print out the CFG with BLOCKS blocks, presumed to be associated with CALLER.
+   This is expected to be optimized out entirely, unless the verbose part of
+   __hardcfr_check_fail is enabled.  */
+static inline void
+__hardcfr_debug_cfg (size_t const blocks,
+		     void const *const caller,
+		     vword const *const cfg)
+{
+  __builtin_printf ("CFG at %p, for %p", cfg, caller);
+  vword const *cfg_it = cfg;
+  for (size_t i = 0; i < blocks; i++)
+    {
+      vword mask; size_t wordidx;
+      block2mask (i, &mask, &wordidx);
+      __builtin_printf ("\nblock %lu (%lu/0x%lx)\npreds: ",
+			(unsigned long)i,
+			(unsigned long)wordidx, (unsigned long)mask);
+      while (next_pair (&cfg_it, &mask, &wordidx))
+	__builtin_printf (" (%lu/0x%lx)",
+			  (unsigned long)wordidx, (unsigned long)mask);
+      __builtin_printf ("\nsuccs: ");
+      while (next_pair (&cfg_it, &mask, &wordidx))
+	__builtin_printf (" (%lu/0x%lx)",
+			  (unsigned long)wordidx, (unsigned long)mask);
+    }
+  __builtin_printf ("\n");
+}
+
+#ifndef ATTRIBUTE_UNUSED
+# define ATTRIBUTE_UNUSED __attribute__ ((__unused__))
+#endif
+
+/* This is called when an out-of-line hardcfr check fails.  All the arguments
+   are ignored, and it just traps, unless HARDCFR_VERBOSE_FAIL is enabled.  IF
+   it is, it prints the PART of the CFG, expected to have BLOCKS blocks, that
+   failed at CALLER's BLOCK, and the VISITED bitmap.  When the verbose mode is
+   enabled, it also forces __hardcfr_debug_cfg (above) to be compiled into an
+   out-of-line function, that could be called from a debugger.
+   */
+static inline void
+__hardcfr_check_fail (size_t const blocks ATTRIBUTE_UNUSED,
+		      vword const *const visited,
+		      vword const *const cfg ATTRIBUTE_UNUSED,
+		      size_t const block ATTRIBUTE_UNUSED,
+		      int const part ATTRIBUTE_UNUSED,
+		      void const *const caller ATTRIBUTE_UNUSED)
+{
+#if HARDCFR_VERBOSE_FAIL
+  static const char *parts[] = { "preds", "succs" };
+
+  vword mask; size_t wordidx;
+  block2mask (block, &mask, &wordidx);
+  __builtin_printf ("hardcfr fail at %p block %lu (%lu/0x%lx), expected %s:",
+		    caller, (unsigned long)block,
+		    (unsigned long)wordidx, (unsigned long)mask,
+		    parts[part]);
+
+  /* Skip data for previous blocks.  */
+  vword const *cfg_it = cfg;
+  for (size_t i = block; i--; )
+    {
+      consume_seq (&cfg_it);
+      consume_seq (&cfg_it);
+    }
+  for (size_t i = part; i--; )
+    consume_seq (&cfg_it);
+
+  while (next_pair (&cfg_it, &mask, &wordidx))
+    __builtin_printf (" (%lu/0x%lx)",
+		      (unsigned long)wordidx, (unsigned long)mask);
+
+  __builtin_printf ("\nvisited:");
+  block2mask (blocks, &mask, &wordidx);
+  for (size_t i = 0; i <= wordidx; i++)
+    __builtin_printf (" (%lu/0x%lx)",
+		      (unsigned long)i, (unsigned long)visited[i]);
+  __builtin_printf ("\n");
+
+  /* Reference __hardcfr_debug_cfg so that it's output out-of-line, so that it
+     can be called from a debugger.  */
+  if (!caller || caller == __hardcfr_debug_cfg)
+    return;
+#endif
+  __builtin_trap ();
 }
 
 /* Check that, for each of the BLOCKS basic blocks, if its bit is set in
@@ -168,9 +267,13 @@ __hardcfr_check (size_t const blocks,
       else
 	{
 	  /* Check predecessors.  */
-	  check_seq (visited, &cfg_it);
+	  if (!check_seq (visited, &cfg_it))
+	    __hardcfr_check_fail (blocks, visited, cfg, i, 0,
+				  __builtin_return_address (0));
 	  /* Check successors.  */
-	  check_seq (visited, &cfg_it);
+	  if (!check_seq (visited, &cfg_it))
+	    __hardcfr_check_fail (blocks, visited, cfg, i, 1,
+				  __builtin_return_address (0));
 	}
     }
 }

^ permalink raw reply	[flat|nested] 13+ messages in thread
* [gcc(refs/users/aoliva/heads/testme)] hardcfr: add optional checkpoints
@ 2023-06-08 10:42 Alexandre Oliva
  0 siblings, 0 replies; 13+ messages in thread
From: Alexandre Oliva @ 2023-06-08 10:42 UTC (permalink / raw)
  To: gcc-cvs

https://gcc.gnu.org/g:a0263378ae0f4aa6f7abb7f89fe3da71cb6f96ff

commit a0263378ae0f4aa6f7abb7f89fe3da71cb6f96ff
Author: Alexandre Oliva <oliva@adacore.com>
Date:   Thu Jun 8 01:34:57 2023 -0300

    hardcfr: add optional checkpoints
    
    Previously, control flow redundancy only checked the visited bitmap
    against the control flow graph at return points and before mandatory
    tail calls, missing various other possibilities of exiting a
    subprogram, such as by raising or propagating exceptions, and calling
    noreturn functions.  The checks inserted before returns also prevented
    potential tail-call optimizations.
    
    This incremental change introduces options to control checking at each
    of these previously-missed checkpoints.  Unless disabled, a cleanup is
    introduced to check when an exceptions escapes a subprogram.  To avoid
    disrupting sibcall optimizations, when they are enabled, checks are
    introduced before calls whose results are immediately returned,
    whether or not they are ultimately optimized.  If enabled, checks are
    introduced before noreturn calls and exception raises, or only before
    nothrow noreturn calls.
    
    
    for  gcc/ChangeLog
    
            * common.opt (-fhardcfr-check-returning-calls): New.
            (-fhardcfr-check-exceptions): New.
            (-fhardcfr-check-noreturn-calls=*): New.
            (Enum hardcfr_check_noreturn_calls): New.
            * doc/invoke.texi: Document them.
            * flag-types.h (enum hardcfr_noret): New.
            * gimple-harden-control-flow.cc: Include more headers.
            (pass_data_harden_control_flow_redundancy): Move
            properties_finish to pass return.
            (pass_harden_control_flow_redundancy::gate): Move
            no-edge-to-exit checking to execute, after exception
            transformations.  Use symbolic NUM_FIXED_BLOCKS.  Test for CFG
            before testing block count.
            (check_returning_calls_p): New.
            (hardcfr_scan_block): New.
            (returnign_call_p): New.
            (chk_edges_t): New.
            (hardcfr_sibcall_search_block): New.
            (hardcfr_sibcall_search_preds): New.
            (rt_bb_visited::num2idx): Use symbolic NUM_FIXED_BLOCKS.
            (rt_bb_visited::rt_bb_visited): Likewise.  Take checkpoints
            count, test it to select out-of-line checks.
            (rt_bb_visited::insert_exit_check): Removed.
            (rt_bb_visited::insert_exit_check_in_blocks): New.
            (rt_bb_visited::insert_exit_check_on_edge): New.
            (rt_bb_visited::check): Take checkpoint edges and blocks.
            Move initialization insertion to the end.  Insert checks on
            edges, and before calls in select blocks.  Clobber the visited
            array to the check sequence.  Output actions to dump_file.
            (rt_bb_visited::visit): Take checkpoint and postcheck.  Assume
            an edge to exit if checkpoint, and skip testing if postcheck.
            (always_throwing_noreturn_call_p): New, dummy.
            (pass_harden_control_flow_redundancy::execute): Introduce
            cleanup block if requested and needed.  Scan blocks for
            noreturn and returning calls if checkpoints before them were
            requested.  Log actions to dump_file.  Visit blocks in index
            order.
    
    for  gcc/testsuite/ChangeLog
    
            * c-c++-common/harden-cfr-noret-never-O0.c: New.
            * c-c++-common/torture/harden-cfr-noret-never.c: New.
            * c-c++-common/torture/harden-cfr-noret-noexcept.c: New.
            * c-c++-common/torture/harden-cfr-noret-nothrow.c: New.
            * c-c++-common/torture/harden-cfr-noret.c: New.
            * c-c++-common/torture/harden-cfr-notail.c: New.
            * c-c++-common/torture/harden-cfr-returning.c: New.
            * c-c++-common/torture/harden-cfr-tail.c: Extend.
            * c-c++-common/torture/harden-cfr-bret-always.c: New.
            * c-c++-common/torture/harden-cfr-bret-never.c: New.
            * c-c++-common/torture/harden-cfr-bret-noopt.c: New.
            * c-c++-common/torture/harden-cfr-bret-noret.c: New.
            * c-c++-common/torture/harden-cfr-bret-nothrow.c: New.
            * c-c++-common/torture/harden-cfr-bret-retcl.c: New.
            * c-c++-common/torture/harden-cfr-bret.c (g): New.
            * g++.dg/harden-cfr-throw-always-O0.C: New.
            * g++.dg/harden-cfr-throw-returning-O0.C: New.
            * g++.dg/torture/harden-cfr-noret-always-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-noret-never-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-noret-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-throw-always.C: New.
            * g++.dg/torture/harden-cfr-throw-nocleanup.C: New.
            * g++.dg/torture/harden-cfr-throw-returning.C: New.
            * g++.dg/torture/harden-cfr-throw.C: New.
            * gcc.dg/torture/harden-cfr-noret-no-nothrow.c: New.
            * gcc.dg/torture/harden-cfr-tail-ub.c: New.
            * gnat.dg/hardcfr.adb: Disable checking at exceptions.
    
    for  libgcc/ChangeLog
    
            * hardcfr.c (block2mask): Split out of...
            (visited_p): ... this.
            (check_seq): Move trapping out, return result instead.
            (__hardcfr_debug_cfg): New.
            (__hardcfr_check_fail): New, with optional verbose error
            printing before trapping.
            (__hardcfr_check): Call __hardcfr_check_fail when check_seq
            fails.

Diff:
---
 gcc/common.opt                                     |  33 +
 gcc/doc/invoke.texi                                |  74 +-
 gcc/flag-types.h                                   |  10 +
 gcc/gimple-harden-control-flow.cc                  | 926 ++++++++++++++++++---
 .../c-c++-common/harden-cfr-noret-never-O0.c       |  12 +
 .../c-c++-common/torture/harden-cfr-bret-always.c  |  13 +
 .../c-c++-common/torture/harden-cfr-bret-never.c   |  13 +
 .../c-c++-common/torture/harden-cfr-bret-noopt.c   |  12 +
 .../c-c++-common/torture/harden-cfr-bret-noret.c   |  12 +
 .../c-c++-common/torture/harden-cfr-bret-nothrow.c |  13 +
 .../c-c++-common/torture/harden-cfr-bret-retcl.c   |  12 +
 .../c-c++-common/torture/harden-cfr-bret.c         |   8 +-
 .../c-c++-common/torture/harden-cfr-noret-never.c  |  18 +
 .../torture/harden-cfr-noret-noexcept.c            |  16 +
 .../torture/harden-cfr-noret-nothrow.c             |  13 +
 .../c-c++-common/torture/harden-cfr-noret.c        |  38 +
 .../c-c++-common/torture/harden-cfr-notail.c       |   8 +
 .../c-c++-common/torture/harden-cfr-returning.c    |  35 +
 .../c-c++-common/torture/harden-cfr-tail.c         |  55 +-
 gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C  |  11 +
 .../g++.dg/harden-cfr-throw-returning-O0.C         |  10 +
 .../torture/harden-cfr-noret-always-no-nothrow.C   |  16 +
 .../torture/harden-cfr-noret-never-no-nothrow.C    |  18 +
 .../g++.dg/torture/harden-cfr-noret-no-nothrow.C   |  23 +
 .../g++.dg/torture/harden-cfr-throw-always.C       |  20 +
 .../g++.dg/torture/harden-cfr-throw-nocleanup.C    |  11 +
 .../g++.dg/torture/harden-cfr-throw-returning.C    |  31 +
 gcc/testsuite/g++.dg/torture/harden-cfr-throw.C    |  65 ++
 .../gcc.dg/torture/harden-cfr-noret-no-nothrow.c   |  15 +
 gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c  |  40 +
 gcc/testsuite/gnat.dg/hardcfr.adb                  |   2 +-
 libgcc/hardcfr.c                                   | 119 ++-
 32 files changed, 1568 insertions(+), 134 deletions(-)

diff --git a/gcc/common.opt b/gcc/common.opt
index ff89b2d1140..c2855cc19a9 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -1805,6 +1805,39 @@ fharden-control-flow-redundancy
 Common Var(flag_harden_control_flow_redundancy) Optimization
 Harden control flow by recording and checking execution paths.
 
+fhardcfr-check-returning-calls
+Common Var(flag_harden_control_flow_redundancy_check_returning_calls) Init(-1) Optimization
+Check CFR execution paths also before calls followed by returns of their results.
+
+fhardcfr-check-exceptions
+Common Var(flag_harden_control_flow_redundancy_check_exceptions) Init(-1) Optimization
+Check CFR execution paths also when exiting a function through an exception.
+
+fhardcfr-check-noreturn-calls=
+Common Joined RejectNegative Enum(hardcfr_check_noreturn_calls) Var(flag_harden_control_flow_redundancy_check_noreturn) Init(HCFRNR_UNSPECIFIED) Optimization
+-fhardcfr-check-noreturn-calls=[always|nothrow|never]	Check CFR execution paths also before calling noreturn functions.
+
+Enum
+Name(hardcfr_check_noreturn_calls) Type(enum hardcfr_noret) UnknownError(unknown hardcfr noreturn checking level %qs)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(never) Value(HCFRNR_NEVER)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(nothrow) Value(HCFRNR_NOTHROW)
+
+; ??? There could be yet another option here, that checked before
+; noreturn calls, except for those known to always throw, if we had
+; means to distinguish noreturn functions known to always throw, such
+; as those used to (re)raise exceptions, from those that merely might
+; throw.  "not always" stands for "not always-throwing", but it also
+; contrasts with "always" below.
+; EnumValue
+; Enum(hardcfr_check_noreturn_calls) String(not-always) Value(HCFRNR_NOT_ALWAYS)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(always) Value(HCFRNR_ALWAYS)
+
 ; Nonzero means ignore `#ident' directives.  0 means handle them.
 ; Generate position-independent code for executables if possible
 ; On SVR4 targets, it also controls whether or not to emit a
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index 87a33b037c1..7742d21f142 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -635,7 +635,9 @@ Objective-C and Objective-C++ Dialects}.
 -fsanitize-undefined-trap-on-error  -fbounds-check
 -fcf-protection=@r{[}full@r{|}branch@r{|}return@r{|}none@r{|}check@r{]}
 -fharden-compares -fharden-conditional-branches
--fharden-control-flow-redundancy
+-fharden-control-flow-redundancy  -fhardcfr-check-exceptions
+-fhardcfr-check-returning-calls
+-fhardcfr-check-noreturn-calls=@r{[}always@r{|}nothrow@r{|}never@r{]}
 -fstack-protector  -fstack-protector-all  -fstack-protector-strong
 -fstack-protector-explicit  -fstack-check
 -fstack-limit-register=@var{reg}  -fstack-limit-symbol=@var{sym}
@@ -17243,11 +17245,75 @@ conditionals.
 @opindex fharden-control-flow-redundancy
 @item -fharden-control-flow-redundancy
 Emit extra code to set booleans when entering basic blocks, and to
-verify, at function exits, that they amount to an execution path that is
-consistent with the control flow graph, trapping otherwise.  Tuning
-options @option{--param hardcfr-max-blocks} and @option{--param
+verify and trap, at function exits, when the booleans do not form an
+execution path that is compatible with the control flow graph.
+
+Verification takes place before returns, before mandatory tail calls
+(see below) and, optionally, before escaping exceptions with
+@option{-fhardcfr-check-exceptions}, before returning calls with
+@option{-fhardcfr-check-returning-calls}, and before noreturn calls with
+@option{-fhardcfr-check-noreturn-calls}).  Tuning options
+@option{--param hardcfr-max-blocks} and @option{--param
 hardcfr-max-inline-blocks} are available.
 
+Tail call optimization takes place too late to affect control flow
+redundancy, but calls annotated as mandatory tail calls by language
+front-ends, and any calls marked early enough as potential tail calls
+would also have verification issued before the call, but these
+possibilities are merely theoretical, as these conditions can only be
+met when using custom compiler plugins.
+
+@opindex fhardcfr-check-exceptions
+@opindex fno-hardcfr-check-exceptions
+@item -fhardcfr-check-exceptions
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph at exception
+escape points, as if the function body was wrapped with a cleanup
+handler that performed the check and reraised.  This option is enabled
+by default; use @option{-fno-hardcfr-check-exceptions} to disable it.
+
+@opindex fhardcfr-check-returning-calls
+@opindex fno-hardcfr-check-returning-calls
+@item -fhardcfr-check-returning-calls
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph before any
+function call immediately followed by a return of its result, if any, so
+as to not prevent tail-call optimization, whether or not it is
+ultimately optimized to a tail call.
+
+This option is enabled by default whenever sibling call optimizations
+are enabled (see @option{-foptimize-sibling-calls}), but it can be
+enabled (or disabled, using its negated form) explicitly, regardless of
+the optimizations.
+
+@opindex fhardcfr-check-noreturn-calls
+@item -fhardcfr-check-noreturn-calls=@r{[}always@r{|}nothrow@r{|}never@r{]}
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph before
+@code{noreturn} calls, either all of them (@option{always}), those that
+may not return control to the caller through an exception either
+(@option{nothrow}), or none of them (@option{never}, the default).
+
+Checking before a @code{noreturn} function that may return control to
+the caller through an exception may cause checking to be performed more
+than once, if the exception is caught in the caller, whether by a
+handler or a cleanup.  When @option{-fhardcfr-check-exceptions} is also
+enabled, the compiler will avoid associating a @code{noreturn} call with
+the implicitly-added cleanup handler, since it would be redundant with
+the check performed before the call, but other handlers or cleanups in
+the function, if activated, will modify the recorded execution path and
+check it again when another checkpoint is hit.  The checkpoint may even
+be another @code{noreturn} call, so checking may end up performed
+multiple times.
+
+Various optimizers may cause calls to be marked as @code{noreturn}
+and/or @code{nothrow}, even in the absence of the corresponding
+attributes, which may affect the placement of checks before calls, as
+well as the addition of implicit cleanup handlers for them.  This
+unpredictability, and the fact that raising and reraising exceptions
+frequently amounts to implicitly calling @code{noreturn} functions, have
+made @option{never} the default setting for this option.
+
 @opindex fstack-protector
 @item -fstack-protector
 Emit extra code to check for buffer overflows, such as stack smashing
diff --git a/gcc/flag-types.h b/gcc/flag-types.h
index 78dbdbe90dd..6d6fd19fcd6 100644
--- a/gcc/flag-types.h
+++ b/gcc/flag-types.h
@@ -157,6 +157,16 @@ enum stack_reuse_level
   SR_ALL
 };
 
+/* Control Flow Redundancy hardening options for noreturn calls.  */
+enum hardcfr_noret
+{
+  HCFRNR_NEVER,
+  HCFRNR_NOTHROW,
+  HCFRNR_NOT_ALWAYS, /* Reserved for future use.  */
+  HCFRNR_ALWAYS,
+  HCFRNR_UNSPECIFIED = -1
+};
+
 /* The live patching level.  */
 enum live_patching_level
 {
diff --git a/gcc/gimple-harden-control-flow.cc b/gcc/gimple-harden-control-flow.cc
index 6b08846dbb1..5c66762d448 100644
--- a/gcc/gimple-harden-control-flow.cc
+++ b/gcc/gimple-harden-control-flow.cc
@@ -29,7 +29,12 @@ along with GCC; see the file COPYING3.  If not see
 #include "tree-pass.h"
 #include "ssa.h"
 #include "gimple-iterator.h"
+#include "gimple-pretty-print.h"
 #include "tree-cfg.h"
+#include "tree-cfgcleanup.h"
+#include "tree-eh.h"
+#include "except.h"
+#include "sbitmap.h"
 #include "basic-block.h"
 #include "cfghooks.h"
 #include "cfgloop.h"
@@ -60,9 +65,7 @@ const pass_data pass_data_harden_control_flow_redundancy = {
   0,	    // properties_provided
   0,	    // properties_destroyed
   TODO_cleanup_cfg, // properties_start
-  TODO_update_ssa
-  | TODO_cleanup_cfg
-  | TODO_verify_il, // properties_finish
+  0,        // properties_finish
 };
 
 class pass_harden_control_flow_redundancy : public gimple_opt_pass
@@ -79,16 +82,6 @@ public:
     if (!flag_harden_control_flow_redundancy)
       return false;
 
-    /* We don't verify when an exception escapes, propagated or raised
-       by the function itself, so we're only concerned with edges to
-       the exit block.  If there aren't any, the function doesn't
-       return normally, so there won't be any checking point, so
-       there's no point in running the pass.  Should we add
-       verification at exception escapes, we should at least look at
-       !flag_exceptions here.  */
-    if (EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (fun)->preds) == 0)
-      return false;
-
     /* Functions that return more than once, like setjmp and vfork
        (that also gets this flag set), will start recording a path
        after the first return, and then may take another path when
@@ -117,8 +110,9 @@ public:
 	return false;
       }
 
-    if (param_hardcfr_max_blocks > 0
-	&& n_basic_blocks_for_fn (fun) - 2 > param_hardcfr_max_blocks)
+    if (fun->cfg && param_hardcfr_max_blocks > 0
+	&& (n_basic_blocks_for_fn (fun) - NUM_FIXED_BLOCKS
+	    > param_hardcfr_max_blocks))
       {
 	warning_at (DECL_SOURCE_LOCATION (fun->decl), 0,
 		    "%qD has more than %u blocks, the requested"
@@ -134,6 +128,293 @@ public:
 
 }
 
+/* Return TRUE iff CFR checks should be inserted before returning
+   calls.  */
+
+static bool
+check_returning_calls_p ()
+{
+  return
+    flag_harden_control_flow_redundancy_check_returning_calls > 0
+    || (flag_harden_control_flow_redundancy_check_returning_calls < 0
+	/* Gates pass_tail_calls.  */
+	&& flag_optimize_sibling_calls
+	/* Gates pass_all_optimizations.  */
+	&& optimize >= 1 && !optimize_debug);
+}
+
+/* Scan BB from the end, updating *RETPTR if given as return stmts and
+   copies are found.  Return a call or a stmt that cannot appear after
+   a tail call, or NULL if the top of the block is reached without
+   finding any.  */
+
+static gimple *
+hardcfr_scan_block (basic_block bb, tree **retptr)
+{
+  gimple_stmt_iterator gsi;
+  for (gsi = gsi_last_bb (bb); !gsi_end_p (gsi); gsi_prev (&gsi))
+    {
+      gimple *stmt = gsi_stmt (gsi);
+
+      /* Ignore labels, returns, nops, clobbers and debug stmts.  */
+      if (gimple_code (stmt) == GIMPLE_LABEL
+	  || gimple_code (stmt) == GIMPLE_NOP
+	  || gimple_code (stmt) == GIMPLE_PREDICT
+	  || gimple_clobber_p (stmt)
+	  || is_gimple_debug (stmt))
+	continue;
+
+      if (gimple_code (stmt) == GIMPLE_RETURN)
+	{
+	  greturn *gret = as_a <greturn *> (stmt);
+	  if (retptr)
+	    {
+	      gcc_checking_assert (!*retptr);
+	      *retptr = gimple_return_retval_ptr (gret);
+	    }
+	  continue;
+	}
+
+      /* Check for a call.  */
+      if (is_gimple_call (stmt))
+	return stmt;
+
+      /* Allow simple copies to the return value, updating the return
+	 value to be found in earlier assignments.  */
+      if (retptr && *retptr && gimple_assign_single_p (stmt)
+	  && **retptr == gimple_assign_lhs (stmt))
+	{
+	  *retptr = gimple_assign_rhs1_ptr (stmt);
+	  continue;
+	}
+
+      return stmt;
+    }
+
+  /* Any other kind of stmt will prevent a tail call.  */
+  return NULL;
+}
+
+/* Return TRUE iff CALL is to be preceded by a CFR checkpoint, i.e.,
+   if it's a returning call (one whose result is ultimately returned
+   without intervening non-copy statements) and we're checking
+   returning calls, a __builtin_return call (noreturn with a path to
+   the exit block), a must-tail call, or a tail call.  */
+
+static bool
+returning_call_p (gcall *call)
+{
+  if (!(gimple_call_noreturn_p (call)
+	|| gimple_call_must_tail_p (call)
+	|| gimple_call_tail_p (call)
+	|| check_returning_calls_p ()))
+    return false;
+
+  /* Quickly check that there's a path to exit compatible with a
+     returning call.  Detect infinite loops through the counter.  */
+  basic_block bb = gimple_bb (call);
+  auto_vec<basic_block, 10> path;
+  for (int i = n_basic_blocks_for_fn (cfun);
+       bb != EXIT_BLOCK_PTR_FOR_FN (cfun) && i--;
+       bb = single_succ (bb))
+    if (!single_succ_p (bb)
+	|| (single_succ_edge (bb)->flags & EDGE_EH) != 0)
+      return false;
+    else
+      path.safe_push (bb);
+
+  /* Check the stmts in the blocks and trace the return value.  */
+  tree *retptr = NULL;
+  for (;;)
+    {
+      gcc_checking_assert (!path.is_empty ());
+      basic_block bb = path.pop ();
+      gimple *stop = hardcfr_scan_block (bb, &retptr);
+      if (stop)
+	{
+	  if (stop != call)
+	    return false;
+	  gcc_checking_assert (path.is_empty ());
+	  break;
+	}
+
+      gphi *retphi = NULL;
+      if (retptr && *retptr && TREE_CODE (*retptr) == SSA_NAME
+	  && !SSA_NAME_IS_DEFAULT_DEF (*retptr)
+	  && SSA_NAME_DEF_STMT (*retptr)
+	  && is_a <gphi *> (SSA_NAME_DEF_STMT (*retptr))
+	  && gimple_bb (SSA_NAME_DEF_STMT (*retptr)) == bb)
+	{
+	  retphi = as_a <gphi *> (SSA_NAME_DEF_STMT (*retptr));
+	  gcc_checking_assert (gimple_phi_result (retphi) == *retptr);
+	}
+      else
+	continue;
+
+      gcc_checking_assert (!path.is_empty ());
+      edge e = single_succ_edge (path.last ());
+      int i = EDGE_COUNT (bb->preds);
+      while (i--)
+	if (EDGE_PRED (bb, i) == e)
+	  break;
+      gcc_checking_assert (i >= 0);
+      retptr = gimple_phi_arg_def_ptr (retphi, i);
+    }
+
+  return (gimple_call_noreturn_p (call)
+	  || gimple_call_must_tail_p (call)
+	  || gimple_call_tail_p (call)
+	  || (gimple_call_lhs (call) == (retptr ? *retptr : NULL)
+	      && check_returning_calls_p ()));
+}
+
+typedef auto_vec<edge, 10> chk_edges_t;
+
+/* Declare for mutual recursion.  */
+static bool hardcfr_sibcall_search_preds (basic_block bb,
+					  chk_edges_t &chk_edges,
+					  int &count_chkcall,
+					  auto_sbitmap &chkcall_blocks,
+					  int &count_postchk,
+					  auto_sbitmap &postchk_blocks,
+					  tree *retptr);
+
+/* Search backwards from the end of BB for a mandatory or potential
+   sibcall.  Schedule the block to be handled sort-of like noreturn if
+   so.  Recurse to preds, with updated RETPTR, if the block only
+   contains stmts that may follow such a call, scheduling checking at
+   edges and marking blocks as post-check as needed.  Return true iff,
+   at the end of the block, a check will have already been
+   performed.  */
+
+static bool
+hardcfr_sibcall_search_block (basic_block bb,
+			      chk_edges_t &chk_edges,
+			      int &count_chkcall,
+			      auto_sbitmap &chkcall_blocks,
+			      int &count_postchk,
+			      auto_sbitmap &postchk_blocks,
+			      tree *retptr)
+{
+  /* Conditionals and internal exceptions rule out tail calls.  */
+  if (!single_succ_p (bb)
+      || (single_succ_edge (bb)->flags & EDGE_EH) != 0)
+    return false;
+
+  gimple *stmt = hardcfr_scan_block (bb, &retptr);
+  if (!stmt)
+    return hardcfr_sibcall_search_preds (bb, chk_edges,
+					 count_chkcall, chkcall_blocks,
+					 count_postchk, postchk_blocks,
+					 retptr);
+
+  if (!is_a <gcall *> (stmt))
+    return false;
+
+  /* Avoid disrupting mandatory or early-marked tail calls,
+     inserting the check before them.  This works for
+     must-tail calls, but tail calling as an optimization is
+     detected too late for us.
+
+     Also check for noreturn calls here.  Noreturn calls won't
+     normally have edges to exit, so they won't be found here,
+     but __builtin_return does, and we must check before
+     it, so handle it like a tail call.  */
+  gcall *call = as_a <gcall *> (stmt);
+  if (!(gimple_call_noreturn_p (call)
+	|| gimple_call_must_tail_p (call)
+	|| gimple_call_tail_p (call)
+	|| (gimple_call_lhs (call) == (retptr ? *retptr : NULL)
+	    && check_returning_calls_p ())))
+    return false;
+
+  gcc_checking_assert (returning_call_p (call));
+
+  /* We found a call that is to be preceded by checking.  */
+  if (bitmap_set_bit (chkcall_blocks, bb->index))
+    ++count_chkcall;
+  else
+    gcc_unreachable ();
+  return true;
+}
+
+
+/* Search preds of BB for a mandatory or potential sibcall or
+   returning call, and arrange for the blocks containing them to have
+   a check inserted before the call, like noreturn calls.  If any
+   preds are found to perform checking, schedule checks at the edges
+   of those that don't, and mark BB as postcheck..  */
+
+static bool
+hardcfr_sibcall_search_preds (basic_block bb,
+			      chk_edges_t &chk_edges,
+			      int &count_chkcall,
+			      auto_sbitmap &chkcall_blocks,
+			      int &count_postchk,
+			      auto_sbitmap &postchk_blocks,
+			      tree *retptr)
+{
+  /* For the exit block, we wish to force a check at every
+     predecessor, so pretend we've already found a pred that had
+     checking, so that we schedule checking at every one of its pred
+     edges.  */
+  bool first = bb->index >= NUM_FIXED_BLOCKS;
+  bool postchecked = true;
+
+  gphi *retphi = NULL;
+  if (retptr && *retptr && TREE_CODE (*retptr) == SSA_NAME
+      && !SSA_NAME_IS_DEFAULT_DEF (*retptr)
+      && SSA_NAME_DEF_STMT (*retptr)
+      && is_a <gphi *> (SSA_NAME_DEF_STMT (*retptr))
+      && gimple_bb (SSA_NAME_DEF_STMT (*retptr)) == bb)
+    {
+      retphi = as_a <gphi *> (SSA_NAME_DEF_STMT (*retptr));
+      gcc_checking_assert (gimple_phi_result (retphi) == *retptr);
+    }
+
+  for (int i = EDGE_COUNT (bb->preds); i--; first = false)
+    {
+      edge e = EDGE_PRED (bb, i);
+
+      bool checked
+	= hardcfr_sibcall_search_block (e->src, chk_edges,
+					count_chkcall, chkcall_blocks,
+					count_postchk, postchk_blocks,
+					!retphi ? retptr
+					: gimple_phi_arg_def_ptr (retphi, i));
+
+      if (first)
+	{
+	  postchecked = checked;
+	  continue;
+	}
+
+      /* When we first find a checked block, force a check at every
+	 other incoming edge we've already visited, and those we
+	 visit afterwards that don't have their own check, so that
+	 when we reach BB, the check has already been performed.  */
+      if (!postchecked && checked)
+	{
+	  for (int j = EDGE_COUNT (bb->preds); --j > i; )
+	    chk_edges.safe_push (EDGE_PRED (bb, j));
+	  postchecked = true;
+	}
+      if (postchecked && !checked)
+	chk_edges.safe_push (EDGE_PRED (bb, i));
+    }
+
+  if (postchecked && bb->index >= NUM_FIXED_BLOCKS)
+    {
+      if (bitmap_set_bit (postchk_blocks, bb->index))
+	count_postchk++;
+      else
+	gcc_unreachable ();
+    }
+
+  return postchecked;
+}
+
+
 class rt_bb_visited
 {
   /* Use a sufficiently wide unsigned type to hold basic block numbers.  */
@@ -176,8 +457,8 @@ class rt_bb_visited
      neither ENTRY nor EXIT, but maybe one-past-the-end, to compute
      the visited array length.  */
   blknum num2idx (blknum n) {
-    gcc_checking_assert (n >= 2 && n <= nblocks);
-    return (n - 2);
+    gcc_checking_assert (n >= NUM_FIXED_BLOCKS && n <= nblocks);
+    return (n - NUM_FIXED_BLOCKS);
   }
   /* Return the block vindex for BB, that must not be ENTRY or
      EXIT.  */
@@ -249,8 +530,7 @@ class rt_bb_visited
   }
 
   /* Set the bit corresponding to BB in VISITED.  Add to SEQ any
-     required gimple statements, and return SEQ, possibly
-     modified.  */
+     required gimple stmts, and return SEQ, possibly modified.  */
   gimple_seq vset (basic_block bb, gimple_seq seq = NULL)
   {
     tree bit, setme = vword (bb, &bit);
@@ -270,7 +550,7 @@ class rt_bb_visited
 
 public:
   /* Prepare to add control flow redundancy testing to CFUN.  */
-  rt_bb_visited ()
+  rt_bb_visited (int checkpoints)
     : nblocks (n_basic_blocks_for_fn (cfun)),
       vword_type (NULL), ckseq (NULL), rtcfg (NULL)
   {
@@ -353,8 +633,8 @@ public:
 					 NULL, NULL);
     gimple_seq_add_stmt (&ckseq, detach);
 
-    if (nblocks - 2 > blknum (param_hardcfr_max_inline_blocks)
-	|| !single_pred_p (EXIT_BLOCK_PTR_FOR_FN (cfun)))
+    if (nblocks - NUM_FIXED_BLOCKS > blknum (param_hardcfr_max_inline_blocks)
+	|| checkpoints > 1)
       {
 	/* Make sure vword_bits is wide enough for the representation
 	   of nblocks in rtcfg.  Compare with vword_bits << vword_bits,
@@ -379,65 +659,34 @@ public:
     gimple_seq_add_stmt (&ckseq, ckfail_init);
   }
 
-  /* Insert SEQ on E, or close enough (e.g., before a noreturn or tail
-     call at the end of E->src).  */
-  void insert_exit_check (gimple_seq seq, edge e)
+  /* Insert SEQ before a resx or a call in INSBB.  */
+  void insert_exit_check_in_block (gimple_seq seq, basic_block insbb)
   {
-    basic_block insbb = e->src;
-
-    /* If the returning block ends with a noreturn call, insert
-       checking before it.  This is particularly important for
-       __builtin_return.  Other noreturn calls won't have an edge to
-       the exit block, so they won't even be considered as exit paths.
-
-       Insert-on-edge inserts before other return stmts, but not
-       before calls, and if a single-block function had the check
-       sequence inserted after a noreturn call, it would be useless,
-       but checking would still flag it as malformed if block 2 has a
-       fallthru edge to the exit block.
-
-       Also avoid disrupting tail calls, inserting the check before
-       them.  This works for must-tail calls, but tail calling as an
-       optimization is detected too late for us.  */
     gimple_stmt_iterator gsi = gsi_last_bb (insbb);
-    gimple *ret = gsi_stmt (gsi);
-    if (ret && is_a <greturn *> (ret))
-      {
+
+    while (!gsi_end_p (gsi))
+      if (is_a <gresx *> (gsi_stmt (gsi))
+	  || is_a <gcall *> (gsi_stmt (gsi)))
+	break;
+      else
 	gsi_prev (&gsi);
-	if (!gsi_end_p (gsi))
-	  ret = gsi_stmt (gsi);
-      }
-    if (ret && is_a <gcall *> (ret)
-	&& (gimple_call_noreturn_p (ret)
-	    || gimple_call_must_tail_p (as_a <gcall *> (ret))
-	    || gimple_call_tail_p (as_a <gcall *> (ret))))
-      gsi_insert_seq_before (&gsi, seq, GSI_SAME_STMT);
-    else
-      gsi_insert_seq_on_edge_immediate (e, seq);
+
+    gsi_insert_seq_before (&gsi, seq, GSI_SAME_STMT);
   }
 
-  /* Add checking code on every exit edge, and initialization code on
-     the entry edge.  Before this point, the CFG has been undisturbed,
-     and all the needed data has been collected and safely stowed.  */
-  void check ()
+  /* Insert SEQ on E.  */
+  void insert_exit_check_on_edge (gimple_seq seq, edge e)
   {
-    /* Insert initializers for visited at the entry.  */
-    gimple_seq iseq = NULL;
-
-    gcall *vinit = gimple_build_call (builtin_decl_explicit
-				      (BUILT_IN_MEMSET), 3,
-				      build1 (ADDR_EXPR,
-					      build_pointer_type
-					      (TREE_TYPE (visited)),
-					      visited),
-				      integer_zero_node,
-				      TYPE_SIZE_UNIT (TREE_TYPE (visited)));
-    gimple_seq_add_stmt (&iseq, vinit);
-
-    gsi_insert_seq_on_edge_immediate (single_succ_edge
-				      (ENTRY_BLOCK_PTR_FOR_FN (cfun)),
-				      iseq);
+    gsi_insert_seq_on_edge_immediate (e, seq);
+  }
 
+  /* Add checking code to CHK_EDGES and CHKCALL_BLOCKS, and
+     initialization code on the entry edge.  Before this point, the
+     CFG has been undisturbed, and all the needed data has been
+     collected and safely stowed.  */
+  void check (chk_edges_t &chk_edges,
+	      int count_chkcall, auto_sbitmap const &chkcall_blocks)
+  {
     /* If we're using out-of-line checking, create and statically
        initialize the CFG checking representation, generate the
        checker call for the checking sequence, and insert it in all
@@ -498,28 +747,116 @@ public:
 						     rtcfg));
 	gimple_seq_add_stmt (&ckseq, call_chk);
 
+	gimple *clobber = gimple_build_assign (visited,
+					       build_clobber
+					       (TREE_TYPE (visited)));
+	gimple_seq_add_stmt (&ckseq, clobber);
+
 	/* If we have multiple exit edges, insert (copies of)
 	   ckseq in all of them.  */
-	for (int i = EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (cfun)->preds);
-	     i--; )
+	for (int i = chk_edges.length (); i--; )
 	  {
 	    gimple_seq seq = ckseq;
 	    /* Copy the sequence, unless we're dealing with the
 	       last edge (we're counting down to zero).  */
-	    if (i)
+	    if (i || count_chkcall)
+	      seq = gimple_seq_copy (seq);
+
+	    edge e = chk_edges[i];
+
+	    if (dump_file)
+	      {
+		if (e->dest == EXIT_BLOCK_PTR_FOR_FN (cfun))
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to exit.\n",
+			   e->src->index);
+		else
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to postcheck block %i.\n",
+			   e->src->index, e->dest->index);
+	      }
+
+	    insert_exit_check_on_edge (seq, e);
+
+	    gcc_checking_assert (!bitmap_bit_p (chkcall_blocks, e->src->index));
+	  }
+
+	sbitmap_iterator it;
+	unsigned i;
+	EXECUTE_IF_SET_IN_BITMAP (chkcall_blocks, 0, i, it)
+	  {
+	    basic_block bb = BASIC_BLOCK_FOR_FN (cfun, i);
+
+	    gimple_seq seq = ckseq;
+	    gcc_checking_assert (count_chkcall > 0);
+	    if (--count_chkcall)
 	      seq = gimple_seq_copy (seq);
 
-	    insert_exit_check (seq,
-			       EDGE_PRED (EXIT_BLOCK_PTR_FOR_FN (cfun), i));
+	    if (dump_file)
+	      fprintf (dump_file,
+		       "Inserting out-of-line check before stmt in block %i.\n",
+		       bb->index);
+
+	    insert_exit_check_in_block (seq, bb);
 	  }
+
+	gcc_checking_assert (count_chkcall == 0);
       }
     else
       {
 	/* Inline checking requires a single exit edge.  */
-	gimple *last = gsi_stmt (gsi_last (ckseq));
+	gimple *last = gimple_build_assign (visited,
+					    build_clobber
+					    (TREE_TYPE (visited)));
+	gimple_seq_add_stmt (&ckseq, last);
 
-	insert_exit_check (ckseq,
-			   single_pred_edge (EXIT_BLOCK_PTR_FOR_FN (cfun)));
+	if (!count_chkcall)
+	  {
+	    edge e = single_pred_edge (EXIT_BLOCK_PTR_FOR_FN (cfun));
+
+	    if (dump_file)
+	      {
+		if (e->dest == EXIT_BLOCK_PTR_FOR_FN (cfun))
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to postcheck block %i.\n",
+			   e->src->index, e->dest->index);
+		else
+		  fprintf (dump_file,
+			   "Inserting inline check in"
+			   " block %i's edge to exit.\n",
+			   e->src->index);
+	      }
+
+	    insert_exit_check_on_edge (ckseq, e);
+	  }
+	else
+	  {
+	    gcc_checking_assert (count_chkcall == 1);
+
+	    sbitmap_iterator it;
+	    unsigned i;
+	    EXECUTE_IF_SET_IN_BITMAP (chkcall_blocks, 0, i, it)
+	      {
+		basic_block bb = BASIC_BLOCK_FOR_FN (cfun, i);
+
+		gimple_seq seq = ckseq;
+		gcc_checking_assert (count_chkcall > 0);
+		if (--count_chkcall)
+		  seq = gimple_seq_copy (seq);
+
+		if (dump_file)
+		  fprintf (dump_file,
+			   "Inserting inline check before stmt in block %i.\n",
+			   bb->index);
+
+		insert_exit_check_in_block (seq, bb);
+	      }
+
+	    gcc_checking_assert (count_chkcall == 0);
+	  }
 
 	/* The inserted ckseq computes CKFAIL at LAST.  Now we have to
 	   conditionally trap on it.  */
@@ -540,8 +877,7 @@ public:
 	  add_bb_to_loop (trp, current_loops->tree_root);
 
 	/* Insert a conditional branch to the trap block.  If the
-	   conditional wouldn't be the last statement, split the
-	   block.  */
+	   conditional wouldn't be the last stmt, split the block.  */
 	gimple_stmt_iterator gsi = gsi_for_stmt (last);
 	if (!gsi_one_before_end_p (gsi))
 	  split_block (gsi_bb (gsi), gsi_stmt (gsi));
@@ -564,6 +900,24 @@ public:
 	if (dom_info_available_p (CDI_DOMINATORS))
 	  set_immediate_dominator (CDI_DOMINATORS, trp, gimple_bb (last));
       }
+
+    /* Insert initializers for visited at the entry.  Do this after
+       other insertions, to avoid messing with block numbers.  */
+    gimple_seq iseq = NULL;
+
+    gcall *vinit = gimple_build_call (builtin_decl_explicit
+				      (BUILT_IN_MEMSET), 3,
+				      build1 (ADDR_EXPR,
+					      build_pointer_type
+					      (TREE_TYPE (visited)),
+					      visited),
+				      integer_zero_node,
+				      TYPE_SIZE_UNIT (TREE_TYPE (visited)));
+    gimple_seq_add_stmt (&iseq, vinit);
+
+    gsi_insert_seq_on_edge_immediate (single_succ_edge
+				      (ENTRY_BLOCK_PTR_FOR_FN (cfun)),
+				      iseq);
   }
 
   /* Push onto RTCFG a (mask, index) pair to test for IBB when BB is
@@ -607,7 +961,7 @@ public:
     return false;
   }
 
-  /* Add to CKSEQ statements to clear CKPART if OBB is visited.  */
+  /* Add to CKSEQ stmts to clear CKPART if OBB is visited.  */
   void
   build_block_check (basic_block obb)
   {
@@ -632,34 +986,48 @@ public:
 
   /* Add to BB code to set its bit in VISITED, and add to RTCFG or
      CKSEQ the data or code needed to check BB's predecessors and
-     successors.  Do NOT change the CFG.  */
-  void visit (basic_block bb)
+     successors.  If CHECKPOINT, assume the block is a checkpoint,
+     whether or not it has an edge to EXIT.  If POSTCHECK, assume the
+     block post-dominates checkpoints and therefore no bitmap setting
+     or checks are to be performed in or for it.  Do NOT change the
+     CFG.  */
+  void visit (basic_block bb, bool checkpoint, bool postcheck)
   {
     /* Set the bit in VISITED when entering the block.  */
     gimple_stmt_iterator gsi = gsi_after_labels (bb);
-    gsi_insert_seq_before (&gsi, vset (bb), GSI_SAME_STMT);
+    if (!postcheck)
+      gsi_insert_seq_before (&gsi, vset (bb), GSI_SAME_STMT);
 
     if (rtcfg)
       {
-	/* Build a list of (index, mask) terminated by (NULL, 0).
-	   Consolidate masks with the same index when they're
-	   adjacent.  First, predecessors.  Count backwards, because
-	   we're going to reverse the list.  The order shouldn't
-	   matter, but let's not make it surprising.  */
-	for (int i = EDGE_COUNT (bb->preds); i--; )
-	  if (push_rtcfg_pair (EDGE_PRED (bb, i)->src, bb,
-			       ENTRY_BLOCK_PTR_FOR_FN (cfun)))
-	    break;
+	if (!postcheck)
+	  {
+	    /* Build a list of (index, mask) terminated by (NULL, 0).
+	       Consolidate masks with the same index when they're
+	       adjacent.  First, predecessors.  Count backwards, because
+	       we're going to reverse the list.  The order shouldn't
+	       matter, but let's not make it surprising.  */
+	    for (int i = EDGE_COUNT (bb->preds); i--; )
+	      if (push_rtcfg_pair (EDGE_PRED (bb, i)->src, bb,
+				   ENTRY_BLOCK_PTR_FOR_FN (cfun)))
+		break;
+	  }
 	rtcfg = tree_cons (NULL_TREE, build_int_cst (vword_type, 0), rtcfg);
 
-	/* Then, successors.  */
-	for (int i = EDGE_COUNT (bb->succs); i--; )
-	  if (push_rtcfg_pair (EDGE_SUCC (bb, i)->dest, bb,
-			       EXIT_BLOCK_PTR_FOR_FN (cfun)))
-	    break;
+	if (!postcheck)
+	  {
+	    /* Then, successors.  */
+	    if (!checkpoint
+		|| !push_rtcfg_pair (EXIT_BLOCK_PTR_FOR_FN (cfun),
+				     bb, EXIT_BLOCK_PTR_FOR_FN (cfun)))
+	      for (int i = EDGE_COUNT (bb->succs); i--; )
+		if (push_rtcfg_pair (EDGE_SUCC (bb, i)->dest, bb,
+				     EXIT_BLOCK_PTR_FOR_FN (cfun)))
+		  break;
+	  }
 	rtcfg = tree_cons (NULL_TREE, build_int_cst (vword_type, 0), rtcfg);
       }
-    else
+    else if (!postcheck)
       {
 	/* Schedule test to fail if the block was reached but somehow none
 	   of its predecessors were.  */
@@ -677,6 +1045,8 @@ public:
 	gassign *blkruns = gimple_build_assign (ckpart, unshare_expr (bit));
 	gimple_seq_add_stmt (&ckseq, blkruns);
 
+	if (checkpoint)
+	  build_block_check (EXIT_BLOCK_PTR_FOR_FN (cfun));
 	for (int i = 0, e = EDGE_COUNT (bb->succs); i < e; i++)
 	  build_block_check (EDGE_SUCC (bb, i)->dest);
 
@@ -687,21 +1057,357 @@ public:
   }
 };
 
+/* It might be useful to avoid checking before noreturn calls that are
+   known to always finish by throwing an exception, rather than by
+   ending the program or looping forever.  Such functions would have
+   to be annotated somehow, with an attribute or flag, so that
+   exception-raising functions, such as C++'s __cxa_throw,
+   __cxa_rethrow, and Ada's gnat_rcheck_*, gnat_reraise*,
+   ada.exception.raise_exception*, and the language-independent
+   unwinders could be detected here and handled differently from other
+   noreturn functions.  */
+static bool
+always_throwing_noreturn_call_p (gimple *)
+{
+  return false;
+}
+
 /* Control flow redundancy hardening: record the execution path, and
    verify at exit that an expect path was taken.  */
 
 unsigned int
-pass_harden_control_flow_redundancy::execute (function *)
+pass_harden_control_flow_redundancy::execute (function *fun)
 {
-  rt_bb_visited vstd;
-
+  bool const check_at_escaping_exceptions
+    = (flag_exceptions
+       && flag_harden_control_flow_redundancy_check_exceptions);
+  bool const check_before_noreturn_calls
+    = flag_harden_control_flow_redundancy_check_noreturn > HCFRNR_NEVER;
+  bool const check_before_nothrow_noreturn_calls
+    = (check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn >= HCFRNR_NOTHROW);
+  bool const check_before_throwing_noreturn_calls
+    = (flag_exceptions
+       && check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn > HCFRNR_NOTHROW);
+  bool const check_before_always_throwing_noreturn_calls
+    = (flag_exceptions
+       && check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn >= HCFRNR_ALWAYS);
   basic_block bb;
-  FOR_EACH_BB_FN (bb, cfun)
-    vstd.visit (bb);
+  basic_block bb_eh_cleanup = NULL;
+
+  if (check_at_escaping_exceptions)
+    {
+      int lp_eh_cleanup = -1;
+
+      /* Record the preexisting blocks, to avoid visiting newly-created
+	 blocks.  */
+      auto_sbitmap to_visit (last_basic_block_for_fn (fun));
+      bitmap_clear (to_visit);
+
+      FOR_EACH_BB_FN (bb, fun)
+	bitmap_set_bit (to_visit, bb->index);
+
+      /* Scan the blocks for stmts with escaping exceptions, that
+	 wouldn't be denoted in the CFG, and associate them with an
+	 empty cleanup handler around the whole function.  Walk
+	 backwards, so that even when we split the block, */
+      sbitmap_iterator it;
+      unsigned i;
+      EXECUTE_IF_SET_IN_BITMAP (to_visit, 0, i, it)
+	{
+	  bb = BASIC_BLOCK_FOR_FN (fun, i);
+
+	  for (gimple_stmt_iterator gsi = gsi_last_bb (bb);
+	       !gsi_end_p (gsi); gsi_prev (&gsi))
+	    {
+	      gimple *stmt = gsi_stmt (gsi);
+	      if (!stmt_could_throw_p (fun, stmt))
+		continue;
+
+	      /* If it must not throw, or if it already has a handler,
+		 we need not worry about it.  */
+	      if (lookup_stmt_eh_lp (stmt) != 0)
+		continue;
+
+	      /* Don't split blocks at, nor add EH edges to, tail
+		 calls, we will add verification before the call
+		 anyway.  */
+	      if (is_a <gcall *> (stmt)
+		  && (gimple_call_must_tail_p (as_a <gcall *> (stmt))
+		      || gimple_call_tail_p (as_a <gcall *> (stmt))
+		      || returning_call_p (as_a <gcall *> (stmt))))
+		continue;
+
+	      if (!gsi_one_before_end_p (gsi))
+		split_block (bb, stmt);
+	      /* A resx or noreturn call needs not be associated with
+		 the cleanup handler if we're going to add checking
+		 before it.  We only test cases that didn't require
+		 block splitting because noreturn calls would always
+		 be at the end of blocks, and we test for zero
+		 successors because if there is an edge, it's not
+		 noreturn, as any EH edges would have already been
+		 caught by the lookup_stmt_eh_lp test above.  */
+	      else if (check_before_noreturn_calls
+		       && EDGE_COUNT (bb->succs) == 0
+		       && (is_a <gresx *> (stmt)
+			   ? check_before_always_throwing_noreturn_calls
+			   : (!is_a <gcall *> (stmt)
+			      || !gimple_call_noreturn_p (stmt))
+			   ? (gcc_unreachable (), false)
+			   : (!flag_exceptions
+			      || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+			   ? check_before_nothrow_noreturn_calls
+			   : always_throwing_noreturn_call_p (stmt)
+			   ? check_before_always_throwing_noreturn_calls
+			   : check_before_throwing_noreturn_calls))
+		{
+		  if (dump_file)
+		    {
+		      fprintf (dump_file,
+			       "Bypassing cleanup for noreturn stmt"
+			       " in block %i:\n",
+			       bb->index);
+		      print_gimple_stmt (dump_file, stmt, 0);
+		    }
+		  continue;
+		}
+
+	      if (!bb_eh_cleanup)
+		{
+		  bb_eh_cleanup = create_empty_bb (bb);
+		  if (dom_info_available_p (CDI_DOMINATORS))
+		    set_immediate_dominator (CDI_DOMINATORS, bb_eh_cleanup, bb);
+		  if (current_loops)
+		    add_bb_to_loop (bb_eh_cleanup, current_loops->tree_root);
+
+		  /* Make the new block an EH cleanup for the call.  */
+		  eh_region new_r = gen_eh_region_cleanup (NULL);
+		  eh_landing_pad lp = gen_eh_landing_pad (new_r);
+		  tree label = gimple_block_label (bb_eh_cleanup);
+		  lp->post_landing_pad = label;
+		  EH_LANDING_PAD_NR (label) = lp_eh_cleanup = lp->index;
+
+		  /* Just propagate the exception.
+		     We will later insert the verifier call.  */
+		  gimple_stmt_iterator ehgsi;
+		  ehgsi = gsi_after_labels (bb_eh_cleanup);
+		  gresx *resx = gimple_build_resx (new_r->index);
+		  gsi_insert_before (&ehgsi, resx, GSI_SAME_STMT);
+
+		  if (dump_file)
+		    fprintf (dump_file,
+			     "Created cleanup block %i:\n",
+			     bb_eh_cleanup->index);
+		}
+	      else if (dom_info_available_p (CDI_DOMINATORS))
+		{
+		  basic_block immdom;
+		  immdom = get_immediate_dominator (CDI_DOMINATORS,
+						    bb_eh_cleanup);
+		  if (!dominated_by_p (CDI_DOMINATORS, bb, immdom))
+		    {
+		      immdom = nearest_common_dominator (CDI_DOMINATORS,
+							 immdom, bb);
+		      set_immediate_dominator (CDI_DOMINATORS,
+					       bb_eh_cleanup, immdom);
+		    }
+		}
+
+	      if (dump_file)
+		{
+		  fprintf (dump_file,
+			   "Associated cleanup block with stmt in block %i:\n",
+			   bb->index);
+		  print_gimple_stmt (dump_file, stmt, 0);
+		}
+
+	      add_stmt_to_eh_lp (stmt, lp_eh_cleanup);
+	      /* Finally, wire the EH cleanup block into the CFG.  */
+	      make_eh_edges (stmt);
+	    }
+	}
+
+      if (bb_eh_cleanup)
+	{
+	  /* A cfg_cleanup after bb_eh_cleanup makes for a more compact
+	     rtcfg, and it avoids bb numbering differences when we split
+	     blocks because of trailing debug insns only.  */
+	  cleanup_tree_cfg ();
+	  gcc_checking_assert (EDGE_COUNT (bb_eh_cleanup->succs) == 0);
+	}
+    }
+
+  /* These record blocks with calls that are to be preceded by
+     checkpoints, such as noreturn calls (if so chosen), must-tail
+     calls, potential early-marked tail calls, and returning calls (if
+     so chosen).  */
+  int count_chkcall = 0;
+  auto_sbitmap chkcall_blocks (last_basic_block_for_fn (fun));
+  bitmap_clear (chkcall_blocks);
+
+  /* We wish to add verification at blocks without successors, such as
+     noreturn calls (raising or not) and the reraise at the cleanup
+     block, but not other reraises: they will go through the cleanup
+     block.  */
+  if (check_before_noreturn_calls)
+    FOR_EACH_BB_FN (bb, fun)
+      {
+	gimple_stmt_iterator gsi = gsi_last_bb (bb);
+	if (gsi_end_p (gsi))
+	  continue;
+	gimple *stmt = gsi_stmt (gsi);
 
-  vstd.check ();
+	if (EDGE_COUNT (bb->succs) == 0)
+	  {
+	    /* A stmt at the end of a block without any successors is
+	       either a resx or a noreturn call without a local
+	       handler.  Check that it's one of the desired
+	       checkpoints.  */
+	    if (flag_exceptions && is_a <gresx *> (stmt)
+		? (check_before_always_throwing_noreturn_calls
+		   || bb == bb_eh_cleanup)
+		: (!is_a <gcall *> (stmt)
+		   || !gimple_call_noreturn_p (stmt))
+		? (/* Catch cases in which successors would be
+		      expected.  */
+		   gcc_unreachable (), false)
+		: (!flag_exceptions
+		   || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+		? check_before_nothrow_noreturn_calls
+		: always_throwing_noreturn_call_p (stmt)
+		? check_before_always_throwing_noreturn_calls
+		: check_before_throwing_noreturn_calls)
+	      {
+		if (dump_file)
+		  {
+		    fprintf (dump_file,
+			     "Scheduling check before stmt"
+			     " in succ-less block %i:\n",
+			     bb->index);
+		    print_gimple_stmt (dump_file, stmt, 0);
+		  }
+
+		if (bitmap_set_bit (chkcall_blocks, bb->index))
+		  count_chkcall++;
+		else
+		  gcc_unreachable ();
+	      }
+	    continue;
+	  }
 
-  return 0;
+	/* If there are no exceptions, it would seem like any noreturn
+	   call must have zero successor edges, but __builtin_return
+	   gets successor edges.  We don't want to handle it here, it
+	   will be dealt with in sibcall_search_preds.  Otherwise,
+	   check for blocks without non-EH successors, but skip those
+	   with resx stmts and edges (i.e., those other than that in
+	   bb_eh_cleanup), since those will go through bb_eh_cleanup,
+	   that will have been counted as noreturn above because it
+	   has no successors.  */
+	gcc_checking_assert (bb != bb_eh_cleanup
+			     || !check_at_escaping_exceptions);
+	if (flag_exceptions && is_a <gresx *> (stmt)
+	    ? check_before_always_throwing_noreturn_calls
+	    : (!is_a <gcall *> (stmt)
+	       || !gimple_call_noreturn_p (stmt))
+	    ? false
+	    : (!flag_exceptions
+	       || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+	    ? false /* rather than check_before_nothrow_noreturn_calls */
+	    : always_throwing_noreturn_call_p (stmt)
+	    ? check_before_always_throwing_noreturn_calls
+	    : check_before_throwing_noreturn_calls)
+	  {
+	    gcc_checking_assert (single_succ_p (bb)
+				 && (single_succ_edge (bb)->flags & EDGE_EH));
+
+	    if (dump_file)
+	      {
+		fprintf (dump_file,
+			 "Scheduling check before stmt"
+			 " in EH-succ block %i:\n",
+			 bb->index);
+		print_gimple_stmt (dump_file, stmt, 0);
+	      }
+
+	    if (bitmap_set_bit (chkcall_blocks, bb->index))
+	      count_chkcall++;
+	    else
+	      gcc_unreachable ();
+	  }
+      }
+  else if (bb_eh_cleanup)
+    {
+      if (bitmap_set_bit (chkcall_blocks, bb_eh_cleanup->index))
+	count_chkcall++;
+      else
+	gcc_unreachable ();
+    }
+
+  gcc_checking_assert (!bb_eh_cleanup
+		       || bitmap_bit_p (chkcall_blocks, bb_eh_cleanup->index));
+
+  /* If we don't have edges to exit nor noreturn calls (including the
+     cleanup reraise), then we may skip instrumentation: that would
+     amount to a function that ends with an infinite loop.  */
+  if (!count_chkcall
+      && EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (fun)->preds) == 0)
+    {
+      if (dump_file)
+	fprintf (dump_file,
+		 "Disabling CFR, no exit paths to check\n");
+
+      return 0;
+    }
+
+  /* Search for must-tail calls, early-marked potential tail calls,
+     and, if requested, returning calls.  As we introduce early
+     checks, */
+  int count_postchk = 0;
+  auto_sbitmap postchk_blocks (last_basic_block_for_fn (fun));
+  bitmap_clear (postchk_blocks);
+  chk_edges_t chk_edges;
+  hardcfr_sibcall_search_preds (EXIT_BLOCK_PTR_FOR_FN (fun), chk_edges,
+				count_chkcall, chkcall_blocks,
+				count_postchk, postchk_blocks,
+				NULL);
+
+  rt_bb_visited vstd (chk_edges.length () + count_chkcall);
+
+  auto_sbitmap combined_blocks (last_basic_block_for_fn (fun));
+  bitmap_copy (combined_blocks, chkcall_blocks);
+  int i;
+  edge *e;
+  FOR_EACH_VEC_ELT (chk_edges, i, e)
+    if (!bitmap_set_bit (combined_blocks, (*e)->src->index))
+      /* There may be multiple chk_edges with the same src block;
+	 guard againt overlaps with chkcall_blocks only.  */
+      gcc_assert (!bitmap_bit_p (chkcall_blocks, (*e)->src->index));
+
+  /* Visit blocks in index order, because building rtcfg depends on
+     that.  Blocks must be compact, which the cleanup_cfg requirement
+     ensures.  This would also enable FOR_EACH_BB_FN to be used to
+     iterate in index order, but bb_eh_cleanup block splits and
+     insertions changes that.  */
+  gcc_checking_assert (n_basic_blocks_for_fn (fun)
+		       == last_basic_block_for_fn (fun));
+  for (int i = NUM_FIXED_BLOCKS; i < n_basic_blocks_for_fn (fun); i++)
+    {
+      bb = BASIC_BLOCK_FOR_FN (fun, i);
+      gcc_checking_assert (bb->index == i);
+      vstd.visit (bb, bitmap_bit_p (combined_blocks, i),
+		  bitmap_bit_p (postchk_blocks, i));
+    }
+
+  vstd.check (chk_edges, count_chkcall, chkcall_blocks);
+
+  return
+    TODO_update_ssa
+    | TODO_cleanup_cfg
+    | TODO_verify_il;
 }
 
 /* Instantiate a hardcfr pass.  */
diff --git a/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c b/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c
new file mode 100644
index 00000000000..a6992eb9f8e
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -O0 -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we don't insert checking before noreturn calls.  -O0 is tested
+   separately because h is not found to be noreturn without optimization.  */
+
+#include "torture/harden-cfr-noret.c"
+
+/* No out-of-line checks.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* Only one inline check at the end of f and of h2.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-always.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-always.c
new file mode 100644
index 00000000000..779896c60e8
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-always.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even enabling all checks before noreturn calls (leaving
+   returning calls enabled), we get checks before __builtin_return without
+   duplication (__builtin_return is both noreturn and a returning call).  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-never.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-never.c
new file mode 100644
index 00000000000..49ce17f5b93
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-never.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even enabling checks before never noreturn calls (leaving
+   returning calls enabled), we get checks before __builtin_return without
+   duplication (__builtin_return is both noreturn and a returning call).  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noopt.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noopt.c
new file mode 100644
index 00000000000..1512614791f
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noopt.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even disabling checks before both noreturn and returning
+   calls, we still get checks before __builtin_return.  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noret.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noret.c
new file mode 100644
index 00000000000..fd95bb7e3e3
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noret.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even disabling checks before returning calls (leaving noreturn
+   calls enabled), we still get checks before __builtin_return.  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-nothrow.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-nothrow.c
new file mode 100644
index 00000000000..c5c361234c4
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-nothrow.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even enabling checks before nothrow noreturn calls (leaving
+   returning calls enabled), we get checks before __builtin_return without
+   duplication (__builtin_return is both noreturn and a returning call).  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-retcl.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-retcl.c
new file mode 100644
index 00000000000..137dfbb95d6
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-retcl.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even disabling checks before noreturn calls (leaving returning
+   calls enabled), we still get checks before __builtin_return.  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c
index 70acdc95f25..b459ff6b864 100644
--- a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c
@@ -7,5 +7,11 @@ int f(int i) {
   return i;
 }
 
-/* Out-of-line checking, before both builtin_return and return.  */
+int g(int i) {
+  __builtin_return (&i);
+}
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
 /* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c
new file mode 100644
index 00000000000..8bd2d13ac18
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c
@@ -0,0 +1,18 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we don't insert checking before noreturn calls.  -O0 is tested
+   separately because h is not found to be noreturn without optimization, which
+   affects codegen for h2, so h2 is omitted here at -O0.  */
+
+#if !__OPTIMIZE__
+# define OMIT_H2
+#endif
+
+#include "harden-cfr-noret.c"
+
+
+/* No out-of-line checks.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* Only one inline check at the end of f.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c
new file mode 100644
index 00000000000..a804a6cfe59
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fno-exceptions -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that -fno-exceptions makes for implicit nothrow in noreturn
+   handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#include "harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c
new file mode 100644
index 00000000000..f390cfdbc59
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checking before nothrow noreturn calls.  */
+
+#include "harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c
new file mode 100644
index 00000000000..fdd803109a4
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c
@@ -0,0 +1,38 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checking before all noreturn calls.  */
+
+#ifndef ATTR_NOTHROW_OPT /* Overridden in harden-cfr-noret-noexcept.  */
+#define ATTR_NOTHROW_OPT __attribute__ ((__nothrow__))
+#endif
+
+extern void __attribute__ ((__noreturn__)) ATTR_NOTHROW_OPT g (void);
+
+void f(int i) {
+  if (i)
+    /* Out-of-line checks here...  */
+    g ();
+  /* ... and here.  */
+}
+
+void __attribute__ ((__noinline__, __noclone__))
+h(void) {
+  /* Inline check here.  */
+  g ();
+}
+
+#ifndef OMIT_H2 /* from harden-cfr-noret-never.  */
+void h2(void) {
+  /* Inline check either here, whether because of noreturn or tail call...  */
+  h ();
+  /* ... or here, if not optimizing.  */
+}
+#endif
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c
new file mode 100644
index 00000000000..6d11487bbba
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c
@@ -0,0 +1,8 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+#include "harden-cfr-tail.c"
+
+/* Inline checking after the calls, disabling tail calling.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 5 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 0 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c
new file mode 100644
index 00000000000..550b02ca088
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c
@@ -0,0 +1,35 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-exceptions -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checks before returning calls and alternate paths, even
+   at -O0, because of the explicit command-line flag.  */
+
+void g (void);
+void g2 (void);
+void g3 (void);
+
+void f (int i) {
+  if (!i)
+    /* Out-of-line checks here...  */
+    g ();
+  else if (i > 0)
+    /* here...  */
+    g2 ();
+  /* else */
+    /* and in the implicit else here.  */
+}
+
+void f2 (int i) {
+  if (!i)
+    /* Out-of-line check here...  */
+    g ();
+  else if (i > 0)
+    /* here...  */
+    g2 ();
+  else
+    /* and here.  */
+    g3 ();
+}
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 6 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 0 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
index 40d76c5c163..d5467eafa9f 100644
--- a/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
@@ -1,17 +1,52 @@
 /* { dg-do compile } */
-/* { dg-options "-fharden-control-flow-redundancy -fdump-tree-hardcfr -ffat-lto-objects" } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-hardcfr-check-exceptions -fdump-tree-hardcfr -ffat-lto-objects -Wno-return-type" } */
 
-/* We'd like to check that we insert checking so as to not disrupt tail calls,
-   but unfortunately mandatory tail calls are not available in C, and optimizing
-   calls as tail calls only takes place after hardcfr.  */
+/* Check that we insert CFR checking so as to not disrupt tail calls.
+   Mandatory tail calls are not available in C, and optimizing calls as tail
+   calls only takes place after hardcfr, so we insert checking before calls
+   followed by copies and return stmts with the same return value, that might
+   (or might not) end up optimized to tail calls.  */
 
-extern int g(int i);
+extern int g (int i);
 
-int f(int i) {
+int f1(int i) {
+  /* Inline check before the returning call.  */
   return g (i);
 }
 
-/* Inline checking before the tail call.  */
-/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
-/* Inline checking before the tail call.  */
-/* { dg-final { scan-tree-dump-times "\\\[tail\]" 1 "hardcfr" { xfail *-*-* } } } */
+extern void g2 (int i);
+
+void f2(int i) {
+  /* Inline check before the returning call, that ignores the returned value,
+     matching the value-less return.  */
+  g2 (i);
+  return;
+}
+
+void f3(int i) {
+  /* Inline check before the returning call.  */
+  g (i);
+}
+
+void f4(int i) {
+  if (i)
+    /* Out-of-line check before the returning call.  */
+    return g2 (i);
+  /* Out-of-line check before implicit return.  */
+}
+
+int f5(int i) {
+  /* Not regarded as a returning call, returning value other than callee's
+     returned value.  */
+  g (i);
+  /* Inline check after the non-returning call.  */
+  return i;
+}
+
+/* Out-of-line checks in f4, before returning calls and before return.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking in all other functions.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 4 "hardcfr" } } */
+/* Check before tail-call in all but f5, but f4 is out-of-line.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 3 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C b/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C
new file mode 100644
index 00000000000..17ea79f7cfb
--- /dev/null
+++ b/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C
@@ -0,0 +1,11 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects -O0" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions, and also checking before noreturn
+   calls.  h2 and h2b get an extra resx without ehcleanup.  */
+
+#include "torture/harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 16 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C b/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C
new file mode 100644
index 00000000000..f0338ccc361
--- /dev/null
+++ b/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C
@@ -0,0 +1,10 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -foptimize-sibling-calls -fdump-tree-hardcfr -O0" } */
+
+/* -fhardcfr-check-returning-calls gets implicitly disabled because,
+   -at O0, -foptimize-sibling-calls has no effect.  */
+
+#include "torture/harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 12 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C
new file mode 100644
index 00000000000..0d35920c7ee
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  */
+
+#include "harden-cfr-noret-no-nothrow.C"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Bypassing cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C
new file mode 100644
index 00000000000..b7d247ff43c
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C
@@ -0,0 +1,18 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  Expected results for =never and =nothrow are the same,
+   since the functions are not nothrow.  */
+
+#include "harden-cfr-noret-no-nothrow.C"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Associated cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C
new file mode 100644
index 00000000000..62c58cfd406
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C
@@ -0,0 +1,23 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#if ! __OPTIMIZE__
+void __attribute__ ((__noreturn__)) h (void);
+#endif
+
+#include "../../c-c++-common/torture/harden-cfr-noret.c"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Associated cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C
new file mode 100644
index 00000000000..0286f6e6d3f
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C
@@ -0,0 +1,20 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions, and also checking before noreturn
+   calls.  */
+
+#if ! __OPTIMIZE__
+/* Without optimization, functions with cleanups end up with an extra
+   resx that is not optimized out, so arrange to optimize them.  */
+void __attribute__ ((__optimize__ (1))) h2(void);
+void __attribute__ ((__optimize__ (1))) h2b(void);
+#endif
+
+#include "harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 14 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
+/* h, h2, h2b, and h4.  */
+/* { dg-final { scan-tree-dump-times "Bypassing" 4 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C
new file mode 100644
index 00000000000..885b0b236af
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C
@@ -0,0 +1,11 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions  -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we do not insert cleanups for checking around the bodies
+   of maybe-throwing functions.  h4 doesn't get any checks, because we
+   don't have noreturn checking enabled.  */
+
+#include "harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 6 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C
new file mode 100644
index 00000000000..32def637255
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C
@@ -0,0 +1,31 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -foptimize-sibling-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions.  These results depend on checking before
+   returning calls, which is only enabled when sibcall optimizations
+   are enabled, so change the optimization mode to -O1 for f and f2,
+   so that -foptimize-sibling-calls can take effect and enable
+   -fhardcfr-check-returning-calls, so that we get the same results.
+   There is a separate test for -O0.  */
+
+#if ! __OPTIMIZE__
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) f(int i);
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) f2(int i);
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) h3(void);
+#endif
+
+#include "harden-cfr-throw.C"
+
+/* f gets out-of-line checks before the unwrapped tail call and in the
+   else edge.  */
+/* f2 gets out-of-line checks before both unwrapped tail calls.  */
+/* h gets out-of-line checks before the implicit return and in the
+   cleanup block.  */
+/* h2 and h2b get out-of-line checks before the cleanup returning
+   call, and in the cleanup block.  */
+/* h3 gets an inline check before the __cxa_end_catch returning call.  */
+/* h4 gets an inline check in the cleanup block.  */
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 10 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C
new file mode 100644
index 00000000000..992fbdad381
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C
@@ -0,0 +1,65 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions.  */
+
+extern void g (void);
+extern void g2 (void);
+
+void f(int i) {
+  if (i)
+    g ();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void f2(int i) {
+  if (i)
+    g ();
+  else
+    g2 ();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h(void) {
+  try {
+    g ();
+  } catch (...) {
+    throw;
+  }
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+struct needs_cleanup {
+  ~needs_cleanup();
+};
+
+void h2(void) {
+  needs_cleanup y; /* No check in the cleanup handler.  */
+  g();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+extern void __attribute__ ((__nothrow__)) another_cleanup (void*);
+
+void h2b(void) {
+  int x __attribute__ ((cleanup (another_cleanup)));
+  g();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h3(void) {
+  try {
+    throw 1;
+  } catch (...) {
+  }
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h4(void) {
+  throw 1;
+  /* Inline check in the cleanup around the __cxa_throw noreturn call.  */
+}
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 12 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c b/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c
new file mode 100644
index 00000000000..8e4ee1fab08
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c
@@ -0,0 +1,15 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C makes for implicit nothrow in noreturn handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#include "../../c-c++-common/torture/harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c b/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c
new file mode 100644
index 00000000000..634d98f1ffc
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c
@@ -0,0 +1,40 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-hardcfr-check-exceptions -fdump-tree-hardcfr -ffat-lto-objects -Wno-return-type" } */
+
+/* In C only, check some additional cases (comparing with
+   c-c++-common/torture/harden-cfr-tail.c) of falling off the end of non-void
+   function.  C++ would issue an unreachable call in these cases.  */
+
+extern int g (int i);
+
+int f1(int i) {
+  /* Inline check before the returning call, that doesn't return anything.  */
+  g (i);
+  /* Implicit return without value, despite the return type; this combination
+     enables tail-calling of g, and is recognized as a returning call.  */
+}
+
+extern void g2 (int i);
+
+int f2(int i) {
+  /* Inline check before the returning call, that disregards its return
+     value.  */
+  g2 (i);
+  /* Implicit return without value, despite the return type; this combination
+     enables tail-calling of g2, and is recognized as a returning call.  */
+}
+
+int f3(int i) {
+  if (i)
+    /* Out-of-line check before the returning call.  */
+    return g (i);
+  /* Out-of-line check before implicit return.  */
+}
+
+/* Out-of-line checks in f3, before returning calls and before return.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking in all other functions.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
+/* Check before tail-call in all functions, but f3 is out-of-line.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/gnat.dg/hardcfr.adb b/gcc/testsuite/gnat.dg/hardcfr.adb
index b578a913d75..abe1605c029 100644
--- a/gcc/testsuite/gnat.dg/hardcfr.adb
+++ b/gcc/testsuite/gnat.dg/hardcfr.adb
@@ -1,5 +1,5 @@
 --  { dg-do run }
---  { dg-options "-fharden-control-flow-redundancy -fdump-tree-hardcfr --param=hardcfr-max-blocks=22 --param=hardcfr-max-inline-blocks=12 -O0" }
+--  { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions -fdump-tree-hardcfr --param=hardcfr-max-blocks=22 --param=hardcfr-max-inline-blocks=12 -O0" }
 
 procedure HardCFR is
    function F (I, J : Integer) return Integer is
diff --git a/libgcc/hardcfr.c b/libgcc/hardcfr.c
index 8ef29428111..55f6b995f2f 100644
--- a/libgcc/hardcfr.c
+++ b/libgcc/hardcfr.c
@@ -60,14 +60,25 @@ extern void __hardcfr_check (size_t blocks,
 			     vword const *visited,
 			     vword const *cfg);
 
+/* Compute the MASK for the bit representing BLOCK in WORDIDX's vword in a
+   visited blocks bit array.  */
+static inline void
+block2mask (size_t const block, vword *const mask, size_t *const wordidx)
+{
+  size_t wbits = __CHAR_BIT__ * sizeof (vword);
+  *wordidx = block / wbits;
+  *mask = (vword)1 << (block % wbits);
+}
 
 /* Check whether the bit corresponding to BLOCK is set in VISITED.  */
 static inline bool
 visited_p (size_t const block, vword const *const visited)
 {
-  size_t wbits = __CHAR_BIT__ * sizeof (vword);
-  vword w = visited[block / wbits];
-  return (w & ((vword)1 << (block % wbits))) != 0;
+  vword mask;
+  size_t wordidx;
+  block2mask (block, &mask, &wordidx);
+  vword w = visited[wordidx];
+  return (w & mask) != 0;
 }
 
 /* Read and consume a mask from **CFG_IT.  (Consume meaning advancing the
@@ -118,19 +129,19 @@ consume_seq (vword const **const cfg_it)
    we reach the terminator without finding any.  Consume the entire sequence
    otherwise, so that *CFG_IT points just past the terminator, which may be the
    beginning of the next sequence.  */
-static inline void
+static inline bool
 check_seq (vword const *const visited, vword const **const cfg_it)
 {
   vword mask;
   size_t wordidx;
 
   /* If the block was visited, check that at least one of the
-     preds was also visited.  */
+     preds/succs was also visited.  */
   do
     /* If we get to the end of the sequence without finding any
        match, something is amiss.  */
     if (!next_pair (cfg_it, &mask, &wordidx))
-      __builtin_trap ();
+      return false;
   /* Keep searching until we find a match, at which point the
      condition is satisfied.  */
   while (!test_mask (visited, mask, wordidx));
@@ -139,6 +150,94 @@ check_seq (vword const *const visited, vword const **const cfg_it)
      skipped the block, so as to position the iterator at the beginning of the
      next .  */
   consume_seq (cfg_it);
+
+  return true;
+}
+
+/* Print out the CFG with BLOCKS blocks, presumed to be associated with CALLER.
+   This is expected to be optimized out entirely, unless the verbose part of
+   __hardcfr_check_fail is enabled.  */
+static inline void
+__hardcfr_debug_cfg (size_t const blocks,
+		     void const *const caller,
+		     vword const *const cfg)
+{
+  __builtin_printf ("CFG at %p, for %p", cfg, caller);
+  vword const *cfg_it = cfg;
+  for (size_t i = 0; i < blocks; i++)
+    {
+      vword mask; size_t wordidx;
+      block2mask (i, &mask, &wordidx);
+      __builtin_printf ("\nblock %lu (%lu/0x%lx)\npreds: ",
+			(unsigned long)i,
+			(unsigned long)wordidx, (unsigned long)mask);
+      while (next_pair (&cfg_it, &mask, &wordidx))
+	__builtin_printf (" (%lu/0x%lx)",
+			  (unsigned long)wordidx, (unsigned long)mask);
+      __builtin_printf ("\nsuccs: ");
+      while (next_pair (&cfg_it, &mask, &wordidx))
+	__builtin_printf (" (%lu/0x%lx)",
+			  (unsigned long)wordidx, (unsigned long)mask);
+    }
+  __builtin_printf ("\n");
+}
+
+#ifndef ATTRIBUTE_UNUSED
+# define ATTRIBUTE_UNUSED __attribute__ ((__unused__))
+#endif
+
+/* This is called when an out-of-line hardcfr check fails.  All the arguments
+   are ignored, and it just traps, unless HARDCFR_VERBOSE_FAIL is enabled.  IF
+   it is, it prints the PART of the CFG, expected to have BLOCKS blocks, that
+   failed at CALLER's BLOCK, and the VISITED bitmap.  When the verbose mode is
+   enabled, it also forces __hardcfr_debug_cfg (above) to be compiled into an
+   out-of-line function, that could be called from a debugger.
+   */
+static inline void
+__hardcfr_check_fail (size_t const blocks ATTRIBUTE_UNUSED,
+		      vword const *const visited,
+		      vword const *const cfg ATTRIBUTE_UNUSED,
+		      size_t const block ATTRIBUTE_UNUSED,
+		      int const part ATTRIBUTE_UNUSED,
+		      void const *const caller ATTRIBUTE_UNUSED)
+{
+#if HARDCFR_VERBOSE_FAIL
+  static const char *parts[] = { "preds", "succs" };
+
+  vword mask; size_t wordidx;
+  block2mask (block, &mask, &wordidx);
+  __builtin_printf ("hardcfr fail at %p block %lu (%lu/0x%lx), expected %s:",
+		    caller, (unsigned long)block,
+		    (unsigned long)wordidx, (unsigned long)mask,
+		    parts[part]);
+
+  /* Skip data for previous blocks.  */
+  vword const *cfg_it = cfg;
+  for (size_t i = block; i--; )
+    {
+      consume_seq (&cfg_it);
+      consume_seq (&cfg_it);
+    }
+  for (size_t i = part; i--; )
+    consume_seq (&cfg_it);
+
+  while (next_pair (&cfg_it, &mask, &wordidx))
+    __builtin_printf (" (%lu/0x%lx)",
+		      (unsigned long)wordidx, (unsigned long)mask);
+
+  __builtin_printf ("\nvisited:");
+  block2mask (blocks, &mask, &wordidx);
+  for (size_t i = 0; i <= wordidx; i++)
+    __builtin_printf (" (%lu/0x%lx)",
+		      (unsigned long)i, (unsigned long)visited[i]);
+  __builtin_printf ("\n");
+
+  /* Reference __hardcfr_debug_cfg so that it's output out-of-line, so that it
+     can be called from a debugger.  */
+  if (!caller || caller == __hardcfr_debug_cfg)
+    return;
+#endif
+  __builtin_trap ();
 }
 
 /* Check that, for each of the BLOCKS basic blocks, if its bit is set in
@@ -168,9 +267,13 @@ __hardcfr_check (size_t const blocks,
       else
 	{
 	  /* Check predecessors.  */
-	  check_seq (visited, &cfg_it);
+	  if (!check_seq (visited, &cfg_it))
+	    __hardcfr_check_fail (blocks, visited, cfg, i, 0,
+				  __builtin_return_address (0));
 	  /* Check successors.  */
-	  check_seq (visited, &cfg_it);
+	  if (!check_seq (visited, &cfg_it))
+	    __hardcfr_check_fail (blocks, visited, cfg, i, 1,
+				  __builtin_return_address (0));
 	}
     }
 }

^ permalink raw reply	[flat|nested] 13+ messages in thread
* [gcc(refs/users/aoliva/heads/testme)] hardcfr: add optional checkpoints
@ 2023-06-08  9:16 Alexandre Oliva
  0 siblings, 0 replies; 13+ messages in thread
From: Alexandre Oliva @ 2023-06-08  9:16 UTC (permalink / raw)
  To: gcc-cvs

https://gcc.gnu.org/g:f2bb0e3aa7cdf9c7adb34224aeda2947ce04df7e

commit f2bb0e3aa7cdf9c7adb34224aeda2947ce04df7e
Author: Alexandre Oliva <oliva@adacore.com>
Date:   Wed Oct 19 20:36:18 2022 -0300

    hardcfr: add optional checkpoints
    
    Previously, control flow redundancy only checked the visited bitmap
    against the control flow graph at return points and before mandatory
    tail calls, missing various other possibilities of exiting a
    subprogram, such as by raising or propagating exceptions, and calling
    noreturn functions.  The checks inserted before returns also prevented
    potential tail-call optimizations.
    
    This incremental change introduces options to control checking at each
    of these previously-missed checkpoints.  Unless disabled, a cleanup is
    introduced to check when an exceptions escapes a subprogram.  To avoid
    disrupting sibcall optimizations, when they are enabled, checks are
    introduced before calls whose results are immediately returned,
    whether or not they are ultimately optimized.  If enabled, checks are
    introduced before noreturn calls and exception raises, or only before
    nothrow noreturn calls.
    
    
    for  gcc/ChangeLog
    
            * common.opt (-fhardcfr-check-returning-calls): New.
            (-fhardcfr-check-exceptions): New.
            (-fhardcfr-check-noreturn-calls=*): New.
            (Enum hardcfr_check_noreturn_calls): New.
            * doc/invoke.texi: Document them.
            * flag-types.h (enum hardcfr_noret): New.
            * gimple-harden-control-flow.cc: Include more headers.
            (pass_data_harden_control_flow_redundancy): Move
            properties_finish to pass return.
            (pass_harden_control_flow_redundancy::gate): Move
            no-edge-to-exit checking to execute, after exception
            transformations.  Use symbolic NUM_FIXED_BLOCKS.  Test for CFG
            before testing block count.
            (check_returning_calls_p): New.
            (hardcfr_scan_block): New.
            (returnign_call_p): New.
            (chk_edges_t): New.
            (hardcfr_sibcall_search_block): New.
            (hardcfr_sibcall_search_preds): New.
            (rt_bb_visited::num2idx): Use symbolic NUM_FIXED_BLOCKS.
            (rt_bb_visited::rt_bb_visited): Likewise.  Take checkpoints
            count, test it to select out-of-line checks.
            (rt_bb_visited::insert_exit_check): Removed.
            (rt_bb_visited::insert_exit_check_in_blocks): New.
            (rt_bb_visited::insert_exit_check_on_edge): New.
            (rt_bb_visited::check): Take checkpoint edges and blocks.
            Move initialization insertion to the end.  Insert checks on
            edges, and before calls in select blocks.  Clobber the visited
            array to the check sequence.  Output actions to dump_file.
            (rt_bb_visited::visit): Take checkpoint and postcheck.  Assume
            an edge to exit if checkpoint, and skip testing if postcheck.
            (always_throwing_noreturn_call_p): New, dummy.
            (pass_harden_control_flow_redundancy::execute): Introduce
            cleanup block if requested and needed.  Scan blocks for
            noreturn and returning calls if checkpoints before them were
            requested.  Log actions to dump_file.  Visit blocks in index
            order.
    
    for  gcc/testsuite/ChangeLog
    
            * c-c++-common/harden-cfr-noret-never-O0.c: New.
            * c-c++-common/torture/harden-cfr-noret-never.c: New.
            * c-c++-common/torture/harden-cfr-noret-noexcept.c: New.
            * c-c++-common/torture/harden-cfr-noret-nothrow.c: New.
            * c-c++-common/torture/harden-cfr-noret.c: New.
            * c-c++-common/torture/harden-cfr-notail.c: New.
            * c-c++-common/torture/harden-cfr-returning.c: New.
            * c-c++-common/torture/harden-cfr-tail.c: Extend.
            * c-c++-common/torture/harden-cfr-bret-always.c: New.
            * c-c++-common/torture/harden-cfr-bret-never.c: New.
            * c-c++-common/torture/harden-cfr-bret-noopt.c: New.
            * c-c++-common/torture/harden-cfr-bret-noret.c: New.
            * c-c++-common/torture/harden-cfr-bret-nothrow.c: New.
            * c-c++-common/torture/harden-cfr-bret-retcl.c: New.
            * c-c++-common/torture/harden-cfr-bret.c (g): New.
            * g++.dg/harden-cfr-throw-always-O0.C: New.
            * g++.dg/harden-cfr-throw-returning-O0.C: New.
            * g++.dg/torture/harden-cfr-noret-always-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-noret-never-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-noret-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-throw-always.C: New.
            * g++.dg/torture/harden-cfr-throw-nocleanup.C: New.
            * g++.dg/torture/harden-cfr-throw-returning.C: New.
            * g++.dg/torture/harden-cfr-throw.C: New.
            * gcc.dg/torture/harden-cfr-noret-no-nothrow.c: New.
            * gcc.dg/torture/harden-cfr-tail-ub.c: New.
            * gnat.dg/hardcfr.adb: Disable checking at exceptions.
    
    for  libgcc/ChangeLog
    
            * hardcfr.c (block2mask): Split out of...
            (visited_p): ... this.
            (check_seq): Move trapping out, return result instead.
            (__hardcfr_debug_cfg): New.
            (__hardcfr_check_fail): New, with optional verbose error
            printing before trapping.
            (__hardcfr_check): Call __hardcfr_check_fail when check_seq
            fails.

Diff:
---
 gcc/common.opt                                     |  33 +
 gcc/doc/invoke.texi                                |  74 +-
 gcc/flag-types.h                                   |  10 +
 gcc/gimple-harden-control-flow.cc                  | 926 ++++++++++++++++++---
 .../c-c++-common/harden-cfr-noret-never-O0.c       |  12 +
 .../c-c++-common/torture/harden-cfr-bret-always.c  |  13 +
 .../c-c++-common/torture/harden-cfr-bret-never.c   |  13 +
 .../c-c++-common/torture/harden-cfr-bret-noopt.c   |  12 +
 .../c-c++-common/torture/harden-cfr-bret-noret.c   |  12 +
 .../c-c++-common/torture/harden-cfr-bret-nothrow.c |  13 +
 .../c-c++-common/torture/harden-cfr-bret-retcl.c   |  12 +
 .../c-c++-common/torture/harden-cfr-bret.c         |   8 +-
 .../c-c++-common/torture/harden-cfr-noret-never.c  |  18 +
 .../torture/harden-cfr-noret-noexcept.c            |  16 +
 .../torture/harden-cfr-noret-nothrow.c             |  13 +
 .../c-c++-common/torture/harden-cfr-noret.c        |  38 +
 .../c-c++-common/torture/harden-cfr-notail.c       |   8 +
 .../c-c++-common/torture/harden-cfr-returning.c    |  35 +
 .../c-c++-common/torture/harden-cfr-tail.c         |  55 +-
 gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C  |  11 +
 .../g++.dg/harden-cfr-throw-returning-O0.C         |  10 +
 .../torture/harden-cfr-noret-always-no-nothrow.C   |  16 +
 .../torture/harden-cfr-noret-never-no-nothrow.C    |  18 +
 .../g++.dg/torture/harden-cfr-noret-no-nothrow.C   |  23 +
 .../g++.dg/torture/harden-cfr-throw-always.C       |  20 +
 .../g++.dg/torture/harden-cfr-throw-nocleanup.C    |  11 +
 .../g++.dg/torture/harden-cfr-throw-returning.C    |  31 +
 gcc/testsuite/g++.dg/torture/harden-cfr-throw.C    |  65 ++
 .../gcc.dg/torture/harden-cfr-noret-no-nothrow.c   |  15 +
 gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c  |  40 +
 gcc/testsuite/gnat.dg/hardcfr.adb                  |   2 +-
 libgcc/hardcfr.c                                   | 119 ++-
 32 files changed, 1568 insertions(+), 134 deletions(-)

diff --git a/gcc/common.opt b/gcc/common.opt
index ff89b2d1140..c2855cc19a9 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -1805,6 +1805,39 @@ fharden-control-flow-redundancy
 Common Var(flag_harden_control_flow_redundancy) Optimization
 Harden control flow by recording and checking execution paths.
 
+fhardcfr-check-returning-calls
+Common Var(flag_harden_control_flow_redundancy_check_returning_calls) Init(-1) Optimization
+Check CFR execution paths also before calls followed by returns of their results.
+
+fhardcfr-check-exceptions
+Common Var(flag_harden_control_flow_redundancy_check_exceptions) Init(-1) Optimization
+Check CFR execution paths also when exiting a function through an exception.
+
+fhardcfr-check-noreturn-calls=
+Common Joined RejectNegative Enum(hardcfr_check_noreturn_calls) Var(flag_harden_control_flow_redundancy_check_noreturn) Init(HCFRNR_UNSPECIFIED) Optimization
+-fhardcfr-check-noreturn-calls=[always|nothrow|never]	Check CFR execution paths also before calling noreturn functions.
+
+Enum
+Name(hardcfr_check_noreturn_calls) Type(enum hardcfr_noret) UnknownError(unknown hardcfr noreturn checking level %qs)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(never) Value(HCFRNR_NEVER)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(nothrow) Value(HCFRNR_NOTHROW)
+
+; ??? There could be yet another option here, that checked before
+; noreturn calls, except for those known to always throw, if we had
+; means to distinguish noreturn functions known to always throw, such
+; as those used to (re)raise exceptions, from those that merely might
+; throw.  "not always" stands for "not always-throwing", but it also
+; contrasts with "always" below.
+; EnumValue
+; Enum(hardcfr_check_noreturn_calls) String(not-always) Value(HCFRNR_NOT_ALWAYS)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(always) Value(HCFRNR_ALWAYS)
+
 ; Nonzero means ignore `#ident' directives.  0 means handle them.
 ; Generate position-independent code for executables if possible
 ; On SVR4 targets, it also controls whether or not to emit a
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index 87a33b037c1..7742d21f142 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -635,7 +635,9 @@ Objective-C and Objective-C++ Dialects}.
 -fsanitize-undefined-trap-on-error  -fbounds-check
 -fcf-protection=@r{[}full@r{|}branch@r{|}return@r{|}none@r{|}check@r{]}
 -fharden-compares -fharden-conditional-branches
--fharden-control-flow-redundancy
+-fharden-control-flow-redundancy  -fhardcfr-check-exceptions
+-fhardcfr-check-returning-calls
+-fhardcfr-check-noreturn-calls=@r{[}always@r{|}nothrow@r{|}never@r{]}
 -fstack-protector  -fstack-protector-all  -fstack-protector-strong
 -fstack-protector-explicit  -fstack-check
 -fstack-limit-register=@var{reg}  -fstack-limit-symbol=@var{sym}
@@ -17243,11 +17245,75 @@ conditionals.
 @opindex fharden-control-flow-redundancy
 @item -fharden-control-flow-redundancy
 Emit extra code to set booleans when entering basic blocks, and to
-verify, at function exits, that they amount to an execution path that is
-consistent with the control flow graph, trapping otherwise.  Tuning
-options @option{--param hardcfr-max-blocks} and @option{--param
+verify and trap, at function exits, when the booleans do not form an
+execution path that is compatible with the control flow graph.
+
+Verification takes place before returns, before mandatory tail calls
+(see below) and, optionally, before escaping exceptions with
+@option{-fhardcfr-check-exceptions}, before returning calls with
+@option{-fhardcfr-check-returning-calls}, and before noreturn calls with
+@option{-fhardcfr-check-noreturn-calls}).  Tuning options
+@option{--param hardcfr-max-blocks} and @option{--param
 hardcfr-max-inline-blocks} are available.
 
+Tail call optimization takes place too late to affect control flow
+redundancy, but calls annotated as mandatory tail calls by language
+front-ends, and any calls marked early enough as potential tail calls
+would also have verification issued before the call, but these
+possibilities are merely theoretical, as these conditions can only be
+met when using custom compiler plugins.
+
+@opindex fhardcfr-check-exceptions
+@opindex fno-hardcfr-check-exceptions
+@item -fhardcfr-check-exceptions
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph at exception
+escape points, as if the function body was wrapped with a cleanup
+handler that performed the check and reraised.  This option is enabled
+by default; use @option{-fno-hardcfr-check-exceptions} to disable it.
+
+@opindex fhardcfr-check-returning-calls
+@opindex fno-hardcfr-check-returning-calls
+@item -fhardcfr-check-returning-calls
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph before any
+function call immediately followed by a return of its result, if any, so
+as to not prevent tail-call optimization, whether or not it is
+ultimately optimized to a tail call.
+
+This option is enabled by default whenever sibling call optimizations
+are enabled (see @option{-foptimize-sibling-calls}), but it can be
+enabled (or disabled, using its negated form) explicitly, regardless of
+the optimizations.
+
+@opindex fhardcfr-check-noreturn-calls
+@item -fhardcfr-check-noreturn-calls=@r{[}always@r{|}nothrow@r{|}never@r{]}
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph before
+@code{noreturn} calls, either all of them (@option{always}), those that
+may not return control to the caller through an exception either
+(@option{nothrow}), or none of them (@option{never}, the default).
+
+Checking before a @code{noreturn} function that may return control to
+the caller through an exception may cause checking to be performed more
+than once, if the exception is caught in the caller, whether by a
+handler or a cleanup.  When @option{-fhardcfr-check-exceptions} is also
+enabled, the compiler will avoid associating a @code{noreturn} call with
+the implicitly-added cleanup handler, since it would be redundant with
+the check performed before the call, but other handlers or cleanups in
+the function, if activated, will modify the recorded execution path and
+check it again when another checkpoint is hit.  The checkpoint may even
+be another @code{noreturn} call, so checking may end up performed
+multiple times.
+
+Various optimizers may cause calls to be marked as @code{noreturn}
+and/or @code{nothrow}, even in the absence of the corresponding
+attributes, which may affect the placement of checks before calls, as
+well as the addition of implicit cleanup handlers for them.  This
+unpredictability, and the fact that raising and reraising exceptions
+frequently amounts to implicitly calling @code{noreturn} functions, have
+made @option{never} the default setting for this option.
+
 @opindex fstack-protector
 @item -fstack-protector
 Emit extra code to check for buffer overflows, such as stack smashing
diff --git a/gcc/flag-types.h b/gcc/flag-types.h
index 78dbdbe90dd..6d6fd19fcd6 100644
--- a/gcc/flag-types.h
+++ b/gcc/flag-types.h
@@ -157,6 +157,16 @@ enum stack_reuse_level
   SR_ALL
 };
 
+/* Control Flow Redundancy hardening options for noreturn calls.  */
+enum hardcfr_noret
+{
+  HCFRNR_NEVER,
+  HCFRNR_NOTHROW,
+  HCFRNR_NOT_ALWAYS, /* Reserved for future use.  */
+  HCFRNR_ALWAYS,
+  HCFRNR_UNSPECIFIED = -1
+};
+
 /* The live patching level.  */
 enum live_patching_level
 {
diff --git a/gcc/gimple-harden-control-flow.cc b/gcc/gimple-harden-control-flow.cc
index 6b08846dbb1..5c66762d448 100644
--- a/gcc/gimple-harden-control-flow.cc
+++ b/gcc/gimple-harden-control-flow.cc
@@ -29,7 +29,12 @@ along with GCC; see the file COPYING3.  If not see
 #include "tree-pass.h"
 #include "ssa.h"
 #include "gimple-iterator.h"
+#include "gimple-pretty-print.h"
 #include "tree-cfg.h"
+#include "tree-cfgcleanup.h"
+#include "tree-eh.h"
+#include "except.h"
+#include "sbitmap.h"
 #include "basic-block.h"
 #include "cfghooks.h"
 #include "cfgloop.h"
@@ -60,9 +65,7 @@ const pass_data pass_data_harden_control_flow_redundancy = {
   0,	    // properties_provided
   0,	    // properties_destroyed
   TODO_cleanup_cfg, // properties_start
-  TODO_update_ssa
-  | TODO_cleanup_cfg
-  | TODO_verify_il, // properties_finish
+  0,        // properties_finish
 };
 
 class pass_harden_control_flow_redundancy : public gimple_opt_pass
@@ -79,16 +82,6 @@ public:
     if (!flag_harden_control_flow_redundancy)
       return false;
 
-    /* We don't verify when an exception escapes, propagated or raised
-       by the function itself, so we're only concerned with edges to
-       the exit block.  If there aren't any, the function doesn't
-       return normally, so there won't be any checking point, so
-       there's no point in running the pass.  Should we add
-       verification at exception escapes, we should at least look at
-       !flag_exceptions here.  */
-    if (EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (fun)->preds) == 0)
-      return false;
-
     /* Functions that return more than once, like setjmp and vfork
        (that also gets this flag set), will start recording a path
        after the first return, and then may take another path when
@@ -117,8 +110,9 @@ public:
 	return false;
       }
 
-    if (param_hardcfr_max_blocks > 0
-	&& n_basic_blocks_for_fn (fun) - 2 > param_hardcfr_max_blocks)
+    if (fun->cfg && param_hardcfr_max_blocks > 0
+	&& (n_basic_blocks_for_fn (fun) - NUM_FIXED_BLOCKS
+	    > param_hardcfr_max_blocks))
       {
 	warning_at (DECL_SOURCE_LOCATION (fun->decl), 0,
 		    "%qD has more than %u blocks, the requested"
@@ -134,6 +128,293 @@ public:
 
 }
 
+/* Return TRUE iff CFR checks should be inserted before returning
+   calls.  */
+
+static bool
+check_returning_calls_p ()
+{
+  return
+    flag_harden_control_flow_redundancy_check_returning_calls > 0
+    || (flag_harden_control_flow_redundancy_check_returning_calls < 0
+	/* Gates pass_tail_calls.  */
+	&& flag_optimize_sibling_calls
+	/* Gates pass_all_optimizations.  */
+	&& optimize >= 1 && !optimize_debug);
+}
+
+/* Scan BB from the end, updating *RETPTR if given as return stmts and
+   copies are found.  Return a call or a stmt that cannot appear after
+   a tail call, or NULL if the top of the block is reached without
+   finding any.  */
+
+static gimple *
+hardcfr_scan_block (basic_block bb, tree **retptr)
+{
+  gimple_stmt_iterator gsi;
+  for (gsi = gsi_last_bb (bb); !gsi_end_p (gsi); gsi_prev (&gsi))
+    {
+      gimple *stmt = gsi_stmt (gsi);
+
+      /* Ignore labels, returns, nops, clobbers and debug stmts.  */
+      if (gimple_code (stmt) == GIMPLE_LABEL
+	  || gimple_code (stmt) == GIMPLE_NOP
+	  || gimple_code (stmt) == GIMPLE_PREDICT
+	  || gimple_clobber_p (stmt)
+	  || is_gimple_debug (stmt))
+	continue;
+
+      if (gimple_code (stmt) == GIMPLE_RETURN)
+	{
+	  greturn *gret = as_a <greturn *> (stmt);
+	  if (retptr)
+	    {
+	      gcc_checking_assert (!*retptr);
+	      *retptr = gimple_return_retval_ptr (gret);
+	    }
+	  continue;
+	}
+
+      /* Check for a call.  */
+      if (is_gimple_call (stmt))
+	return stmt;
+
+      /* Allow simple copies to the return value, updating the return
+	 value to be found in earlier assignments.  */
+      if (retptr && *retptr && gimple_assign_single_p (stmt)
+	  && **retptr == gimple_assign_lhs (stmt))
+	{
+	  *retptr = gimple_assign_rhs1_ptr (stmt);
+	  continue;
+	}
+
+      return stmt;
+    }
+
+  /* Any other kind of stmt will prevent a tail call.  */
+  return NULL;
+}
+
+/* Return TRUE iff CALL is to be preceded by a CFR checkpoint, i.e.,
+   if it's a returning call (one whose result is ultimately returned
+   without intervening non-copy statements) and we're checking
+   returning calls, a __builtin_return call (noreturn with a path to
+   the exit block), a must-tail call, or a tail call.  */
+
+static bool
+returning_call_p (gcall *call)
+{
+  if (!(gimple_call_noreturn_p (call)
+	|| gimple_call_must_tail_p (call)
+	|| gimple_call_tail_p (call)
+	|| check_returning_calls_p ()))
+    return false;
+
+  /* Quickly check that there's a path to exit compatible with a
+     returning call.  Detect infinite loops through the counter.  */
+  basic_block bb = gimple_bb (call);
+  auto_vec<basic_block, 10> path;
+  for (int i = n_basic_blocks_for_fn (cfun);
+       bb != EXIT_BLOCK_PTR_FOR_FN (cfun) && i--;
+       bb = single_succ (bb))
+    if (!single_succ_p (bb)
+	|| (single_succ_edge (bb)->flags & EDGE_EH) != 0)
+      return false;
+    else
+      path.safe_push (bb);
+
+  /* Check the stmts in the blocks and trace the return value.  */
+  tree *retptr = NULL;
+  for (;;)
+    {
+      gcc_checking_assert (!path.is_empty ());
+      basic_block bb = path.pop ();
+      gimple *stop = hardcfr_scan_block (bb, &retptr);
+      if (stop)
+	{
+	  if (stop != call)
+	    return false;
+	  gcc_checking_assert (path.is_empty ());
+	  break;
+	}
+
+      gphi *retphi = NULL;
+      if (retptr && *retptr && TREE_CODE (*retptr) == SSA_NAME
+	  && !SSA_NAME_IS_DEFAULT_DEF (*retptr)
+	  && SSA_NAME_DEF_STMT (*retptr)
+	  && is_a <gphi *> (SSA_NAME_DEF_STMT (*retptr))
+	  && gimple_bb (SSA_NAME_DEF_STMT (*retptr)) == bb)
+	{
+	  retphi = as_a <gphi *> (SSA_NAME_DEF_STMT (*retptr));
+	  gcc_checking_assert (gimple_phi_result (retphi) == *retptr);
+	}
+      else
+	continue;
+
+      gcc_checking_assert (!path.is_empty ());
+      edge e = single_succ_edge (path.last ());
+      int i = EDGE_COUNT (bb->preds);
+      while (i--)
+	if (EDGE_PRED (bb, i) == e)
+	  break;
+      gcc_checking_assert (i >= 0);
+      retptr = gimple_phi_arg_def_ptr (retphi, i);
+    }
+
+  return (gimple_call_noreturn_p (call)
+	  || gimple_call_must_tail_p (call)
+	  || gimple_call_tail_p (call)
+	  || (gimple_call_lhs (call) == (retptr ? *retptr : NULL)
+	      && check_returning_calls_p ()));
+}
+
+typedef auto_vec<edge, 10> chk_edges_t;
+
+/* Declare for mutual recursion.  */
+static bool hardcfr_sibcall_search_preds (basic_block bb,
+					  chk_edges_t &chk_edges,
+					  int &count_chkcall,
+					  auto_sbitmap &chkcall_blocks,
+					  int &count_postchk,
+					  auto_sbitmap &postchk_blocks,
+					  tree *retptr);
+
+/* Search backwards from the end of BB for a mandatory or potential
+   sibcall.  Schedule the block to be handled sort-of like noreturn if
+   so.  Recurse to preds, with updated RETPTR, if the block only
+   contains stmts that may follow such a call, scheduling checking at
+   edges and marking blocks as post-check as needed.  Return true iff,
+   at the end of the block, a check will have already been
+   performed.  */
+
+static bool
+hardcfr_sibcall_search_block (basic_block bb,
+			      chk_edges_t &chk_edges,
+			      int &count_chkcall,
+			      auto_sbitmap &chkcall_blocks,
+			      int &count_postchk,
+			      auto_sbitmap &postchk_blocks,
+			      tree *retptr)
+{
+  /* Conditionals and internal exceptions rule out tail calls.  */
+  if (!single_succ_p (bb)
+      || (single_succ_edge (bb)->flags & EDGE_EH) != 0)
+    return false;
+
+  gimple *stmt = hardcfr_scan_block (bb, &retptr);
+  if (!stmt)
+    return hardcfr_sibcall_search_preds (bb, chk_edges,
+					 count_chkcall, chkcall_blocks,
+					 count_postchk, postchk_blocks,
+					 retptr);
+
+  if (!is_a <gcall *> (stmt))
+    return false;
+
+  /* Avoid disrupting mandatory or early-marked tail calls,
+     inserting the check before them.  This works for
+     must-tail calls, but tail calling as an optimization is
+     detected too late for us.
+
+     Also check for noreturn calls here.  Noreturn calls won't
+     normally have edges to exit, so they won't be found here,
+     but __builtin_return does, and we must check before
+     it, so handle it like a tail call.  */
+  gcall *call = as_a <gcall *> (stmt);
+  if (!(gimple_call_noreturn_p (call)
+	|| gimple_call_must_tail_p (call)
+	|| gimple_call_tail_p (call)
+	|| (gimple_call_lhs (call) == (retptr ? *retptr : NULL)
+	    && check_returning_calls_p ())))
+    return false;
+
+  gcc_checking_assert (returning_call_p (call));
+
+  /* We found a call that is to be preceded by checking.  */
+  if (bitmap_set_bit (chkcall_blocks, bb->index))
+    ++count_chkcall;
+  else
+    gcc_unreachable ();
+  return true;
+}
+
+
+/* Search preds of BB for a mandatory or potential sibcall or
+   returning call, and arrange for the blocks containing them to have
+   a check inserted before the call, like noreturn calls.  If any
+   preds are found to perform checking, schedule checks at the edges
+   of those that don't, and mark BB as postcheck..  */
+
+static bool
+hardcfr_sibcall_search_preds (basic_block bb,
+			      chk_edges_t &chk_edges,
+			      int &count_chkcall,
+			      auto_sbitmap &chkcall_blocks,
+			      int &count_postchk,
+			      auto_sbitmap &postchk_blocks,
+			      tree *retptr)
+{
+  /* For the exit block, we wish to force a check at every
+     predecessor, so pretend we've already found a pred that had
+     checking, so that we schedule checking at every one of its pred
+     edges.  */
+  bool first = bb->index >= NUM_FIXED_BLOCKS;
+  bool postchecked = true;
+
+  gphi *retphi = NULL;
+  if (retptr && *retptr && TREE_CODE (*retptr) == SSA_NAME
+      && !SSA_NAME_IS_DEFAULT_DEF (*retptr)
+      && SSA_NAME_DEF_STMT (*retptr)
+      && is_a <gphi *> (SSA_NAME_DEF_STMT (*retptr))
+      && gimple_bb (SSA_NAME_DEF_STMT (*retptr)) == bb)
+    {
+      retphi = as_a <gphi *> (SSA_NAME_DEF_STMT (*retptr));
+      gcc_checking_assert (gimple_phi_result (retphi) == *retptr);
+    }
+
+  for (int i = EDGE_COUNT (bb->preds); i--; first = false)
+    {
+      edge e = EDGE_PRED (bb, i);
+
+      bool checked
+	= hardcfr_sibcall_search_block (e->src, chk_edges,
+					count_chkcall, chkcall_blocks,
+					count_postchk, postchk_blocks,
+					!retphi ? retptr
+					: gimple_phi_arg_def_ptr (retphi, i));
+
+      if (first)
+	{
+	  postchecked = checked;
+	  continue;
+	}
+
+      /* When we first find a checked block, force a check at every
+	 other incoming edge we've already visited, and those we
+	 visit afterwards that don't have their own check, so that
+	 when we reach BB, the check has already been performed.  */
+      if (!postchecked && checked)
+	{
+	  for (int j = EDGE_COUNT (bb->preds); --j > i; )
+	    chk_edges.safe_push (EDGE_PRED (bb, j));
+	  postchecked = true;
+	}
+      if (postchecked && !checked)
+	chk_edges.safe_push (EDGE_PRED (bb, i));
+    }
+
+  if (postchecked && bb->index >= NUM_FIXED_BLOCKS)
+    {
+      if (bitmap_set_bit (postchk_blocks, bb->index))
+	count_postchk++;
+      else
+	gcc_unreachable ();
+    }
+
+  return postchecked;
+}
+
+
 class rt_bb_visited
 {
   /* Use a sufficiently wide unsigned type to hold basic block numbers.  */
@@ -176,8 +457,8 @@ class rt_bb_visited
      neither ENTRY nor EXIT, but maybe one-past-the-end, to compute
      the visited array length.  */
   blknum num2idx (blknum n) {
-    gcc_checking_assert (n >= 2 && n <= nblocks);
-    return (n - 2);
+    gcc_checking_assert (n >= NUM_FIXED_BLOCKS && n <= nblocks);
+    return (n - NUM_FIXED_BLOCKS);
   }
   /* Return the block vindex for BB, that must not be ENTRY or
      EXIT.  */
@@ -249,8 +530,7 @@ class rt_bb_visited
   }
 
   /* Set the bit corresponding to BB in VISITED.  Add to SEQ any
-     required gimple statements, and return SEQ, possibly
-     modified.  */
+     required gimple stmts, and return SEQ, possibly modified.  */
   gimple_seq vset (basic_block bb, gimple_seq seq = NULL)
   {
     tree bit, setme = vword (bb, &bit);
@@ -270,7 +550,7 @@ class rt_bb_visited
 
 public:
   /* Prepare to add control flow redundancy testing to CFUN.  */
-  rt_bb_visited ()
+  rt_bb_visited (int checkpoints)
     : nblocks (n_basic_blocks_for_fn (cfun)),
       vword_type (NULL), ckseq (NULL), rtcfg (NULL)
   {
@@ -353,8 +633,8 @@ public:
 					 NULL, NULL);
     gimple_seq_add_stmt (&ckseq, detach);
 
-    if (nblocks - 2 > blknum (param_hardcfr_max_inline_blocks)
-	|| !single_pred_p (EXIT_BLOCK_PTR_FOR_FN (cfun)))
+    if (nblocks - NUM_FIXED_BLOCKS > blknum (param_hardcfr_max_inline_blocks)
+	|| checkpoints > 1)
       {
 	/* Make sure vword_bits is wide enough for the representation
 	   of nblocks in rtcfg.  Compare with vword_bits << vword_bits,
@@ -379,65 +659,34 @@ public:
     gimple_seq_add_stmt (&ckseq, ckfail_init);
   }
 
-  /* Insert SEQ on E, or close enough (e.g., before a noreturn or tail
-     call at the end of E->src).  */
-  void insert_exit_check (gimple_seq seq, edge e)
+  /* Insert SEQ before a resx or a call in INSBB.  */
+  void insert_exit_check_in_block (gimple_seq seq, basic_block insbb)
   {
-    basic_block insbb = e->src;
-
-    /* If the returning block ends with a noreturn call, insert
-       checking before it.  This is particularly important for
-       __builtin_return.  Other noreturn calls won't have an edge to
-       the exit block, so they won't even be considered as exit paths.
-
-       Insert-on-edge inserts before other return stmts, but not
-       before calls, and if a single-block function had the check
-       sequence inserted after a noreturn call, it would be useless,
-       but checking would still flag it as malformed if block 2 has a
-       fallthru edge to the exit block.
-
-       Also avoid disrupting tail calls, inserting the check before
-       them.  This works for must-tail calls, but tail calling as an
-       optimization is detected too late for us.  */
     gimple_stmt_iterator gsi = gsi_last_bb (insbb);
-    gimple *ret = gsi_stmt (gsi);
-    if (ret && is_a <greturn *> (ret))
-      {
+
+    while (!gsi_end_p (gsi))
+      if (is_a <gresx *> (gsi_stmt (gsi))
+	  || is_a <gcall *> (gsi_stmt (gsi)))
+	break;
+      else
 	gsi_prev (&gsi);
-	if (!gsi_end_p (gsi))
-	  ret = gsi_stmt (gsi);
-      }
-    if (ret && is_a <gcall *> (ret)
-	&& (gimple_call_noreturn_p (ret)
-	    || gimple_call_must_tail_p (as_a <gcall *> (ret))
-	    || gimple_call_tail_p (as_a <gcall *> (ret))))
-      gsi_insert_seq_before (&gsi, seq, GSI_SAME_STMT);
-    else
-      gsi_insert_seq_on_edge_immediate (e, seq);
+
+    gsi_insert_seq_before (&gsi, seq, GSI_SAME_STMT);
   }
 
-  /* Add checking code on every exit edge, and initialization code on
-     the entry edge.  Before this point, the CFG has been undisturbed,
-     and all the needed data has been collected and safely stowed.  */
-  void check ()
+  /* Insert SEQ on E.  */
+  void insert_exit_check_on_edge (gimple_seq seq, edge e)
   {
-    /* Insert initializers for visited at the entry.  */
-    gimple_seq iseq = NULL;
-
-    gcall *vinit = gimple_build_call (builtin_decl_explicit
-				      (BUILT_IN_MEMSET), 3,
-				      build1 (ADDR_EXPR,
-					      build_pointer_type
-					      (TREE_TYPE (visited)),
-					      visited),
-				      integer_zero_node,
-				      TYPE_SIZE_UNIT (TREE_TYPE (visited)));
-    gimple_seq_add_stmt (&iseq, vinit);
-
-    gsi_insert_seq_on_edge_immediate (single_succ_edge
-				      (ENTRY_BLOCK_PTR_FOR_FN (cfun)),
-				      iseq);
+    gsi_insert_seq_on_edge_immediate (e, seq);
+  }
 
+  /* Add checking code to CHK_EDGES and CHKCALL_BLOCKS, and
+     initialization code on the entry edge.  Before this point, the
+     CFG has been undisturbed, and all the needed data has been
+     collected and safely stowed.  */
+  void check (chk_edges_t &chk_edges,
+	      int count_chkcall, auto_sbitmap const &chkcall_blocks)
+  {
     /* If we're using out-of-line checking, create and statically
        initialize the CFG checking representation, generate the
        checker call for the checking sequence, and insert it in all
@@ -498,28 +747,116 @@ public:
 						     rtcfg));
 	gimple_seq_add_stmt (&ckseq, call_chk);
 
+	gimple *clobber = gimple_build_assign (visited,
+					       build_clobber
+					       (TREE_TYPE (visited)));
+	gimple_seq_add_stmt (&ckseq, clobber);
+
 	/* If we have multiple exit edges, insert (copies of)
 	   ckseq in all of them.  */
-	for (int i = EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (cfun)->preds);
-	     i--; )
+	for (int i = chk_edges.length (); i--; )
 	  {
 	    gimple_seq seq = ckseq;
 	    /* Copy the sequence, unless we're dealing with the
 	       last edge (we're counting down to zero).  */
-	    if (i)
+	    if (i || count_chkcall)
+	      seq = gimple_seq_copy (seq);
+
+	    edge e = chk_edges[i];
+
+	    if (dump_file)
+	      {
+		if (e->dest == EXIT_BLOCK_PTR_FOR_FN (cfun))
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to exit.\n",
+			   e->src->index);
+		else
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to postcheck block %i.\n",
+			   e->src->index, e->dest->index);
+	      }
+
+	    insert_exit_check_on_edge (seq, e);
+
+	    gcc_checking_assert (!bitmap_bit_p (chkcall_blocks, e->src->index));
+	  }
+
+	sbitmap_iterator it;
+	unsigned i;
+	EXECUTE_IF_SET_IN_BITMAP (chkcall_blocks, 0, i, it)
+	  {
+	    basic_block bb = BASIC_BLOCK_FOR_FN (cfun, i);
+
+	    gimple_seq seq = ckseq;
+	    gcc_checking_assert (count_chkcall > 0);
+	    if (--count_chkcall)
 	      seq = gimple_seq_copy (seq);
 
-	    insert_exit_check (seq,
-			       EDGE_PRED (EXIT_BLOCK_PTR_FOR_FN (cfun), i));
+	    if (dump_file)
+	      fprintf (dump_file,
+		       "Inserting out-of-line check before stmt in block %i.\n",
+		       bb->index);
+
+	    insert_exit_check_in_block (seq, bb);
 	  }
+
+	gcc_checking_assert (count_chkcall == 0);
       }
     else
       {
 	/* Inline checking requires a single exit edge.  */
-	gimple *last = gsi_stmt (gsi_last (ckseq));
+	gimple *last = gimple_build_assign (visited,
+					    build_clobber
+					    (TREE_TYPE (visited)));
+	gimple_seq_add_stmt (&ckseq, last);
 
-	insert_exit_check (ckseq,
-			   single_pred_edge (EXIT_BLOCK_PTR_FOR_FN (cfun)));
+	if (!count_chkcall)
+	  {
+	    edge e = single_pred_edge (EXIT_BLOCK_PTR_FOR_FN (cfun));
+
+	    if (dump_file)
+	      {
+		if (e->dest == EXIT_BLOCK_PTR_FOR_FN (cfun))
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to postcheck block %i.\n",
+			   e->src->index, e->dest->index);
+		else
+		  fprintf (dump_file,
+			   "Inserting inline check in"
+			   " block %i's edge to exit.\n",
+			   e->src->index);
+	      }
+
+	    insert_exit_check_on_edge (ckseq, e);
+	  }
+	else
+	  {
+	    gcc_checking_assert (count_chkcall == 1);
+
+	    sbitmap_iterator it;
+	    unsigned i;
+	    EXECUTE_IF_SET_IN_BITMAP (chkcall_blocks, 0, i, it)
+	      {
+		basic_block bb = BASIC_BLOCK_FOR_FN (cfun, i);
+
+		gimple_seq seq = ckseq;
+		gcc_checking_assert (count_chkcall > 0);
+		if (--count_chkcall)
+		  seq = gimple_seq_copy (seq);
+
+		if (dump_file)
+		  fprintf (dump_file,
+			   "Inserting inline check before stmt in block %i.\n",
+			   bb->index);
+
+		insert_exit_check_in_block (seq, bb);
+	      }
+
+	    gcc_checking_assert (count_chkcall == 0);
+	  }
 
 	/* The inserted ckseq computes CKFAIL at LAST.  Now we have to
 	   conditionally trap on it.  */
@@ -540,8 +877,7 @@ public:
 	  add_bb_to_loop (trp, current_loops->tree_root);
 
 	/* Insert a conditional branch to the trap block.  If the
-	   conditional wouldn't be the last statement, split the
-	   block.  */
+	   conditional wouldn't be the last stmt, split the block.  */
 	gimple_stmt_iterator gsi = gsi_for_stmt (last);
 	if (!gsi_one_before_end_p (gsi))
 	  split_block (gsi_bb (gsi), gsi_stmt (gsi));
@@ -564,6 +900,24 @@ public:
 	if (dom_info_available_p (CDI_DOMINATORS))
 	  set_immediate_dominator (CDI_DOMINATORS, trp, gimple_bb (last));
       }
+
+    /* Insert initializers for visited at the entry.  Do this after
+       other insertions, to avoid messing with block numbers.  */
+    gimple_seq iseq = NULL;
+
+    gcall *vinit = gimple_build_call (builtin_decl_explicit
+				      (BUILT_IN_MEMSET), 3,
+				      build1 (ADDR_EXPR,
+					      build_pointer_type
+					      (TREE_TYPE (visited)),
+					      visited),
+				      integer_zero_node,
+				      TYPE_SIZE_UNIT (TREE_TYPE (visited)));
+    gimple_seq_add_stmt (&iseq, vinit);
+
+    gsi_insert_seq_on_edge_immediate (single_succ_edge
+				      (ENTRY_BLOCK_PTR_FOR_FN (cfun)),
+				      iseq);
   }
 
   /* Push onto RTCFG a (mask, index) pair to test for IBB when BB is
@@ -607,7 +961,7 @@ public:
     return false;
   }
 
-  /* Add to CKSEQ statements to clear CKPART if OBB is visited.  */
+  /* Add to CKSEQ stmts to clear CKPART if OBB is visited.  */
   void
   build_block_check (basic_block obb)
   {
@@ -632,34 +986,48 @@ public:
 
   /* Add to BB code to set its bit in VISITED, and add to RTCFG or
      CKSEQ the data or code needed to check BB's predecessors and
-     successors.  Do NOT change the CFG.  */
-  void visit (basic_block bb)
+     successors.  If CHECKPOINT, assume the block is a checkpoint,
+     whether or not it has an edge to EXIT.  If POSTCHECK, assume the
+     block post-dominates checkpoints and therefore no bitmap setting
+     or checks are to be performed in or for it.  Do NOT change the
+     CFG.  */
+  void visit (basic_block bb, bool checkpoint, bool postcheck)
   {
     /* Set the bit in VISITED when entering the block.  */
     gimple_stmt_iterator gsi = gsi_after_labels (bb);
-    gsi_insert_seq_before (&gsi, vset (bb), GSI_SAME_STMT);
+    if (!postcheck)
+      gsi_insert_seq_before (&gsi, vset (bb), GSI_SAME_STMT);
 
     if (rtcfg)
       {
-	/* Build a list of (index, mask) terminated by (NULL, 0).
-	   Consolidate masks with the same index when they're
-	   adjacent.  First, predecessors.  Count backwards, because
-	   we're going to reverse the list.  The order shouldn't
-	   matter, but let's not make it surprising.  */
-	for (int i = EDGE_COUNT (bb->preds); i--; )
-	  if (push_rtcfg_pair (EDGE_PRED (bb, i)->src, bb,
-			       ENTRY_BLOCK_PTR_FOR_FN (cfun)))
-	    break;
+	if (!postcheck)
+	  {
+	    /* Build a list of (index, mask) terminated by (NULL, 0).
+	       Consolidate masks with the same index when they're
+	       adjacent.  First, predecessors.  Count backwards, because
+	       we're going to reverse the list.  The order shouldn't
+	       matter, but let's not make it surprising.  */
+	    for (int i = EDGE_COUNT (bb->preds); i--; )
+	      if (push_rtcfg_pair (EDGE_PRED (bb, i)->src, bb,
+				   ENTRY_BLOCK_PTR_FOR_FN (cfun)))
+		break;
+	  }
 	rtcfg = tree_cons (NULL_TREE, build_int_cst (vword_type, 0), rtcfg);
 
-	/* Then, successors.  */
-	for (int i = EDGE_COUNT (bb->succs); i--; )
-	  if (push_rtcfg_pair (EDGE_SUCC (bb, i)->dest, bb,
-			       EXIT_BLOCK_PTR_FOR_FN (cfun)))
-	    break;
+	if (!postcheck)
+	  {
+	    /* Then, successors.  */
+	    if (!checkpoint
+		|| !push_rtcfg_pair (EXIT_BLOCK_PTR_FOR_FN (cfun),
+				     bb, EXIT_BLOCK_PTR_FOR_FN (cfun)))
+	      for (int i = EDGE_COUNT (bb->succs); i--; )
+		if (push_rtcfg_pair (EDGE_SUCC (bb, i)->dest, bb,
+				     EXIT_BLOCK_PTR_FOR_FN (cfun)))
+		  break;
+	  }
 	rtcfg = tree_cons (NULL_TREE, build_int_cst (vword_type, 0), rtcfg);
       }
-    else
+    else if (!postcheck)
       {
 	/* Schedule test to fail if the block was reached but somehow none
 	   of its predecessors were.  */
@@ -677,6 +1045,8 @@ public:
 	gassign *blkruns = gimple_build_assign (ckpart, unshare_expr (bit));
 	gimple_seq_add_stmt (&ckseq, blkruns);
 
+	if (checkpoint)
+	  build_block_check (EXIT_BLOCK_PTR_FOR_FN (cfun));
 	for (int i = 0, e = EDGE_COUNT (bb->succs); i < e; i++)
 	  build_block_check (EDGE_SUCC (bb, i)->dest);
 
@@ -687,21 +1057,357 @@ public:
   }
 };
 
+/* It might be useful to avoid checking before noreturn calls that are
+   known to always finish by throwing an exception, rather than by
+   ending the program or looping forever.  Such functions would have
+   to be annotated somehow, with an attribute or flag, so that
+   exception-raising functions, such as C++'s __cxa_throw,
+   __cxa_rethrow, and Ada's gnat_rcheck_*, gnat_reraise*,
+   ada.exception.raise_exception*, and the language-independent
+   unwinders could be detected here and handled differently from other
+   noreturn functions.  */
+static bool
+always_throwing_noreturn_call_p (gimple *)
+{
+  return false;
+}
+
 /* Control flow redundancy hardening: record the execution path, and
    verify at exit that an expect path was taken.  */
 
 unsigned int
-pass_harden_control_flow_redundancy::execute (function *)
+pass_harden_control_flow_redundancy::execute (function *fun)
 {
-  rt_bb_visited vstd;
-
+  bool const check_at_escaping_exceptions
+    = (flag_exceptions
+       && flag_harden_control_flow_redundancy_check_exceptions);
+  bool const check_before_noreturn_calls
+    = flag_harden_control_flow_redundancy_check_noreturn > HCFRNR_NEVER;
+  bool const check_before_nothrow_noreturn_calls
+    = (check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn >= HCFRNR_NOTHROW);
+  bool const check_before_throwing_noreturn_calls
+    = (flag_exceptions
+       && check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn > HCFRNR_NOTHROW);
+  bool const check_before_always_throwing_noreturn_calls
+    = (flag_exceptions
+       && check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn >= HCFRNR_ALWAYS);
   basic_block bb;
-  FOR_EACH_BB_FN (bb, cfun)
-    vstd.visit (bb);
+  basic_block bb_eh_cleanup = NULL;
+
+  if (check_at_escaping_exceptions)
+    {
+      int lp_eh_cleanup = -1;
+
+      /* Record the preexisting blocks, to avoid visiting newly-created
+	 blocks.  */
+      auto_sbitmap to_visit (last_basic_block_for_fn (fun));
+      bitmap_clear (to_visit);
+
+      FOR_EACH_BB_FN (bb, fun)
+	bitmap_set_bit (to_visit, bb->index);
+
+      /* Scan the blocks for stmts with escaping exceptions, that
+	 wouldn't be denoted in the CFG, and associate them with an
+	 empty cleanup handler around the whole function.  Walk
+	 backwards, so that even when we split the block, */
+      sbitmap_iterator it;
+      unsigned i;
+      EXECUTE_IF_SET_IN_BITMAP (to_visit, 0, i, it)
+	{
+	  bb = BASIC_BLOCK_FOR_FN (fun, i);
+
+	  for (gimple_stmt_iterator gsi = gsi_last_bb (bb);
+	       !gsi_end_p (gsi); gsi_prev (&gsi))
+	    {
+	      gimple *stmt = gsi_stmt (gsi);
+	      if (!stmt_could_throw_p (fun, stmt))
+		continue;
+
+	      /* If it must not throw, or if it already has a handler,
+		 we need not worry about it.  */
+	      if (lookup_stmt_eh_lp (stmt) != 0)
+		continue;
+
+	      /* Don't split blocks at, nor add EH edges to, tail
+		 calls, we will add verification before the call
+		 anyway.  */
+	      if (is_a <gcall *> (stmt)
+		  && (gimple_call_must_tail_p (as_a <gcall *> (stmt))
+		      || gimple_call_tail_p (as_a <gcall *> (stmt))
+		      || returning_call_p (as_a <gcall *> (stmt))))
+		continue;
+
+	      if (!gsi_one_before_end_p (gsi))
+		split_block (bb, stmt);
+	      /* A resx or noreturn call needs not be associated with
+		 the cleanup handler if we're going to add checking
+		 before it.  We only test cases that didn't require
+		 block splitting because noreturn calls would always
+		 be at the end of blocks, and we test for zero
+		 successors because if there is an edge, it's not
+		 noreturn, as any EH edges would have already been
+		 caught by the lookup_stmt_eh_lp test above.  */
+	      else if (check_before_noreturn_calls
+		       && EDGE_COUNT (bb->succs) == 0
+		       && (is_a <gresx *> (stmt)
+			   ? check_before_always_throwing_noreturn_calls
+			   : (!is_a <gcall *> (stmt)
+			      || !gimple_call_noreturn_p (stmt))
+			   ? (gcc_unreachable (), false)
+			   : (!flag_exceptions
+			      || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+			   ? check_before_nothrow_noreturn_calls
+			   : always_throwing_noreturn_call_p (stmt)
+			   ? check_before_always_throwing_noreturn_calls
+			   : check_before_throwing_noreturn_calls))
+		{
+		  if (dump_file)
+		    {
+		      fprintf (dump_file,
+			       "Bypassing cleanup for noreturn stmt"
+			       " in block %i:\n",
+			       bb->index);
+		      print_gimple_stmt (dump_file, stmt, 0);
+		    }
+		  continue;
+		}
+
+	      if (!bb_eh_cleanup)
+		{
+		  bb_eh_cleanup = create_empty_bb (bb);
+		  if (dom_info_available_p (CDI_DOMINATORS))
+		    set_immediate_dominator (CDI_DOMINATORS, bb_eh_cleanup, bb);
+		  if (current_loops)
+		    add_bb_to_loop (bb_eh_cleanup, current_loops->tree_root);
+
+		  /* Make the new block an EH cleanup for the call.  */
+		  eh_region new_r = gen_eh_region_cleanup (NULL);
+		  eh_landing_pad lp = gen_eh_landing_pad (new_r);
+		  tree label = gimple_block_label (bb_eh_cleanup);
+		  lp->post_landing_pad = label;
+		  EH_LANDING_PAD_NR (label) = lp_eh_cleanup = lp->index;
+
+		  /* Just propagate the exception.
+		     We will later insert the verifier call.  */
+		  gimple_stmt_iterator ehgsi;
+		  ehgsi = gsi_after_labels (bb_eh_cleanup);
+		  gresx *resx = gimple_build_resx (new_r->index);
+		  gsi_insert_before (&ehgsi, resx, GSI_SAME_STMT);
+
+		  if (dump_file)
+		    fprintf (dump_file,
+			     "Created cleanup block %i:\n",
+			     bb_eh_cleanup->index);
+		}
+	      else if (dom_info_available_p (CDI_DOMINATORS))
+		{
+		  basic_block immdom;
+		  immdom = get_immediate_dominator (CDI_DOMINATORS,
+						    bb_eh_cleanup);
+		  if (!dominated_by_p (CDI_DOMINATORS, bb, immdom))
+		    {
+		      immdom = nearest_common_dominator (CDI_DOMINATORS,
+							 immdom, bb);
+		      set_immediate_dominator (CDI_DOMINATORS,
+					       bb_eh_cleanup, immdom);
+		    }
+		}
+
+	      if (dump_file)
+		{
+		  fprintf (dump_file,
+			   "Associated cleanup block with stmt in block %i:\n",
+			   bb->index);
+		  print_gimple_stmt (dump_file, stmt, 0);
+		}
+
+	      add_stmt_to_eh_lp (stmt, lp_eh_cleanup);
+	      /* Finally, wire the EH cleanup block into the CFG.  */
+	      make_eh_edges (stmt);
+	    }
+	}
+
+      if (bb_eh_cleanup)
+	{
+	  /* A cfg_cleanup after bb_eh_cleanup makes for a more compact
+	     rtcfg, and it avoids bb numbering differences when we split
+	     blocks because of trailing debug insns only.  */
+	  cleanup_tree_cfg ();
+	  gcc_checking_assert (EDGE_COUNT (bb_eh_cleanup->succs) == 0);
+	}
+    }
+
+  /* These record blocks with calls that are to be preceded by
+     checkpoints, such as noreturn calls (if so chosen), must-tail
+     calls, potential early-marked tail calls, and returning calls (if
+     so chosen).  */
+  int count_chkcall = 0;
+  auto_sbitmap chkcall_blocks (last_basic_block_for_fn (fun));
+  bitmap_clear (chkcall_blocks);
+
+  /* We wish to add verification at blocks without successors, such as
+     noreturn calls (raising or not) and the reraise at the cleanup
+     block, but not other reraises: they will go through the cleanup
+     block.  */
+  if (check_before_noreturn_calls)
+    FOR_EACH_BB_FN (bb, fun)
+      {
+	gimple_stmt_iterator gsi = gsi_last_bb (bb);
+	if (gsi_end_p (gsi))
+	  continue;
+	gimple *stmt = gsi_stmt (gsi);
 
-  vstd.check ();
+	if (EDGE_COUNT (bb->succs) == 0)
+	  {
+	    /* A stmt at the end of a block without any successors is
+	       either a resx or a noreturn call without a local
+	       handler.  Check that it's one of the desired
+	       checkpoints.  */
+	    if (flag_exceptions && is_a <gresx *> (stmt)
+		? (check_before_always_throwing_noreturn_calls
+		   || bb == bb_eh_cleanup)
+		: (!is_a <gcall *> (stmt)
+		   || !gimple_call_noreturn_p (stmt))
+		? (/* Catch cases in which successors would be
+		      expected.  */
+		   gcc_unreachable (), false)
+		: (!flag_exceptions
+		   || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+		? check_before_nothrow_noreturn_calls
+		: always_throwing_noreturn_call_p (stmt)
+		? check_before_always_throwing_noreturn_calls
+		: check_before_throwing_noreturn_calls)
+	      {
+		if (dump_file)
+		  {
+		    fprintf (dump_file,
+			     "Scheduling check before stmt"
+			     " in succ-less block %i:\n",
+			     bb->index);
+		    print_gimple_stmt (dump_file, stmt, 0);
+		  }
+
+		if (bitmap_set_bit (chkcall_blocks, bb->index))
+		  count_chkcall++;
+		else
+		  gcc_unreachable ();
+	      }
+	    continue;
+	  }
 
-  return 0;
+	/* If there are no exceptions, it would seem like any noreturn
+	   call must have zero successor edges, but __builtin_return
+	   gets successor edges.  We don't want to handle it here, it
+	   will be dealt with in sibcall_search_preds.  Otherwise,
+	   check for blocks without non-EH successors, but skip those
+	   with resx stmts and edges (i.e., those other than that in
+	   bb_eh_cleanup), since those will go through bb_eh_cleanup,
+	   that will have been counted as noreturn above because it
+	   has no successors.  */
+	gcc_checking_assert (bb != bb_eh_cleanup
+			     || !check_at_escaping_exceptions);
+	if (flag_exceptions && is_a <gresx *> (stmt)
+	    ? check_before_always_throwing_noreturn_calls
+	    : (!is_a <gcall *> (stmt)
+	       || !gimple_call_noreturn_p (stmt))
+	    ? false
+	    : (!flag_exceptions
+	       || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+	    ? false /* rather than check_before_nothrow_noreturn_calls */
+	    : always_throwing_noreturn_call_p (stmt)
+	    ? check_before_always_throwing_noreturn_calls
+	    : check_before_throwing_noreturn_calls)
+	  {
+	    gcc_checking_assert (single_succ_p (bb)
+				 && (single_succ_edge (bb)->flags & EDGE_EH));
+
+	    if (dump_file)
+	      {
+		fprintf (dump_file,
+			 "Scheduling check before stmt"
+			 " in EH-succ block %i:\n",
+			 bb->index);
+		print_gimple_stmt (dump_file, stmt, 0);
+	      }
+
+	    if (bitmap_set_bit (chkcall_blocks, bb->index))
+	      count_chkcall++;
+	    else
+	      gcc_unreachable ();
+	  }
+      }
+  else if (bb_eh_cleanup)
+    {
+      if (bitmap_set_bit (chkcall_blocks, bb_eh_cleanup->index))
+	count_chkcall++;
+      else
+	gcc_unreachable ();
+    }
+
+  gcc_checking_assert (!bb_eh_cleanup
+		       || bitmap_bit_p (chkcall_blocks, bb_eh_cleanup->index));
+
+  /* If we don't have edges to exit nor noreturn calls (including the
+     cleanup reraise), then we may skip instrumentation: that would
+     amount to a function that ends with an infinite loop.  */
+  if (!count_chkcall
+      && EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (fun)->preds) == 0)
+    {
+      if (dump_file)
+	fprintf (dump_file,
+		 "Disabling CFR, no exit paths to check\n");
+
+      return 0;
+    }
+
+  /* Search for must-tail calls, early-marked potential tail calls,
+     and, if requested, returning calls.  As we introduce early
+     checks, */
+  int count_postchk = 0;
+  auto_sbitmap postchk_blocks (last_basic_block_for_fn (fun));
+  bitmap_clear (postchk_blocks);
+  chk_edges_t chk_edges;
+  hardcfr_sibcall_search_preds (EXIT_BLOCK_PTR_FOR_FN (fun), chk_edges,
+				count_chkcall, chkcall_blocks,
+				count_postchk, postchk_blocks,
+				NULL);
+
+  rt_bb_visited vstd (chk_edges.length () + count_chkcall);
+
+  auto_sbitmap combined_blocks (last_basic_block_for_fn (fun));
+  bitmap_copy (combined_blocks, chkcall_blocks);
+  int i;
+  edge *e;
+  FOR_EACH_VEC_ELT (chk_edges, i, e)
+    if (!bitmap_set_bit (combined_blocks, (*e)->src->index))
+      /* There may be multiple chk_edges with the same src block;
+	 guard againt overlaps with chkcall_blocks only.  */
+      gcc_assert (!bitmap_bit_p (chkcall_blocks, (*e)->src->index));
+
+  /* Visit blocks in index order, because building rtcfg depends on
+     that.  Blocks must be compact, which the cleanup_cfg requirement
+     ensures.  This would also enable FOR_EACH_BB_FN to be used to
+     iterate in index order, but bb_eh_cleanup block splits and
+     insertions changes that.  */
+  gcc_checking_assert (n_basic_blocks_for_fn (fun)
+		       == last_basic_block_for_fn (fun));
+  for (int i = NUM_FIXED_BLOCKS; i < n_basic_blocks_for_fn (fun); i++)
+    {
+      bb = BASIC_BLOCK_FOR_FN (fun, i);
+      gcc_checking_assert (bb->index == i);
+      vstd.visit (bb, bitmap_bit_p (combined_blocks, i),
+		  bitmap_bit_p (postchk_blocks, i));
+    }
+
+  vstd.check (chk_edges, count_chkcall, chkcall_blocks);
+
+  return
+    TODO_update_ssa
+    | TODO_cleanup_cfg
+    | TODO_verify_il;
 }
 
 /* Instantiate a hardcfr pass.  */
diff --git a/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c b/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c
new file mode 100644
index 00000000000..a6992eb9f8e
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -O0 -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we don't insert checking before noreturn calls.  -O0 is tested
+   separately because h is not found to be noreturn without optimization.  */
+
+#include "torture/harden-cfr-noret.c"
+
+/* No out-of-line checks.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* Only one inline check at the end of f and of h2.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-always.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-always.c
new file mode 100644
index 00000000000..779896c60e8
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-always.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even enabling all checks before noreturn calls (leaving
+   returning calls enabled), we get checks before __builtin_return without
+   duplication (__builtin_return is both noreturn and a returning call).  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-never.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-never.c
new file mode 100644
index 00000000000..49ce17f5b93
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-never.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even enabling checks before never noreturn calls (leaving
+   returning calls enabled), we get checks before __builtin_return without
+   duplication (__builtin_return is both noreturn and a returning call).  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noopt.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noopt.c
new file mode 100644
index 00000000000..1512614791f
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noopt.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even disabling checks before both noreturn and returning
+   calls, we still get checks before __builtin_return.  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noret.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noret.c
new file mode 100644
index 00000000000..fd95bb7e3e3
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noret.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even disabling checks before returning calls (leaving noreturn
+   calls enabled), we still get checks before __builtin_return.  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-nothrow.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-nothrow.c
new file mode 100644
index 00000000000..c5c361234c4
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-nothrow.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even enabling checks before nothrow noreturn calls (leaving
+   returning calls enabled), we get checks before __builtin_return without
+   duplication (__builtin_return is both noreturn and a returning call).  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-retcl.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-retcl.c
new file mode 100644
index 00000000000..137dfbb95d6
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-retcl.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even disabling checks before noreturn calls (leaving returning
+   calls enabled), we still get checks before __builtin_return.  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c
index 70acdc95f25..b459ff6b864 100644
--- a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c
@@ -7,5 +7,11 @@ int f(int i) {
   return i;
 }
 
-/* Out-of-line checking, before both builtin_return and return.  */
+int g(int i) {
+  __builtin_return (&i);
+}
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
 /* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c
new file mode 100644
index 00000000000..8bd2d13ac18
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c
@@ -0,0 +1,18 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we don't insert checking before noreturn calls.  -O0 is tested
+   separately because h is not found to be noreturn without optimization, which
+   affects codegen for h2, so h2 is omitted here at -O0.  */
+
+#if !__OPTIMIZE__
+# define OMIT_H2
+#endif
+
+#include "harden-cfr-noret.c"
+
+
+/* No out-of-line checks.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* Only one inline check at the end of f.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c
new file mode 100644
index 00000000000..a804a6cfe59
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fno-exceptions -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that -fno-exceptions makes for implicit nothrow in noreturn
+   handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#include "harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c
new file mode 100644
index 00000000000..f390cfdbc59
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checking before nothrow noreturn calls.  */
+
+#include "harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c
new file mode 100644
index 00000000000..fdd803109a4
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c
@@ -0,0 +1,38 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checking before all noreturn calls.  */
+
+#ifndef ATTR_NOTHROW_OPT /* Overridden in harden-cfr-noret-noexcept.  */
+#define ATTR_NOTHROW_OPT __attribute__ ((__nothrow__))
+#endif
+
+extern void __attribute__ ((__noreturn__)) ATTR_NOTHROW_OPT g (void);
+
+void f(int i) {
+  if (i)
+    /* Out-of-line checks here...  */
+    g ();
+  /* ... and here.  */
+}
+
+void __attribute__ ((__noinline__, __noclone__))
+h(void) {
+  /* Inline check here.  */
+  g ();
+}
+
+#ifndef OMIT_H2 /* from harden-cfr-noret-never.  */
+void h2(void) {
+  /* Inline check either here, whether because of noreturn or tail call...  */
+  h ();
+  /* ... or here, if not optimizing.  */
+}
+#endif
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c
new file mode 100644
index 00000000000..6d11487bbba
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c
@@ -0,0 +1,8 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+#include "harden-cfr-tail.c"
+
+/* Inline checking after the calls, disabling tail calling.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 5 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 0 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c
new file mode 100644
index 00000000000..550b02ca088
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c
@@ -0,0 +1,35 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-exceptions -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checks before returning calls and alternate paths, even
+   at -O0, because of the explicit command-line flag.  */
+
+void g (void);
+void g2 (void);
+void g3 (void);
+
+void f (int i) {
+  if (!i)
+    /* Out-of-line checks here...  */
+    g ();
+  else if (i > 0)
+    /* here...  */
+    g2 ();
+  /* else */
+    /* and in the implicit else here.  */
+}
+
+void f2 (int i) {
+  if (!i)
+    /* Out-of-line check here...  */
+    g ();
+  else if (i > 0)
+    /* here...  */
+    g2 ();
+  else
+    /* and here.  */
+    g3 ();
+}
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 6 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 0 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
index 40d76c5c163..d5467eafa9f 100644
--- a/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
@@ -1,17 +1,52 @@
 /* { dg-do compile } */
-/* { dg-options "-fharden-control-flow-redundancy -fdump-tree-hardcfr -ffat-lto-objects" } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-hardcfr-check-exceptions -fdump-tree-hardcfr -ffat-lto-objects -Wno-return-type" } */
 
-/* We'd like to check that we insert checking so as to not disrupt tail calls,
-   but unfortunately mandatory tail calls are not available in C, and optimizing
-   calls as tail calls only takes place after hardcfr.  */
+/* Check that we insert CFR checking so as to not disrupt tail calls.
+   Mandatory tail calls are not available in C, and optimizing calls as tail
+   calls only takes place after hardcfr, so we insert checking before calls
+   followed by copies and return stmts with the same return value, that might
+   (or might not) end up optimized to tail calls.  */
 
-extern int g(int i);
+extern int g (int i);
 
-int f(int i) {
+int f1(int i) {
+  /* Inline check before the returning call.  */
   return g (i);
 }
 
-/* Inline checking before the tail call.  */
-/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
-/* Inline checking before the tail call.  */
-/* { dg-final { scan-tree-dump-times "\\\[tail\]" 1 "hardcfr" { xfail *-*-* } } } */
+extern void g2 (int i);
+
+void f2(int i) {
+  /* Inline check before the returning call, that ignores the returned value,
+     matching the value-less return.  */
+  g2 (i);
+  return;
+}
+
+void f3(int i) {
+  /* Inline check before the returning call.  */
+  g (i);
+}
+
+void f4(int i) {
+  if (i)
+    /* Out-of-line check before the returning call.  */
+    return g2 (i);
+  /* Out-of-line check before implicit return.  */
+}
+
+int f5(int i) {
+  /* Not regarded as a returning call, returning value other than callee's
+     returned value.  */
+  g (i);
+  /* Inline check after the non-returning call.  */
+  return i;
+}
+
+/* Out-of-line checks in f4, before returning calls and before return.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking in all other functions.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 4 "hardcfr" } } */
+/* Check before tail-call in all but f5, but f4 is out-of-line.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 3 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C b/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C
new file mode 100644
index 00000000000..17ea79f7cfb
--- /dev/null
+++ b/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C
@@ -0,0 +1,11 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects -O0" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions, and also checking before noreturn
+   calls.  h2 and h2b get an extra resx without ehcleanup.  */
+
+#include "torture/harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 16 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C b/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C
new file mode 100644
index 00000000000..f0338ccc361
--- /dev/null
+++ b/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C
@@ -0,0 +1,10 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -foptimize-sibling-calls -fdump-tree-hardcfr -O0" } */
+
+/* -fhardcfr-check-returning-calls gets implicitly disabled because,
+   -at O0, -foptimize-sibling-calls has no effect.  */
+
+#include "torture/harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 12 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C
new file mode 100644
index 00000000000..0d35920c7ee
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  */
+
+#include "harden-cfr-noret-no-nothrow.C"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Bypassing cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C
new file mode 100644
index 00000000000..b7d247ff43c
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C
@@ -0,0 +1,18 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  Expected results for =never and =nothrow are the same,
+   since the functions are not nothrow.  */
+
+#include "harden-cfr-noret-no-nothrow.C"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Associated cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C
new file mode 100644
index 00000000000..62c58cfd406
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C
@@ -0,0 +1,23 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#if ! __OPTIMIZE__
+void __attribute__ ((__noreturn__)) h (void);
+#endif
+
+#include "../../c-c++-common/torture/harden-cfr-noret.c"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Associated cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C
new file mode 100644
index 00000000000..0286f6e6d3f
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C
@@ -0,0 +1,20 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions, and also checking before noreturn
+   calls.  */
+
+#if ! __OPTIMIZE__
+/* Without optimization, functions with cleanups end up with an extra
+   resx that is not optimized out, so arrange to optimize them.  */
+void __attribute__ ((__optimize__ (1))) h2(void);
+void __attribute__ ((__optimize__ (1))) h2b(void);
+#endif
+
+#include "harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 14 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
+/* h, h2, h2b, and h4.  */
+/* { dg-final { scan-tree-dump-times "Bypassing" 4 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C
new file mode 100644
index 00000000000..885b0b236af
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C
@@ -0,0 +1,11 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions  -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we do not insert cleanups for checking around the bodies
+   of maybe-throwing functions.  h4 doesn't get any checks, because we
+   don't have noreturn checking enabled.  */
+
+#include "harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 6 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C
new file mode 100644
index 00000000000..32def637255
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C
@@ -0,0 +1,31 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -foptimize-sibling-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions.  These results depend on checking before
+   returning calls, which is only enabled when sibcall optimizations
+   are enabled, so change the optimization mode to -O1 for f and f2,
+   so that -foptimize-sibling-calls can take effect and enable
+   -fhardcfr-check-returning-calls, so that we get the same results.
+   There is a separate test for -O0.  */
+
+#if ! __OPTIMIZE__
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) f(int i);
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) f2(int i);
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) h3(void);
+#endif
+
+#include "harden-cfr-throw.C"
+
+/* f gets out-of-line checks before the unwrapped tail call and in the
+   else edge.  */
+/* f2 gets out-of-line checks before both unwrapped tail calls.  */
+/* h gets out-of-line checks before the implicit return and in the
+   cleanup block.  */
+/* h2 and h2b get out-of-line checks before the cleanup returning
+   call, and in the cleanup block.  */
+/* h3 gets an inline check before the __cxa_end_catch returning call.  */
+/* h4 gets an inline check in the cleanup block.  */
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 10 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C
new file mode 100644
index 00000000000..992fbdad381
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C
@@ -0,0 +1,65 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions.  */
+
+extern void g (void);
+extern void g2 (void);
+
+void f(int i) {
+  if (i)
+    g ();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void f2(int i) {
+  if (i)
+    g ();
+  else
+    g2 ();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h(void) {
+  try {
+    g ();
+  } catch (...) {
+    throw;
+  }
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+struct needs_cleanup {
+  ~needs_cleanup();
+};
+
+void h2(void) {
+  needs_cleanup y; /* No check in the cleanup handler.  */
+  g();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+extern void __attribute__ ((__nothrow__)) another_cleanup (void*);
+
+void h2b(void) {
+  int x __attribute__ ((cleanup (another_cleanup)));
+  g();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h3(void) {
+  try {
+    throw 1;
+  } catch (...) {
+  }
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h4(void) {
+  throw 1;
+  /* Inline check in the cleanup around the __cxa_throw noreturn call.  */
+}
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 12 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c b/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c
new file mode 100644
index 00000000000..8e4ee1fab08
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c
@@ -0,0 +1,15 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C makes for implicit nothrow in noreturn handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#include "../../c-c++-common/torture/harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c b/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c
new file mode 100644
index 00000000000..634d98f1ffc
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c
@@ -0,0 +1,40 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-hardcfr-check-exceptions -fdump-tree-hardcfr -ffat-lto-objects -Wno-return-type" } */
+
+/* In C only, check some additional cases (comparing with
+   c-c++-common/torture/harden-cfr-tail.c) of falling off the end of non-void
+   function.  C++ would issue an unreachable call in these cases.  */
+
+extern int g (int i);
+
+int f1(int i) {
+  /* Inline check before the returning call, that doesn't return anything.  */
+  g (i);
+  /* Implicit return without value, despite the return type; this combination
+     enables tail-calling of g, and is recognized as a returning call.  */
+}
+
+extern void g2 (int i);
+
+int f2(int i) {
+  /* Inline check before the returning call, that disregards its return
+     value.  */
+  g2 (i);
+  /* Implicit return without value, despite the return type; this combination
+     enables tail-calling of g2, and is recognized as a returning call.  */
+}
+
+int f3(int i) {
+  if (i)
+    /* Out-of-line check before the returning call.  */
+    return g (i);
+  /* Out-of-line check before implicit return.  */
+}
+
+/* Out-of-line checks in f3, before returning calls and before return.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking in all other functions.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
+/* Check before tail-call in all functions, but f3 is out-of-line.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/gnat.dg/hardcfr.adb b/gcc/testsuite/gnat.dg/hardcfr.adb
index b578a913d75..abe1605c029 100644
--- a/gcc/testsuite/gnat.dg/hardcfr.adb
+++ b/gcc/testsuite/gnat.dg/hardcfr.adb
@@ -1,5 +1,5 @@
 --  { dg-do run }
---  { dg-options "-fharden-control-flow-redundancy -fdump-tree-hardcfr --param=hardcfr-max-blocks=22 --param=hardcfr-max-inline-blocks=12 -O0" }
+--  { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions -fdump-tree-hardcfr --param=hardcfr-max-blocks=22 --param=hardcfr-max-inline-blocks=12 -O0" }
 
 procedure HardCFR is
    function F (I, J : Integer) return Integer is
diff --git a/libgcc/hardcfr.c b/libgcc/hardcfr.c
index 8ef29428111..55f6b995f2f 100644
--- a/libgcc/hardcfr.c
+++ b/libgcc/hardcfr.c
@@ -60,14 +60,25 @@ extern void __hardcfr_check (size_t blocks,
 			     vword const *visited,
 			     vword const *cfg);
 
+/* Compute the MASK for the bit representing BLOCK in WORDIDX's vword in a
+   visited blocks bit array.  */
+static inline void
+block2mask (size_t const block, vword *const mask, size_t *const wordidx)
+{
+  size_t wbits = __CHAR_BIT__ * sizeof (vword);
+  *wordidx = block / wbits;
+  *mask = (vword)1 << (block % wbits);
+}
 
 /* Check whether the bit corresponding to BLOCK is set in VISITED.  */
 static inline bool
 visited_p (size_t const block, vword const *const visited)
 {
-  size_t wbits = __CHAR_BIT__ * sizeof (vword);
-  vword w = visited[block / wbits];
-  return (w & ((vword)1 << (block % wbits))) != 0;
+  vword mask;
+  size_t wordidx;
+  block2mask (block, &mask, &wordidx);
+  vword w = visited[wordidx];
+  return (w & mask) != 0;
 }
 
 /* Read and consume a mask from **CFG_IT.  (Consume meaning advancing the
@@ -118,19 +129,19 @@ consume_seq (vword const **const cfg_it)
    we reach the terminator without finding any.  Consume the entire sequence
    otherwise, so that *CFG_IT points just past the terminator, which may be the
    beginning of the next sequence.  */
-static inline void
+static inline bool
 check_seq (vword const *const visited, vword const **const cfg_it)
 {
   vword mask;
   size_t wordidx;
 
   /* If the block was visited, check that at least one of the
-     preds was also visited.  */
+     preds/succs was also visited.  */
   do
     /* If we get to the end of the sequence without finding any
        match, something is amiss.  */
     if (!next_pair (cfg_it, &mask, &wordidx))
-      __builtin_trap ();
+      return false;
   /* Keep searching until we find a match, at which point the
      condition is satisfied.  */
   while (!test_mask (visited, mask, wordidx));
@@ -139,6 +150,94 @@ check_seq (vword const *const visited, vword const **const cfg_it)
      skipped the block, so as to position the iterator at the beginning of the
      next .  */
   consume_seq (cfg_it);
+
+  return true;
+}
+
+/* Print out the CFG with BLOCKS blocks, presumed to be associated with CALLER.
+   This is expected to be optimized out entirely, unless the verbose part of
+   __hardcfr_check_fail is enabled.  */
+static inline void
+__hardcfr_debug_cfg (size_t const blocks,
+		     void const *const caller,
+		     vword const *const cfg)
+{
+  __builtin_printf ("CFG at %p, for %p", cfg, caller);
+  vword const *cfg_it = cfg;
+  for (size_t i = 0; i < blocks; i++)
+    {
+      vword mask; size_t wordidx;
+      block2mask (i, &mask, &wordidx);
+      __builtin_printf ("\nblock %lu (%lu/0x%lx)\npreds: ",
+			(unsigned long)i,
+			(unsigned long)wordidx, (unsigned long)mask);
+      while (next_pair (&cfg_it, &mask, &wordidx))
+	__builtin_printf (" (%lu/0x%lx)",
+			  (unsigned long)wordidx, (unsigned long)mask);
+      __builtin_printf ("\nsuccs: ");
+      while (next_pair (&cfg_it, &mask, &wordidx))
+	__builtin_printf (" (%lu/0x%lx)",
+			  (unsigned long)wordidx, (unsigned long)mask);
+    }
+  __builtin_printf ("\n");
+}
+
+#ifndef ATTRIBUTE_UNUSED
+# define ATTRIBUTE_UNUSED __attribute__ ((__unused__))
+#endif
+
+/* This is called when an out-of-line hardcfr check fails.  All the arguments
+   are ignored, and it just traps, unless HARDCFR_VERBOSE_FAIL is enabled.  IF
+   it is, it prints the PART of the CFG, expected to have BLOCKS blocks, that
+   failed at CALLER's BLOCK, and the VISITED bitmap.  When the verbose mode is
+   enabled, it also forces __hardcfr_debug_cfg (above) to be compiled into an
+   out-of-line function, that could be called from a debugger.
+   */
+static inline void
+__hardcfr_check_fail (size_t const blocks ATTRIBUTE_UNUSED,
+		      vword const *const visited,
+		      vword const *const cfg ATTRIBUTE_UNUSED,
+		      size_t const block ATTRIBUTE_UNUSED,
+		      int const part ATTRIBUTE_UNUSED,
+		      void const *const caller ATTRIBUTE_UNUSED)
+{
+#if HARDCFR_VERBOSE_FAIL
+  static const char *parts[] = { "preds", "succs" };
+
+  vword mask; size_t wordidx;
+  block2mask (block, &mask, &wordidx);
+  __builtin_printf ("hardcfr fail at %p block %lu (%lu/0x%lx), expected %s:",
+		    caller, (unsigned long)block,
+		    (unsigned long)wordidx, (unsigned long)mask,
+		    parts[part]);
+
+  /* Skip data for previous blocks.  */
+  vword const *cfg_it = cfg;
+  for (size_t i = block; i--; )
+    {
+      consume_seq (&cfg_it);
+      consume_seq (&cfg_it);
+    }
+  for (size_t i = part; i--; )
+    consume_seq (&cfg_it);
+
+  while (next_pair (&cfg_it, &mask, &wordidx))
+    __builtin_printf (" (%lu/0x%lx)",
+		      (unsigned long)wordidx, (unsigned long)mask);
+
+  __builtin_printf ("\nvisited:");
+  block2mask (blocks, &mask, &wordidx);
+  for (size_t i = 0; i <= wordidx; i++)
+    __builtin_printf (" (%lu/0x%lx)",
+		      (unsigned long)i, (unsigned long)visited[i]);
+  __builtin_printf ("\n");
+
+  /* Reference __hardcfr_debug_cfg so that it's output out-of-line, so that it
+     can be called from a debugger.  */
+  if (!caller || caller == __hardcfr_debug_cfg)
+    return;
+#endif
+  __builtin_trap ();
 }
 
 /* Check that, for each of the BLOCKS basic blocks, if its bit is set in
@@ -168,9 +267,13 @@ __hardcfr_check (size_t const blocks,
       else
 	{
 	  /* Check predecessors.  */
-	  check_seq (visited, &cfg_it);
+	  if (!check_seq (visited, &cfg_it))
+	    __hardcfr_check_fail (blocks, visited, cfg, i, 0,
+				  __builtin_return_address (0));
 	  /* Check successors.  */
-	  check_seq (visited, &cfg_it);
+	  if (!check_seq (visited, &cfg_it))
+	    __hardcfr_check_fail (blocks, visited, cfg, i, 1,
+				  __builtin_return_address (0));
 	}
     }
 }

^ permalink raw reply	[flat|nested] 13+ messages in thread
* [gcc(refs/users/aoliva/heads/testme)] hardcfr: add optional checkpoints
@ 2023-06-08  4:47 Alexandre Oliva
  0 siblings, 0 replies; 13+ messages in thread
From: Alexandre Oliva @ 2023-06-08  4:47 UTC (permalink / raw)
  To: gcc-cvs

https://gcc.gnu.org/g:6c914d07ef926b53d7c6ffeca352537e43b47dfc

commit 6c914d07ef926b53d7c6ffeca352537e43b47dfc
Author: Alexandre Oliva <oliva@adacore.com>
Date:   Wed Oct 19 20:36:18 2022 -0300

    hardcfr: add optional checkpoints
    
    Previously, control flow redundancy only checked the visited bitmap
    against the control flow graph at return points and before mandatory
    tail calls, missing various other possibilities of exiting a
    subprogram, such as by raising or propagating exceptions, and calling
    noreturn functions.  The checks inserted before returns also prevented
    potential tail-call optimizations.
    
    This incremental change introduces options to control checking at each
    of these previously-missed checkpoints.  Unless disabled, a cleanup is
    introduced to check when an exceptions escapes a subprogram.  To avoid
    disrupting sibcall optimizations, when they are enabled, checks are
    introduced before calls whose results are immediately returned,
    whether or not they are ultimately optimized.  If enabled, checks are
    introduced before noreturn calls and exception raises, or only before
    nothrow noreturn calls.
    
    
    for  gcc/ChangeLog
    
            * common.opt (-fhardcfr-check-returning-calls): New.
            (-fhardcfr-check-exceptions): New.
            (-fhardcfr-check-noreturn-calls=*): New.
            (Enum hardcfr_check_noreturn_calls): New.
            * doc/invoke.texi: Document them.
            * flag-types.h (enum hardcfr_noret): New.
            * gimple-harden-control-flow.cc: Include more headers.
            (pass_data_harden_control_flow_redundancy): Move
            properties_finish to pass return.
            (pass_harden_control_flow_redundancy::gate): Move
            no-edge-to-exit checking to execute, after exception
            transformations.  Use symbolic NUM_FIXED_BLOCKS.  Test for CFG
            before testing block count.
            (check_returning_calls_p): New.
            (hardcfr_scan_block): New.
            (returnign_call_p): New.
            (chk_edges_t): New.
            (hardcfr_sibcall_search_block): New.
            (hardcfr_sibcall_search_preds): New.
            (rt_bb_visited::num2idx): Use symbolic NUM_FIXED_BLOCKS.
            (rt_bb_visited::rt_bb_visited): Likewise.  Take checkpoints
            count, test it to select out-of-line checks.
            (rt_bb_visited::insert_exit_check): Removed.
            (rt_bb_visited::insert_exit_check_in_blocks): New.
            (rt_bb_visited::insert_exit_check_on_edge): New.
            (rt_bb_visited::check): Take checkpoint edges and blocks.
            Move initialization insertion to the end.  Insert checks on
            edges, and before calls in select blocks.  Clobber the visited
            array to the check sequence.  Output actions to dump_file.
            (rt_bb_visited::visit): Take checkpoint and postcheck.  Assume
            an edge to exit if checkpoint, and skip testing if postcheck.
            (always_throwing_noreturn_call_p): New, dummy.
            (pass_harden_control_flow_redundancy::execute): Introduce
            cleanup block if requested and needed.  Scan blocks for
            noreturn and returning calls if checkpoints before them were
            requested.  Log actions to dump_file.  Visit blocks in index
            order.
    
    for  gcc/testsuite/ChangeLog
    
            * c-c++-common/harden-cfr-noret-never-O0.c: New.
            * c-c++-common/torture/harden-cfr-noret-never.c: New.
            * c-c++-common/torture/harden-cfr-noret-noexcept.c: New.
            * c-c++-common/torture/harden-cfr-noret-nothrow.c: New.
            * c-c++-common/torture/harden-cfr-noret.c: New.
            * c-c++-common/torture/harden-cfr-notail.c: New.
            * c-c++-common/torture/harden-cfr-returning.c: New.
            * c-c++-common/torture/harden-cfr-tail.c: Extend.
            * c-c++-common/torture/harden-cfr-bret-always.c: New.
            * c-c++-common/torture/harden-cfr-bret-never.c: New.
            * c-c++-common/torture/harden-cfr-bret-noopt.c: New.
            * c-c++-common/torture/harden-cfr-bret-noret.c: New.
            * c-c++-common/torture/harden-cfr-bret-nothrow.c: New.
            * c-c++-common/torture/harden-cfr-bret-retcl.c: New.
            * c-c++-common/torture/harden-cfr-bret.c (g): New.
            * g++.dg/harden-cfr-throw-always-O0.C: New.
            * g++.dg/harden-cfr-throw-returning-O0.C: New.
            * g++.dg/torture/harden-cfr-noret-always-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-noret-never-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-noret-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-throw-always.C: New.
            * g++.dg/torture/harden-cfr-throw-nocleanup.C: New.
            * g++.dg/torture/harden-cfr-throw-returning.C: New.
            * g++.dg/torture/harden-cfr-throw.C: New.
            * gcc.dg/torture/harden-cfr-noret-no-nothrow.c: New.
            * gcc.dg/torture/harden-cfr-tail-ub.c: New.
            * gnat.dg/hardcfr.adb: Disable checking at exceptions.
    
    for  libgcc/ChangeLog
    
            * hardcfr.c (block2mask): Split out of...
            (visited_p): ... this.
            (check_seq): Move trapping out, return result instead.
            (__hardcfr_debug_cfg): New.
            (__hardcfr_check_fail): New, with optional verbose error
            printing before trapping.
            (__hardcfr_check): Call __hardcfr_check_fail when check_seq
            fails.

Diff:
---
 gcc/common.opt                                     |  33 +
 gcc/doc/invoke.texi                                |  74 +-
 gcc/flag-types.h                                   |  10 +
 gcc/gimple-harden-control-flow.cc                  | 926 ++++++++++++++++++---
 .../c-c++-common/harden-cfr-noret-never-O0.c       |  12 +
 .../c-c++-common/torture/harden-cfr-bret-always.c  |  13 +
 .../c-c++-common/torture/harden-cfr-bret-never.c   |  13 +
 .../c-c++-common/torture/harden-cfr-bret-noopt.c   |  12 +
 .../c-c++-common/torture/harden-cfr-bret-noret.c   |  12 +
 .../c-c++-common/torture/harden-cfr-bret-nothrow.c |  13 +
 .../c-c++-common/torture/harden-cfr-bret-retcl.c   |  12 +
 .../c-c++-common/torture/harden-cfr-bret.c         |   8 +-
 .../c-c++-common/torture/harden-cfr-noret-never.c  |  18 +
 .../torture/harden-cfr-noret-noexcept.c            |  16 +
 .../torture/harden-cfr-noret-nothrow.c             |  13 +
 .../c-c++-common/torture/harden-cfr-noret.c        |  38 +
 .../c-c++-common/torture/harden-cfr-notail.c       |   8 +
 .../c-c++-common/torture/harden-cfr-returning.c    |  35 +
 .../c-c++-common/torture/harden-cfr-tail.c         |  55 +-
 gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C  |  11 +
 .../g++.dg/harden-cfr-throw-returning-O0.C         |  10 +
 .../torture/harden-cfr-noret-always-no-nothrow.C   |  16 +
 .../torture/harden-cfr-noret-never-no-nothrow.C    |  18 +
 .../g++.dg/torture/harden-cfr-noret-no-nothrow.C   |  23 +
 .../g++.dg/torture/harden-cfr-throw-always.C       |  20 +
 .../g++.dg/torture/harden-cfr-throw-nocleanup.C    |  11 +
 .../g++.dg/torture/harden-cfr-throw-returning.C    |  31 +
 gcc/testsuite/g++.dg/torture/harden-cfr-throw.C    |  65 ++
 .../gcc.dg/torture/harden-cfr-noret-no-nothrow.c   |  15 +
 gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c  |  40 +
 gcc/testsuite/gnat.dg/hardcfr.adb                  |   2 +-
 libgcc/hardcfr.c                                   | 119 ++-
 32 files changed, 1568 insertions(+), 134 deletions(-)

diff --git a/gcc/common.opt b/gcc/common.opt
index ff89b2d1140..c2855cc19a9 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -1805,6 +1805,39 @@ fharden-control-flow-redundancy
 Common Var(flag_harden_control_flow_redundancy) Optimization
 Harden control flow by recording and checking execution paths.
 
+fhardcfr-check-returning-calls
+Common Var(flag_harden_control_flow_redundancy_check_returning_calls) Init(-1) Optimization
+Check CFR execution paths also before calls followed by returns of their results.
+
+fhardcfr-check-exceptions
+Common Var(flag_harden_control_flow_redundancy_check_exceptions) Init(-1) Optimization
+Check CFR execution paths also when exiting a function through an exception.
+
+fhardcfr-check-noreturn-calls=
+Common Joined RejectNegative Enum(hardcfr_check_noreturn_calls) Var(flag_harden_control_flow_redundancy_check_noreturn) Init(HCFRNR_UNSPECIFIED) Optimization
+-fhardcfr-check-noreturn-calls=[always|nothrow|never]	Check CFR execution paths also before calling noreturn functions.
+
+Enum
+Name(hardcfr_check_noreturn_calls) Type(enum hardcfr_noret) UnknownError(unknown hardcfr noreturn checking level %qs)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(never) Value(HCFRNR_NEVER)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(nothrow) Value(HCFRNR_NOTHROW)
+
+; ??? There could be yet another option here, that checked before
+; noreturn calls, except for those known to always throw, if we had
+; means to distinguish noreturn functions known to always throw, such
+; as those used to (re)raise exceptions, from those that merely might
+; throw.  "not always" stands for "not always-throwing", but it also
+; contrasts with "always" below.
+; EnumValue
+; Enum(hardcfr_check_noreturn_calls) String(not-always) Value(HCFRNR_NOT_ALWAYS)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(always) Value(HCFRNR_ALWAYS)
+
 ; Nonzero means ignore `#ident' directives.  0 means handle them.
 ; Generate position-independent code for executables if possible
 ; On SVR4 targets, it also controls whether or not to emit a
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index 87a33b037c1..7742d21f142 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -635,7 +635,9 @@ Objective-C and Objective-C++ Dialects}.
 -fsanitize-undefined-trap-on-error  -fbounds-check
 -fcf-protection=@r{[}full@r{|}branch@r{|}return@r{|}none@r{|}check@r{]}
 -fharden-compares -fharden-conditional-branches
--fharden-control-flow-redundancy
+-fharden-control-flow-redundancy  -fhardcfr-check-exceptions
+-fhardcfr-check-returning-calls
+-fhardcfr-check-noreturn-calls=@r{[}always@r{|}nothrow@r{|}never@r{]}
 -fstack-protector  -fstack-protector-all  -fstack-protector-strong
 -fstack-protector-explicit  -fstack-check
 -fstack-limit-register=@var{reg}  -fstack-limit-symbol=@var{sym}
@@ -17243,11 +17245,75 @@ conditionals.
 @opindex fharden-control-flow-redundancy
 @item -fharden-control-flow-redundancy
 Emit extra code to set booleans when entering basic blocks, and to
-verify, at function exits, that they amount to an execution path that is
-consistent with the control flow graph, trapping otherwise.  Tuning
-options @option{--param hardcfr-max-blocks} and @option{--param
+verify and trap, at function exits, when the booleans do not form an
+execution path that is compatible with the control flow graph.
+
+Verification takes place before returns, before mandatory tail calls
+(see below) and, optionally, before escaping exceptions with
+@option{-fhardcfr-check-exceptions}, before returning calls with
+@option{-fhardcfr-check-returning-calls}, and before noreturn calls with
+@option{-fhardcfr-check-noreturn-calls}).  Tuning options
+@option{--param hardcfr-max-blocks} and @option{--param
 hardcfr-max-inline-blocks} are available.
 
+Tail call optimization takes place too late to affect control flow
+redundancy, but calls annotated as mandatory tail calls by language
+front-ends, and any calls marked early enough as potential tail calls
+would also have verification issued before the call, but these
+possibilities are merely theoretical, as these conditions can only be
+met when using custom compiler plugins.
+
+@opindex fhardcfr-check-exceptions
+@opindex fno-hardcfr-check-exceptions
+@item -fhardcfr-check-exceptions
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph at exception
+escape points, as if the function body was wrapped with a cleanup
+handler that performed the check and reraised.  This option is enabled
+by default; use @option{-fno-hardcfr-check-exceptions} to disable it.
+
+@opindex fhardcfr-check-returning-calls
+@opindex fno-hardcfr-check-returning-calls
+@item -fhardcfr-check-returning-calls
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph before any
+function call immediately followed by a return of its result, if any, so
+as to not prevent tail-call optimization, whether or not it is
+ultimately optimized to a tail call.
+
+This option is enabled by default whenever sibling call optimizations
+are enabled (see @option{-foptimize-sibling-calls}), but it can be
+enabled (or disabled, using its negated form) explicitly, regardless of
+the optimizations.
+
+@opindex fhardcfr-check-noreturn-calls
+@item -fhardcfr-check-noreturn-calls=@r{[}always@r{|}nothrow@r{|}never@r{]}
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph before
+@code{noreturn} calls, either all of them (@option{always}), those that
+may not return control to the caller through an exception either
+(@option{nothrow}), or none of them (@option{never}, the default).
+
+Checking before a @code{noreturn} function that may return control to
+the caller through an exception may cause checking to be performed more
+than once, if the exception is caught in the caller, whether by a
+handler or a cleanup.  When @option{-fhardcfr-check-exceptions} is also
+enabled, the compiler will avoid associating a @code{noreturn} call with
+the implicitly-added cleanup handler, since it would be redundant with
+the check performed before the call, but other handlers or cleanups in
+the function, if activated, will modify the recorded execution path and
+check it again when another checkpoint is hit.  The checkpoint may even
+be another @code{noreturn} call, so checking may end up performed
+multiple times.
+
+Various optimizers may cause calls to be marked as @code{noreturn}
+and/or @code{nothrow}, even in the absence of the corresponding
+attributes, which may affect the placement of checks before calls, as
+well as the addition of implicit cleanup handlers for them.  This
+unpredictability, and the fact that raising and reraising exceptions
+frequently amounts to implicitly calling @code{noreturn} functions, have
+made @option{never} the default setting for this option.
+
 @opindex fstack-protector
 @item -fstack-protector
 Emit extra code to check for buffer overflows, such as stack smashing
diff --git a/gcc/flag-types.h b/gcc/flag-types.h
index 36305de589e..d2084dd1931 100644
--- a/gcc/flag-types.h
+++ b/gcc/flag-types.h
@@ -157,6 +157,16 @@ enum stack_reuse_level
   SR_ALL
 };
 
+/* Control Flow Redundancy hardening options for noreturn calls.  */
+enum hardcfr_noret
+{
+  HCFRNR_NEVER,
+  HCFRNR_NOTHROW,
+  HCFRNR_NOT_ALWAYS, /* Reserved for future use.  */
+  HCFRNR_ALWAYS,
+  HCFRNR_UNSPECIFIED = -1
+};
+
 /* The live patching level.  */
 enum live_patching_level
 {
diff --git a/gcc/gimple-harden-control-flow.cc b/gcc/gimple-harden-control-flow.cc
index 6b08846dbb1..5c66762d448 100644
--- a/gcc/gimple-harden-control-flow.cc
+++ b/gcc/gimple-harden-control-flow.cc
@@ -29,7 +29,12 @@ along with GCC; see the file COPYING3.  If not see
 #include "tree-pass.h"
 #include "ssa.h"
 #include "gimple-iterator.h"
+#include "gimple-pretty-print.h"
 #include "tree-cfg.h"
+#include "tree-cfgcleanup.h"
+#include "tree-eh.h"
+#include "except.h"
+#include "sbitmap.h"
 #include "basic-block.h"
 #include "cfghooks.h"
 #include "cfgloop.h"
@@ -60,9 +65,7 @@ const pass_data pass_data_harden_control_flow_redundancy = {
   0,	    // properties_provided
   0,	    // properties_destroyed
   TODO_cleanup_cfg, // properties_start
-  TODO_update_ssa
-  | TODO_cleanup_cfg
-  | TODO_verify_il, // properties_finish
+  0,        // properties_finish
 };
 
 class pass_harden_control_flow_redundancy : public gimple_opt_pass
@@ -79,16 +82,6 @@ public:
     if (!flag_harden_control_flow_redundancy)
       return false;
 
-    /* We don't verify when an exception escapes, propagated or raised
-       by the function itself, so we're only concerned with edges to
-       the exit block.  If there aren't any, the function doesn't
-       return normally, so there won't be any checking point, so
-       there's no point in running the pass.  Should we add
-       verification at exception escapes, we should at least look at
-       !flag_exceptions here.  */
-    if (EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (fun)->preds) == 0)
-      return false;
-
     /* Functions that return more than once, like setjmp and vfork
        (that also gets this flag set), will start recording a path
        after the first return, and then may take another path when
@@ -117,8 +110,9 @@ public:
 	return false;
       }
 
-    if (param_hardcfr_max_blocks > 0
-	&& n_basic_blocks_for_fn (fun) - 2 > param_hardcfr_max_blocks)
+    if (fun->cfg && param_hardcfr_max_blocks > 0
+	&& (n_basic_blocks_for_fn (fun) - NUM_FIXED_BLOCKS
+	    > param_hardcfr_max_blocks))
       {
 	warning_at (DECL_SOURCE_LOCATION (fun->decl), 0,
 		    "%qD has more than %u blocks, the requested"
@@ -134,6 +128,293 @@ public:
 
 }
 
+/* Return TRUE iff CFR checks should be inserted before returning
+   calls.  */
+
+static bool
+check_returning_calls_p ()
+{
+  return
+    flag_harden_control_flow_redundancy_check_returning_calls > 0
+    || (flag_harden_control_flow_redundancy_check_returning_calls < 0
+	/* Gates pass_tail_calls.  */
+	&& flag_optimize_sibling_calls
+	/* Gates pass_all_optimizations.  */
+	&& optimize >= 1 && !optimize_debug);
+}
+
+/* Scan BB from the end, updating *RETPTR if given as return stmts and
+   copies are found.  Return a call or a stmt that cannot appear after
+   a tail call, or NULL if the top of the block is reached without
+   finding any.  */
+
+static gimple *
+hardcfr_scan_block (basic_block bb, tree **retptr)
+{
+  gimple_stmt_iterator gsi;
+  for (gsi = gsi_last_bb (bb); !gsi_end_p (gsi); gsi_prev (&gsi))
+    {
+      gimple *stmt = gsi_stmt (gsi);
+
+      /* Ignore labels, returns, nops, clobbers and debug stmts.  */
+      if (gimple_code (stmt) == GIMPLE_LABEL
+	  || gimple_code (stmt) == GIMPLE_NOP
+	  || gimple_code (stmt) == GIMPLE_PREDICT
+	  || gimple_clobber_p (stmt)
+	  || is_gimple_debug (stmt))
+	continue;
+
+      if (gimple_code (stmt) == GIMPLE_RETURN)
+	{
+	  greturn *gret = as_a <greturn *> (stmt);
+	  if (retptr)
+	    {
+	      gcc_checking_assert (!*retptr);
+	      *retptr = gimple_return_retval_ptr (gret);
+	    }
+	  continue;
+	}
+
+      /* Check for a call.  */
+      if (is_gimple_call (stmt))
+	return stmt;
+
+      /* Allow simple copies to the return value, updating the return
+	 value to be found in earlier assignments.  */
+      if (retptr && *retptr && gimple_assign_single_p (stmt)
+	  && **retptr == gimple_assign_lhs (stmt))
+	{
+	  *retptr = gimple_assign_rhs1_ptr (stmt);
+	  continue;
+	}
+
+      return stmt;
+    }
+
+  /* Any other kind of stmt will prevent a tail call.  */
+  return NULL;
+}
+
+/* Return TRUE iff CALL is to be preceded by a CFR checkpoint, i.e.,
+   if it's a returning call (one whose result is ultimately returned
+   without intervening non-copy statements) and we're checking
+   returning calls, a __builtin_return call (noreturn with a path to
+   the exit block), a must-tail call, or a tail call.  */
+
+static bool
+returning_call_p (gcall *call)
+{
+  if (!(gimple_call_noreturn_p (call)
+	|| gimple_call_must_tail_p (call)
+	|| gimple_call_tail_p (call)
+	|| check_returning_calls_p ()))
+    return false;
+
+  /* Quickly check that there's a path to exit compatible with a
+     returning call.  Detect infinite loops through the counter.  */
+  basic_block bb = gimple_bb (call);
+  auto_vec<basic_block, 10> path;
+  for (int i = n_basic_blocks_for_fn (cfun);
+       bb != EXIT_BLOCK_PTR_FOR_FN (cfun) && i--;
+       bb = single_succ (bb))
+    if (!single_succ_p (bb)
+	|| (single_succ_edge (bb)->flags & EDGE_EH) != 0)
+      return false;
+    else
+      path.safe_push (bb);
+
+  /* Check the stmts in the blocks and trace the return value.  */
+  tree *retptr = NULL;
+  for (;;)
+    {
+      gcc_checking_assert (!path.is_empty ());
+      basic_block bb = path.pop ();
+      gimple *stop = hardcfr_scan_block (bb, &retptr);
+      if (stop)
+	{
+	  if (stop != call)
+	    return false;
+	  gcc_checking_assert (path.is_empty ());
+	  break;
+	}
+
+      gphi *retphi = NULL;
+      if (retptr && *retptr && TREE_CODE (*retptr) == SSA_NAME
+	  && !SSA_NAME_IS_DEFAULT_DEF (*retptr)
+	  && SSA_NAME_DEF_STMT (*retptr)
+	  && is_a <gphi *> (SSA_NAME_DEF_STMT (*retptr))
+	  && gimple_bb (SSA_NAME_DEF_STMT (*retptr)) == bb)
+	{
+	  retphi = as_a <gphi *> (SSA_NAME_DEF_STMT (*retptr));
+	  gcc_checking_assert (gimple_phi_result (retphi) == *retptr);
+	}
+      else
+	continue;
+
+      gcc_checking_assert (!path.is_empty ());
+      edge e = single_succ_edge (path.last ());
+      int i = EDGE_COUNT (bb->preds);
+      while (i--)
+	if (EDGE_PRED (bb, i) == e)
+	  break;
+      gcc_checking_assert (i >= 0);
+      retptr = gimple_phi_arg_def_ptr (retphi, i);
+    }
+
+  return (gimple_call_noreturn_p (call)
+	  || gimple_call_must_tail_p (call)
+	  || gimple_call_tail_p (call)
+	  || (gimple_call_lhs (call) == (retptr ? *retptr : NULL)
+	      && check_returning_calls_p ()));
+}
+
+typedef auto_vec<edge, 10> chk_edges_t;
+
+/* Declare for mutual recursion.  */
+static bool hardcfr_sibcall_search_preds (basic_block bb,
+					  chk_edges_t &chk_edges,
+					  int &count_chkcall,
+					  auto_sbitmap &chkcall_blocks,
+					  int &count_postchk,
+					  auto_sbitmap &postchk_blocks,
+					  tree *retptr);
+
+/* Search backwards from the end of BB for a mandatory or potential
+   sibcall.  Schedule the block to be handled sort-of like noreturn if
+   so.  Recurse to preds, with updated RETPTR, if the block only
+   contains stmts that may follow such a call, scheduling checking at
+   edges and marking blocks as post-check as needed.  Return true iff,
+   at the end of the block, a check will have already been
+   performed.  */
+
+static bool
+hardcfr_sibcall_search_block (basic_block bb,
+			      chk_edges_t &chk_edges,
+			      int &count_chkcall,
+			      auto_sbitmap &chkcall_blocks,
+			      int &count_postchk,
+			      auto_sbitmap &postchk_blocks,
+			      tree *retptr)
+{
+  /* Conditionals and internal exceptions rule out tail calls.  */
+  if (!single_succ_p (bb)
+      || (single_succ_edge (bb)->flags & EDGE_EH) != 0)
+    return false;
+
+  gimple *stmt = hardcfr_scan_block (bb, &retptr);
+  if (!stmt)
+    return hardcfr_sibcall_search_preds (bb, chk_edges,
+					 count_chkcall, chkcall_blocks,
+					 count_postchk, postchk_blocks,
+					 retptr);
+
+  if (!is_a <gcall *> (stmt))
+    return false;
+
+  /* Avoid disrupting mandatory or early-marked tail calls,
+     inserting the check before them.  This works for
+     must-tail calls, but tail calling as an optimization is
+     detected too late for us.
+
+     Also check for noreturn calls here.  Noreturn calls won't
+     normally have edges to exit, so they won't be found here,
+     but __builtin_return does, and we must check before
+     it, so handle it like a tail call.  */
+  gcall *call = as_a <gcall *> (stmt);
+  if (!(gimple_call_noreturn_p (call)
+	|| gimple_call_must_tail_p (call)
+	|| gimple_call_tail_p (call)
+	|| (gimple_call_lhs (call) == (retptr ? *retptr : NULL)
+	    && check_returning_calls_p ())))
+    return false;
+
+  gcc_checking_assert (returning_call_p (call));
+
+  /* We found a call that is to be preceded by checking.  */
+  if (bitmap_set_bit (chkcall_blocks, bb->index))
+    ++count_chkcall;
+  else
+    gcc_unreachable ();
+  return true;
+}
+
+
+/* Search preds of BB for a mandatory or potential sibcall or
+   returning call, and arrange for the blocks containing them to have
+   a check inserted before the call, like noreturn calls.  If any
+   preds are found to perform checking, schedule checks at the edges
+   of those that don't, and mark BB as postcheck..  */
+
+static bool
+hardcfr_sibcall_search_preds (basic_block bb,
+			      chk_edges_t &chk_edges,
+			      int &count_chkcall,
+			      auto_sbitmap &chkcall_blocks,
+			      int &count_postchk,
+			      auto_sbitmap &postchk_blocks,
+			      tree *retptr)
+{
+  /* For the exit block, we wish to force a check at every
+     predecessor, so pretend we've already found a pred that had
+     checking, so that we schedule checking at every one of its pred
+     edges.  */
+  bool first = bb->index >= NUM_FIXED_BLOCKS;
+  bool postchecked = true;
+
+  gphi *retphi = NULL;
+  if (retptr && *retptr && TREE_CODE (*retptr) == SSA_NAME
+      && !SSA_NAME_IS_DEFAULT_DEF (*retptr)
+      && SSA_NAME_DEF_STMT (*retptr)
+      && is_a <gphi *> (SSA_NAME_DEF_STMT (*retptr))
+      && gimple_bb (SSA_NAME_DEF_STMT (*retptr)) == bb)
+    {
+      retphi = as_a <gphi *> (SSA_NAME_DEF_STMT (*retptr));
+      gcc_checking_assert (gimple_phi_result (retphi) == *retptr);
+    }
+
+  for (int i = EDGE_COUNT (bb->preds); i--; first = false)
+    {
+      edge e = EDGE_PRED (bb, i);
+
+      bool checked
+	= hardcfr_sibcall_search_block (e->src, chk_edges,
+					count_chkcall, chkcall_blocks,
+					count_postchk, postchk_blocks,
+					!retphi ? retptr
+					: gimple_phi_arg_def_ptr (retphi, i));
+
+      if (first)
+	{
+	  postchecked = checked;
+	  continue;
+	}
+
+      /* When we first find a checked block, force a check at every
+	 other incoming edge we've already visited, and those we
+	 visit afterwards that don't have their own check, so that
+	 when we reach BB, the check has already been performed.  */
+      if (!postchecked && checked)
+	{
+	  for (int j = EDGE_COUNT (bb->preds); --j > i; )
+	    chk_edges.safe_push (EDGE_PRED (bb, j));
+	  postchecked = true;
+	}
+      if (postchecked && !checked)
+	chk_edges.safe_push (EDGE_PRED (bb, i));
+    }
+
+  if (postchecked && bb->index >= NUM_FIXED_BLOCKS)
+    {
+      if (bitmap_set_bit (postchk_blocks, bb->index))
+	count_postchk++;
+      else
+	gcc_unreachable ();
+    }
+
+  return postchecked;
+}
+
+
 class rt_bb_visited
 {
   /* Use a sufficiently wide unsigned type to hold basic block numbers.  */
@@ -176,8 +457,8 @@ class rt_bb_visited
      neither ENTRY nor EXIT, but maybe one-past-the-end, to compute
      the visited array length.  */
   blknum num2idx (blknum n) {
-    gcc_checking_assert (n >= 2 && n <= nblocks);
-    return (n - 2);
+    gcc_checking_assert (n >= NUM_FIXED_BLOCKS && n <= nblocks);
+    return (n - NUM_FIXED_BLOCKS);
   }
   /* Return the block vindex for BB, that must not be ENTRY or
      EXIT.  */
@@ -249,8 +530,7 @@ class rt_bb_visited
   }
 
   /* Set the bit corresponding to BB in VISITED.  Add to SEQ any
-     required gimple statements, and return SEQ, possibly
-     modified.  */
+     required gimple stmts, and return SEQ, possibly modified.  */
   gimple_seq vset (basic_block bb, gimple_seq seq = NULL)
   {
     tree bit, setme = vword (bb, &bit);
@@ -270,7 +550,7 @@ class rt_bb_visited
 
 public:
   /* Prepare to add control flow redundancy testing to CFUN.  */
-  rt_bb_visited ()
+  rt_bb_visited (int checkpoints)
     : nblocks (n_basic_blocks_for_fn (cfun)),
       vword_type (NULL), ckseq (NULL), rtcfg (NULL)
   {
@@ -353,8 +633,8 @@ public:
 					 NULL, NULL);
     gimple_seq_add_stmt (&ckseq, detach);
 
-    if (nblocks - 2 > blknum (param_hardcfr_max_inline_blocks)
-	|| !single_pred_p (EXIT_BLOCK_PTR_FOR_FN (cfun)))
+    if (nblocks - NUM_FIXED_BLOCKS > blknum (param_hardcfr_max_inline_blocks)
+	|| checkpoints > 1)
       {
 	/* Make sure vword_bits is wide enough for the representation
 	   of nblocks in rtcfg.  Compare with vword_bits << vword_bits,
@@ -379,65 +659,34 @@ public:
     gimple_seq_add_stmt (&ckseq, ckfail_init);
   }
 
-  /* Insert SEQ on E, or close enough (e.g., before a noreturn or tail
-     call at the end of E->src).  */
-  void insert_exit_check (gimple_seq seq, edge e)
+  /* Insert SEQ before a resx or a call in INSBB.  */
+  void insert_exit_check_in_block (gimple_seq seq, basic_block insbb)
   {
-    basic_block insbb = e->src;
-
-    /* If the returning block ends with a noreturn call, insert
-       checking before it.  This is particularly important for
-       __builtin_return.  Other noreturn calls won't have an edge to
-       the exit block, so they won't even be considered as exit paths.
-
-       Insert-on-edge inserts before other return stmts, but not
-       before calls, and if a single-block function had the check
-       sequence inserted after a noreturn call, it would be useless,
-       but checking would still flag it as malformed if block 2 has a
-       fallthru edge to the exit block.
-
-       Also avoid disrupting tail calls, inserting the check before
-       them.  This works for must-tail calls, but tail calling as an
-       optimization is detected too late for us.  */
     gimple_stmt_iterator gsi = gsi_last_bb (insbb);
-    gimple *ret = gsi_stmt (gsi);
-    if (ret && is_a <greturn *> (ret))
-      {
+
+    while (!gsi_end_p (gsi))
+      if (is_a <gresx *> (gsi_stmt (gsi))
+	  || is_a <gcall *> (gsi_stmt (gsi)))
+	break;
+      else
 	gsi_prev (&gsi);
-	if (!gsi_end_p (gsi))
-	  ret = gsi_stmt (gsi);
-      }
-    if (ret && is_a <gcall *> (ret)
-	&& (gimple_call_noreturn_p (ret)
-	    || gimple_call_must_tail_p (as_a <gcall *> (ret))
-	    || gimple_call_tail_p (as_a <gcall *> (ret))))
-      gsi_insert_seq_before (&gsi, seq, GSI_SAME_STMT);
-    else
-      gsi_insert_seq_on_edge_immediate (e, seq);
+
+    gsi_insert_seq_before (&gsi, seq, GSI_SAME_STMT);
   }
 
-  /* Add checking code on every exit edge, and initialization code on
-     the entry edge.  Before this point, the CFG has been undisturbed,
-     and all the needed data has been collected and safely stowed.  */
-  void check ()
+  /* Insert SEQ on E.  */
+  void insert_exit_check_on_edge (gimple_seq seq, edge e)
   {
-    /* Insert initializers for visited at the entry.  */
-    gimple_seq iseq = NULL;
-
-    gcall *vinit = gimple_build_call (builtin_decl_explicit
-				      (BUILT_IN_MEMSET), 3,
-				      build1 (ADDR_EXPR,
-					      build_pointer_type
-					      (TREE_TYPE (visited)),
-					      visited),
-				      integer_zero_node,
-				      TYPE_SIZE_UNIT (TREE_TYPE (visited)));
-    gimple_seq_add_stmt (&iseq, vinit);
-
-    gsi_insert_seq_on_edge_immediate (single_succ_edge
-				      (ENTRY_BLOCK_PTR_FOR_FN (cfun)),
-				      iseq);
+    gsi_insert_seq_on_edge_immediate (e, seq);
+  }
 
+  /* Add checking code to CHK_EDGES and CHKCALL_BLOCKS, and
+     initialization code on the entry edge.  Before this point, the
+     CFG has been undisturbed, and all the needed data has been
+     collected and safely stowed.  */
+  void check (chk_edges_t &chk_edges,
+	      int count_chkcall, auto_sbitmap const &chkcall_blocks)
+  {
     /* If we're using out-of-line checking, create and statically
        initialize the CFG checking representation, generate the
        checker call for the checking sequence, and insert it in all
@@ -498,28 +747,116 @@ public:
 						     rtcfg));
 	gimple_seq_add_stmt (&ckseq, call_chk);
 
+	gimple *clobber = gimple_build_assign (visited,
+					       build_clobber
+					       (TREE_TYPE (visited)));
+	gimple_seq_add_stmt (&ckseq, clobber);
+
 	/* If we have multiple exit edges, insert (copies of)
 	   ckseq in all of them.  */
-	for (int i = EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (cfun)->preds);
-	     i--; )
+	for (int i = chk_edges.length (); i--; )
 	  {
 	    gimple_seq seq = ckseq;
 	    /* Copy the sequence, unless we're dealing with the
 	       last edge (we're counting down to zero).  */
-	    if (i)
+	    if (i || count_chkcall)
+	      seq = gimple_seq_copy (seq);
+
+	    edge e = chk_edges[i];
+
+	    if (dump_file)
+	      {
+		if (e->dest == EXIT_BLOCK_PTR_FOR_FN (cfun))
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to exit.\n",
+			   e->src->index);
+		else
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to postcheck block %i.\n",
+			   e->src->index, e->dest->index);
+	      }
+
+	    insert_exit_check_on_edge (seq, e);
+
+	    gcc_checking_assert (!bitmap_bit_p (chkcall_blocks, e->src->index));
+	  }
+
+	sbitmap_iterator it;
+	unsigned i;
+	EXECUTE_IF_SET_IN_BITMAP (chkcall_blocks, 0, i, it)
+	  {
+	    basic_block bb = BASIC_BLOCK_FOR_FN (cfun, i);
+
+	    gimple_seq seq = ckseq;
+	    gcc_checking_assert (count_chkcall > 0);
+	    if (--count_chkcall)
 	      seq = gimple_seq_copy (seq);
 
-	    insert_exit_check (seq,
-			       EDGE_PRED (EXIT_BLOCK_PTR_FOR_FN (cfun), i));
+	    if (dump_file)
+	      fprintf (dump_file,
+		       "Inserting out-of-line check before stmt in block %i.\n",
+		       bb->index);
+
+	    insert_exit_check_in_block (seq, bb);
 	  }
+
+	gcc_checking_assert (count_chkcall == 0);
       }
     else
       {
 	/* Inline checking requires a single exit edge.  */
-	gimple *last = gsi_stmt (gsi_last (ckseq));
+	gimple *last = gimple_build_assign (visited,
+					    build_clobber
+					    (TREE_TYPE (visited)));
+	gimple_seq_add_stmt (&ckseq, last);
 
-	insert_exit_check (ckseq,
-			   single_pred_edge (EXIT_BLOCK_PTR_FOR_FN (cfun)));
+	if (!count_chkcall)
+	  {
+	    edge e = single_pred_edge (EXIT_BLOCK_PTR_FOR_FN (cfun));
+
+	    if (dump_file)
+	      {
+		if (e->dest == EXIT_BLOCK_PTR_FOR_FN (cfun))
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to postcheck block %i.\n",
+			   e->src->index, e->dest->index);
+		else
+		  fprintf (dump_file,
+			   "Inserting inline check in"
+			   " block %i's edge to exit.\n",
+			   e->src->index);
+	      }
+
+	    insert_exit_check_on_edge (ckseq, e);
+	  }
+	else
+	  {
+	    gcc_checking_assert (count_chkcall == 1);
+
+	    sbitmap_iterator it;
+	    unsigned i;
+	    EXECUTE_IF_SET_IN_BITMAP (chkcall_blocks, 0, i, it)
+	      {
+		basic_block bb = BASIC_BLOCK_FOR_FN (cfun, i);
+
+		gimple_seq seq = ckseq;
+		gcc_checking_assert (count_chkcall > 0);
+		if (--count_chkcall)
+		  seq = gimple_seq_copy (seq);
+
+		if (dump_file)
+		  fprintf (dump_file,
+			   "Inserting inline check before stmt in block %i.\n",
+			   bb->index);
+
+		insert_exit_check_in_block (seq, bb);
+	      }
+
+	    gcc_checking_assert (count_chkcall == 0);
+	  }
 
 	/* The inserted ckseq computes CKFAIL at LAST.  Now we have to
 	   conditionally trap on it.  */
@@ -540,8 +877,7 @@ public:
 	  add_bb_to_loop (trp, current_loops->tree_root);
 
 	/* Insert a conditional branch to the trap block.  If the
-	   conditional wouldn't be the last statement, split the
-	   block.  */
+	   conditional wouldn't be the last stmt, split the block.  */
 	gimple_stmt_iterator gsi = gsi_for_stmt (last);
 	if (!gsi_one_before_end_p (gsi))
 	  split_block (gsi_bb (gsi), gsi_stmt (gsi));
@@ -564,6 +900,24 @@ public:
 	if (dom_info_available_p (CDI_DOMINATORS))
 	  set_immediate_dominator (CDI_DOMINATORS, trp, gimple_bb (last));
       }
+
+    /* Insert initializers for visited at the entry.  Do this after
+       other insertions, to avoid messing with block numbers.  */
+    gimple_seq iseq = NULL;
+
+    gcall *vinit = gimple_build_call (builtin_decl_explicit
+				      (BUILT_IN_MEMSET), 3,
+				      build1 (ADDR_EXPR,
+					      build_pointer_type
+					      (TREE_TYPE (visited)),
+					      visited),
+				      integer_zero_node,
+				      TYPE_SIZE_UNIT (TREE_TYPE (visited)));
+    gimple_seq_add_stmt (&iseq, vinit);
+
+    gsi_insert_seq_on_edge_immediate (single_succ_edge
+				      (ENTRY_BLOCK_PTR_FOR_FN (cfun)),
+				      iseq);
   }
 
   /* Push onto RTCFG a (mask, index) pair to test for IBB when BB is
@@ -607,7 +961,7 @@ public:
     return false;
   }
 
-  /* Add to CKSEQ statements to clear CKPART if OBB is visited.  */
+  /* Add to CKSEQ stmts to clear CKPART if OBB is visited.  */
   void
   build_block_check (basic_block obb)
   {
@@ -632,34 +986,48 @@ public:
 
   /* Add to BB code to set its bit in VISITED, and add to RTCFG or
      CKSEQ the data or code needed to check BB's predecessors and
-     successors.  Do NOT change the CFG.  */
-  void visit (basic_block bb)
+     successors.  If CHECKPOINT, assume the block is a checkpoint,
+     whether or not it has an edge to EXIT.  If POSTCHECK, assume the
+     block post-dominates checkpoints and therefore no bitmap setting
+     or checks are to be performed in or for it.  Do NOT change the
+     CFG.  */
+  void visit (basic_block bb, bool checkpoint, bool postcheck)
   {
     /* Set the bit in VISITED when entering the block.  */
     gimple_stmt_iterator gsi = gsi_after_labels (bb);
-    gsi_insert_seq_before (&gsi, vset (bb), GSI_SAME_STMT);
+    if (!postcheck)
+      gsi_insert_seq_before (&gsi, vset (bb), GSI_SAME_STMT);
 
     if (rtcfg)
       {
-	/* Build a list of (index, mask) terminated by (NULL, 0).
-	   Consolidate masks with the same index when they're
-	   adjacent.  First, predecessors.  Count backwards, because
-	   we're going to reverse the list.  The order shouldn't
-	   matter, but let's not make it surprising.  */
-	for (int i = EDGE_COUNT (bb->preds); i--; )
-	  if (push_rtcfg_pair (EDGE_PRED (bb, i)->src, bb,
-			       ENTRY_BLOCK_PTR_FOR_FN (cfun)))
-	    break;
+	if (!postcheck)
+	  {
+	    /* Build a list of (index, mask) terminated by (NULL, 0).
+	       Consolidate masks with the same index when they're
+	       adjacent.  First, predecessors.  Count backwards, because
+	       we're going to reverse the list.  The order shouldn't
+	       matter, but let's not make it surprising.  */
+	    for (int i = EDGE_COUNT (bb->preds); i--; )
+	      if (push_rtcfg_pair (EDGE_PRED (bb, i)->src, bb,
+				   ENTRY_BLOCK_PTR_FOR_FN (cfun)))
+		break;
+	  }
 	rtcfg = tree_cons (NULL_TREE, build_int_cst (vword_type, 0), rtcfg);
 
-	/* Then, successors.  */
-	for (int i = EDGE_COUNT (bb->succs); i--; )
-	  if (push_rtcfg_pair (EDGE_SUCC (bb, i)->dest, bb,
-			       EXIT_BLOCK_PTR_FOR_FN (cfun)))
-	    break;
+	if (!postcheck)
+	  {
+	    /* Then, successors.  */
+	    if (!checkpoint
+		|| !push_rtcfg_pair (EXIT_BLOCK_PTR_FOR_FN (cfun),
+				     bb, EXIT_BLOCK_PTR_FOR_FN (cfun)))
+	      for (int i = EDGE_COUNT (bb->succs); i--; )
+		if (push_rtcfg_pair (EDGE_SUCC (bb, i)->dest, bb,
+				     EXIT_BLOCK_PTR_FOR_FN (cfun)))
+		  break;
+	  }
 	rtcfg = tree_cons (NULL_TREE, build_int_cst (vword_type, 0), rtcfg);
       }
-    else
+    else if (!postcheck)
       {
 	/* Schedule test to fail if the block was reached but somehow none
 	   of its predecessors were.  */
@@ -677,6 +1045,8 @@ public:
 	gassign *blkruns = gimple_build_assign (ckpart, unshare_expr (bit));
 	gimple_seq_add_stmt (&ckseq, blkruns);
 
+	if (checkpoint)
+	  build_block_check (EXIT_BLOCK_PTR_FOR_FN (cfun));
 	for (int i = 0, e = EDGE_COUNT (bb->succs); i < e; i++)
 	  build_block_check (EDGE_SUCC (bb, i)->dest);
 
@@ -687,21 +1057,357 @@ public:
   }
 };
 
+/* It might be useful to avoid checking before noreturn calls that are
+   known to always finish by throwing an exception, rather than by
+   ending the program or looping forever.  Such functions would have
+   to be annotated somehow, with an attribute or flag, so that
+   exception-raising functions, such as C++'s __cxa_throw,
+   __cxa_rethrow, and Ada's gnat_rcheck_*, gnat_reraise*,
+   ada.exception.raise_exception*, and the language-independent
+   unwinders could be detected here and handled differently from other
+   noreturn functions.  */
+static bool
+always_throwing_noreturn_call_p (gimple *)
+{
+  return false;
+}
+
 /* Control flow redundancy hardening: record the execution path, and
    verify at exit that an expect path was taken.  */
 
 unsigned int
-pass_harden_control_flow_redundancy::execute (function *)
+pass_harden_control_flow_redundancy::execute (function *fun)
 {
-  rt_bb_visited vstd;
-
+  bool const check_at_escaping_exceptions
+    = (flag_exceptions
+       && flag_harden_control_flow_redundancy_check_exceptions);
+  bool const check_before_noreturn_calls
+    = flag_harden_control_flow_redundancy_check_noreturn > HCFRNR_NEVER;
+  bool const check_before_nothrow_noreturn_calls
+    = (check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn >= HCFRNR_NOTHROW);
+  bool const check_before_throwing_noreturn_calls
+    = (flag_exceptions
+       && check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn > HCFRNR_NOTHROW);
+  bool const check_before_always_throwing_noreturn_calls
+    = (flag_exceptions
+       && check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn >= HCFRNR_ALWAYS);
   basic_block bb;
-  FOR_EACH_BB_FN (bb, cfun)
-    vstd.visit (bb);
+  basic_block bb_eh_cleanup = NULL;
+
+  if (check_at_escaping_exceptions)
+    {
+      int lp_eh_cleanup = -1;
+
+      /* Record the preexisting blocks, to avoid visiting newly-created
+	 blocks.  */
+      auto_sbitmap to_visit (last_basic_block_for_fn (fun));
+      bitmap_clear (to_visit);
+
+      FOR_EACH_BB_FN (bb, fun)
+	bitmap_set_bit (to_visit, bb->index);
+
+      /* Scan the blocks for stmts with escaping exceptions, that
+	 wouldn't be denoted in the CFG, and associate them with an
+	 empty cleanup handler around the whole function.  Walk
+	 backwards, so that even when we split the block, */
+      sbitmap_iterator it;
+      unsigned i;
+      EXECUTE_IF_SET_IN_BITMAP (to_visit, 0, i, it)
+	{
+	  bb = BASIC_BLOCK_FOR_FN (fun, i);
+
+	  for (gimple_stmt_iterator gsi = gsi_last_bb (bb);
+	       !gsi_end_p (gsi); gsi_prev (&gsi))
+	    {
+	      gimple *stmt = gsi_stmt (gsi);
+	      if (!stmt_could_throw_p (fun, stmt))
+		continue;
+
+	      /* If it must not throw, or if it already has a handler,
+		 we need not worry about it.  */
+	      if (lookup_stmt_eh_lp (stmt) != 0)
+		continue;
+
+	      /* Don't split blocks at, nor add EH edges to, tail
+		 calls, we will add verification before the call
+		 anyway.  */
+	      if (is_a <gcall *> (stmt)
+		  && (gimple_call_must_tail_p (as_a <gcall *> (stmt))
+		      || gimple_call_tail_p (as_a <gcall *> (stmt))
+		      || returning_call_p (as_a <gcall *> (stmt))))
+		continue;
+
+	      if (!gsi_one_before_end_p (gsi))
+		split_block (bb, stmt);
+	      /* A resx or noreturn call needs not be associated with
+		 the cleanup handler if we're going to add checking
+		 before it.  We only test cases that didn't require
+		 block splitting because noreturn calls would always
+		 be at the end of blocks, and we test for zero
+		 successors because if there is an edge, it's not
+		 noreturn, as any EH edges would have already been
+		 caught by the lookup_stmt_eh_lp test above.  */
+	      else if (check_before_noreturn_calls
+		       && EDGE_COUNT (bb->succs) == 0
+		       && (is_a <gresx *> (stmt)
+			   ? check_before_always_throwing_noreturn_calls
+			   : (!is_a <gcall *> (stmt)
+			      || !gimple_call_noreturn_p (stmt))
+			   ? (gcc_unreachable (), false)
+			   : (!flag_exceptions
+			      || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+			   ? check_before_nothrow_noreturn_calls
+			   : always_throwing_noreturn_call_p (stmt)
+			   ? check_before_always_throwing_noreturn_calls
+			   : check_before_throwing_noreturn_calls))
+		{
+		  if (dump_file)
+		    {
+		      fprintf (dump_file,
+			       "Bypassing cleanup for noreturn stmt"
+			       " in block %i:\n",
+			       bb->index);
+		      print_gimple_stmt (dump_file, stmt, 0);
+		    }
+		  continue;
+		}
+
+	      if (!bb_eh_cleanup)
+		{
+		  bb_eh_cleanup = create_empty_bb (bb);
+		  if (dom_info_available_p (CDI_DOMINATORS))
+		    set_immediate_dominator (CDI_DOMINATORS, bb_eh_cleanup, bb);
+		  if (current_loops)
+		    add_bb_to_loop (bb_eh_cleanup, current_loops->tree_root);
+
+		  /* Make the new block an EH cleanup for the call.  */
+		  eh_region new_r = gen_eh_region_cleanup (NULL);
+		  eh_landing_pad lp = gen_eh_landing_pad (new_r);
+		  tree label = gimple_block_label (bb_eh_cleanup);
+		  lp->post_landing_pad = label;
+		  EH_LANDING_PAD_NR (label) = lp_eh_cleanup = lp->index;
+
+		  /* Just propagate the exception.
+		     We will later insert the verifier call.  */
+		  gimple_stmt_iterator ehgsi;
+		  ehgsi = gsi_after_labels (bb_eh_cleanup);
+		  gresx *resx = gimple_build_resx (new_r->index);
+		  gsi_insert_before (&ehgsi, resx, GSI_SAME_STMT);
+
+		  if (dump_file)
+		    fprintf (dump_file,
+			     "Created cleanup block %i:\n",
+			     bb_eh_cleanup->index);
+		}
+	      else if (dom_info_available_p (CDI_DOMINATORS))
+		{
+		  basic_block immdom;
+		  immdom = get_immediate_dominator (CDI_DOMINATORS,
+						    bb_eh_cleanup);
+		  if (!dominated_by_p (CDI_DOMINATORS, bb, immdom))
+		    {
+		      immdom = nearest_common_dominator (CDI_DOMINATORS,
+							 immdom, bb);
+		      set_immediate_dominator (CDI_DOMINATORS,
+					       bb_eh_cleanup, immdom);
+		    }
+		}
+
+	      if (dump_file)
+		{
+		  fprintf (dump_file,
+			   "Associated cleanup block with stmt in block %i:\n",
+			   bb->index);
+		  print_gimple_stmt (dump_file, stmt, 0);
+		}
+
+	      add_stmt_to_eh_lp (stmt, lp_eh_cleanup);
+	      /* Finally, wire the EH cleanup block into the CFG.  */
+	      make_eh_edges (stmt);
+	    }
+	}
+
+      if (bb_eh_cleanup)
+	{
+	  /* A cfg_cleanup after bb_eh_cleanup makes for a more compact
+	     rtcfg, and it avoids bb numbering differences when we split
+	     blocks because of trailing debug insns only.  */
+	  cleanup_tree_cfg ();
+	  gcc_checking_assert (EDGE_COUNT (bb_eh_cleanup->succs) == 0);
+	}
+    }
+
+  /* These record blocks with calls that are to be preceded by
+     checkpoints, such as noreturn calls (if so chosen), must-tail
+     calls, potential early-marked tail calls, and returning calls (if
+     so chosen).  */
+  int count_chkcall = 0;
+  auto_sbitmap chkcall_blocks (last_basic_block_for_fn (fun));
+  bitmap_clear (chkcall_blocks);
+
+  /* We wish to add verification at blocks without successors, such as
+     noreturn calls (raising or not) and the reraise at the cleanup
+     block, but not other reraises: they will go through the cleanup
+     block.  */
+  if (check_before_noreturn_calls)
+    FOR_EACH_BB_FN (bb, fun)
+      {
+	gimple_stmt_iterator gsi = gsi_last_bb (bb);
+	if (gsi_end_p (gsi))
+	  continue;
+	gimple *stmt = gsi_stmt (gsi);
 
-  vstd.check ();
+	if (EDGE_COUNT (bb->succs) == 0)
+	  {
+	    /* A stmt at the end of a block without any successors is
+	       either a resx or a noreturn call without a local
+	       handler.  Check that it's one of the desired
+	       checkpoints.  */
+	    if (flag_exceptions && is_a <gresx *> (stmt)
+		? (check_before_always_throwing_noreturn_calls
+		   || bb == bb_eh_cleanup)
+		: (!is_a <gcall *> (stmt)
+		   || !gimple_call_noreturn_p (stmt))
+		? (/* Catch cases in which successors would be
+		      expected.  */
+		   gcc_unreachable (), false)
+		: (!flag_exceptions
+		   || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+		? check_before_nothrow_noreturn_calls
+		: always_throwing_noreturn_call_p (stmt)
+		? check_before_always_throwing_noreturn_calls
+		: check_before_throwing_noreturn_calls)
+	      {
+		if (dump_file)
+		  {
+		    fprintf (dump_file,
+			     "Scheduling check before stmt"
+			     " in succ-less block %i:\n",
+			     bb->index);
+		    print_gimple_stmt (dump_file, stmt, 0);
+		  }
+
+		if (bitmap_set_bit (chkcall_blocks, bb->index))
+		  count_chkcall++;
+		else
+		  gcc_unreachable ();
+	      }
+	    continue;
+	  }
 
-  return 0;
+	/* If there are no exceptions, it would seem like any noreturn
+	   call must have zero successor edges, but __builtin_return
+	   gets successor edges.  We don't want to handle it here, it
+	   will be dealt with in sibcall_search_preds.  Otherwise,
+	   check for blocks without non-EH successors, but skip those
+	   with resx stmts and edges (i.e., those other than that in
+	   bb_eh_cleanup), since those will go through bb_eh_cleanup,
+	   that will have been counted as noreturn above because it
+	   has no successors.  */
+	gcc_checking_assert (bb != bb_eh_cleanup
+			     || !check_at_escaping_exceptions);
+	if (flag_exceptions && is_a <gresx *> (stmt)
+	    ? check_before_always_throwing_noreturn_calls
+	    : (!is_a <gcall *> (stmt)
+	       || !gimple_call_noreturn_p (stmt))
+	    ? false
+	    : (!flag_exceptions
+	       || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+	    ? false /* rather than check_before_nothrow_noreturn_calls */
+	    : always_throwing_noreturn_call_p (stmt)
+	    ? check_before_always_throwing_noreturn_calls
+	    : check_before_throwing_noreturn_calls)
+	  {
+	    gcc_checking_assert (single_succ_p (bb)
+				 && (single_succ_edge (bb)->flags & EDGE_EH));
+
+	    if (dump_file)
+	      {
+		fprintf (dump_file,
+			 "Scheduling check before stmt"
+			 " in EH-succ block %i:\n",
+			 bb->index);
+		print_gimple_stmt (dump_file, stmt, 0);
+	      }
+
+	    if (bitmap_set_bit (chkcall_blocks, bb->index))
+	      count_chkcall++;
+	    else
+	      gcc_unreachable ();
+	  }
+      }
+  else if (bb_eh_cleanup)
+    {
+      if (bitmap_set_bit (chkcall_blocks, bb_eh_cleanup->index))
+	count_chkcall++;
+      else
+	gcc_unreachable ();
+    }
+
+  gcc_checking_assert (!bb_eh_cleanup
+		       || bitmap_bit_p (chkcall_blocks, bb_eh_cleanup->index));
+
+  /* If we don't have edges to exit nor noreturn calls (including the
+     cleanup reraise), then we may skip instrumentation: that would
+     amount to a function that ends with an infinite loop.  */
+  if (!count_chkcall
+      && EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (fun)->preds) == 0)
+    {
+      if (dump_file)
+	fprintf (dump_file,
+		 "Disabling CFR, no exit paths to check\n");
+
+      return 0;
+    }
+
+  /* Search for must-tail calls, early-marked potential tail calls,
+     and, if requested, returning calls.  As we introduce early
+     checks, */
+  int count_postchk = 0;
+  auto_sbitmap postchk_blocks (last_basic_block_for_fn (fun));
+  bitmap_clear (postchk_blocks);
+  chk_edges_t chk_edges;
+  hardcfr_sibcall_search_preds (EXIT_BLOCK_PTR_FOR_FN (fun), chk_edges,
+				count_chkcall, chkcall_blocks,
+				count_postchk, postchk_blocks,
+				NULL);
+
+  rt_bb_visited vstd (chk_edges.length () + count_chkcall);
+
+  auto_sbitmap combined_blocks (last_basic_block_for_fn (fun));
+  bitmap_copy (combined_blocks, chkcall_blocks);
+  int i;
+  edge *e;
+  FOR_EACH_VEC_ELT (chk_edges, i, e)
+    if (!bitmap_set_bit (combined_blocks, (*e)->src->index))
+      /* There may be multiple chk_edges with the same src block;
+	 guard againt overlaps with chkcall_blocks only.  */
+      gcc_assert (!bitmap_bit_p (chkcall_blocks, (*e)->src->index));
+
+  /* Visit blocks in index order, because building rtcfg depends on
+     that.  Blocks must be compact, which the cleanup_cfg requirement
+     ensures.  This would also enable FOR_EACH_BB_FN to be used to
+     iterate in index order, but bb_eh_cleanup block splits and
+     insertions changes that.  */
+  gcc_checking_assert (n_basic_blocks_for_fn (fun)
+		       == last_basic_block_for_fn (fun));
+  for (int i = NUM_FIXED_BLOCKS; i < n_basic_blocks_for_fn (fun); i++)
+    {
+      bb = BASIC_BLOCK_FOR_FN (fun, i);
+      gcc_checking_assert (bb->index == i);
+      vstd.visit (bb, bitmap_bit_p (combined_blocks, i),
+		  bitmap_bit_p (postchk_blocks, i));
+    }
+
+  vstd.check (chk_edges, count_chkcall, chkcall_blocks);
+
+  return
+    TODO_update_ssa
+    | TODO_cleanup_cfg
+    | TODO_verify_il;
 }
 
 /* Instantiate a hardcfr pass.  */
diff --git a/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c b/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c
new file mode 100644
index 00000000000..a6992eb9f8e
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -O0 -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we don't insert checking before noreturn calls.  -O0 is tested
+   separately because h is not found to be noreturn without optimization.  */
+
+#include "torture/harden-cfr-noret.c"
+
+/* No out-of-line checks.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* Only one inline check at the end of f and of h2.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-always.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-always.c
new file mode 100644
index 00000000000..779896c60e8
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-always.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even enabling all checks before noreturn calls (leaving
+   returning calls enabled), we get checks before __builtin_return without
+   duplication (__builtin_return is both noreturn and a returning call).  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-never.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-never.c
new file mode 100644
index 00000000000..49ce17f5b93
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-never.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even enabling checks before never noreturn calls (leaving
+   returning calls enabled), we get checks before __builtin_return without
+   duplication (__builtin_return is both noreturn and a returning call).  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noopt.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noopt.c
new file mode 100644
index 00000000000..1512614791f
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noopt.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even disabling checks before both noreturn and returning
+   calls, we still get checks before __builtin_return.  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noret.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noret.c
new file mode 100644
index 00000000000..fd95bb7e3e3
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noret.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even disabling checks before returning calls (leaving noreturn
+   calls enabled), we still get checks before __builtin_return.  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-nothrow.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-nothrow.c
new file mode 100644
index 00000000000..c5c361234c4
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-nothrow.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even enabling checks before nothrow noreturn calls (leaving
+   returning calls enabled), we get checks before __builtin_return without
+   duplication (__builtin_return is both noreturn and a returning call).  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-retcl.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-retcl.c
new file mode 100644
index 00000000000..137dfbb95d6
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-retcl.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even disabling checks before noreturn calls (leaving returning
+   calls enabled), we still get checks before __builtin_return.  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c
index 70acdc95f25..b459ff6b864 100644
--- a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c
@@ -7,5 +7,11 @@ int f(int i) {
   return i;
 }
 
-/* Out-of-line checking, before both builtin_return and return.  */
+int g(int i) {
+  __builtin_return (&i);
+}
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
 /* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c
new file mode 100644
index 00000000000..8bd2d13ac18
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c
@@ -0,0 +1,18 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we don't insert checking before noreturn calls.  -O0 is tested
+   separately because h is not found to be noreturn without optimization, which
+   affects codegen for h2, so h2 is omitted here at -O0.  */
+
+#if !__OPTIMIZE__
+# define OMIT_H2
+#endif
+
+#include "harden-cfr-noret.c"
+
+
+/* No out-of-line checks.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* Only one inline check at the end of f.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c
new file mode 100644
index 00000000000..a804a6cfe59
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fno-exceptions -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that -fno-exceptions makes for implicit nothrow in noreturn
+   handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#include "harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c
new file mode 100644
index 00000000000..f390cfdbc59
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checking before nothrow noreturn calls.  */
+
+#include "harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c
new file mode 100644
index 00000000000..fdd803109a4
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c
@@ -0,0 +1,38 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checking before all noreturn calls.  */
+
+#ifndef ATTR_NOTHROW_OPT /* Overridden in harden-cfr-noret-noexcept.  */
+#define ATTR_NOTHROW_OPT __attribute__ ((__nothrow__))
+#endif
+
+extern void __attribute__ ((__noreturn__)) ATTR_NOTHROW_OPT g (void);
+
+void f(int i) {
+  if (i)
+    /* Out-of-line checks here...  */
+    g ();
+  /* ... and here.  */
+}
+
+void __attribute__ ((__noinline__, __noclone__))
+h(void) {
+  /* Inline check here.  */
+  g ();
+}
+
+#ifndef OMIT_H2 /* from harden-cfr-noret-never.  */
+void h2(void) {
+  /* Inline check either here, whether because of noreturn or tail call...  */
+  h ();
+  /* ... or here, if not optimizing.  */
+}
+#endif
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c
new file mode 100644
index 00000000000..6d11487bbba
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c
@@ -0,0 +1,8 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+#include "harden-cfr-tail.c"
+
+/* Inline checking after the calls, disabling tail calling.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 5 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 0 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c
new file mode 100644
index 00000000000..550b02ca088
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c
@@ -0,0 +1,35 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-exceptions -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checks before returning calls and alternate paths, even
+   at -O0, because of the explicit command-line flag.  */
+
+void g (void);
+void g2 (void);
+void g3 (void);
+
+void f (int i) {
+  if (!i)
+    /* Out-of-line checks here...  */
+    g ();
+  else if (i > 0)
+    /* here...  */
+    g2 ();
+  /* else */
+    /* and in the implicit else here.  */
+}
+
+void f2 (int i) {
+  if (!i)
+    /* Out-of-line check here...  */
+    g ();
+  else if (i > 0)
+    /* here...  */
+    g2 ();
+  else
+    /* and here.  */
+    g3 ();
+}
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 6 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 0 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
index 40d76c5c163..d5467eafa9f 100644
--- a/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
@@ -1,17 +1,52 @@
 /* { dg-do compile } */
-/* { dg-options "-fharden-control-flow-redundancy -fdump-tree-hardcfr -ffat-lto-objects" } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-hardcfr-check-exceptions -fdump-tree-hardcfr -ffat-lto-objects -Wno-return-type" } */
 
-/* We'd like to check that we insert checking so as to not disrupt tail calls,
-   but unfortunately mandatory tail calls are not available in C, and optimizing
-   calls as tail calls only takes place after hardcfr.  */
+/* Check that we insert CFR checking so as to not disrupt tail calls.
+   Mandatory tail calls are not available in C, and optimizing calls as tail
+   calls only takes place after hardcfr, so we insert checking before calls
+   followed by copies and return stmts with the same return value, that might
+   (or might not) end up optimized to tail calls.  */
 
-extern int g(int i);
+extern int g (int i);
 
-int f(int i) {
+int f1(int i) {
+  /* Inline check before the returning call.  */
   return g (i);
 }
 
-/* Inline checking before the tail call.  */
-/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
-/* Inline checking before the tail call.  */
-/* { dg-final { scan-tree-dump-times "\\\[tail\]" 1 "hardcfr" { xfail *-*-* } } } */
+extern void g2 (int i);
+
+void f2(int i) {
+  /* Inline check before the returning call, that ignores the returned value,
+     matching the value-less return.  */
+  g2 (i);
+  return;
+}
+
+void f3(int i) {
+  /* Inline check before the returning call.  */
+  g (i);
+}
+
+void f4(int i) {
+  if (i)
+    /* Out-of-line check before the returning call.  */
+    return g2 (i);
+  /* Out-of-line check before implicit return.  */
+}
+
+int f5(int i) {
+  /* Not regarded as a returning call, returning value other than callee's
+     returned value.  */
+  g (i);
+  /* Inline check after the non-returning call.  */
+  return i;
+}
+
+/* Out-of-line checks in f4, before returning calls and before return.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking in all other functions.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 4 "hardcfr" } } */
+/* Check before tail-call in all but f5, but f4 is out-of-line.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 3 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C b/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C
new file mode 100644
index 00000000000..17ea79f7cfb
--- /dev/null
+++ b/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C
@@ -0,0 +1,11 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects -O0" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions, and also checking before noreturn
+   calls.  h2 and h2b get an extra resx without ehcleanup.  */
+
+#include "torture/harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 16 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C b/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C
new file mode 100644
index 00000000000..f0338ccc361
--- /dev/null
+++ b/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C
@@ -0,0 +1,10 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -foptimize-sibling-calls -fdump-tree-hardcfr -O0" } */
+
+/* -fhardcfr-check-returning-calls gets implicitly disabled because,
+   -at O0, -foptimize-sibling-calls has no effect.  */
+
+#include "torture/harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 12 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C
new file mode 100644
index 00000000000..0d35920c7ee
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  */
+
+#include "harden-cfr-noret-no-nothrow.C"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Bypassing cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C
new file mode 100644
index 00000000000..b7d247ff43c
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C
@@ -0,0 +1,18 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  Expected results for =never and =nothrow are the same,
+   since the functions are not nothrow.  */
+
+#include "harden-cfr-noret-no-nothrow.C"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Associated cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C
new file mode 100644
index 00000000000..62c58cfd406
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C
@@ -0,0 +1,23 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#if ! __OPTIMIZE__
+void __attribute__ ((__noreturn__)) h (void);
+#endif
+
+#include "../../c-c++-common/torture/harden-cfr-noret.c"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Associated cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C
new file mode 100644
index 00000000000..0286f6e6d3f
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C
@@ -0,0 +1,20 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions, and also checking before noreturn
+   calls.  */
+
+#if ! __OPTIMIZE__
+/* Without optimization, functions with cleanups end up with an extra
+   resx that is not optimized out, so arrange to optimize them.  */
+void __attribute__ ((__optimize__ (1))) h2(void);
+void __attribute__ ((__optimize__ (1))) h2b(void);
+#endif
+
+#include "harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 14 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
+/* h, h2, h2b, and h4.  */
+/* { dg-final { scan-tree-dump-times "Bypassing" 4 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C
new file mode 100644
index 00000000000..885b0b236af
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C
@@ -0,0 +1,11 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions  -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we do not insert cleanups for checking around the bodies
+   of maybe-throwing functions.  h4 doesn't get any checks, because we
+   don't have noreturn checking enabled.  */
+
+#include "harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 6 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C
new file mode 100644
index 00000000000..32def637255
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C
@@ -0,0 +1,31 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -foptimize-sibling-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions.  These results depend on checking before
+   returning calls, which is only enabled when sibcall optimizations
+   are enabled, so change the optimization mode to -O1 for f and f2,
+   so that -foptimize-sibling-calls can take effect and enable
+   -fhardcfr-check-returning-calls, so that we get the same results.
+   There is a separate test for -O0.  */
+
+#if ! __OPTIMIZE__
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) f(int i);
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) f2(int i);
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) h3(void);
+#endif
+
+#include "harden-cfr-throw.C"
+
+/* f gets out-of-line checks before the unwrapped tail call and in the
+   else edge.  */
+/* f2 gets out-of-line checks before both unwrapped tail calls.  */
+/* h gets out-of-line checks before the implicit return and in the
+   cleanup block.  */
+/* h2 and h2b get out-of-line checks before the cleanup returning
+   call, and in the cleanup block.  */
+/* h3 gets an inline check before the __cxa_end_catch returning call.  */
+/* h4 gets an inline check in the cleanup block.  */
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 10 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C
new file mode 100644
index 00000000000..992fbdad381
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C
@@ -0,0 +1,65 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions.  */
+
+extern void g (void);
+extern void g2 (void);
+
+void f(int i) {
+  if (i)
+    g ();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void f2(int i) {
+  if (i)
+    g ();
+  else
+    g2 ();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h(void) {
+  try {
+    g ();
+  } catch (...) {
+    throw;
+  }
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+struct needs_cleanup {
+  ~needs_cleanup();
+};
+
+void h2(void) {
+  needs_cleanup y; /* No check in the cleanup handler.  */
+  g();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+extern void __attribute__ ((__nothrow__)) another_cleanup (void*);
+
+void h2b(void) {
+  int x __attribute__ ((cleanup (another_cleanup)));
+  g();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h3(void) {
+  try {
+    throw 1;
+  } catch (...) {
+  }
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h4(void) {
+  throw 1;
+  /* Inline check in the cleanup around the __cxa_throw noreturn call.  */
+}
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 12 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c b/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c
new file mode 100644
index 00000000000..8e4ee1fab08
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c
@@ -0,0 +1,15 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C makes for implicit nothrow in noreturn handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#include "../../c-c++-common/torture/harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c b/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c
new file mode 100644
index 00000000000..634d98f1ffc
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c
@@ -0,0 +1,40 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-hardcfr-check-exceptions -fdump-tree-hardcfr -ffat-lto-objects -Wno-return-type" } */
+
+/* In C only, check some additional cases (comparing with
+   c-c++-common/torture/harden-cfr-tail.c) of falling off the end of non-void
+   function.  C++ would issue an unreachable call in these cases.  */
+
+extern int g (int i);
+
+int f1(int i) {
+  /* Inline check before the returning call, that doesn't return anything.  */
+  g (i);
+  /* Implicit return without value, despite the return type; this combination
+     enables tail-calling of g, and is recognized as a returning call.  */
+}
+
+extern void g2 (int i);
+
+int f2(int i) {
+  /* Inline check before the returning call, that disregards its return
+     value.  */
+  g2 (i);
+  /* Implicit return without value, despite the return type; this combination
+     enables tail-calling of g2, and is recognized as a returning call.  */
+}
+
+int f3(int i) {
+  if (i)
+    /* Out-of-line check before the returning call.  */
+    return g (i);
+  /* Out-of-line check before implicit return.  */
+}
+
+/* Out-of-line checks in f3, before returning calls and before return.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking in all other functions.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
+/* Check before tail-call in all functions, but f3 is out-of-line.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/gnat.dg/hardcfr.adb b/gcc/testsuite/gnat.dg/hardcfr.adb
index b578a913d75..abe1605c029 100644
--- a/gcc/testsuite/gnat.dg/hardcfr.adb
+++ b/gcc/testsuite/gnat.dg/hardcfr.adb
@@ -1,5 +1,5 @@
 --  { dg-do run }
---  { dg-options "-fharden-control-flow-redundancy -fdump-tree-hardcfr --param=hardcfr-max-blocks=22 --param=hardcfr-max-inline-blocks=12 -O0" }
+--  { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions -fdump-tree-hardcfr --param=hardcfr-max-blocks=22 --param=hardcfr-max-inline-blocks=12 -O0" }
 
 procedure HardCFR is
    function F (I, J : Integer) return Integer is
diff --git a/libgcc/hardcfr.c b/libgcc/hardcfr.c
index 8ef29428111..55f6b995f2f 100644
--- a/libgcc/hardcfr.c
+++ b/libgcc/hardcfr.c
@@ -60,14 +60,25 @@ extern void __hardcfr_check (size_t blocks,
 			     vword const *visited,
 			     vword const *cfg);
 
+/* Compute the MASK for the bit representing BLOCK in WORDIDX's vword in a
+   visited blocks bit array.  */
+static inline void
+block2mask (size_t const block, vword *const mask, size_t *const wordidx)
+{
+  size_t wbits = __CHAR_BIT__ * sizeof (vword);
+  *wordidx = block / wbits;
+  *mask = (vword)1 << (block % wbits);
+}
 
 /* Check whether the bit corresponding to BLOCK is set in VISITED.  */
 static inline bool
 visited_p (size_t const block, vword const *const visited)
 {
-  size_t wbits = __CHAR_BIT__ * sizeof (vword);
-  vword w = visited[block / wbits];
-  return (w & ((vword)1 << (block % wbits))) != 0;
+  vword mask;
+  size_t wordidx;
+  block2mask (block, &mask, &wordidx);
+  vword w = visited[wordidx];
+  return (w & mask) != 0;
 }
 
 /* Read and consume a mask from **CFG_IT.  (Consume meaning advancing the
@@ -118,19 +129,19 @@ consume_seq (vword const **const cfg_it)
    we reach the terminator without finding any.  Consume the entire sequence
    otherwise, so that *CFG_IT points just past the terminator, which may be the
    beginning of the next sequence.  */
-static inline void
+static inline bool
 check_seq (vword const *const visited, vword const **const cfg_it)
 {
   vword mask;
   size_t wordidx;
 
   /* If the block was visited, check that at least one of the
-     preds was also visited.  */
+     preds/succs was also visited.  */
   do
     /* If we get to the end of the sequence without finding any
        match, something is amiss.  */
     if (!next_pair (cfg_it, &mask, &wordidx))
-      __builtin_trap ();
+      return false;
   /* Keep searching until we find a match, at which point the
      condition is satisfied.  */
   while (!test_mask (visited, mask, wordidx));
@@ -139,6 +150,94 @@ check_seq (vword const *const visited, vword const **const cfg_it)
      skipped the block, so as to position the iterator at the beginning of the
      next .  */
   consume_seq (cfg_it);
+
+  return true;
+}
+
+/* Print out the CFG with BLOCKS blocks, presumed to be associated with CALLER.
+   This is expected to be optimized out entirely, unless the verbose part of
+   __hardcfr_check_fail is enabled.  */
+static inline void
+__hardcfr_debug_cfg (size_t const blocks,
+		     void const *const caller,
+		     vword const *const cfg)
+{
+  __builtin_printf ("CFG at %p, for %p", cfg, caller);
+  vword const *cfg_it = cfg;
+  for (size_t i = 0; i < blocks; i++)
+    {
+      vword mask; size_t wordidx;
+      block2mask (i, &mask, &wordidx);
+      __builtin_printf ("\nblock %lu (%lu/0x%lx)\npreds: ",
+			(unsigned long)i,
+			(unsigned long)wordidx, (unsigned long)mask);
+      while (next_pair (&cfg_it, &mask, &wordidx))
+	__builtin_printf (" (%lu/0x%lx)",
+			  (unsigned long)wordidx, (unsigned long)mask);
+      __builtin_printf ("\nsuccs: ");
+      while (next_pair (&cfg_it, &mask, &wordidx))
+	__builtin_printf (" (%lu/0x%lx)",
+			  (unsigned long)wordidx, (unsigned long)mask);
+    }
+  __builtin_printf ("\n");
+}
+
+#ifndef ATTRIBUTE_UNUSED
+# define ATTRIBUTE_UNUSED __attribute__ ((__unused__))
+#endif
+
+/* This is called when an out-of-line hardcfr check fails.  All the arguments
+   are ignored, and it just traps, unless HARDCFR_VERBOSE_FAIL is enabled.  IF
+   it is, it prints the PART of the CFG, expected to have BLOCKS blocks, that
+   failed at CALLER's BLOCK, and the VISITED bitmap.  When the verbose mode is
+   enabled, it also forces __hardcfr_debug_cfg (above) to be compiled into an
+   out-of-line function, that could be called from a debugger.
+   */
+static inline void
+__hardcfr_check_fail (size_t const blocks ATTRIBUTE_UNUSED,
+		      vword const *const visited,
+		      vword const *const cfg ATTRIBUTE_UNUSED,
+		      size_t const block ATTRIBUTE_UNUSED,
+		      int const part ATTRIBUTE_UNUSED,
+		      void const *const caller ATTRIBUTE_UNUSED)
+{
+#if HARDCFR_VERBOSE_FAIL
+  static const char *parts[] = { "preds", "succs" };
+
+  vword mask; size_t wordidx;
+  block2mask (block, &mask, &wordidx);
+  __builtin_printf ("hardcfr fail at %p block %lu (%lu/0x%lx), expected %s:",
+		    caller, (unsigned long)block,
+		    (unsigned long)wordidx, (unsigned long)mask,
+		    parts[part]);
+
+  /* Skip data for previous blocks.  */
+  vword const *cfg_it = cfg;
+  for (size_t i = block; i--; )
+    {
+      consume_seq (&cfg_it);
+      consume_seq (&cfg_it);
+    }
+  for (size_t i = part; i--; )
+    consume_seq (&cfg_it);
+
+  while (next_pair (&cfg_it, &mask, &wordidx))
+    __builtin_printf (" (%lu/0x%lx)",
+		      (unsigned long)wordidx, (unsigned long)mask);
+
+  __builtin_printf ("\nvisited:");
+  block2mask (blocks, &mask, &wordidx);
+  for (size_t i = 0; i <= wordidx; i++)
+    __builtin_printf (" (%lu/0x%lx)",
+		      (unsigned long)i, (unsigned long)visited[i]);
+  __builtin_printf ("\n");
+
+  /* Reference __hardcfr_debug_cfg so that it's output out-of-line, so that it
+     can be called from a debugger.  */
+  if (!caller || caller == __hardcfr_debug_cfg)
+    return;
+#endif
+  __builtin_trap ();
 }
 
 /* Check that, for each of the BLOCKS basic blocks, if its bit is set in
@@ -168,9 +267,13 @@ __hardcfr_check (size_t const blocks,
       else
 	{
 	  /* Check predecessors.  */
-	  check_seq (visited, &cfg_it);
+	  if (!check_seq (visited, &cfg_it))
+	    __hardcfr_check_fail (blocks, visited, cfg, i, 0,
+				  __builtin_return_address (0));
 	  /* Check successors.  */
-	  check_seq (visited, &cfg_it);
+	  if (!check_seq (visited, &cfg_it))
+	    __hardcfr_check_fail (blocks, visited, cfg, i, 1,
+				  __builtin_return_address (0));
 	}
     }
 }

^ permalink raw reply	[flat|nested] 13+ messages in thread
* [gcc(refs/users/aoliva/heads/testme)] hardcfr: add optional checkpoints
@ 2022-10-25  2:51 Alexandre Oliva
  0 siblings, 0 replies; 13+ messages in thread
From: Alexandre Oliva @ 2022-10-25  2:51 UTC (permalink / raw)
  To: gcc-cvs

https://gcc.gnu.org/g:813d61d3f9dcdb51f9130fcfebac67f249aef4aa

commit 813d61d3f9dcdb51f9130fcfebac67f249aef4aa
Author: Alexandre Oliva <oliva@adacore.com>
Date:   Wed Oct 19 20:36:18 2022 -0300

    hardcfr: add optional checkpoints
    
    Previously, control flow redundancy only checked the visited bitmap
    against the control flow graph at return points and before mandatory
    tail calls, missing various other possibilities of exiting a
    subprogram, such as by raising or propagating exceptions, and calling
    noreturn functions.  The checks inserted before returns also prevented
    potential tail-call optimizations.
    
    This incremental change introduces options to control checking at each
    of these previously-missed checkpoints.  Unless disabled, a cleanup is
    introduced to check when an exceptions escapes a subprogram.  To avoid
    disrupting sibcall optimizations, when they are enabled, checks are
    introduced before calls whose results are immediately returned,
    whether or not they are ultimately optimized.  If enabled, checks are
    introduced before noreturn calls and exception raises, or only before
    nothrow noreturn calls.
    
    
    for  gcc/ChangeLog
    
            * common.opt (-fhardcfr-check-returning-calls): New.
            (-fhardcfr-check-exceptions): New.
            (-fhardcfr-check-noreturn-calls=*): New.
            (Enum hardcfr_check_noreturn_calls): New.
            * doc/invoke.texi: Document them.
            * flag-types.h (enum hardcfr_noret): New.
            * gimple-harden-control-flow.cc: Include more headers.
            (pass_data_harden_control_flow_redundancy): Move
            properties_finish to pass return.
            (pass_harden_control_flow_redundancy::gate): Move
            no-edge-to-exit checking to execute, after exception
            transformations.  Use symbolic NUM_FIXED_BLOCKS.  Test for CFG
            before testing block count.
            (check_returning_calls_p): New.
            (hardcfr_scan_block): New.
            (returnign_call_p): New.
            (chk_edges_t): New.
            (hardcfr_sibcall_search_block): New.
            (hardcfr_sibcall_search_preds): New.
            (rt_bb_visited::num2idx): Use symbolic NUM_FIXED_BLOCKS.
            (rt_bb_visited::rt_bb_visited): Likewise.  Take checkpoints
            count, test it to select out-of-line checks.
            (rt_bb_visited::insert_exit_check): Removed.
            (rt_bb_visited::insert_exit_check_in_blocks): New.
            (rt_bb_visited::insert_exit_check_on_edge): New.
            (rt_bb_visited::check): Take checkpoint edges and blocks.
            Move initialization insertion to the end.  Insert checks on
            edges, and before calls in select blocks.  Clobber the visited
            array to the check sequence.  Output actions to dump_file.
            (rt_bb_visited::visit): Take checkpoint and postcheck.  Assume
            an edge to exit if checkpoint, and skip testing if postcheck.
            (always_throwing_noreturn_call_p): New, dummy.
            (pass_harden_control_flow_redundancy::execute): Introduce
            cleanup block if requested and needed.  Scan blocks for
            noreturn and returning calls if checkpoints before them were
            requested.  Log actions to dump_file.  Visit blocks in index
            order.
    
    for  gcc/testsuite/ChangeLog
    
            * c-c++-common/harden-cfr-noret-never-O0.c: New.
            * c-c++-common/torture/harden-cfr-noret-never.c: New.
            * c-c++-common/torture/harden-cfr-noret-noexcept.c: New.
            * c-c++-common/torture/harden-cfr-noret-nothrow.c: New.
            * c-c++-common/torture/harden-cfr-noret.c: New.
            * c-c++-common/torture/harden-cfr-notail.c: New.
            * c-c++-common/torture/harden-cfr-returning.c: New.
            * c-c++-common/torture/harden-cfr-tail.c: Extend.
            * c-c++-common/torture/harden-cfr-bret-always.c: New.
            * c-c++-common/torture/harden-cfr-bret-never.c: New.
            * c-c++-common/torture/harden-cfr-bret-noopt.c: New.
            * c-c++-common/torture/harden-cfr-bret-noret.c: New.
            * c-c++-common/torture/harden-cfr-bret-nothrow.c: New.
            * c-c++-common/torture/harden-cfr-bret-retcl.c: New.
            * c-c++-common/torture/harden-cfr-bret.c (g): New.
            * g++.dg/harden-cfr-throw-always-O0.C: New.
            * g++.dg/harden-cfr-throw-returning-O0.C: New.
            * g++.dg/torture/harden-cfr-noret-always-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-noret-never-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-noret-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-throw-always.C: New.
            * g++.dg/torture/harden-cfr-throw-nocleanup.C: New.
            * g++.dg/torture/harden-cfr-throw-returning.C: New.
            * g++.dg/torture/harden-cfr-throw.C: New.
            * gcc.dg/torture/harden-cfr-noret-no-nothrow.c: New.
            * gcc.dg/torture/harden-cfr-tail-ub.c: New.
            * gnat.dg/hardcfr.adb: Disable checking at exceptions.
    
    for  libgcc/ChangeLog
    
            * hardcfr.c (block2mask): Split out of...
            (visited_p): ... this.
            (check_seq): Move trapping out, return result instead.
            (__hardcfr_debug_cfg): New.
            (__hardcfr_check_fail): New, with optional verbose error
            printing before trapping.
            (__hardcfr_check): Call __hardcfr_check_fail when check_seq
            fails.

Diff:
---
 gcc/common.opt                                     |  33 +
 gcc/doc/invoke.texi                                |  74 +-
 gcc/flag-types.h                                   |  10 +
 gcc/gimple-harden-control-flow.cc                  | 921 ++++++++++++++++++---
 .../c-c++-common/harden-cfr-noret-never-O0.c       |  12 +
 .../c-c++-common/torture/harden-cfr-bret-always.c  |  13 +
 .../c-c++-common/torture/harden-cfr-bret-never.c   |  13 +
 .../c-c++-common/torture/harden-cfr-bret-noopt.c   |  12 +
 .../c-c++-common/torture/harden-cfr-bret-noret.c   |  12 +
 .../c-c++-common/torture/harden-cfr-bret-nothrow.c |  13 +
 .../c-c++-common/torture/harden-cfr-bret-retcl.c   |  12 +
 .../c-c++-common/torture/harden-cfr-bret.c         |   8 +-
 .../c-c++-common/torture/harden-cfr-noret-never.c  |  18 +
 .../torture/harden-cfr-noret-noexcept.c            |  16 +
 .../torture/harden-cfr-noret-nothrow.c             |  13 +
 .../c-c++-common/torture/harden-cfr-noret.c        |  38 +
 .../c-c++-common/torture/harden-cfr-notail.c       |   8 +
 .../c-c++-common/torture/harden-cfr-returning.c    |  35 +
 .../c-c++-common/torture/harden-cfr-tail.c         |  55 +-
 gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C  |  11 +
 .../g++.dg/harden-cfr-throw-returning-O0.C         |  10 +
 .../torture/harden-cfr-noret-always-no-nothrow.C   |  16 +
 .../torture/harden-cfr-noret-never-no-nothrow.C    |  18 +
 .../g++.dg/torture/harden-cfr-noret-no-nothrow.C   |  23 +
 .../g++.dg/torture/harden-cfr-throw-always.C       |  20 +
 .../g++.dg/torture/harden-cfr-throw-nocleanup.C    |  11 +
 .../g++.dg/torture/harden-cfr-throw-returning.C    |  31 +
 gcc/testsuite/g++.dg/torture/harden-cfr-throw.C    |  65 ++
 .../gcc.dg/torture/harden-cfr-noret-no-nothrow.c   |  15 +
 gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c  |  40 +
 gcc/testsuite/gnat.dg/hardcfr.adb                  |   2 +-
 libgcc/hardcfr.c                                   | 119 ++-
 32 files changed, 1564 insertions(+), 133 deletions(-)

diff --git a/gcc/common.opt b/gcc/common.opt
index c2512651294..2f37d2fcbd3 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -1798,6 +1798,39 @@ fharden-control-flow-redundancy
 Common Var(flag_harden_control_flow_redundancy) Optimization
 Harden control flow by recording and checking execution paths.
 
+fhardcfr-check-returning-calls
+Common Var(flag_harden_control_flow_redundancy_check_returning_calls) Init(-1) Optimization
+Check CFR execution paths also before calls followed by returns of their results.
+
+fhardcfr-check-exceptions
+Common Var(flag_harden_control_flow_redundancy_check_exceptions) Init(-1) Optimization
+Check CFR execution paths also when exiting a function through an exception.
+
+fhardcfr-check-noreturn-calls=
+Common Joined RejectNegative Enum(hardcfr_check_noreturn_calls) Var(flag_harden_control_flow_redundancy_check_noreturn) Init(HCFRNR_UNSPECIFIED) Optimization
+-fhardcfr-check-noreturn-calls=[always|nothrow|never]	Check CFR execution paths also before calling noreturn functions.
+
+Enum
+Name(hardcfr_check_noreturn_calls) Type(enum hardcfr_noret) UnknownError(unknown hardcfr noreturn checking level %qs)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(never) Value(HCFRNR_NEVER)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(nothrow) Value(HCFRNR_NOTHROW)
+
+; ??? There could be yet another option here, that checked before
+; noreturn calls, except for those known to always throw, if we had
+; means to distinguish noreturn functions known to always throw, such
+; as those used to (re)raise exceptions, from those that merely might
+; throw.  "not always" stands for "not always-throwing", but it also
+; contrasts with "always" below.
+; EnumValue
+; Enum(hardcfr_check_noreturn_calls) String(not-always) Value(HCFRNR_NOT_ALWAYS)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(always) Value(HCFRNR_ALWAYS)
+
 ; Nonzero means ignore `#ident' directives.  0 means handle them.
 ; Generate position-independent code for executables if possible
 ; On SVR4 targets, it also controls whether or not to emit a
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index ea5f22162bb..1ea9525c78d 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -627,7 +627,9 @@ Objective-C and Objective-C++ Dialects}.
 -fsanitize-undefined-trap-on-error  -fbounds-check @gol
 -fcf-protection=@r{[}full@r{|}branch@r{|}return@r{|}none@r{|}check@r{]} @gol
 -fharden-compares -fharden-conditional-branches @gol
--fharden-control-flow-redundancy @gol
+-fharden-control-flow-redundancy  -fhardcfr-check-exceptions  @gol
+-fhardcfr-check-returning-calls  @gol
+-fhardcfr-check-noreturn-calls=@r{[}always@r{|}nothrow@r{|}never@r{]}  @gol
 -fstack-protector  -fstack-protector-all  -fstack-protector-strong @gol
 -fstack-protector-explicit  -fstack-check @gol
 -fstack-limit-register=@var{reg}  -fstack-limit-symbol=@var{sym} @gol
@@ -16644,11 +16646,75 @@ conditionals.
 @item -fharden-control-flow-redundancy
 @opindex fharden-control-flow-redundancy
 Emit extra code to set booleans when entering basic blocks, and to
-verify, at function exits, that they amount to an execution path that is
-consistent with the control flow graph, trapping otherwise.  Tuning
-options @option{--param hardcfr-max-blocks} and @option{--param
+verify and trap, at function exits, when the booleans do not form an
+execution path that is compatible with the control flow graph.
+
+Verification takes place before returns, before mandatory tail calls
+(see below) and, optionally, before escaping exceptions with
+@option{-fhardcfr-check-exceptions}, before returning calls with
+@option{-fhardcfr-check-returning-calls}, and before noreturn calls with
+@option{-fhardcfr-check-noreturn-calls}).  Tuning options
+@option{--param hardcfr-max-blocks} and @option{--param
 hardcfr-max-inline-blocks} are available.
 
+Tail call optimization takes place too late to affect control flow
+redundancy, but calls annotated as mandatory tail calls by language
+front-ends, and any calls marked early enough as potential tail calls
+would also have verification issued before the call, but these
+possibilities are merely theoretical, as these conditions can only be
+met when using custom compiler plugins.
+
+@item -fhardcfr-check-exceptions
+@opindex fhardcfr-check-exceptions
+@opindex fno-hardcfr-check-exceptions
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph at exception
+escape points, as if the function body was wrapped with a cleanup
+handler that performed the check and reraised.  This option is enabled
+by default; use @option{-fno-hardcfr-check-exceptions} to disable it.
+
+@item -fhardcfr-check-returning-calls
+@opindex fhardcfr-check-returning-calls
+@opindex fno-hardcfr-check-returning-calls
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph before any
+function call immediately followed by a return of its result, if any, so
+as to not prevent tail-call optimization, whether or not it is
+ultimately optimized to a tail call.
+
+This option is enabled by default whenever sibling call optimizations
+are enabled (see @option{-foptimize-sibling-calls}), but it can be
+enabled (or disabled, using its negated form) explicitly, regardless of
+the optimizations.
+
+@item -fhardcfr-check-noreturn-calls=@r{[}always@r{|}nothrow@r{|}never@r{]}
+@opindex fhardcfr-check-noreturn-calls
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph before
+@code{noreturn} calls, either all of them (@option{always}), those that
+may not return control to the caller through an exception either
+(@option{nothrow}), or none of them (@option{never}, the default).
+
+Checking before a @code{noreturn} function that may return control to
+the caller through an exception may cause checking to be performed more
+than once, if the exception is caught in the caller, whether by a
+handler or a cleanup.  When @option{-fhardcfr-check-exceptions} is also
+enabled, the compiler will avoid associating a @code{noreturn} call with
+the implicitly-added cleanup handler, since it would be redundant with
+the check performed before the call, but other handlers or cleanups in
+the function, if activated, will modify the recorded execution path and
+check it again when another checkpoint is hit.  The checkpoint may even
+be another @code{noreturn} call, so checking may end up performed
+multiple times.
+
+Various optimizers may cause calls to be marked as @code{noreturn}
+and/or @code{nothrow}, even in the absence of the corresponding
+attributes, which may affect the placement of checks before calls, as
+well as the addition of implicit cleanup handlers for them.  This
+unpredictability, and the fact that raising and reraising exceptions
+frequently amounts to implicitly calling @code{noreturn} functions, have
+made @option{never} the default setting for this option.
+
 @item -fstack-protector
 @opindex fstack-protector
 Emit extra code to check for buffer overflows, such as stack smashing
diff --git a/gcc/flag-types.h b/gcc/flag-types.h
index b90c85167dc..b6d2294d2d1 100644
--- a/gcc/flag-types.h
+++ b/gcc/flag-types.h
@@ -157,6 +157,16 @@ enum stack_reuse_level
   SR_ALL
 };
 
+/* Control Flow Redundancy hardening options for noreturn calls.  */
+enum hardcfr_noret
+{
+  HCFRNR_NEVER,
+  HCFRNR_NOTHROW,
+  HCFRNR_NOT_ALWAYS, /* Reserved for future use.  */
+  HCFRNR_ALWAYS,
+  HCFRNR_UNSPECIFIED = -1
+};
+
 /* The live patching level.  */
 enum live_patching_level
 {
diff --git a/gcc/gimple-harden-control-flow.cc b/gcc/gimple-harden-control-flow.cc
index 6b08846dbb1..fe322f021ee 100644
--- a/gcc/gimple-harden-control-flow.cc
+++ b/gcc/gimple-harden-control-flow.cc
@@ -29,7 +29,12 @@ along with GCC; see the file COPYING3.  If not see
 #include "tree-pass.h"
 #include "ssa.h"
 #include "gimple-iterator.h"
+#include "gimple-pretty-print.h"
 #include "tree-cfg.h"
+#include "tree-cfgcleanup.h"
+#include "tree-eh.h"
+#include "except.h"
+#include "sbitmap.h"
 #include "basic-block.h"
 #include "cfghooks.h"
 #include "cfgloop.h"
@@ -60,9 +65,7 @@ const pass_data pass_data_harden_control_flow_redundancy = {
   0,	    // properties_provided
   0,	    // properties_destroyed
   TODO_cleanup_cfg, // properties_start
-  TODO_update_ssa
-  | TODO_cleanup_cfg
-  | TODO_verify_il, // properties_finish
+  0,        // properties_finish
 };
 
 class pass_harden_control_flow_redundancy : public gimple_opt_pass
@@ -79,16 +82,6 @@ public:
     if (!flag_harden_control_flow_redundancy)
       return false;
 
-    /* We don't verify when an exception escapes, propagated or raised
-       by the function itself, so we're only concerned with edges to
-       the exit block.  If there aren't any, the function doesn't
-       return normally, so there won't be any checking point, so
-       there's no point in running the pass.  Should we add
-       verification at exception escapes, we should at least look at
-       !flag_exceptions here.  */
-    if (EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (fun)->preds) == 0)
-      return false;
-
     /* Functions that return more than once, like setjmp and vfork
        (that also gets this flag set), will start recording a path
        after the first return, and then may take another path when
@@ -117,8 +110,9 @@ public:
 	return false;
       }
 
-    if (param_hardcfr_max_blocks > 0
-	&& n_basic_blocks_for_fn (fun) - 2 > param_hardcfr_max_blocks)
+    if (fun->cfg && param_hardcfr_max_blocks > 0
+	&& (n_basic_blocks_for_fn (fun) - NUM_FIXED_BLOCKS
+	    > param_hardcfr_max_blocks))
       {
 	warning_at (DECL_SOURCE_LOCATION (fun->decl), 0,
 		    "%qD has more than %u blocks, the requested"
@@ -134,6 +128,293 @@ public:
 
 }
 
+/* Return TRUE iff CFR checks should be inserted before returning
+   calls.  */
+
+static bool
+check_returning_calls_p ()
+{
+  return
+    flag_harden_control_flow_redundancy_check_returning_calls > 0
+    || (flag_harden_control_flow_redundancy_check_returning_calls < 0
+	/* Gates pass_tail_calls.  */
+	&& flag_optimize_sibling_calls
+	/* Gates pass_all_optimizations.  */
+	&& optimize >= 1 && !optimize_debug);
+}
+
+/* Scan BB from the end, updating *RETPTR if given as return stmts and
+   copies are found.  Return a call or a stmt that cannot appear after
+   a tail call, or NULL if the top of the block is reached without
+   finding any.  */
+
+static gimple *
+hardcfr_scan_block (basic_block bb, tree **retptr)
+{
+  gimple_stmt_iterator gsi;
+  for (gsi = gsi_last_bb (bb); !gsi_end_p (gsi); gsi_prev (&gsi))
+    {
+      gimple *stmt = gsi_stmt (gsi);
+
+      /* Ignore labels, returns, nops, clobbers and debug stmts.  */
+      if (gimple_code (stmt) == GIMPLE_LABEL
+	  || gimple_code (stmt) == GIMPLE_NOP
+	  || gimple_code (stmt) == GIMPLE_PREDICT
+	  || gimple_clobber_p (stmt)
+	  || is_gimple_debug (stmt))
+	continue;
+
+      if (gimple_code (stmt) == GIMPLE_RETURN)
+	{
+	  greturn *gret = as_a <greturn *> (stmt);
+	  if (retptr)
+	    {
+	      gcc_checking_assert (!*retptr);
+	      *retptr = gimple_return_retval_ptr (gret);
+	    }
+	  continue;
+	}
+
+      /* Check for a call.  */
+      if (is_gimple_call (stmt))
+	return stmt;
+
+      /* Allow simple copies to the return value, updating the return
+	 value to be found in earlier assignments.  */
+      if (retptr && *retptr && gimple_assign_single_p (stmt)
+	  && **retptr == gimple_assign_lhs (stmt))
+	{
+	  *retptr = gimple_assign_rhs1_ptr (stmt);
+	  continue;
+	}
+
+      return stmt;
+    }
+
+  /* Any other kind of stmt will prevent a tail call.  */
+  return NULL;
+}
+
+/* Return TRUE iff CALL is to be preceded by a CFR checkpoint, i.e.,
+   if it's a returning call (one whose result is ultimately returned
+   without intervening non-copy statements) and we're checking
+   returning calls, a __builtin_return call (noreturn with a path to
+   the exit block), a must-tail call, or a tail call.  */
+
+static bool
+returning_call_p (gcall *call)
+{
+  if (!(gimple_call_noreturn_p (call)
+	|| gimple_call_must_tail_p (call)
+	|| gimple_call_tail_p (call)
+	|| check_returning_calls_p ()))
+    return false;
+
+  /* Quickly check that there's a path to exit compatible with a
+     returning call.  Detect infinite loops through the counter.  */
+  basic_block bb = gimple_bb (call);
+  auto_vec<basic_block, 10> path;
+  for (int i = n_basic_blocks_for_fn (cfun);
+       bb != EXIT_BLOCK_PTR_FOR_FN (cfun) && i--;
+       bb = single_succ (bb))
+    if (!single_succ_p (bb)
+	|| (single_succ_edge (bb)->flags & EDGE_EH) != 0)
+      return false;
+    else
+      path.safe_push (bb);
+
+  /* Check the stmts in the blocks and trace the return value.  */
+  tree *retptr = NULL;
+  for (;;)
+    {
+      gcc_checking_assert (!path.is_empty ());
+      basic_block bb = path.pop ();
+      gimple *stop = hardcfr_scan_block (bb, &retptr);
+      if (stop)
+	{
+	  if (stop != call)
+	    return false;
+	  gcc_checking_assert (path.is_empty ());
+	  break;
+	}
+
+      gphi *retphi = NULL;
+      if (retptr && *retptr && TREE_CODE (*retptr) == SSA_NAME
+	  && !SSA_NAME_IS_DEFAULT_DEF (*retptr)
+	  && SSA_NAME_DEF_STMT (*retptr)
+	  && is_a <gphi *> (SSA_NAME_DEF_STMT (*retptr))
+	  && gimple_bb (SSA_NAME_DEF_STMT (*retptr)) == bb)
+	{
+	  retphi = as_a <gphi *> (SSA_NAME_DEF_STMT (*retptr));
+	  gcc_checking_assert (gimple_phi_result (retphi) == *retptr);
+	}
+      else
+	continue;
+
+      gcc_checking_assert (!path.is_empty ());
+      edge e = single_succ_edge (path.last ());
+      int i = EDGE_COUNT (bb->preds);
+      while (i--)
+	if (EDGE_PRED (bb, i) == e)
+	  break;
+      gcc_checking_assert (i >= 0);
+      retptr = gimple_phi_arg_def_ptr (retphi, i);
+    }
+
+  return (gimple_call_noreturn_p (call)
+	  || gimple_call_must_tail_p (call)
+	  || gimple_call_tail_p (call)
+	  || (gimple_call_lhs (call) == (retptr ? *retptr : NULL)
+	      && check_returning_calls_p ()));
+}
+
+typedef auto_vec<edge, 10> chk_edges_t;
+
+/* Declare for mutual recursion.  */
+static bool hardcfr_sibcall_search_preds (basic_block bb,
+					  chk_edges_t &chk_edges,
+					  int &count_chkcall,
+					  auto_sbitmap &chkcall_blocks,
+					  int &count_postchk,
+					  auto_sbitmap &postchk_blocks,
+					  tree *retptr);
+
+/* Search backwards from the end of BB for a mandatory or potential
+   sibcall.  Schedule the block to be handled sort-of like noreturn if
+   so.  Recurse to preds, with updated RETPTR, if the block only
+   contains stmts that may follow such a call, scheduling checking at
+   edges and marking blocks as post-check as needed.  Return true iff,
+   at the end of the block, a check will have already been
+   performed.  */
+
+static bool
+hardcfr_sibcall_search_block (basic_block bb,
+			      chk_edges_t &chk_edges,
+			      int &count_chkcall,
+			      auto_sbitmap &chkcall_blocks,
+			      int &count_postchk,
+			      auto_sbitmap &postchk_blocks,
+			      tree *retptr)
+{
+  /* Conditionals and internal exceptions rule out tail calls.  */
+  if (!single_succ_p (bb)
+      || (single_succ_edge (bb)->flags & EDGE_EH) != 0)
+    return false;
+
+  gimple *stmt = hardcfr_scan_block (bb, &retptr);
+  if (!stmt)
+    return hardcfr_sibcall_search_preds (bb, chk_edges,
+					 count_chkcall, chkcall_blocks,
+					 count_postchk, postchk_blocks,
+					 retptr);
+
+  if (!is_a <gcall *> (stmt))
+    return false;
+
+  /* Avoid disrupting mandatory or early-marked tail calls,
+     inserting the check before them.  This works for
+     must-tail calls, but tail calling as an optimization is
+     detected too late for us.
+
+     Also check for noreturn calls here.  Noreturn calls won't
+     normally have edges to exit, so they won't be found here,
+     but __builtin_return does, and we must check before
+     it, so handle it like a tail call.  */
+  gcall *call = as_a <gcall *> (stmt);
+  if (!(gimple_call_noreturn_p (call)
+	|| gimple_call_must_tail_p (call)
+	|| gimple_call_tail_p (call)
+	|| (gimple_call_lhs (call) == (retptr ? *retptr : NULL)
+	    && check_returning_calls_p ())))
+    return false;
+
+  gcc_checking_assert (returning_call_p (call));
+
+  /* We found a call that is to be preceded by checking.  */
+  if (bitmap_set_bit (chkcall_blocks, bb->index))
+    ++count_chkcall;
+  else
+    gcc_unreachable ();
+  return true;
+}
+
+
+/* Search preds of BB for a mandatory or potential sibcall or
+   returning call, and arrange for the blocks containing them to have
+   a check inserted before the call, like noreturn calls.  If any
+   preds are found to perform checking, schedule checks at the edges
+   of those that don't, and mark BB as postcheck..  */
+
+static bool
+hardcfr_sibcall_search_preds (basic_block bb,
+			      chk_edges_t &chk_edges,
+			      int &count_chkcall,
+			      auto_sbitmap &chkcall_blocks,
+			      int &count_postchk,
+			      auto_sbitmap &postchk_blocks,
+			      tree *retptr)
+{
+  /* For the exit block, we wish to force a check at every
+     predecessor, so pretend we've already found a pred that had
+     checking, so that we schedule checking at every one of its pred
+     edges.  */
+  bool first = bb->index >= NUM_FIXED_BLOCKS;
+  bool postchecked = true;
+
+  gphi *retphi = NULL;
+  if (retptr && *retptr && TREE_CODE (*retptr) == SSA_NAME
+      && !SSA_NAME_IS_DEFAULT_DEF (*retptr)
+      && SSA_NAME_DEF_STMT (*retptr)
+      && is_a <gphi *> (SSA_NAME_DEF_STMT (*retptr))
+      && gimple_bb (SSA_NAME_DEF_STMT (*retptr)) == bb)
+    {
+      retphi = as_a <gphi *> (SSA_NAME_DEF_STMT (*retptr));
+      gcc_checking_assert (gimple_phi_result (retphi) == *retptr);
+    }
+
+  for (int i = EDGE_COUNT (bb->preds); i--; first = false)
+    {
+      edge e = EDGE_PRED (bb, i);
+
+      bool checked
+	= hardcfr_sibcall_search_block (e->src, chk_edges,
+					count_chkcall, chkcall_blocks,
+					count_postchk, postchk_blocks,
+					!retphi ? retptr
+					: gimple_phi_arg_def_ptr (retphi, i));
+
+      if (first)
+	{
+	  postchecked = checked;
+	  continue;
+	}
+
+      /* When we first find a checked block, force a check at every
+	 other incoming edge we've already visited, and those we
+	 visit afterwards that don't have their own check, so that
+	 when we reach BB, the check has already been performed.  */
+      if (!postchecked && checked)
+	{
+	  for (int j = EDGE_COUNT (bb->preds); --j > i; )
+	    chk_edges.safe_push (EDGE_PRED (bb, j));
+	  postchecked = true;
+	}
+      if (postchecked && !checked)
+	chk_edges.safe_push (EDGE_PRED (bb, i));
+    }
+
+  if (postchecked && bb->index >= NUM_FIXED_BLOCKS)
+    {
+      if (bitmap_set_bit (postchk_blocks, bb->index))
+	count_postchk++;
+      else
+	gcc_unreachable ();
+    }
+
+  return postchecked;
+}
+
+
 class rt_bb_visited
 {
   /* Use a sufficiently wide unsigned type to hold basic block numbers.  */
@@ -176,8 +457,8 @@ class rt_bb_visited
      neither ENTRY nor EXIT, but maybe one-past-the-end, to compute
      the visited array length.  */
   blknum num2idx (blknum n) {
-    gcc_checking_assert (n >= 2 && n <= nblocks);
-    return (n - 2);
+    gcc_checking_assert (n >= NUM_FIXED_BLOCKS && n <= nblocks);
+    return (n - NUM_FIXED_BLOCKS);
   }
   /* Return the block vindex for BB, that must not be ENTRY or
      EXIT.  */
@@ -249,8 +530,7 @@ class rt_bb_visited
   }
 
   /* Set the bit corresponding to BB in VISITED.  Add to SEQ any
-     required gimple statements, and return SEQ, possibly
-     modified.  */
+     required gimple stmts, and return SEQ, possibly modified.  */
   gimple_seq vset (basic_block bb, gimple_seq seq = NULL)
   {
     tree bit, setme = vword (bb, &bit);
@@ -270,7 +550,7 @@ class rt_bb_visited
 
 public:
   /* Prepare to add control flow redundancy testing to CFUN.  */
-  rt_bb_visited ()
+  rt_bb_visited (int checkpoints)
     : nblocks (n_basic_blocks_for_fn (cfun)),
       vword_type (NULL), ckseq (NULL), rtcfg (NULL)
   {
@@ -353,8 +633,8 @@ public:
 					 NULL, NULL);
     gimple_seq_add_stmt (&ckseq, detach);
 
-    if (nblocks - 2 > blknum (param_hardcfr_max_inline_blocks)
-	|| !single_pred_p (EXIT_BLOCK_PTR_FOR_FN (cfun)))
+    if (nblocks - NUM_FIXED_BLOCKS > blknum (param_hardcfr_max_inline_blocks)
+	|| checkpoints > 1)
       {
 	/* Make sure vword_bits is wide enough for the representation
 	   of nblocks in rtcfg.  Compare with vword_bits << vword_bits,
@@ -379,65 +659,33 @@ public:
     gimple_seq_add_stmt (&ckseq, ckfail_init);
   }
 
-  /* Insert SEQ on E, or close enough (e.g., before a noreturn or tail
-     call at the end of E->src).  */
-  void insert_exit_check (gimple_seq seq, edge e)
+  /* Insert SEQ before a resx or a call in INSBB.  */
+  void insert_exit_check_in_block (gimple_seq seq, basic_block insbb)
   {
-    basic_block insbb = e->src;
-
-    /* If the returning block ends with a noreturn call, insert
-       checking before it.  This is particularly important for
-       __builtin_return.  Other noreturn calls won't have an edge to
-       the exit block, so they won't even be considered as exit paths.
-
-       Insert-on-edge inserts before other return stmts, but not
-       before calls, and if a single-block function had the check
-       sequence inserted after a noreturn call, it would be useless,
-       but checking would still flag it as malformed if block 2 has a
-       fallthru edge to the exit block.
-
-       Also avoid disrupting tail calls, inserting the check before
-       them.  This works for must-tail calls, but tail calling as an
-       optimization is detected too late for us.  */
     gimple_stmt_iterator gsi = gsi_last_bb (insbb);
-    gimple *ret = gsi_stmt (gsi);
-    if (ret && is_a <greturn *> (ret))
-      {
+
+    while (!gsi_end_p (gsi))
+      if (is_a <gresx *> (gsi_stmt (gsi))
+	  || is_a <gcall *> (gsi_stmt (gsi)))
+	break;
+      else
 	gsi_prev (&gsi);
-	if (!gsi_end_p (gsi))
-	  ret = gsi_stmt (gsi);
-      }
-    if (ret && is_a <gcall *> (ret)
-	&& (gimple_call_noreturn_p (ret)
-	    || gimple_call_must_tail_p (as_a <gcall *> (ret))
-	    || gimple_call_tail_p (as_a <gcall *> (ret))))
-      gsi_insert_seq_before (&gsi, seq, GSI_SAME_STMT);
-    else
-      gsi_insert_seq_on_edge_immediate (e, seq);
+
+    gsi_insert_seq_before (&gsi, seq, GSI_SAME_STMT);
+  }
+
+  /* Insert SEQ on E.  */
+  void insert_exit_check_on_edge (gimple_seq seq, edge e)
+  {
+    gsi_insert_seq_on_edge_immediate (e, seq);
   }
 
-  /* Add checking code on every exit edge, and initialization code on
+  /* Add checking code to CHK_EDGES, and initialization code on
      the entry edge.  Before this point, the CFG has been undisturbed,
      and all the needed data has been collected and safely stowed.  */
-  void check ()
+  void check (chk_edges_t &chk_edges,
+	      int count_chkcall, auto_sbitmap const &chkcall_blocks)
   {
-    /* Insert initializers for visited at the entry.  */
-    gimple_seq iseq = NULL;
-
-    gcall *vinit = gimple_build_call (builtin_decl_explicit
-				      (BUILT_IN_MEMSET), 3,
-				      build1 (ADDR_EXPR,
-					      build_pointer_type
-					      (TREE_TYPE (visited)),
-					      visited),
-				      integer_zero_node,
-				      TYPE_SIZE_UNIT (TREE_TYPE (visited)));
-    gimple_seq_add_stmt (&iseq, vinit);
-
-    gsi_insert_seq_on_edge_immediate (single_succ_edge
-				      (ENTRY_BLOCK_PTR_FOR_FN (cfun)),
-				      iseq);
-
     /* If we're using out-of-line checking, create and statically
        initialize the CFG checking representation, generate the
        checker call for the checking sequence, and insert it in all
@@ -498,28 +746,116 @@ public:
 						     rtcfg));
 	gimple_seq_add_stmt (&ckseq, call_chk);
 
+	gimple *clobber = gimple_build_assign (visited,
+					       build_clobber
+					       (TREE_TYPE (visited)));
+	gimple_seq_add_stmt (&ckseq, clobber);
+
 	/* If we have multiple exit edges, insert (copies of)
 	   ckseq in all of them.  */
-	for (int i = EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (cfun)->preds);
-	     i--; )
+	for (int i = chk_edges.length (); i--; )
 	  {
 	    gimple_seq seq = ckseq;
 	    /* Copy the sequence, unless we're dealing with the
 	       last edge (we're counting down to zero).  */
-	    if (i)
+	    if (i || count_chkcall)
+	      seq = gimple_seq_copy (seq);
+
+	    edge e = chk_edges[i];
+
+	    if (dump_file)
+	      {
+		if (e->dest == EXIT_BLOCK_PTR_FOR_FN (cfun))
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to exit.\n",
+			   e->src->index);
+		else
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to postcheck block %i.\n",
+			   e->src->index, e->dest->index);
+	      }
+
+	    insert_exit_check_on_edge (seq, e);
+
+	    gcc_checking_assert (!bitmap_bit_p (chkcall_blocks, e->src->index));
+	  }
+
+	sbitmap_iterator it;
+	unsigned i;
+	EXECUTE_IF_SET_IN_BITMAP (chkcall_blocks, 0, i, it)
+	  {
+	    basic_block bb = BASIC_BLOCK_FOR_FN (cfun, i);
+
+	    gimple_seq seq = ckseq;
+	    gcc_checking_assert (count_chkcall > 0);
+	    if (--count_chkcall)
 	      seq = gimple_seq_copy (seq);
 
-	    insert_exit_check (seq,
-			       EDGE_PRED (EXIT_BLOCK_PTR_FOR_FN (cfun), i));
+	    if (dump_file)
+	      fprintf (dump_file,
+		       "Inserting out-of-line check before stmt in block %i.\n",
+		       bb->index);
+
+	    insert_exit_check_in_block (seq, bb);
 	  }
+
+	gcc_checking_assert (count_chkcall == 0);
       }
     else
       {
 	/* Inline checking requires a single exit edge.  */
-	gimple *last = gsi_stmt (gsi_last (ckseq));
+	gimple *last = gimple_build_assign (visited,
+					       build_clobber
+					       (TREE_TYPE (visited)));
+	gimple_seq_add_stmt (&ckseq, last);
 
-	insert_exit_check (ckseq,
-			   single_pred_edge (EXIT_BLOCK_PTR_FOR_FN (cfun)));
+	if (!count_chkcall)
+	  {
+	    edge e = single_pred_edge (EXIT_BLOCK_PTR_FOR_FN (cfun));
+
+	    if (dump_file)
+	      {
+		if (e->dest == EXIT_BLOCK_PTR_FOR_FN (cfun))
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to postcheck block %i.\n",
+			   e->src->index, e->dest->index);
+		else
+		  fprintf (dump_file,
+			   "Inserting inline check in"
+			   " block %i's edge to exit.\n",
+			   e->src->index);
+	      }
+
+	    insert_exit_check_on_edge (ckseq, e);
+	  }
+	else
+	  {
+	    gcc_checking_assert (count_chkcall == 1);
+
+	    sbitmap_iterator it;
+	    unsigned i;
+	    EXECUTE_IF_SET_IN_BITMAP (chkcall_blocks, 0, i, it)
+	      {
+		basic_block bb = BASIC_BLOCK_FOR_FN (cfun, i);
+
+		gimple_seq seq = ckseq;
+		gcc_checking_assert (count_chkcall > 0);
+		if (--count_chkcall)
+		  seq = gimple_seq_copy (seq);
+
+		if (dump_file)
+		  fprintf (dump_file,
+			   "Inserting inline check before stmt in block %i.\n",
+			   bb->index);
+
+		insert_exit_check_in_block (seq, bb);
+	      }
+
+	    gcc_checking_assert (count_chkcall == 0);
+	  }
 
 	/* The inserted ckseq computes CKFAIL at LAST.  Now we have to
 	   conditionally trap on it.  */
@@ -540,8 +876,7 @@ public:
 	  add_bb_to_loop (trp, current_loops->tree_root);
 
 	/* Insert a conditional branch to the trap block.  If the
-	   conditional wouldn't be the last statement, split the
-	   block.  */
+	   conditional wouldn't be the last stmt, split the block.  */
 	gimple_stmt_iterator gsi = gsi_for_stmt (last);
 	if (!gsi_one_before_end_p (gsi))
 	  split_block (gsi_bb (gsi), gsi_stmt (gsi));
@@ -564,6 +899,24 @@ public:
 	if (dom_info_available_p (CDI_DOMINATORS))
 	  set_immediate_dominator (CDI_DOMINATORS, trp, gimple_bb (last));
       }
+
+    /* Insert initializers for visited at the entry.  Do this after
+       other insertions, to avoid messing with block numbers.  */
+    gimple_seq iseq = NULL;
+
+    gcall *vinit = gimple_build_call (builtin_decl_explicit
+				      (BUILT_IN_MEMSET), 3,
+				      build1 (ADDR_EXPR,
+					      build_pointer_type
+					      (TREE_TYPE (visited)),
+					      visited),
+				      integer_zero_node,
+				      TYPE_SIZE_UNIT (TREE_TYPE (visited)));
+    gimple_seq_add_stmt (&iseq, vinit);
+
+    gsi_insert_seq_on_edge_immediate (single_succ_edge
+				      (ENTRY_BLOCK_PTR_FOR_FN (cfun)),
+				      iseq);
   }
 
   /* Push onto RTCFG a (mask, index) pair to test for IBB when BB is
@@ -607,7 +960,7 @@ public:
     return false;
   }
 
-  /* Add to CKSEQ statements to clear CKPART if OBB is visited.  */
+  /* Add to CKSEQ stmts to clear CKPART if OBB is visited.  */
   void
   build_block_check (basic_block obb)
   {
@@ -632,34 +985,48 @@ public:
 
   /* Add to BB code to set its bit in VISITED, and add to RTCFG or
      CKSEQ the data or code needed to check BB's predecessors and
-     successors.  Do NOT change the CFG.  */
-  void visit (basic_block bb)
+     successors.  If CHECKPOINT, assume the block is a checkpoint,
+     whether or not it has an edge to EXIT.  If POSTCHECK, assume the
+     block post-dominates checkpoints and therefore no bitmap setting
+     or checks are to be performed in or for it.  Do NOT change the
+     CFG.  */
+  void visit (basic_block bb, bool checkpoint, bool postcheck)
   {
     /* Set the bit in VISITED when entering the block.  */
     gimple_stmt_iterator gsi = gsi_after_labels (bb);
-    gsi_insert_seq_before (&gsi, vset (bb), GSI_SAME_STMT);
+    if (!postcheck)
+      gsi_insert_seq_before (&gsi, vset (bb), GSI_SAME_STMT);
 
     if (rtcfg)
       {
-	/* Build a list of (index, mask) terminated by (NULL, 0).
-	   Consolidate masks with the same index when they're
-	   adjacent.  First, predecessors.  Count backwards, because
-	   we're going to reverse the list.  The order shouldn't
-	   matter, but let's not make it surprising.  */
-	for (int i = EDGE_COUNT (bb->preds); i--; )
-	  if (push_rtcfg_pair (EDGE_PRED (bb, i)->src, bb,
-			       ENTRY_BLOCK_PTR_FOR_FN (cfun)))
-	    break;
+	if (!postcheck)
+	  {
+	    /* Build a list of (index, mask) terminated by (NULL, 0).
+	       Consolidate masks with the same index when they're
+	       adjacent.  First, predecessors.  Count backwards, because
+	       we're going to reverse the list.  The order shouldn't
+	       matter, but let's not make it surprising.  */
+	    for (int i = EDGE_COUNT (bb->preds); i--; )
+	      if (push_rtcfg_pair (EDGE_PRED (bb, i)->src, bb,
+				   ENTRY_BLOCK_PTR_FOR_FN (cfun)))
+		break;
+	  }
 	rtcfg = tree_cons (NULL_TREE, build_int_cst (vword_type, 0), rtcfg);
 
-	/* Then, successors.  */
-	for (int i = EDGE_COUNT (bb->succs); i--; )
-	  if (push_rtcfg_pair (EDGE_SUCC (bb, i)->dest, bb,
-			       EXIT_BLOCK_PTR_FOR_FN (cfun)))
-	    break;
+	if (!postcheck)
+	  {
+	    /* Then, successors.  */
+	    if (!checkpoint
+		|| !push_rtcfg_pair (EXIT_BLOCK_PTR_FOR_FN (cfun),
+				     bb, EXIT_BLOCK_PTR_FOR_FN (cfun)))
+	      for (int i = EDGE_COUNT (bb->succs); i--; )
+		if (push_rtcfg_pair (EDGE_SUCC (bb, i)->dest, bb,
+				     EXIT_BLOCK_PTR_FOR_FN (cfun)))
+		  break;
+	  }
 	rtcfg = tree_cons (NULL_TREE, build_int_cst (vword_type, 0), rtcfg);
       }
-    else
+    else if (!postcheck)
       {
 	/* Schedule test to fail if the block was reached but somehow none
 	   of its predecessors were.  */
@@ -677,6 +1044,8 @@ public:
 	gassign *blkruns = gimple_build_assign (ckpart, unshare_expr (bit));
 	gimple_seq_add_stmt (&ckseq, blkruns);
 
+	if (checkpoint)
+	  build_block_check (EXIT_BLOCK_PTR_FOR_FN (cfun));
 	for (int i = 0, e = EDGE_COUNT (bb->succs); i < e; i++)
 	  build_block_check (EDGE_SUCC (bb, i)->dest);
 
@@ -687,21 +1056,355 @@ public:
   }
 };
 
+/* It might be useful to avoid checking before noreturn calls that are
+   known to always finish by throwing an exception, rather than by
+   ending the program or looping forever.  Such functions would have
+   to be annotated somehow, with an attribute or flag, so that
+   exception-raising functions, such as C++'s __cxa_throw,
+   __cxa_rethrow, and Ada's gnat_rcheck_*, gnat_reraise*,
+   ada.exception.raise_exception*, and the language-independent
+   unwinders could be detected here and handled differently from other
+   noreturn functions.  */
+static bool
+always_throwing_noreturn_call_p (gimple *)
+{
+  return false;
+}
+
 /* Control flow redundancy hardening: record the execution path, and
    verify at exit that an expect path was taken.  */
 
 unsigned int
-pass_harden_control_flow_redundancy::execute (function *)
+pass_harden_control_flow_redundancy::execute (function *fun)
 {
-  rt_bb_visited vstd;
-
+  bool const check_at_escaping_exceptions
+    = (flag_exceptions
+       && flag_harden_control_flow_redundancy_check_exceptions);
+  bool const check_before_noreturn_calls
+    = flag_harden_control_flow_redundancy_check_noreturn > HCFRNR_NEVER;
+  bool const check_before_nothrow_noreturn_calls
+    = (check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn >= HCFRNR_NOTHROW);
+  bool const check_before_throwing_noreturn_calls
+    = (flag_exceptions
+       && check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn > HCFRNR_NOTHROW);
+  bool const check_before_always_throwing_noreturn_calls
+    = (flag_exceptions
+       && check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn >= HCFRNR_ALWAYS);
   basic_block bb;
-  FOR_EACH_BB_FN (bb, cfun)
-    vstd.visit (bb);
+  basic_block bb_eh_cleanup = NULL;
+
+  if (check_at_escaping_exceptions)
+    {
+      int lp_eh_cleanup = -1;
+
+      /* Record the preexisting blocks, to avoid visiting newly-created
+	 blocks.  */
+      auto_sbitmap to_visit (last_basic_block_for_fn (fun));
+      bitmap_clear (to_visit);
+
+      FOR_EACH_BB_FN (bb, fun)
+	bitmap_set_bit (to_visit, bb->index);
+
+      /* Scan the blocks for stmts with escaping exceptions, that
+	 wouldn't be denoted in the CFG, and associate them with an
+	 empty cleanup handler around the whole function.  Walk
+	 backwards, so that even when we split the block, */
+      sbitmap_iterator it;
+      unsigned i;
+      EXECUTE_IF_SET_IN_BITMAP (to_visit, 0, i, it)
+	{
+	  bb = BASIC_BLOCK_FOR_FN (fun, i);
+
+	  for (gimple_stmt_iterator gsi = gsi_last_bb (bb);
+	       !gsi_end_p (gsi); gsi_prev (&gsi))
+	    {
+	      gimple *stmt = gsi_stmt (gsi);
+	      if (!stmt_could_throw_p (fun, stmt))
+		continue;
+
+	      /* If it must not throw, or if it already has a handler,
+		 we need not worry about it.  */
+	      if (lookup_stmt_eh_lp (stmt) != 0)
+		continue;
+
+	      /* Don't split blocks at, nor add EH edges to, tail
+		 calls, we will add verification before the call
+		 anyway.  */
+	      if (is_a <gcall *> (stmt)
+		  && (gimple_call_must_tail_p (as_a <gcall *> (stmt))
+		      || gimple_call_tail_p (as_a <gcall *> (stmt))
+		      || returning_call_p (as_a <gcall *> (stmt))))
+		continue;
+
+	      if (!gsi_one_before_end_p (gsi))
+		split_block (bb, stmt);
+	      /* A resx or noreturn call needs not be associated with
+		 the cleanup handler if we're going to add checking
+		 before it.  We only test cases that didn't require
+		 block splitting because noreturn calls would always
+		 be at the end of blocks, and we test for zero
+		 successors because if there is an edge, it's not
+		 noreturn, as any EH edges would have already been
+		 caught by the lookup_stmt_eh_lp test above.  */
+	      else if (check_before_noreturn_calls
+		       && EDGE_COUNT (bb->succs) == 0
+		       && (is_a <gresx *> (stmt)
+			   ? check_before_always_throwing_noreturn_calls
+			   : (!is_a <gcall *> (stmt)
+			      || !gimple_call_noreturn_p (stmt))
+			   ? (gcc_unreachable (), false)
+			   : (!flag_exceptions
+			      || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+			   ? check_before_nothrow_noreturn_calls
+			   : always_throwing_noreturn_call_p (stmt)
+			   ? check_before_always_throwing_noreturn_calls
+			   : check_before_throwing_noreturn_calls))
+		{
+		  if (dump_file)
+		    {
+		      fprintf (dump_file,
+			       "Bypassing cleanup for noreturn stmt"
+			       " in block %i:\n",
+			       bb->index);
+		      print_gimple_stmt (dump_file, stmt, 0);
+		    }
+		  continue;
+		}
+
+	      if (!bb_eh_cleanup)
+		{
+		  bb_eh_cleanup = create_empty_bb (bb);
+		  if (dom_info_available_p (CDI_DOMINATORS))
+		    set_immediate_dominator (CDI_DOMINATORS, bb_eh_cleanup, bb);
+		  if (current_loops)
+		    add_bb_to_loop (bb_eh_cleanup, current_loops->tree_root);
+
+		  /* Make the new block an EH cleanup for the call.  */
+		  eh_region new_r = gen_eh_region_cleanup (NULL);
+		  eh_landing_pad lp = gen_eh_landing_pad (new_r);
+		  tree label = gimple_block_label (bb_eh_cleanup);
+		  lp->post_landing_pad = label;
+		  EH_LANDING_PAD_NR (label) = lp_eh_cleanup = lp->index;
+
+		  /* Just propagate the exception.
+		     We will later insert the verifier call.  */
+		  gimple_stmt_iterator ehgsi;
+		  ehgsi = gsi_after_labels (bb_eh_cleanup);
+		  gresx *resx = gimple_build_resx (new_r->index);
+		  gsi_insert_before (&ehgsi, resx, GSI_SAME_STMT);
+
+		  if (dump_file)
+		    fprintf (dump_file,
+			     "Created cleanup block %i:\n",
+			     bb_eh_cleanup->index);
+		}
+	      else if (dom_info_available_p (CDI_DOMINATORS))
+		{
+		  basic_block immdom;
+		  immdom = get_immediate_dominator (CDI_DOMINATORS,
+						    bb_eh_cleanup);
+		  if (!dominated_by_p (CDI_DOMINATORS, bb, immdom))
+		    {
+		      immdom = nearest_common_dominator (CDI_DOMINATORS,
+							 immdom, bb);
+		      set_immediate_dominator (CDI_DOMINATORS,
+					       bb_eh_cleanup, immdom);
+		    }
+		}
+
+	      if (dump_file)
+		{
+		  fprintf (dump_file,
+			   "Associated cleanup block with stmt in block %i:\n",
+			   bb->index);
+		  print_gimple_stmt (dump_file, stmt, 0);
+		}
+
+	      add_stmt_to_eh_lp (stmt, lp_eh_cleanup);
+	      /* Finally, wire the EH cleanup block into the CFG.  */
+	      make_eh_edges (stmt);
+	    }
+	}
+
+      if (bb_eh_cleanup)
+	{
+	  /* A cfg_cleanup after bb_eh_cleanup makes for a more compact
+	     rtcfg, and it avoids bb numbering differences when we split
+	     blocks because of trailing debug insns only.  */
+	  cleanup_tree_cfg ();
+	  gcc_checking_assert (EDGE_COUNT (bb_eh_cleanup->succs) == 0);
+	}
+    }
+
+  /* These record blocks with calls that are to be preceded by
+     checkpoints, such as noreturn calls (if so chosen), must-tail
+     calls, potential early-marked tail calls, and returning calls (if
+     so chosen).  */
+  int count_chkcall = 0;
+  auto_sbitmap chkcall_blocks (last_basic_block_for_fn (fun));
+  bitmap_clear (chkcall_blocks);
+
+  /* We wish to add verification at blocks without successors, such as
+     noreturn calls (raising or not) and the reraise at the cleanup
+     block, but not other reraises: they will go through the cleanup
+     block.  */
+  if (check_before_noreturn_calls)
+    FOR_EACH_BB_FN (bb, fun)
+      {
+	gimple_stmt_iterator gsi = gsi_last_bb (bb);
+	if (gsi_end_p (gsi))
+	  continue;
+	gimple *stmt = gsi_stmt (gsi);
 
-  vstd.check ();
+	if (EDGE_COUNT (bb->succs) == 0)
+	  {
+	    /* A stmt at the end of a block without any successors is
+	       either a resx or a noreturn call without a local
+	       handler.  Check that it's one of the desired
+	       checkpoints.  */
+	    if (flag_exceptions && is_a <gresx *> (stmt)
+		? (check_before_always_throwing_noreturn_calls
+		   || bb == bb_eh_cleanup)
+		: (!is_a <gcall *> (stmt)
+		   || !gimple_call_noreturn_p (stmt))
+		? (/* Catch cases in which successors would be
+		      expected.  */
+		   gcc_unreachable (), false)
+		: (!flag_exceptions
+		   || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+		? check_before_nothrow_noreturn_calls
+		: always_throwing_noreturn_call_p (stmt)
+		? check_before_always_throwing_noreturn_calls
+		: check_before_throwing_noreturn_calls)
+	      {
+		if (dump_file)
+		  {
+		    fprintf (dump_file,
+			     "Scheduling check before stmt"
+			     " in succ-less block %i:\n",
+			     bb->index);
+		    print_gimple_stmt (dump_file, stmt, 0);
+		  }
+
+		if (bitmap_set_bit (chkcall_blocks, bb->index))
+		  count_chkcall++;
+		else
+		  gcc_unreachable ();
+	      }
+	    continue;
+	  }
 
-  return 0;
+	/* If there are no exceptions, it would seem like any noreturn
+	   call must have zero successor edges, but __builtin_return
+	   gets successor edges.  We don't want to handle it here, it
+	   will be dealt with in sibcall_search_preds.  Otherwise,
+	   check for blocks without non-EH successors, but skip those
+	   with resx stmts and edges (i.e., those other than that in
+	   bb_eh_cleanup), since those will go through bb_eh_cleanup,
+	   that will have been counted as noreturn above because it
+	   has no successors.  */
+	gcc_checking_assert (bb != bb_eh_cleanup
+			     || !check_at_escaping_exceptions);
+	if (flag_exceptions && is_a <gresx *> (stmt)
+	    ? check_before_always_throwing_noreturn_calls
+	    : (!is_a <gcall *> (stmt)
+	       || !gimple_call_noreturn_p (stmt))
+	    ? false
+	    : (!flag_exceptions
+	       || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+	    ? false /* rather than check_before_nothrow_noreturn_calls */
+	    : always_throwing_noreturn_call_p (stmt)
+	    ? check_before_always_throwing_noreturn_calls
+	    : check_before_throwing_noreturn_calls)
+	  {
+	    gcc_checking_assert (single_succ_p (bb)
+				 && (single_succ_edge (bb)->flags & EDGE_EH));
+
+	    if (dump_file)
+	      {
+		fprintf (dump_file,
+			 "Scheduling check before stmt"
+			 " in EH-succ block %i:\n",
+			 bb->index);
+		print_gimple_stmt (dump_file, stmt, 0);
+	      }
+
+	    if (bitmap_set_bit (chkcall_blocks, bb->index))
+	      count_chkcall++;
+	    else
+	      gcc_unreachable ();
+	  }
+      }
+  else if (bb_eh_cleanup)
+    {
+      if (bitmap_set_bit (chkcall_blocks, bb_eh_cleanup->index))
+	count_chkcall++;
+      else
+	gcc_unreachable ();
+    }
+
+  gcc_checking_assert (!bb_eh_cleanup
+		       || bitmap_bit_p (chkcall_blocks, bb_eh_cleanup->index));
+
+  /* If we don't have edges to exit nor noreturn calls (including the
+     cleanup reraise), then we may skip instrumentation: that would
+     amount to a function that ends with an infinite loop.  */
+  if (!count_chkcall
+      && EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (fun)->preds) == 0)
+    {
+      if (dump_file)
+	fprintf (dump_file,
+		 "Disabling CFR, no exit paths to check\n");
+
+      return 0;
+    }
+
+  /* Search for must-tail calls, early-marked potential tail calls,
+     and, if requested, returning calls.  As we introduce early
+     checks, */
+  int count_postchk = 0;
+  auto_sbitmap postchk_blocks (last_basic_block_for_fn (fun));
+  bitmap_clear (postchk_blocks);
+  chk_edges_t chk_edges;
+  hardcfr_sibcall_search_preds (EXIT_BLOCK_PTR_FOR_FN (fun), chk_edges,
+				count_chkcall, chkcall_blocks,
+				count_postchk, postchk_blocks,
+				NULL);
+
+  rt_bb_visited vstd (chk_edges.length () + count_chkcall);
+
+  auto_sbitmap combined_blocks (last_basic_block_for_fn (fun));
+  bitmap_copy (combined_blocks, chkcall_blocks);
+  int i;
+  edge *e;
+  FOR_EACH_VEC_ELT (chk_edges, i, e)
+    if (!bitmap_set_bit (combined_blocks, (*e)->src->index))
+      gcc_unreachable ();
+
+  /* Visit blocks in index order, because building rtcfg depends on
+     that.  Blocks must be compact, which the cleanup_cfg requirement
+     ensures.  This would also enable FOR_EACH_BB_FN to be used to
+     iterate in index order, but bb_eh_cleanup block splits and
+     insertions changes that.  */
+  gcc_checking_assert (n_basic_blocks_for_fn (fun)
+		       == last_basic_block_for_fn (fun));
+  for (int i = NUM_FIXED_BLOCKS; i < n_basic_blocks_for_fn (fun); i++)
+    {
+      bb = BASIC_BLOCK_FOR_FN (fun, i);
+      gcc_checking_assert (bb->index == i);
+      vstd.visit (bb, bitmap_bit_p (combined_blocks, i),
+		  bitmap_bit_p (postchk_blocks, i));
+    }
+
+  vstd.check (chk_edges, count_chkcall, chkcall_blocks);
+
+  return
+    TODO_update_ssa
+    | TODO_cleanup_cfg
+    | TODO_verify_il;
 }
 
 /* Instantiate a hardcfr pass.  */
diff --git a/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c b/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c
new file mode 100644
index 00000000000..a6992eb9f8e
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -O0 -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we don't insert checking before noreturn calls.  -O0 is tested
+   separately because h is not found to be noreturn without optimization.  */
+
+#include "torture/harden-cfr-noret.c"
+
+/* No out-of-line checks.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* Only one inline check at the end of f and of h2.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-always.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-always.c
new file mode 100644
index 00000000000..779896c60e8
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-always.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even enabling all checks before noreturn calls (leaving
+   returning calls enabled), we get checks before __builtin_return without
+   duplication (__builtin_return is both noreturn and a returning call).  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-never.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-never.c
new file mode 100644
index 00000000000..49ce17f5b93
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-never.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even enabling checks before never noreturn calls (leaving
+   returning calls enabled), we get checks before __builtin_return without
+   duplication (__builtin_return is both noreturn and a returning call).  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noopt.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noopt.c
new file mode 100644
index 00000000000..1512614791f
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noopt.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even disabling checks before both noreturn and returning
+   calls, we still get checks before __builtin_return.  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noret.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noret.c
new file mode 100644
index 00000000000..fd95bb7e3e3
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noret.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even disabling checks before returning calls (leaving noreturn
+   calls enabled), we still get checks before __builtin_return.  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-nothrow.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-nothrow.c
new file mode 100644
index 00000000000..c5c361234c4
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-nothrow.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even enabling checks before nothrow noreturn calls (leaving
+   returning calls enabled), we get checks before __builtin_return without
+   duplication (__builtin_return is both noreturn and a returning call).  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-retcl.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-retcl.c
new file mode 100644
index 00000000000..137dfbb95d6
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-retcl.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even disabling checks before noreturn calls (leaving returning
+   calls enabled), we still get checks before __builtin_return.  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c
index 70acdc95f25..b459ff6b864 100644
--- a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c
@@ -7,5 +7,11 @@ int f(int i) {
   return i;
 }
 
-/* Out-of-line checking, before both builtin_return and return.  */
+int g(int i) {
+  __builtin_return (&i);
+}
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
 /* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c
new file mode 100644
index 00000000000..8bd2d13ac18
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c
@@ -0,0 +1,18 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we don't insert checking before noreturn calls.  -O0 is tested
+   separately because h is not found to be noreturn without optimization, which
+   affects codegen for h2, so h2 is omitted here at -O0.  */
+
+#if !__OPTIMIZE__
+# define OMIT_H2
+#endif
+
+#include "harden-cfr-noret.c"
+
+
+/* No out-of-line checks.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* Only one inline check at the end of f.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c
new file mode 100644
index 00000000000..a804a6cfe59
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fno-exceptions -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that -fno-exceptions makes for implicit nothrow in noreturn
+   handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#include "harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c
new file mode 100644
index 00000000000..f390cfdbc59
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checking before nothrow noreturn calls.  */
+
+#include "harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c
new file mode 100644
index 00000000000..fdd803109a4
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c
@@ -0,0 +1,38 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checking before all noreturn calls.  */
+
+#ifndef ATTR_NOTHROW_OPT /* Overridden in harden-cfr-noret-noexcept.  */
+#define ATTR_NOTHROW_OPT __attribute__ ((__nothrow__))
+#endif
+
+extern void __attribute__ ((__noreturn__)) ATTR_NOTHROW_OPT g (void);
+
+void f(int i) {
+  if (i)
+    /* Out-of-line checks here...  */
+    g ();
+  /* ... and here.  */
+}
+
+void __attribute__ ((__noinline__, __noclone__))
+h(void) {
+  /* Inline check here.  */
+  g ();
+}
+
+#ifndef OMIT_H2 /* from harden-cfr-noret-never.  */
+void h2(void) {
+  /* Inline check either here, whether because of noreturn or tail call...  */
+  h ();
+  /* ... or here, if not optimizing.  */
+}
+#endif
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c
new file mode 100644
index 00000000000..6d11487bbba
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c
@@ -0,0 +1,8 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+#include "harden-cfr-tail.c"
+
+/* Inline checking after the calls, disabling tail calling.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 5 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 0 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c
new file mode 100644
index 00000000000..550b02ca088
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c
@@ -0,0 +1,35 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-exceptions -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checks before returning calls and alternate paths, even
+   at -O0, because of the explicit command-line flag.  */
+
+void g (void);
+void g2 (void);
+void g3 (void);
+
+void f (int i) {
+  if (!i)
+    /* Out-of-line checks here...  */
+    g ();
+  else if (i > 0)
+    /* here...  */
+    g2 ();
+  /* else */
+    /* and in the implicit else here.  */
+}
+
+void f2 (int i) {
+  if (!i)
+    /* Out-of-line check here...  */
+    g ();
+  else if (i > 0)
+    /* here...  */
+    g2 ();
+  else
+    /* and here.  */
+    g3 ();
+}
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 6 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 0 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
index 40d76c5c163..d5467eafa9f 100644
--- a/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
@@ -1,17 +1,52 @@
 /* { dg-do compile } */
-/* { dg-options "-fharden-control-flow-redundancy -fdump-tree-hardcfr -ffat-lto-objects" } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-hardcfr-check-exceptions -fdump-tree-hardcfr -ffat-lto-objects -Wno-return-type" } */
 
-/* We'd like to check that we insert checking so as to not disrupt tail calls,
-   but unfortunately mandatory tail calls are not available in C, and optimizing
-   calls as tail calls only takes place after hardcfr.  */
+/* Check that we insert CFR checking so as to not disrupt tail calls.
+   Mandatory tail calls are not available in C, and optimizing calls as tail
+   calls only takes place after hardcfr, so we insert checking before calls
+   followed by copies and return stmts with the same return value, that might
+   (or might not) end up optimized to tail calls.  */
 
-extern int g(int i);
+extern int g (int i);
 
-int f(int i) {
+int f1(int i) {
+  /* Inline check before the returning call.  */
   return g (i);
 }
 
-/* Inline checking before the tail call.  */
-/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
-/* Inline checking before the tail call.  */
-/* { dg-final { scan-tree-dump-times "\\\[tail\]" 1 "hardcfr" { xfail *-*-* } } } */
+extern void g2 (int i);
+
+void f2(int i) {
+  /* Inline check before the returning call, that ignores the returned value,
+     matching the value-less return.  */
+  g2 (i);
+  return;
+}
+
+void f3(int i) {
+  /* Inline check before the returning call.  */
+  g (i);
+}
+
+void f4(int i) {
+  if (i)
+    /* Out-of-line check before the returning call.  */
+    return g2 (i);
+  /* Out-of-line check before implicit return.  */
+}
+
+int f5(int i) {
+  /* Not regarded as a returning call, returning value other than callee's
+     returned value.  */
+  g (i);
+  /* Inline check after the non-returning call.  */
+  return i;
+}
+
+/* Out-of-line checks in f4, before returning calls and before return.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking in all other functions.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 4 "hardcfr" } } */
+/* Check before tail-call in all but f5, but f4 is out-of-line.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 3 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C b/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C
new file mode 100644
index 00000000000..17ea79f7cfb
--- /dev/null
+++ b/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C
@@ -0,0 +1,11 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects -O0" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions, and also checking before noreturn
+   calls.  h2 and h2b get an extra resx without ehcleanup.  */
+
+#include "torture/harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 16 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C b/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C
new file mode 100644
index 00000000000..f0338ccc361
--- /dev/null
+++ b/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C
@@ -0,0 +1,10 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -foptimize-sibling-calls -fdump-tree-hardcfr -O0" } */
+
+/* -fhardcfr-check-returning-calls gets implicitly disabled because,
+   -at O0, -foptimize-sibling-calls has no effect.  */
+
+#include "torture/harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 12 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C
new file mode 100644
index 00000000000..0d35920c7ee
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  */
+
+#include "harden-cfr-noret-no-nothrow.C"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Bypassing cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C
new file mode 100644
index 00000000000..b7d247ff43c
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C
@@ -0,0 +1,18 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  Expected results for =never and =nothrow are the same,
+   since the functions are not nothrow.  */
+
+#include "harden-cfr-noret-no-nothrow.C"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Associated cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C
new file mode 100644
index 00000000000..62c58cfd406
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C
@@ -0,0 +1,23 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#if ! __OPTIMIZE__
+void __attribute__ ((__noreturn__)) h (void);
+#endif
+
+#include "../../c-c++-common/torture/harden-cfr-noret.c"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Associated cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C
new file mode 100644
index 00000000000..0286f6e6d3f
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C
@@ -0,0 +1,20 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions, and also checking before noreturn
+   calls.  */
+
+#if ! __OPTIMIZE__
+/* Without optimization, functions with cleanups end up with an extra
+   resx that is not optimized out, so arrange to optimize them.  */
+void __attribute__ ((__optimize__ (1))) h2(void);
+void __attribute__ ((__optimize__ (1))) h2b(void);
+#endif
+
+#include "harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 14 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
+/* h, h2, h2b, and h4.  */
+/* { dg-final { scan-tree-dump-times "Bypassing" 4 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C
new file mode 100644
index 00000000000..885b0b236af
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C
@@ -0,0 +1,11 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions  -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we do not insert cleanups for checking around the bodies
+   of maybe-throwing functions.  h4 doesn't get any checks, because we
+   don't have noreturn checking enabled.  */
+
+#include "harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 6 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C
new file mode 100644
index 00000000000..32def637255
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C
@@ -0,0 +1,31 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -foptimize-sibling-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions.  These results depend on checking before
+   returning calls, which is only enabled when sibcall optimizations
+   are enabled, so change the optimization mode to -O1 for f and f2,
+   so that -foptimize-sibling-calls can take effect and enable
+   -fhardcfr-check-returning-calls, so that we get the same results.
+   There is a separate test for -O0.  */
+
+#if ! __OPTIMIZE__
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) f(int i);
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) f2(int i);
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) h3(void);
+#endif
+
+#include "harden-cfr-throw.C"
+
+/* f gets out-of-line checks before the unwrapped tail call and in the
+   else edge.  */
+/* f2 gets out-of-line checks before both unwrapped tail calls.  */
+/* h gets out-of-line checks before the implicit return and in the
+   cleanup block.  */
+/* h2 and h2b get out-of-line checks before the cleanup returning
+   call, and in the cleanup block.  */
+/* h3 gets an inline check before the __cxa_end_catch returning call.  */
+/* h4 gets an inline check in the cleanup block.  */
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 10 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C
new file mode 100644
index 00000000000..992fbdad381
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C
@@ -0,0 +1,65 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions.  */
+
+extern void g (void);
+extern void g2 (void);
+
+void f(int i) {
+  if (i)
+    g ();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void f2(int i) {
+  if (i)
+    g ();
+  else
+    g2 ();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h(void) {
+  try {
+    g ();
+  } catch (...) {
+    throw;
+  }
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+struct needs_cleanup {
+  ~needs_cleanup();
+};
+
+void h2(void) {
+  needs_cleanup y; /* No check in the cleanup handler.  */
+  g();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+extern void __attribute__ ((__nothrow__)) another_cleanup (void*);
+
+void h2b(void) {
+  int x __attribute__ ((cleanup (another_cleanup)));
+  g();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h3(void) {
+  try {
+    throw 1;
+  } catch (...) {
+  }
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h4(void) {
+  throw 1;
+  /* Inline check in the cleanup around the __cxa_throw noreturn call.  */
+}
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 12 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c b/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c
new file mode 100644
index 00000000000..8e4ee1fab08
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c
@@ -0,0 +1,15 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C makes for implicit nothrow in noreturn handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#include "../../c-c++-common/torture/harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c b/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c
new file mode 100644
index 00000000000..634d98f1ffc
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c
@@ -0,0 +1,40 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-hardcfr-check-exceptions -fdump-tree-hardcfr -ffat-lto-objects -Wno-return-type" } */
+
+/* In C only, check some additional cases (comparing with
+   c-c++-common/torture/harden-cfr-tail.c) of falling off the end of non-void
+   function.  C++ would issue an unreachable call in these cases.  */
+
+extern int g (int i);
+
+int f1(int i) {
+  /* Inline check before the returning call, that doesn't return anything.  */
+  g (i);
+  /* Implicit return without value, despite the return type; this combination
+     enables tail-calling of g, and is recognized as a returning call.  */
+}
+
+extern void g2 (int i);
+
+int f2(int i) {
+  /* Inline check before the returning call, that disregards its return
+     value.  */
+  g2 (i);
+  /* Implicit return without value, despite the return type; this combination
+     enables tail-calling of g2, and is recognized as a returning call.  */
+}
+
+int f3(int i) {
+  if (i)
+    /* Out-of-line check before the returning call.  */
+    return g (i);
+  /* Out-of-line check before implicit return.  */
+}
+
+/* Out-of-line checks in f3, before returning calls and before return.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking in all other functions.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
+/* Check before tail-call in all functions, but f3 is out-of-line.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/gnat.dg/hardcfr.adb b/gcc/testsuite/gnat.dg/hardcfr.adb
index b578a913d75..abe1605c029 100644
--- a/gcc/testsuite/gnat.dg/hardcfr.adb
+++ b/gcc/testsuite/gnat.dg/hardcfr.adb
@@ -1,5 +1,5 @@
 --  { dg-do run }
---  { dg-options "-fharden-control-flow-redundancy -fdump-tree-hardcfr --param=hardcfr-max-blocks=22 --param=hardcfr-max-inline-blocks=12 -O0" }
+--  { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions -fdump-tree-hardcfr --param=hardcfr-max-blocks=22 --param=hardcfr-max-inline-blocks=12 -O0" }
 
 procedure HardCFR is
    function F (I, J : Integer) return Integer is
diff --git a/libgcc/hardcfr.c b/libgcc/hardcfr.c
index 8ef29428111..55f6b995f2f 100644
--- a/libgcc/hardcfr.c
+++ b/libgcc/hardcfr.c
@@ -60,14 +60,25 @@ extern void __hardcfr_check (size_t blocks,
 			     vword const *visited,
 			     vword const *cfg);
 
+/* Compute the MASK for the bit representing BLOCK in WORDIDX's vword in a
+   visited blocks bit array.  */
+static inline void
+block2mask (size_t const block, vword *const mask, size_t *const wordidx)
+{
+  size_t wbits = __CHAR_BIT__ * sizeof (vword);
+  *wordidx = block / wbits;
+  *mask = (vword)1 << (block % wbits);
+}
 
 /* Check whether the bit corresponding to BLOCK is set in VISITED.  */
 static inline bool
 visited_p (size_t const block, vword const *const visited)
 {
-  size_t wbits = __CHAR_BIT__ * sizeof (vword);
-  vword w = visited[block / wbits];
-  return (w & ((vword)1 << (block % wbits))) != 0;
+  vword mask;
+  size_t wordidx;
+  block2mask (block, &mask, &wordidx);
+  vword w = visited[wordidx];
+  return (w & mask) != 0;
 }
 
 /* Read and consume a mask from **CFG_IT.  (Consume meaning advancing the
@@ -118,19 +129,19 @@ consume_seq (vword const **const cfg_it)
    we reach the terminator without finding any.  Consume the entire sequence
    otherwise, so that *CFG_IT points just past the terminator, which may be the
    beginning of the next sequence.  */
-static inline void
+static inline bool
 check_seq (vword const *const visited, vword const **const cfg_it)
 {
   vword mask;
   size_t wordidx;
 
   /* If the block was visited, check that at least one of the
-     preds was also visited.  */
+     preds/succs was also visited.  */
   do
     /* If we get to the end of the sequence without finding any
        match, something is amiss.  */
     if (!next_pair (cfg_it, &mask, &wordidx))
-      __builtin_trap ();
+      return false;
   /* Keep searching until we find a match, at which point the
      condition is satisfied.  */
   while (!test_mask (visited, mask, wordidx));
@@ -139,6 +150,94 @@ check_seq (vword const *const visited, vword const **const cfg_it)
      skipped the block, so as to position the iterator at the beginning of the
      next .  */
   consume_seq (cfg_it);
+
+  return true;
+}
+
+/* Print out the CFG with BLOCKS blocks, presumed to be associated with CALLER.
+   This is expected to be optimized out entirely, unless the verbose part of
+   __hardcfr_check_fail is enabled.  */
+static inline void
+__hardcfr_debug_cfg (size_t const blocks,
+		     void const *const caller,
+		     vword const *const cfg)
+{
+  __builtin_printf ("CFG at %p, for %p", cfg, caller);
+  vword const *cfg_it = cfg;
+  for (size_t i = 0; i < blocks; i++)
+    {
+      vword mask; size_t wordidx;
+      block2mask (i, &mask, &wordidx);
+      __builtin_printf ("\nblock %lu (%lu/0x%lx)\npreds: ",
+			(unsigned long)i,
+			(unsigned long)wordidx, (unsigned long)mask);
+      while (next_pair (&cfg_it, &mask, &wordidx))
+	__builtin_printf (" (%lu/0x%lx)",
+			  (unsigned long)wordidx, (unsigned long)mask);
+      __builtin_printf ("\nsuccs: ");
+      while (next_pair (&cfg_it, &mask, &wordidx))
+	__builtin_printf (" (%lu/0x%lx)",
+			  (unsigned long)wordidx, (unsigned long)mask);
+    }
+  __builtin_printf ("\n");
+}
+
+#ifndef ATTRIBUTE_UNUSED
+# define ATTRIBUTE_UNUSED __attribute__ ((__unused__))
+#endif
+
+/* This is called when an out-of-line hardcfr check fails.  All the arguments
+   are ignored, and it just traps, unless HARDCFR_VERBOSE_FAIL is enabled.  IF
+   it is, it prints the PART of the CFG, expected to have BLOCKS blocks, that
+   failed at CALLER's BLOCK, and the VISITED bitmap.  When the verbose mode is
+   enabled, it also forces __hardcfr_debug_cfg (above) to be compiled into an
+   out-of-line function, that could be called from a debugger.
+   */
+static inline void
+__hardcfr_check_fail (size_t const blocks ATTRIBUTE_UNUSED,
+		      vword const *const visited,
+		      vword const *const cfg ATTRIBUTE_UNUSED,
+		      size_t const block ATTRIBUTE_UNUSED,
+		      int const part ATTRIBUTE_UNUSED,
+		      void const *const caller ATTRIBUTE_UNUSED)
+{
+#if HARDCFR_VERBOSE_FAIL
+  static const char *parts[] = { "preds", "succs" };
+
+  vword mask; size_t wordidx;
+  block2mask (block, &mask, &wordidx);
+  __builtin_printf ("hardcfr fail at %p block %lu (%lu/0x%lx), expected %s:",
+		    caller, (unsigned long)block,
+		    (unsigned long)wordidx, (unsigned long)mask,
+		    parts[part]);
+
+  /* Skip data for previous blocks.  */
+  vword const *cfg_it = cfg;
+  for (size_t i = block; i--; )
+    {
+      consume_seq (&cfg_it);
+      consume_seq (&cfg_it);
+    }
+  for (size_t i = part; i--; )
+    consume_seq (&cfg_it);
+
+  while (next_pair (&cfg_it, &mask, &wordidx))
+    __builtin_printf (" (%lu/0x%lx)",
+		      (unsigned long)wordidx, (unsigned long)mask);
+
+  __builtin_printf ("\nvisited:");
+  block2mask (blocks, &mask, &wordidx);
+  for (size_t i = 0; i <= wordidx; i++)
+    __builtin_printf (" (%lu/0x%lx)",
+		      (unsigned long)i, (unsigned long)visited[i]);
+  __builtin_printf ("\n");
+
+  /* Reference __hardcfr_debug_cfg so that it's output out-of-line, so that it
+     can be called from a debugger.  */
+  if (!caller || caller == __hardcfr_debug_cfg)
+    return;
+#endif
+  __builtin_trap ();
 }
 
 /* Check that, for each of the BLOCKS basic blocks, if its bit is set in
@@ -168,9 +267,13 @@ __hardcfr_check (size_t const blocks,
       else
 	{
 	  /* Check predecessors.  */
-	  check_seq (visited, &cfg_it);
+	  if (!check_seq (visited, &cfg_it))
+	    __hardcfr_check_fail (blocks, visited, cfg, i, 0,
+				  __builtin_return_address (0));
 	  /* Check successors.  */
-	  check_seq (visited, &cfg_it);
+	  if (!check_seq (visited, &cfg_it))
+	    __hardcfr_check_fail (blocks, visited, cfg, i, 1,
+				  __builtin_return_address (0));
 	}
     }
 }

^ permalink raw reply	[flat|nested] 13+ messages in thread
* [gcc(refs/users/aoliva/heads/testme)] hardcfr: add optional checkpoints
@ 2022-10-20  4:09 Alexandre Oliva
  0 siblings, 0 replies; 13+ messages in thread
From: Alexandre Oliva @ 2022-10-20  4:09 UTC (permalink / raw)
  To: gcc-cvs

https://gcc.gnu.org/g:20608bca1241ee84a92475a61920dc924b47f47d

commit 20608bca1241ee84a92475a61920dc924b47f47d
Author: Alexandre Oliva <oliva@adacore.com>
Date:   Wed Oct 19 20:36:18 2022 -0300

    hardcfr: add optional checkpoints
    
    Previously, control flow redundancy only checked the visited bitmap
    against the control flow graph at return points and before mandatory
    tail calls, missing various other possibilities of exiting a
    subprogram, such as by raising or propagating exceptions, and calling
    noreturn functions.  The checks inserted before returns also prevented
    potential tail-call optimizations.
    
    This incremental change introduces options to control checking at each
    of these previously-missed checkpoints.  Unless disabled, a cleanup is
    introduced to check when an exceptions escapes a subprogram.  To avoid
    disrupting sibcall optimizations, when they are enabled, checks are
    introduced before calls whose results are immediately returned,
    whether or not they are ultimately optimized.  If enabled, checks are
    introduced before noreturn calls and exception raises, or only before
    nothrow noreturn calls.
    
    
    for  gcc/ChangeLog
    
            * common.opt (-fhardcfr-check-returning-calls): New.
            (-fhardcfr-check-exceptions): New.
            (-fhardcfr-check-noreturn-calls=*): New.
            (Enum hardcfr_check_noreturn_calls): New.
            * doc/invoke.texi: Document them.
            * flag-types.h (enum hardcfr_noret): New.
            * gimple-harden-control-flow.cc: Include more headers.
            (pass_data_harden_control_flow_redundancy): Move
            properties_finish to pass return.
            (pass_harden_control_flow_redundancy::gate): Move
            no-edge-to-exit checking to execute, after exception
            transformations.  Use symbolic NUM_FIXED_BLOCKS.  Test for CFG
            before testing block count.
            (check_returning_calls_p): New.
            (hardcfr_scan_block): New.
            (returnign_call_p): New.
            (chk_edges_t): New.
            (hardcfr_sibcall_search_block): New.
            (hardcfr_sibcall_search_preds): New.
            (rt_bb_visited::num2idx): Use symbolic NUM_FIXED_BLOCKS.
            (rt_bb_visited::rt_bb_visited): Likewise.  Take checkpoints
            count, test it to select out-of-line checks.
            (rt_bb_visited::insert_exit_check): Removed.
            (rt_bb_visited::insert_exit_check_in_blocks): New.
            (rt_bb_visited::insert_exit_check_on_edge): New.
            (rt_bb_visited::check): Take checkpoint edges and blocks.
            Move initialization insertion to the end.  Insert checks on
            edges, and before calls in select blocks.  Clobber the visited
            array to the check sequence.  Output actions to dump_file.
            (rt_bb_visited::visit): Take checkpoint and postcheck.  Assume
            an edge to exit if checkpoint, and skip testing if postcheck.
            (always_throwing_noreturn_call_p): New, dummy.
            (pass_harden_control_flow_redundancy::execute): Introduce
            cleanup block if requested and needed.  Scan blocks for
            noreturn and returning calls if checkpoints before them were
            requested.  Log actions to dump_file.  Visit blocks in index
            order.
    
    for  gcc/testsuite/ChangeLog
    
            * c-c++-common/harden-cfr-noret-never-O0.c: New.
            * c-c++-common/torture/harden-cfr-noret-never.c: New.
            * c-c++-common/torture/harden-cfr-noret-noexcept.c: New.
            * c-c++-common/torture/harden-cfr-noret-nothrow.c: New.
            * c-c++-common/torture/harden-cfr-noret.c: New.
            * c-c++-common/torture/harden-cfr-notail.c: New.
            * c-c++-common/torture/harden-cfr-returning.c: New.
            * c-c++-common/torture/harden-cfr-tail.c: Extend.
            * c-c++-common/torture/harden-cfr-bret-always.c: New.
            * c-c++-common/torture/harden-cfr-bret-never.c: New.
            * c-c++-common/torture/harden-cfr-bret-noopt.c: New.
            * c-c++-common/torture/harden-cfr-bret-noret.c: New.
            * c-c++-common/torture/harden-cfr-bret-nothrow.c: New.
            * c-c++-common/torture/harden-cfr-bret-retcl.c: New.
            * c-c++-common/torture/harden-cfr-bret.c (g): New.
            * g++.dg/harden-cfr-throw-always-O0.C: New.
            * g++.dg/harden-cfr-throw-returning-O0.C: New.
            * g++.dg/torture/harden-cfr-noret-always-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-noret-never-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-noret-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-throw-always.C: New.
            * g++.dg/torture/harden-cfr-throw-nocleanup.C: New.
            * g++.dg/torture/harden-cfr-throw-returning.C: New.
            * g++.dg/torture/harden-cfr-throw.C: New.
            * gcc.dg/torture/harden-cfr-noret-no-nothrow.c: New.
            * gcc.dg/torture/harden-cfr-tail-ub.c: New.
            * gnat.dg/hardcfr.adb: Disable checking at exceptions.
    
    for  libgcc/ChangeLog
    
            * hardcfr.c (block2mask): Split out of...
            (visited_p): ... this.
            (check_seq): Move trapping out, return result instead.
            (__hardcfr_debug_cfg): New.
            (__hardcfr_check_fail): New, with optional verbose error
            printing before trapping.
            (__hardcfr_check): Call __hardcfr_check_fail when check_seq
            fails.

Diff:
---
 gcc/common.opt                                     |  33 +
 gcc/doc/invoke.texi                                |  74 +-
 gcc/flag-types.h                                   |  10 +
 gcc/gimple-harden-control-flow.cc                  | 921 ++++++++++++++++++---
 .../c-c++-common/harden-cfr-noret-never-O0.c       |  12 +
 .../c-c++-common/torture/harden-cfr-bret-always.c  |  13 +
 .../c-c++-common/torture/harden-cfr-bret-never.c   |  13 +
 .../c-c++-common/torture/harden-cfr-bret-noopt.c   |  12 +
 .../c-c++-common/torture/harden-cfr-bret-noret.c   |  12 +
 .../c-c++-common/torture/harden-cfr-bret-nothrow.c |  13 +
 .../c-c++-common/torture/harden-cfr-bret-retcl.c   |  12 +
 .../c-c++-common/torture/harden-cfr-bret.c         |   8 +-
 .../c-c++-common/torture/harden-cfr-noret-never.c  |  18 +
 .../torture/harden-cfr-noret-noexcept.c            |  16 +
 .../torture/harden-cfr-noret-nothrow.c             |  13 +
 .../c-c++-common/torture/harden-cfr-noret.c        |  38 +
 .../c-c++-common/torture/harden-cfr-notail.c       |   8 +
 .../c-c++-common/torture/harden-cfr-returning.c    |  35 +
 .../c-c++-common/torture/harden-cfr-tail.c         |  55 +-
 gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C  |  11 +
 .../g++.dg/harden-cfr-throw-returning-O0.C         |  10 +
 .../torture/harden-cfr-noret-always-no-nothrow.C   |  16 +
 .../torture/harden-cfr-noret-never-no-nothrow.C    |  18 +
 .../g++.dg/torture/harden-cfr-noret-no-nothrow.C   |  23 +
 .../g++.dg/torture/harden-cfr-throw-always.C       |  20 +
 .../g++.dg/torture/harden-cfr-throw-nocleanup.C    |  11 +
 .../g++.dg/torture/harden-cfr-throw-returning.C    |  31 +
 gcc/testsuite/g++.dg/torture/harden-cfr-throw.C    |  65 ++
 .../gcc.dg/torture/harden-cfr-noret-no-nothrow.c   |  15 +
 gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c  |  40 +
 gcc/testsuite/gnat.dg/hardcfr.adb                  |   2 +-
 libgcc/hardcfr.c                                   | 119 ++-
 32 files changed, 1564 insertions(+), 133 deletions(-)

diff --git a/gcc/common.opt b/gcc/common.opt
index c2512651294..2f37d2fcbd3 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -1798,6 +1798,39 @@ fharden-control-flow-redundancy
 Common Var(flag_harden_control_flow_redundancy) Optimization
 Harden control flow by recording and checking execution paths.
 
+fhardcfr-check-returning-calls
+Common Var(flag_harden_control_flow_redundancy_check_returning_calls) Init(-1) Optimization
+Check CFR execution paths also before calls followed by returns of their results.
+
+fhardcfr-check-exceptions
+Common Var(flag_harden_control_flow_redundancy_check_exceptions) Init(-1) Optimization
+Check CFR execution paths also when exiting a function through an exception.
+
+fhardcfr-check-noreturn-calls=
+Common Joined RejectNegative Enum(hardcfr_check_noreturn_calls) Var(flag_harden_control_flow_redundancy_check_noreturn) Init(HCFRNR_UNSPECIFIED) Optimization
+-fhardcfr-check-noreturn-calls=[always|nothrow|never]	Check CFR execution paths also before calling noreturn functions.
+
+Enum
+Name(hardcfr_check_noreturn_calls) Type(enum hardcfr_noret) UnknownError(unknown hardcfr noreturn checking level %qs)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(never) Value(HCFRNR_NEVER)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(nothrow) Value(HCFRNR_NOTHROW)
+
+; ??? There could be yet another option here, that checked before
+; noreturn calls, except for those known to always throw, if we had
+; means to distinguish noreturn functions known to always throw, such
+; as those used to (re)raise exceptions, from those that merely might
+; throw.  "not always" stands for "not always-throwing", but it also
+; contrasts with "always" below.
+; EnumValue
+; Enum(hardcfr_check_noreturn_calls) String(not-always) Value(HCFRNR_NOT_ALWAYS)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(always) Value(HCFRNR_ALWAYS)
+
 ; Nonzero means ignore `#ident' directives.  0 means handle them.
 ; Generate position-independent code for executables if possible
 ; On SVR4 targets, it also controls whether or not to emit a
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index ea5f22162bb..1ea9525c78d 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -627,7 +627,9 @@ Objective-C and Objective-C++ Dialects}.
 -fsanitize-undefined-trap-on-error  -fbounds-check @gol
 -fcf-protection=@r{[}full@r{|}branch@r{|}return@r{|}none@r{|}check@r{]} @gol
 -fharden-compares -fharden-conditional-branches @gol
--fharden-control-flow-redundancy @gol
+-fharden-control-flow-redundancy  -fhardcfr-check-exceptions  @gol
+-fhardcfr-check-returning-calls  @gol
+-fhardcfr-check-noreturn-calls=@r{[}always@r{|}nothrow@r{|}never@r{]}  @gol
 -fstack-protector  -fstack-protector-all  -fstack-protector-strong @gol
 -fstack-protector-explicit  -fstack-check @gol
 -fstack-limit-register=@var{reg}  -fstack-limit-symbol=@var{sym} @gol
@@ -16644,11 +16646,75 @@ conditionals.
 @item -fharden-control-flow-redundancy
 @opindex fharden-control-flow-redundancy
 Emit extra code to set booleans when entering basic blocks, and to
-verify, at function exits, that they amount to an execution path that is
-consistent with the control flow graph, trapping otherwise.  Tuning
-options @option{--param hardcfr-max-blocks} and @option{--param
+verify and trap, at function exits, when the booleans do not form an
+execution path that is compatible with the control flow graph.
+
+Verification takes place before returns, before mandatory tail calls
+(see below) and, optionally, before escaping exceptions with
+@option{-fhardcfr-check-exceptions}, before returning calls with
+@option{-fhardcfr-check-returning-calls}, and before noreturn calls with
+@option{-fhardcfr-check-noreturn-calls}).  Tuning options
+@option{--param hardcfr-max-blocks} and @option{--param
 hardcfr-max-inline-blocks} are available.
 
+Tail call optimization takes place too late to affect control flow
+redundancy, but calls annotated as mandatory tail calls by language
+front-ends, and any calls marked early enough as potential tail calls
+would also have verification issued before the call, but these
+possibilities are merely theoretical, as these conditions can only be
+met when using custom compiler plugins.
+
+@item -fhardcfr-check-exceptions
+@opindex fhardcfr-check-exceptions
+@opindex fno-hardcfr-check-exceptions
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph at exception
+escape points, as if the function body was wrapped with a cleanup
+handler that performed the check and reraised.  This option is enabled
+by default; use @option{-fno-hardcfr-check-exceptions} to disable it.
+
+@item -fhardcfr-check-returning-calls
+@opindex fhardcfr-check-returning-calls
+@opindex fno-hardcfr-check-returning-calls
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph before any
+function call immediately followed by a return of its result, if any, so
+as to not prevent tail-call optimization, whether or not it is
+ultimately optimized to a tail call.
+
+This option is enabled by default whenever sibling call optimizations
+are enabled (see @option{-foptimize-sibling-calls}), but it can be
+enabled (or disabled, using its negated form) explicitly, regardless of
+the optimizations.
+
+@item -fhardcfr-check-noreturn-calls=@r{[}always@r{|}nothrow@r{|}never@r{]}
+@opindex fhardcfr-check-noreturn-calls
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph before
+@code{noreturn} calls, either all of them (@option{always}), those that
+may not return control to the caller through an exception either
+(@option{nothrow}), or none of them (@option{never}, the default).
+
+Checking before a @code{noreturn} function that may return control to
+the caller through an exception may cause checking to be performed more
+than once, if the exception is caught in the caller, whether by a
+handler or a cleanup.  When @option{-fhardcfr-check-exceptions} is also
+enabled, the compiler will avoid associating a @code{noreturn} call with
+the implicitly-added cleanup handler, since it would be redundant with
+the check performed before the call, but other handlers or cleanups in
+the function, if activated, will modify the recorded execution path and
+check it again when another checkpoint is hit.  The checkpoint may even
+be another @code{noreturn} call, so checking may end up performed
+multiple times.
+
+Various optimizers may cause calls to be marked as @code{noreturn}
+and/or @code{nothrow}, even in the absence of the corresponding
+attributes, which may affect the placement of checks before calls, as
+well as the addition of implicit cleanup handlers for them.  This
+unpredictability, and the fact that raising and reraising exceptions
+frequently amounts to implicitly calling @code{noreturn} functions, have
+made @option{never} the default setting for this option.
+
 @item -fstack-protector
 @opindex fstack-protector
 Emit extra code to check for buffer overflows, such as stack smashing
diff --git a/gcc/flag-types.h b/gcc/flag-types.h
index d2e751060ff..3fae7548cab 100644
--- a/gcc/flag-types.h
+++ b/gcc/flag-types.h
@@ -157,6 +157,16 @@ enum stack_reuse_level
   SR_ALL
 };
 
+/* Control Flow Redundancy hardening options for noreturn calls.  */
+enum hardcfr_noret
+{
+  HCFRNR_NEVER,
+  HCFRNR_NOTHROW,
+  HCFRNR_NOT_ALWAYS, /* Reserved for future use.  */
+  HCFRNR_ALWAYS,
+  HCFRNR_UNSPECIFIED = -1
+};
+
 /* The live patching level.  */
 enum live_patching_level
 {
diff --git a/gcc/gimple-harden-control-flow.cc b/gcc/gimple-harden-control-flow.cc
index 6b08846dbb1..fe322f021ee 100644
--- a/gcc/gimple-harden-control-flow.cc
+++ b/gcc/gimple-harden-control-flow.cc
@@ -29,7 +29,12 @@ along with GCC; see the file COPYING3.  If not see
 #include "tree-pass.h"
 #include "ssa.h"
 #include "gimple-iterator.h"
+#include "gimple-pretty-print.h"
 #include "tree-cfg.h"
+#include "tree-cfgcleanup.h"
+#include "tree-eh.h"
+#include "except.h"
+#include "sbitmap.h"
 #include "basic-block.h"
 #include "cfghooks.h"
 #include "cfgloop.h"
@@ -60,9 +65,7 @@ const pass_data pass_data_harden_control_flow_redundancy = {
   0,	    // properties_provided
   0,	    // properties_destroyed
   TODO_cleanup_cfg, // properties_start
-  TODO_update_ssa
-  | TODO_cleanup_cfg
-  | TODO_verify_il, // properties_finish
+  0,        // properties_finish
 };
 
 class pass_harden_control_flow_redundancy : public gimple_opt_pass
@@ -79,16 +82,6 @@ public:
     if (!flag_harden_control_flow_redundancy)
       return false;
 
-    /* We don't verify when an exception escapes, propagated or raised
-       by the function itself, so we're only concerned with edges to
-       the exit block.  If there aren't any, the function doesn't
-       return normally, so there won't be any checking point, so
-       there's no point in running the pass.  Should we add
-       verification at exception escapes, we should at least look at
-       !flag_exceptions here.  */
-    if (EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (fun)->preds) == 0)
-      return false;
-
     /* Functions that return more than once, like setjmp and vfork
        (that also gets this flag set), will start recording a path
        after the first return, and then may take another path when
@@ -117,8 +110,9 @@ public:
 	return false;
       }
 
-    if (param_hardcfr_max_blocks > 0
-	&& n_basic_blocks_for_fn (fun) - 2 > param_hardcfr_max_blocks)
+    if (fun->cfg && param_hardcfr_max_blocks > 0
+	&& (n_basic_blocks_for_fn (fun) - NUM_FIXED_BLOCKS
+	    > param_hardcfr_max_blocks))
       {
 	warning_at (DECL_SOURCE_LOCATION (fun->decl), 0,
 		    "%qD has more than %u blocks, the requested"
@@ -134,6 +128,293 @@ public:
 
 }
 
+/* Return TRUE iff CFR checks should be inserted before returning
+   calls.  */
+
+static bool
+check_returning_calls_p ()
+{
+  return
+    flag_harden_control_flow_redundancy_check_returning_calls > 0
+    || (flag_harden_control_flow_redundancy_check_returning_calls < 0
+	/* Gates pass_tail_calls.  */
+	&& flag_optimize_sibling_calls
+	/* Gates pass_all_optimizations.  */
+	&& optimize >= 1 && !optimize_debug);
+}
+
+/* Scan BB from the end, updating *RETPTR if given as return stmts and
+   copies are found.  Return a call or a stmt that cannot appear after
+   a tail call, or NULL if the top of the block is reached without
+   finding any.  */
+
+static gimple *
+hardcfr_scan_block (basic_block bb, tree **retptr)
+{
+  gimple_stmt_iterator gsi;
+  for (gsi = gsi_last_bb (bb); !gsi_end_p (gsi); gsi_prev (&gsi))
+    {
+      gimple *stmt = gsi_stmt (gsi);
+
+      /* Ignore labels, returns, nops, clobbers and debug stmts.  */
+      if (gimple_code (stmt) == GIMPLE_LABEL
+	  || gimple_code (stmt) == GIMPLE_NOP
+	  || gimple_code (stmt) == GIMPLE_PREDICT
+	  || gimple_clobber_p (stmt)
+	  || is_gimple_debug (stmt))
+	continue;
+
+      if (gimple_code (stmt) == GIMPLE_RETURN)
+	{
+	  greturn *gret = as_a <greturn *> (stmt);
+	  if (retptr)
+	    {
+	      gcc_checking_assert (!*retptr);
+	      *retptr = gimple_return_retval_ptr (gret);
+	    }
+	  continue;
+	}
+
+      /* Check for a call.  */
+      if (is_gimple_call (stmt))
+	return stmt;
+
+      /* Allow simple copies to the return value, updating the return
+	 value to be found in earlier assignments.  */
+      if (retptr && *retptr && gimple_assign_single_p (stmt)
+	  && **retptr == gimple_assign_lhs (stmt))
+	{
+	  *retptr = gimple_assign_rhs1_ptr (stmt);
+	  continue;
+	}
+
+      return stmt;
+    }
+
+  /* Any other kind of stmt will prevent a tail call.  */
+  return NULL;
+}
+
+/* Return TRUE iff CALL is to be preceded by a CFR checkpoint, i.e.,
+   if it's a returning call (one whose result is ultimately returned
+   without intervening non-copy statements) and we're checking
+   returning calls, a __builtin_return call (noreturn with a path to
+   the exit block), a must-tail call, or a tail call.  */
+
+static bool
+returning_call_p (gcall *call)
+{
+  if (!(gimple_call_noreturn_p (call)
+	|| gimple_call_must_tail_p (call)
+	|| gimple_call_tail_p (call)
+	|| check_returning_calls_p ()))
+    return false;
+
+  /* Quickly check that there's a path to exit compatible with a
+     returning call.  Detect infinite loops through the counter.  */
+  basic_block bb = gimple_bb (call);
+  auto_vec<basic_block, 10> path;
+  for (int i = n_basic_blocks_for_fn (cfun);
+       bb != EXIT_BLOCK_PTR_FOR_FN (cfun) && i--;
+       bb = single_succ (bb))
+    if (!single_succ_p (bb)
+	|| (single_succ_edge (bb)->flags & EDGE_EH) != 0)
+      return false;
+    else
+      path.safe_push (bb);
+
+  /* Check the stmts in the blocks and trace the return value.  */
+  tree *retptr = NULL;
+  for (;;)
+    {
+      gcc_checking_assert (!path.is_empty ());
+      basic_block bb = path.pop ();
+      gimple *stop = hardcfr_scan_block (bb, &retptr);
+      if (stop)
+	{
+	  if (stop != call)
+	    return false;
+	  gcc_checking_assert (path.is_empty ());
+	  break;
+	}
+
+      gphi *retphi = NULL;
+      if (retptr && *retptr && TREE_CODE (*retptr) == SSA_NAME
+	  && !SSA_NAME_IS_DEFAULT_DEF (*retptr)
+	  && SSA_NAME_DEF_STMT (*retptr)
+	  && is_a <gphi *> (SSA_NAME_DEF_STMT (*retptr))
+	  && gimple_bb (SSA_NAME_DEF_STMT (*retptr)) == bb)
+	{
+	  retphi = as_a <gphi *> (SSA_NAME_DEF_STMT (*retptr));
+	  gcc_checking_assert (gimple_phi_result (retphi) == *retptr);
+	}
+      else
+	continue;
+
+      gcc_checking_assert (!path.is_empty ());
+      edge e = single_succ_edge (path.last ());
+      int i = EDGE_COUNT (bb->preds);
+      while (i--)
+	if (EDGE_PRED (bb, i) == e)
+	  break;
+      gcc_checking_assert (i >= 0);
+      retptr = gimple_phi_arg_def_ptr (retphi, i);
+    }
+
+  return (gimple_call_noreturn_p (call)
+	  || gimple_call_must_tail_p (call)
+	  || gimple_call_tail_p (call)
+	  || (gimple_call_lhs (call) == (retptr ? *retptr : NULL)
+	      && check_returning_calls_p ()));
+}
+
+typedef auto_vec<edge, 10> chk_edges_t;
+
+/* Declare for mutual recursion.  */
+static bool hardcfr_sibcall_search_preds (basic_block bb,
+					  chk_edges_t &chk_edges,
+					  int &count_chkcall,
+					  auto_sbitmap &chkcall_blocks,
+					  int &count_postchk,
+					  auto_sbitmap &postchk_blocks,
+					  tree *retptr);
+
+/* Search backwards from the end of BB for a mandatory or potential
+   sibcall.  Schedule the block to be handled sort-of like noreturn if
+   so.  Recurse to preds, with updated RETPTR, if the block only
+   contains stmts that may follow such a call, scheduling checking at
+   edges and marking blocks as post-check as needed.  Return true iff,
+   at the end of the block, a check will have already been
+   performed.  */
+
+static bool
+hardcfr_sibcall_search_block (basic_block bb,
+			      chk_edges_t &chk_edges,
+			      int &count_chkcall,
+			      auto_sbitmap &chkcall_blocks,
+			      int &count_postchk,
+			      auto_sbitmap &postchk_blocks,
+			      tree *retptr)
+{
+  /* Conditionals and internal exceptions rule out tail calls.  */
+  if (!single_succ_p (bb)
+      || (single_succ_edge (bb)->flags & EDGE_EH) != 0)
+    return false;
+
+  gimple *stmt = hardcfr_scan_block (bb, &retptr);
+  if (!stmt)
+    return hardcfr_sibcall_search_preds (bb, chk_edges,
+					 count_chkcall, chkcall_blocks,
+					 count_postchk, postchk_blocks,
+					 retptr);
+
+  if (!is_a <gcall *> (stmt))
+    return false;
+
+  /* Avoid disrupting mandatory or early-marked tail calls,
+     inserting the check before them.  This works for
+     must-tail calls, but tail calling as an optimization is
+     detected too late for us.
+
+     Also check for noreturn calls here.  Noreturn calls won't
+     normally have edges to exit, so they won't be found here,
+     but __builtin_return does, and we must check before
+     it, so handle it like a tail call.  */
+  gcall *call = as_a <gcall *> (stmt);
+  if (!(gimple_call_noreturn_p (call)
+	|| gimple_call_must_tail_p (call)
+	|| gimple_call_tail_p (call)
+	|| (gimple_call_lhs (call) == (retptr ? *retptr : NULL)
+	    && check_returning_calls_p ())))
+    return false;
+
+  gcc_checking_assert (returning_call_p (call));
+
+  /* We found a call that is to be preceded by checking.  */
+  if (bitmap_set_bit (chkcall_blocks, bb->index))
+    ++count_chkcall;
+  else
+    gcc_unreachable ();
+  return true;
+}
+
+
+/* Search preds of BB for a mandatory or potential sibcall or
+   returning call, and arrange for the blocks containing them to have
+   a check inserted before the call, like noreturn calls.  If any
+   preds are found to perform checking, schedule checks at the edges
+   of those that don't, and mark BB as postcheck..  */
+
+static bool
+hardcfr_sibcall_search_preds (basic_block bb,
+			      chk_edges_t &chk_edges,
+			      int &count_chkcall,
+			      auto_sbitmap &chkcall_blocks,
+			      int &count_postchk,
+			      auto_sbitmap &postchk_blocks,
+			      tree *retptr)
+{
+  /* For the exit block, we wish to force a check at every
+     predecessor, so pretend we've already found a pred that had
+     checking, so that we schedule checking at every one of its pred
+     edges.  */
+  bool first = bb->index >= NUM_FIXED_BLOCKS;
+  bool postchecked = true;
+
+  gphi *retphi = NULL;
+  if (retptr && *retptr && TREE_CODE (*retptr) == SSA_NAME
+      && !SSA_NAME_IS_DEFAULT_DEF (*retptr)
+      && SSA_NAME_DEF_STMT (*retptr)
+      && is_a <gphi *> (SSA_NAME_DEF_STMT (*retptr))
+      && gimple_bb (SSA_NAME_DEF_STMT (*retptr)) == bb)
+    {
+      retphi = as_a <gphi *> (SSA_NAME_DEF_STMT (*retptr));
+      gcc_checking_assert (gimple_phi_result (retphi) == *retptr);
+    }
+
+  for (int i = EDGE_COUNT (bb->preds); i--; first = false)
+    {
+      edge e = EDGE_PRED (bb, i);
+
+      bool checked
+	= hardcfr_sibcall_search_block (e->src, chk_edges,
+					count_chkcall, chkcall_blocks,
+					count_postchk, postchk_blocks,
+					!retphi ? retptr
+					: gimple_phi_arg_def_ptr (retphi, i));
+
+      if (first)
+	{
+	  postchecked = checked;
+	  continue;
+	}
+
+      /* When we first find a checked block, force a check at every
+	 other incoming edge we've already visited, and those we
+	 visit afterwards that don't have their own check, so that
+	 when we reach BB, the check has already been performed.  */
+      if (!postchecked && checked)
+	{
+	  for (int j = EDGE_COUNT (bb->preds); --j > i; )
+	    chk_edges.safe_push (EDGE_PRED (bb, j));
+	  postchecked = true;
+	}
+      if (postchecked && !checked)
+	chk_edges.safe_push (EDGE_PRED (bb, i));
+    }
+
+  if (postchecked && bb->index >= NUM_FIXED_BLOCKS)
+    {
+      if (bitmap_set_bit (postchk_blocks, bb->index))
+	count_postchk++;
+      else
+	gcc_unreachable ();
+    }
+
+  return postchecked;
+}
+
+
 class rt_bb_visited
 {
   /* Use a sufficiently wide unsigned type to hold basic block numbers.  */
@@ -176,8 +457,8 @@ class rt_bb_visited
      neither ENTRY nor EXIT, but maybe one-past-the-end, to compute
      the visited array length.  */
   blknum num2idx (blknum n) {
-    gcc_checking_assert (n >= 2 && n <= nblocks);
-    return (n - 2);
+    gcc_checking_assert (n >= NUM_FIXED_BLOCKS && n <= nblocks);
+    return (n - NUM_FIXED_BLOCKS);
   }
   /* Return the block vindex for BB, that must not be ENTRY or
      EXIT.  */
@@ -249,8 +530,7 @@ class rt_bb_visited
   }
 
   /* Set the bit corresponding to BB in VISITED.  Add to SEQ any
-     required gimple statements, and return SEQ, possibly
-     modified.  */
+     required gimple stmts, and return SEQ, possibly modified.  */
   gimple_seq vset (basic_block bb, gimple_seq seq = NULL)
   {
     tree bit, setme = vword (bb, &bit);
@@ -270,7 +550,7 @@ class rt_bb_visited
 
 public:
   /* Prepare to add control flow redundancy testing to CFUN.  */
-  rt_bb_visited ()
+  rt_bb_visited (int checkpoints)
     : nblocks (n_basic_blocks_for_fn (cfun)),
       vword_type (NULL), ckseq (NULL), rtcfg (NULL)
   {
@@ -353,8 +633,8 @@ public:
 					 NULL, NULL);
     gimple_seq_add_stmt (&ckseq, detach);
 
-    if (nblocks - 2 > blknum (param_hardcfr_max_inline_blocks)
-	|| !single_pred_p (EXIT_BLOCK_PTR_FOR_FN (cfun)))
+    if (nblocks - NUM_FIXED_BLOCKS > blknum (param_hardcfr_max_inline_blocks)
+	|| checkpoints > 1)
       {
 	/* Make sure vword_bits is wide enough for the representation
 	   of nblocks in rtcfg.  Compare with vword_bits << vword_bits,
@@ -379,65 +659,33 @@ public:
     gimple_seq_add_stmt (&ckseq, ckfail_init);
   }
 
-  /* Insert SEQ on E, or close enough (e.g., before a noreturn or tail
-     call at the end of E->src).  */
-  void insert_exit_check (gimple_seq seq, edge e)
+  /* Insert SEQ before a resx or a call in INSBB.  */
+  void insert_exit_check_in_block (gimple_seq seq, basic_block insbb)
   {
-    basic_block insbb = e->src;
-
-    /* If the returning block ends with a noreturn call, insert
-       checking before it.  This is particularly important for
-       __builtin_return.  Other noreturn calls won't have an edge to
-       the exit block, so they won't even be considered as exit paths.
-
-       Insert-on-edge inserts before other return stmts, but not
-       before calls, and if a single-block function had the check
-       sequence inserted after a noreturn call, it would be useless,
-       but checking would still flag it as malformed if block 2 has a
-       fallthru edge to the exit block.
-
-       Also avoid disrupting tail calls, inserting the check before
-       them.  This works for must-tail calls, but tail calling as an
-       optimization is detected too late for us.  */
     gimple_stmt_iterator gsi = gsi_last_bb (insbb);
-    gimple *ret = gsi_stmt (gsi);
-    if (ret && is_a <greturn *> (ret))
-      {
+
+    while (!gsi_end_p (gsi))
+      if (is_a <gresx *> (gsi_stmt (gsi))
+	  || is_a <gcall *> (gsi_stmt (gsi)))
+	break;
+      else
 	gsi_prev (&gsi);
-	if (!gsi_end_p (gsi))
-	  ret = gsi_stmt (gsi);
-      }
-    if (ret && is_a <gcall *> (ret)
-	&& (gimple_call_noreturn_p (ret)
-	    || gimple_call_must_tail_p (as_a <gcall *> (ret))
-	    || gimple_call_tail_p (as_a <gcall *> (ret))))
-      gsi_insert_seq_before (&gsi, seq, GSI_SAME_STMT);
-    else
-      gsi_insert_seq_on_edge_immediate (e, seq);
+
+    gsi_insert_seq_before (&gsi, seq, GSI_SAME_STMT);
+  }
+
+  /* Insert SEQ on E.  */
+  void insert_exit_check_on_edge (gimple_seq seq, edge e)
+  {
+    gsi_insert_seq_on_edge_immediate (e, seq);
   }
 
-  /* Add checking code on every exit edge, and initialization code on
+  /* Add checking code to CHK_EDGES, and initialization code on
      the entry edge.  Before this point, the CFG has been undisturbed,
      and all the needed data has been collected and safely stowed.  */
-  void check ()
+  void check (chk_edges_t &chk_edges,
+	      int count_chkcall, auto_sbitmap const &chkcall_blocks)
   {
-    /* Insert initializers for visited at the entry.  */
-    gimple_seq iseq = NULL;
-
-    gcall *vinit = gimple_build_call (builtin_decl_explicit
-				      (BUILT_IN_MEMSET), 3,
-				      build1 (ADDR_EXPR,
-					      build_pointer_type
-					      (TREE_TYPE (visited)),
-					      visited),
-				      integer_zero_node,
-				      TYPE_SIZE_UNIT (TREE_TYPE (visited)));
-    gimple_seq_add_stmt (&iseq, vinit);
-
-    gsi_insert_seq_on_edge_immediate (single_succ_edge
-				      (ENTRY_BLOCK_PTR_FOR_FN (cfun)),
-				      iseq);
-
     /* If we're using out-of-line checking, create and statically
        initialize the CFG checking representation, generate the
        checker call for the checking sequence, and insert it in all
@@ -498,28 +746,116 @@ public:
 						     rtcfg));
 	gimple_seq_add_stmt (&ckseq, call_chk);
 
+	gimple *clobber = gimple_build_assign (visited,
+					       build_clobber
+					       (TREE_TYPE (visited)));
+	gimple_seq_add_stmt (&ckseq, clobber);
+
 	/* If we have multiple exit edges, insert (copies of)
 	   ckseq in all of them.  */
-	for (int i = EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (cfun)->preds);
-	     i--; )
+	for (int i = chk_edges.length (); i--; )
 	  {
 	    gimple_seq seq = ckseq;
 	    /* Copy the sequence, unless we're dealing with the
 	       last edge (we're counting down to zero).  */
-	    if (i)
+	    if (i || count_chkcall)
+	      seq = gimple_seq_copy (seq);
+
+	    edge e = chk_edges[i];
+
+	    if (dump_file)
+	      {
+		if (e->dest == EXIT_BLOCK_PTR_FOR_FN (cfun))
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to exit.\n",
+			   e->src->index);
+		else
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to postcheck block %i.\n",
+			   e->src->index, e->dest->index);
+	      }
+
+	    insert_exit_check_on_edge (seq, e);
+
+	    gcc_checking_assert (!bitmap_bit_p (chkcall_blocks, e->src->index));
+	  }
+
+	sbitmap_iterator it;
+	unsigned i;
+	EXECUTE_IF_SET_IN_BITMAP (chkcall_blocks, 0, i, it)
+	  {
+	    basic_block bb = BASIC_BLOCK_FOR_FN (cfun, i);
+
+	    gimple_seq seq = ckseq;
+	    gcc_checking_assert (count_chkcall > 0);
+	    if (--count_chkcall)
 	      seq = gimple_seq_copy (seq);
 
-	    insert_exit_check (seq,
-			       EDGE_PRED (EXIT_BLOCK_PTR_FOR_FN (cfun), i));
+	    if (dump_file)
+	      fprintf (dump_file,
+		       "Inserting out-of-line check before stmt in block %i.\n",
+		       bb->index);
+
+	    insert_exit_check_in_block (seq, bb);
 	  }
+
+	gcc_checking_assert (count_chkcall == 0);
       }
     else
       {
 	/* Inline checking requires a single exit edge.  */
-	gimple *last = gsi_stmt (gsi_last (ckseq));
+	gimple *last = gimple_build_assign (visited,
+					       build_clobber
+					       (TREE_TYPE (visited)));
+	gimple_seq_add_stmt (&ckseq, last);
 
-	insert_exit_check (ckseq,
-			   single_pred_edge (EXIT_BLOCK_PTR_FOR_FN (cfun)));
+	if (!count_chkcall)
+	  {
+	    edge e = single_pred_edge (EXIT_BLOCK_PTR_FOR_FN (cfun));
+
+	    if (dump_file)
+	      {
+		if (e->dest == EXIT_BLOCK_PTR_FOR_FN (cfun))
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to postcheck block %i.\n",
+			   e->src->index, e->dest->index);
+		else
+		  fprintf (dump_file,
+			   "Inserting inline check in"
+			   " block %i's edge to exit.\n",
+			   e->src->index);
+	      }
+
+	    insert_exit_check_on_edge (ckseq, e);
+	  }
+	else
+	  {
+	    gcc_checking_assert (count_chkcall == 1);
+
+	    sbitmap_iterator it;
+	    unsigned i;
+	    EXECUTE_IF_SET_IN_BITMAP (chkcall_blocks, 0, i, it)
+	      {
+		basic_block bb = BASIC_BLOCK_FOR_FN (cfun, i);
+
+		gimple_seq seq = ckseq;
+		gcc_checking_assert (count_chkcall > 0);
+		if (--count_chkcall)
+		  seq = gimple_seq_copy (seq);
+
+		if (dump_file)
+		  fprintf (dump_file,
+			   "Inserting inline check before stmt in block %i.\n",
+			   bb->index);
+
+		insert_exit_check_in_block (seq, bb);
+	      }
+
+	    gcc_checking_assert (count_chkcall == 0);
+	  }
 
 	/* The inserted ckseq computes CKFAIL at LAST.  Now we have to
 	   conditionally trap on it.  */
@@ -540,8 +876,7 @@ public:
 	  add_bb_to_loop (trp, current_loops->tree_root);
 
 	/* Insert a conditional branch to the trap block.  If the
-	   conditional wouldn't be the last statement, split the
-	   block.  */
+	   conditional wouldn't be the last stmt, split the block.  */
 	gimple_stmt_iterator gsi = gsi_for_stmt (last);
 	if (!gsi_one_before_end_p (gsi))
 	  split_block (gsi_bb (gsi), gsi_stmt (gsi));
@@ -564,6 +899,24 @@ public:
 	if (dom_info_available_p (CDI_DOMINATORS))
 	  set_immediate_dominator (CDI_DOMINATORS, trp, gimple_bb (last));
       }
+
+    /* Insert initializers for visited at the entry.  Do this after
+       other insertions, to avoid messing with block numbers.  */
+    gimple_seq iseq = NULL;
+
+    gcall *vinit = gimple_build_call (builtin_decl_explicit
+				      (BUILT_IN_MEMSET), 3,
+				      build1 (ADDR_EXPR,
+					      build_pointer_type
+					      (TREE_TYPE (visited)),
+					      visited),
+				      integer_zero_node,
+				      TYPE_SIZE_UNIT (TREE_TYPE (visited)));
+    gimple_seq_add_stmt (&iseq, vinit);
+
+    gsi_insert_seq_on_edge_immediate (single_succ_edge
+				      (ENTRY_BLOCK_PTR_FOR_FN (cfun)),
+				      iseq);
   }
 
   /* Push onto RTCFG a (mask, index) pair to test for IBB when BB is
@@ -607,7 +960,7 @@ public:
     return false;
   }
 
-  /* Add to CKSEQ statements to clear CKPART if OBB is visited.  */
+  /* Add to CKSEQ stmts to clear CKPART if OBB is visited.  */
   void
   build_block_check (basic_block obb)
   {
@@ -632,34 +985,48 @@ public:
 
   /* Add to BB code to set its bit in VISITED, and add to RTCFG or
      CKSEQ the data or code needed to check BB's predecessors and
-     successors.  Do NOT change the CFG.  */
-  void visit (basic_block bb)
+     successors.  If CHECKPOINT, assume the block is a checkpoint,
+     whether or not it has an edge to EXIT.  If POSTCHECK, assume the
+     block post-dominates checkpoints and therefore no bitmap setting
+     or checks are to be performed in or for it.  Do NOT change the
+     CFG.  */
+  void visit (basic_block bb, bool checkpoint, bool postcheck)
   {
     /* Set the bit in VISITED when entering the block.  */
     gimple_stmt_iterator gsi = gsi_after_labels (bb);
-    gsi_insert_seq_before (&gsi, vset (bb), GSI_SAME_STMT);
+    if (!postcheck)
+      gsi_insert_seq_before (&gsi, vset (bb), GSI_SAME_STMT);
 
     if (rtcfg)
       {
-	/* Build a list of (index, mask) terminated by (NULL, 0).
-	   Consolidate masks with the same index when they're
-	   adjacent.  First, predecessors.  Count backwards, because
-	   we're going to reverse the list.  The order shouldn't
-	   matter, but let's not make it surprising.  */
-	for (int i = EDGE_COUNT (bb->preds); i--; )
-	  if (push_rtcfg_pair (EDGE_PRED (bb, i)->src, bb,
-			       ENTRY_BLOCK_PTR_FOR_FN (cfun)))
-	    break;
+	if (!postcheck)
+	  {
+	    /* Build a list of (index, mask) terminated by (NULL, 0).
+	       Consolidate masks with the same index when they're
+	       adjacent.  First, predecessors.  Count backwards, because
+	       we're going to reverse the list.  The order shouldn't
+	       matter, but let's not make it surprising.  */
+	    for (int i = EDGE_COUNT (bb->preds); i--; )
+	      if (push_rtcfg_pair (EDGE_PRED (bb, i)->src, bb,
+				   ENTRY_BLOCK_PTR_FOR_FN (cfun)))
+		break;
+	  }
 	rtcfg = tree_cons (NULL_TREE, build_int_cst (vword_type, 0), rtcfg);
 
-	/* Then, successors.  */
-	for (int i = EDGE_COUNT (bb->succs); i--; )
-	  if (push_rtcfg_pair (EDGE_SUCC (bb, i)->dest, bb,
-			       EXIT_BLOCK_PTR_FOR_FN (cfun)))
-	    break;
+	if (!postcheck)
+	  {
+	    /* Then, successors.  */
+	    if (!checkpoint
+		|| !push_rtcfg_pair (EXIT_BLOCK_PTR_FOR_FN (cfun),
+				     bb, EXIT_BLOCK_PTR_FOR_FN (cfun)))
+	      for (int i = EDGE_COUNT (bb->succs); i--; )
+		if (push_rtcfg_pair (EDGE_SUCC (bb, i)->dest, bb,
+				     EXIT_BLOCK_PTR_FOR_FN (cfun)))
+		  break;
+	  }
 	rtcfg = tree_cons (NULL_TREE, build_int_cst (vword_type, 0), rtcfg);
       }
-    else
+    else if (!postcheck)
       {
 	/* Schedule test to fail if the block was reached but somehow none
 	   of its predecessors were.  */
@@ -677,6 +1044,8 @@ public:
 	gassign *blkruns = gimple_build_assign (ckpart, unshare_expr (bit));
 	gimple_seq_add_stmt (&ckseq, blkruns);
 
+	if (checkpoint)
+	  build_block_check (EXIT_BLOCK_PTR_FOR_FN (cfun));
 	for (int i = 0, e = EDGE_COUNT (bb->succs); i < e; i++)
 	  build_block_check (EDGE_SUCC (bb, i)->dest);
 
@@ -687,21 +1056,355 @@ public:
   }
 };
 
+/* It might be useful to avoid checking before noreturn calls that are
+   known to always finish by throwing an exception, rather than by
+   ending the program or looping forever.  Such functions would have
+   to be annotated somehow, with an attribute or flag, so that
+   exception-raising functions, such as C++'s __cxa_throw,
+   __cxa_rethrow, and Ada's gnat_rcheck_*, gnat_reraise*,
+   ada.exception.raise_exception*, and the language-independent
+   unwinders could be detected here and handled differently from other
+   noreturn functions.  */
+static bool
+always_throwing_noreturn_call_p (gimple *)
+{
+  return false;
+}
+
 /* Control flow redundancy hardening: record the execution path, and
    verify at exit that an expect path was taken.  */
 
 unsigned int
-pass_harden_control_flow_redundancy::execute (function *)
+pass_harden_control_flow_redundancy::execute (function *fun)
 {
-  rt_bb_visited vstd;
-
+  bool const check_at_escaping_exceptions
+    = (flag_exceptions
+       && flag_harden_control_flow_redundancy_check_exceptions);
+  bool const check_before_noreturn_calls
+    = flag_harden_control_flow_redundancy_check_noreturn > HCFRNR_NEVER;
+  bool const check_before_nothrow_noreturn_calls
+    = (check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn >= HCFRNR_NOTHROW);
+  bool const check_before_throwing_noreturn_calls
+    = (flag_exceptions
+       && check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn > HCFRNR_NOTHROW);
+  bool const check_before_always_throwing_noreturn_calls
+    = (flag_exceptions
+       && check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn >= HCFRNR_ALWAYS);
   basic_block bb;
-  FOR_EACH_BB_FN (bb, cfun)
-    vstd.visit (bb);
+  basic_block bb_eh_cleanup = NULL;
+
+  if (check_at_escaping_exceptions)
+    {
+      int lp_eh_cleanup = -1;
+
+      /* Record the preexisting blocks, to avoid visiting newly-created
+	 blocks.  */
+      auto_sbitmap to_visit (last_basic_block_for_fn (fun));
+      bitmap_clear (to_visit);
+
+      FOR_EACH_BB_FN (bb, fun)
+	bitmap_set_bit (to_visit, bb->index);
+
+      /* Scan the blocks for stmts with escaping exceptions, that
+	 wouldn't be denoted in the CFG, and associate them with an
+	 empty cleanup handler around the whole function.  Walk
+	 backwards, so that even when we split the block, */
+      sbitmap_iterator it;
+      unsigned i;
+      EXECUTE_IF_SET_IN_BITMAP (to_visit, 0, i, it)
+	{
+	  bb = BASIC_BLOCK_FOR_FN (fun, i);
+
+	  for (gimple_stmt_iterator gsi = gsi_last_bb (bb);
+	       !gsi_end_p (gsi); gsi_prev (&gsi))
+	    {
+	      gimple *stmt = gsi_stmt (gsi);
+	      if (!stmt_could_throw_p (fun, stmt))
+		continue;
+
+	      /* If it must not throw, or if it already has a handler,
+		 we need not worry about it.  */
+	      if (lookup_stmt_eh_lp (stmt) != 0)
+		continue;
+
+	      /* Don't split blocks at, nor add EH edges to, tail
+		 calls, we will add verification before the call
+		 anyway.  */
+	      if (is_a <gcall *> (stmt)
+		  && (gimple_call_must_tail_p (as_a <gcall *> (stmt))
+		      || gimple_call_tail_p (as_a <gcall *> (stmt))
+		      || returning_call_p (as_a <gcall *> (stmt))))
+		continue;
+
+	      if (!gsi_one_before_end_p (gsi))
+		split_block (bb, stmt);
+	      /* A resx or noreturn call needs not be associated with
+		 the cleanup handler if we're going to add checking
+		 before it.  We only test cases that didn't require
+		 block splitting because noreturn calls would always
+		 be at the end of blocks, and we test for zero
+		 successors because if there is an edge, it's not
+		 noreturn, as any EH edges would have already been
+		 caught by the lookup_stmt_eh_lp test above.  */
+	      else if (check_before_noreturn_calls
+		       && EDGE_COUNT (bb->succs) == 0
+		       && (is_a <gresx *> (stmt)
+			   ? check_before_always_throwing_noreturn_calls
+			   : (!is_a <gcall *> (stmt)
+			      || !gimple_call_noreturn_p (stmt))
+			   ? (gcc_unreachable (), false)
+			   : (!flag_exceptions
+			      || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+			   ? check_before_nothrow_noreturn_calls
+			   : always_throwing_noreturn_call_p (stmt)
+			   ? check_before_always_throwing_noreturn_calls
+			   : check_before_throwing_noreturn_calls))
+		{
+		  if (dump_file)
+		    {
+		      fprintf (dump_file,
+			       "Bypassing cleanup for noreturn stmt"
+			       " in block %i:\n",
+			       bb->index);
+		      print_gimple_stmt (dump_file, stmt, 0);
+		    }
+		  continue;
+		}
+
+	      if (!bb_eh_cleanup)
+		{
+		  bb_eh_cleanup = create_empty_bb (bb);
+		  if (dom_info_available_p (CDI_DOMINATORS))
+		    set_immediate_dominator (CDI_DOMINATORS, bb_eh_cleanup, bb);
+		  if (current_loops)
+		    add_bb_to_loop (bb_eh_cleanup, current_loops->tree_root);
+
+		  /* Make the new block an EH cleanup for the call.  */
+		  eh_region new_r = gen_eh_region_cleanup (NULL);
+		  eh_landing_pad lp = gen_eh_landing_pad (new_r);
+		  tree label = gimple_block_label (bb_eh_cleanup);
+		  lp->post_landing_pad = label;
+		  EH_LANDING_PAD_NR (label) = lp_eh_cleanup = lp->index;
+
+		  /* Just propagate the exception.
+		     We will later insert the verifier call.  */
+		  gimple_stmt_iterator ehgsi;
+		  ehgsi = gsi_after_labels (bb_eh_cleanup);
+		  gresx *resx = gimple_build_resx (new_r->index);
+		  gsi_insert_before (&ehgsi, resx, GSI_SAME_STMT);
+
+		  if (dump_file)
+		    fprintf (dump_file,
+			     "Created cleanup block %i:\n",
+			     bb_eh_cleanup->index);
+		}
+	      else if (dom_info_available_p (CDI_DOMINATORS))
+		{
+		  basic_block immdom;
+		  immdom = get_immediate_dominator (CDI_DOMINATORS,
+						    bb_eh_cleanup);
+		  if (!dominated_by_p (CDI_DOMINATORS, bb, immdom))
+		    {
+		      immdom = nearest_common_dominator (CDI_DOMINATORS,
+							 immdom, bb);
+		      set_immediate_dominator (CDI_DOMINATORS,
+					       bb_eh_cleanup, immdom);
+		    }
+		}
+
+	      if (dump_file)
+		{
+		  fprintf (dump_file,
+			   "Associated cleanup block with stmt in block %i:\n",
+			   bb->index);
+		  print_gimple_stmt (dump_file, stmt, 0);
+		}
+
+	      add_stmt_to_eh_lp (stmt, lp_eh_cleanup);
+	      /* Finally, wire the EH cleanup block into the CFG.  */
+	      make_eh_edges (stmt);
+	    }
+	}
+
+      if (bb_eh_cleanup)
+	{
+	  /* A cfg_cleanup after bb_eh_cleanup makes for a more compact
+	     rtcfg, and it avoids bb numbering differences when we split
+	     blocks because of trailing debug insns only.  */
+	  cleanup_tree_cfg ();
+	  gcc_checking_assert (EDGE_COUNT (bb_eh_cleanup->succs) == 0);
+	}
+    }
+
+  /* These record blocks with calls that are to be preceded by
+     checkpoints, such as noreturn calls (if so chosen), must-tail
+     calls, potential early-marked tail calls, and returning calls (if
+     so chosen).  */
+  int count_chkcall = 0;
+  auto_sbitmap chkcall_blocks (last_basic_block_for_fn (fun));
+  bitmap_clear (chkcall_blocks);
+
+  /* We wish to add verification at blocks without successors, such as
+     noreturn calls (raising or not) and the reraise at the cleanup
+     block, but not other reraises: they will go through the cleanup
+     block.  */
+  if (check_before_noreturn_calls)
+    FOR_EACH_BB_FN (bb, fun)
+      {
+	gimple_stmt_iterator gsi = gsi_last_bb (bb);
+	if (gsi_end_p (gsi))
+	  continue;
+	gimple *stmt = gsi_stmt (gsi);
 
-  vstd.check ();
+	if (EDGE_COUNT (bb->succs) == 0)
+	  {
+	    /* A stmt at the end of a block without any successors is
+	       either a resx or a noreturn call without a local
+	       handler.  Check that it's one of the desired
+	       checkpoints.  */
+	    if (flag_exceptions && is_a <gresx *> (stmt)
+		? (check_before_always_throwing_noreturn_calls
+		   || bb == bb_eh_cleanup)
+		: (!is_a <gcall *> (stmt)
+		   || !gimple_call_noreturn_p (stmt))
+		? (/* Catch cases in which successors would be
+		      expected.  */
+		   gcc_unreachable (), false)
+		: (!flag_exceptions
+		   || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+		? check_before_nothrow_noreturn_calls
+		: always_throwing_noreturn_call_p (stmt)
+		? check_before_always_throwing_noreturn_calls
+		: check_before_throwing_noreturn_calls)
+	      {
+		if (dump_file)
+		  {
+		    fprintf (dump_file,
+			     "Scheduling check before stmt"
+			     " in succ-less block %i:\n",
+			     bb->index);
+		    print_gimple_stmt (dump_file, stmt, 0);
+		  }
+
+		if (bitmap_set_bit (chkcall_blocks, bb->index))
+		  count_chkcall++;
+		else
+		  gcc_unreachable ();
+	      }
+	    continue;
+	  }
 
-  return 0;
+	/* If there are no exceptions, it would seem like any noreturn
+	   call must have zero successor edges, but __builtin_return
+	   gets successor edges.  We don't want to handle it here, it
+	   will be dealt with in sibcall_search_preds.  Otherwise,
+	   check for blocks without non-EH successors, but skip those
+	   with resx stmts and edges (i.e., those other than that in
+	   bb_eh_cleanup), since those will go through bb_eh_cleanup,
+	   that will have been counted as noreturn above because it
+	   has no successors.  */
+	gcc_checking_assert (bb != bb_eh_cleanup
+			     || !check_at_escaping_exceptions);
+	if (flag_exceptions && is_a <gresx *> (stmt)
+	    ? check_before_always_throwing_noreturn_calls
+	    : (!is_a <gcall *> (stmt)
+	       || !gimple_call_noreturn_p (stmt))
+	    ? false
+	    : (!flag_exceptions
+	       || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+	    ? false /* rather than check_before_nothrow_noreturn_calls */
+	    : always_throwing_noreturn_call_p (stmt)
+	    ? check_before_always_throwing_noreturn_calls
+	    : check_before_throwing_noreturn_calls)
+	  {
+	    gcc_checking_assert (single_succ_p (bb)
+				 && (single_succ_edge (bb)->flags & EDGE_EH));
+
+	    if (dump_file)
+	      {
+		fprintf (dump_file,
+			 "Scheduling check before stmt"
+			 " in EH-succ block %i:\n",
+			 bb->index);
+		print_gimple_stmt (dump_file, stmt, 0);
+	      }
+
+	    if (bitmap_set_bit (chkcall_blocks, bb->index))
+	      count_chkcall++;
+	    else
+	      gcc_unreachable ();
+	  }
+      }
+  else if (bb_eh_cleanup)
+    {
+      if (bitmap_set_bit (chkcall_blocks, bb_eh_cleanup->index))
+	count_chkcall++;
+      else
+	gcc_unreachable ();
+    }
+
+  gcc_checking_assert (!bb_eh_cleanup
+		       || bitmap_bit_p (chkcall_blocks, bb_eh_cleanup->index));
+
+  /* If we don't have edges to exit nor noreturn calls (including the
+     cleanup reraise), then we may skip instrumentation: that would
+     amount to a function that ends with an infinite loop.  */
+  if (!count_chkcall
+      && EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (fun)->preds) == 0)
+    {
+      if (dump_file)
+	fprintf (dump_file,
+		 "Disabling CFR, no exit paths to check\n");
+
+      return 0;
+    }
+
+  /* Search for must-tail calls, early-marked potential tail calls,
+     and, if requested, returning calls.  As we introduce early
+     checks, */
+  int count_postchk = 0;
+  auto_sbitmap postchk_blocks (last_basic_block_for_fn (fun));
+  bitmap_clear (postchk_blocks);
+  chk_edges_t chk_edges;
+  hardcfr_sibcall_search_preds (EXIT_BLOCK_PTR_FOR_FN (fun), chk_edges,
+				count_chkcall, chkcall_blocks,
+				count_postchk, postchk_blocks,
+				NULL);
+
+  rt_bb_visited vstd (chk_edges.length () + count_chkcall);
+
+  auto_sbitmap combined_blocks (last_basic_block_for_fn (fun));
+  bitmap_copy (combined_blocks, chkcall_blocks);
+  int i;
+  edge *e;
+  FOR_EACH_VEC_ELT (chk_edges, i, e)
+    if (!bitmap_set_bit (combined_blocks, (*e)->src->index))
+      gcc_unreachable ();
+
+  /* Visit blocks in index order, because building rtcfg depends on
+     that.  Blocks must be compact, which the cleanup_cfg requirement
+     ensures.  This would also enable FOR_EACH_BB_FN to be used to
+     iterate in index order, but bb_eh_cleanup block splits and
+     insertions changes that.  */
+  gcc_checking_assert (n_basic_blocks_for_fn (fun)
+		       == last_basic_block_for_fn (fun));
+  for (int i = NUM_FIXED_BLOCKS; i < n_basic_blocks_for_fn (fun); i++)
+    {
+      bb = BASIC_BLOCK_FOR_FN (fun, i);
+      gcc_checking_assert (bb->index == i);
+      vstd.visit (bb, bitmap_bit_p (combined_blocks, i),
+		  bitmap_bit_p (postchk_blocks, i));
+    }
+
+  vstd.check (chk_edges, count_chkcall, chkcall_blocks);
+
+  return
+    TODO_update_ssa
+    | TODO_cleanup_cfg
+    | TODO_verify_il;
 }
 
 /* Instantiate a hardcfr pass.  */
diff --git a/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c b/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c
new file mode 100644
index 00000000000..a6992eb9f8e
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -O0 -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we don't insert checking before noreturn calls.  -O0 is tested
+   separately because h is not found to be noreturn without optimization.  */
+
+#include "torture/harden-cfr-noret.c"
+
+/* No out-of-line checks.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* Only one inline check at the end of f and of h2.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-always.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-always.c
new file mode 100644
index 00000000000..779896c60e8
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-always.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even enabling all checks before noreturn calls (leaving
+   returning calls enabled), we get checks before __builtin_return without
+   duplication (__builtin_return is both noreturn and a returning call).  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-never.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-never.c
new file mode 100644
index 00000000000..49ce17f5b93
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-never.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even enabling checks before never noreturn calls (leaving
+   returning calls enabled), we get checks before __builtin_return without
+   duplication (__builtin_return is both noreturn and a returning call).  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noopt.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noopt.c
new file mode 100644
index 00000000000..1512614791f
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noopt.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even disabling checks before both noreturn and returning
+   calls, we still get checks before __builtin_return.  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noret.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noret.c
new file mode 100644
index 00000000000..fd95bb7e3e3
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noret.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even disabling checks before returning calls (leaving noreturn
+   calls enabled), we still get checks before __builtin_return.  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-nothrow.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-nothrow.c
new file mode 100644
index 00000000000..c5c361234c4
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-nothrow.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even enabling checks before nothrow noreturn calls (leaving
+   returning calls enabled), we get checks before __builtin_return without
+   duplication (__builtin_return is both noreturn and a returning call).  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-retcl.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-retcl.c
new file mode 100644
index 00000000000..137dfbb95d6
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-retcl.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even disabling checks before noreturn calls (leaving returning
+   calls enabled), we still get checks before __builtin_return.  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c
index 70acdc95f25..b459ff6b864 100644
--- a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c
@@ -7,5 +7,11 @@ int f(int i) {
   return i;
 }
 
-/* Out-of-line checking, before both builtin_return and return.  */
+int g(int i) {
+  __builtin_return (&i);
+}
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
 /* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c
new file mode 100644
index 00000000000..8bd2d13ac18
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c
@@ -0,0 +1,18 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we don't insert checking before noreturn calls.  -O0 is tested
+   separately because h is not found to be noreturn without optimization, which
+   affects codegen for h2, so h2 is omitted here at -O0.  */
+
+#if !__OPTIMIZE__
+# define OMIT_H2
+#endif
+
+#include "harden-cfr-noret.c"
+
+
+/* No out-of-line checks.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* Only one inline check at the end of f.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c
new file mode 100644
index 00000000000..a804a6cfe59
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fno-exceptions -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that -fno-exceptions makes for implicit nothrow in noreturn
+   handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#include "harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c
new file mode 100644
index 00000000000..f390cfdbc59
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checking before nothrow noreturn calls.  */
+
+#include "harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c
new file mode 100644
index 00000000000..fdd803109a4
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c
@@ -0,0 +1,38 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checking before all noreturn calls.  */
+
+#ifndef ATTR_NOTHROW_OPT /* Overridden in harden-cfr-noret-noexcept.  */
+#define ATTR_NOTHROW_OPT __attribute__ ((__nothrow__))
+#endif
+
+extern void __attribute__ ((__noreturn__)) ATTR_NOTHROW_OPT g (void);
+
+void f(int i) {
+  if (i)
+    /* Out-of-line checks here...  */
+    g ();
+  /* ... and here.  */
+}
+
+void __attribute__ ((__noinline__, __noclone__))
+h(void) {
+  /* Inline check here.  */
+  g ();
+}
+
+#ifndef OMIT_H2 /* from harden-cfr-noret-never.  */
+void h2(void) {
+  /* Inline check either here, whether because of noreturn or tail call...  */
+  h ();
+  /* ... or here, if not optimizing.  */
+}
+#endif
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c
new file mode 100644
index 00000000000..6d11487bbba
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c
@@ -0,0 +1,8 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+#include "harden-cfr-tail.c"
+
+/* Inline checking after the calls, disabling tail calling.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 5 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 0 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c
new file mode 100644
index 00000000000..550b02ca088
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c
@@ -0,0 +1,35 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-exceptions -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checks before returning calls and alternate paths, even
+   at -O0, because of the explicit command-line flag.  */
+
+void g (void);
+void g2 (void);
+void g3 (void);
+
+void f (int i) {
+  if (!i)
+    /* Out-of-line checks here...  */
+    g ();
+  else if (i > 0)
+    /* here...  */
+    g2 ();
+  /* else */
+    /* and in the implicit else here.  */
+}
+
+void f2 (int i) {
+  if (!i)
+    /* Out-of-line check here...  */
+    g ();
+  else if (i > 0)
+    /* here...  */
+    g2 ();
+  else
+    /* and here.  */
+    g3 ();
+}
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 6 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 0 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
index 40d76c5c163..d5467eafa9f 100644
--- a/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
@@ -1,17 +1,52 @@
 /* { dg-do compile } */
-/* { dg-options "-fharden-control-flow-redundancy -fdump-tree-hardcfr -ffat-lto-objects" } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-hardcfr-check-exceptions -fdump-tree-hardcfr -ffat-lto-objects -Wno-return-type" } */
 
-/* We'd like to check that we insert checking so as to not disrupt tail calls,
-   but unfortunately mandatory tail calls are not available in C, and optimizing
-   calls as tail calls only takes place after hardcfr.  */
+/* Check that we insert CFR checking so as to not disrupt tail calls.
+   Mandatory tail calls are not available in C, and optimizing calls as tail
+   calls only takes place after hardcfr, so we insert checking before calls
+   followed by copies and return stmts with the same return value, that might
+   (or might not) end up optimized to tail calls.  */
 
-extern int g(int i);
+extern int g (int i);
 
-int f(int i) {
+int f1(int i) {
+  /* Inline check before the returning call.  */
   return g (i);
 }
 
-/* Inline checking before the tail call.  */
-/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
-/* Inline checking before the tail call.  */
-/* { dg-final { scan-tree-dump-times "\\\[tail\]" 1 "hardcfr" { xfail *-*-* } } } */
+extern void g2 (int i);
+
+void f2(int i) {
+  /* Inline check before the returning call, that ignores the returned value,
+     matching the value-less return.  */
+  g2 (i);
+  return;
+}
+
+void f3(int i) {
+  /* Inline check before the returning call.  */
+  g (i);
+}
+
+void f4(int i) {
+  if (i)
+    /* Out-of-line check before the returning call.  */
+    return g2 (i);
+  /* Out-of-line check before implicit return.  */
+}
+
+int f5(int i) {
+  /* Not regarded as a returning call, returning value other than callee's
+     returned value.  */
+  g (i);
+  /* Inline check after the non-returning call.  */
+  return i;
+}
+
+/* Out-of-line checks in f4, before returning calls and before return.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking in all other functions.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 4 "hardcfr" } } */
+/* Check before tail-call in all but f5, but f4 is out-of-line.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 3 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C b/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C
new file mode 100644
index 00000000000..17ea79f7cfb
--- /dev/null
+++ b/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C
@@ -0,0 +1,11 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects -O0" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions, and also checking before noreturn
+   calls.  h2 and h2b get an extra resx without ehcleanup.  */
+
+#include "torture/harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 16 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C b/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C
new file mode 100644
index 00000000000..f0338ccc361
--- /dev/null
+++ b/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C
@@ -0,0 +1,10 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -foptimize-sibling-calls -fdump-tree-hardcfr -O0" } */
+
+/* -fhardcfr-check-returning-calls gets implicitly disabled because,
+   -at O0, -foptimize-sibling-calls has no effect.  */
+
+#include "torture/harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 12 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C
new file mode 100644
index 00000000000..0d35920c7ee
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  */
+
+#include "harden-cfr-noret-no-nothrow.C"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Bypassing cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C
new file mode 100644
index 00000000000..b7d247ff43c
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C
@@ -0,0 +1,18 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  Expected results for =never and =nothrow are the same,
+   since the functions are not nothrow.  */
+
+#include "harden-cfr-noret-no-nothrow.C"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Associated cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C
new file mode 100644
index 00000000000..62c58cfd406
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C
@@ -0,0 +1,23 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#if ! __OPTIMIZE__
+void __attribute__ ((__noreturn__)) h (void);
+#endif
+
+#include "../../c-c++-common/torture/harden-cfr-noret.c"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Associated cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C
new file mode 100644
index 00000000000..0286f6e6d3f
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C
@@ -0,0 +1,20 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions, and also checking before noreturn
+   calls.  */
+
+#if ! __OPTIMIZE__
+/* Without optimization, functions with cleanups end up with an extra
+   resx that is not optimized out, so arrange to optimize them.  */
+void __attribute__ ((__optimize__ (1))) h2(void);
+void __attribute__ ((__optimize__ (1))) h2b(void);
+#endif
+
+#include "harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 14 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
+/* h, h2, h2b, and h4.  */
+/* { dg-final { scan-tree-dump-times "Bypassing" 4 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C
new file mode 100644
index 00000000000..885b0b236af
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C
@@ -0,0 +1,11 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions  -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we do not insert cleanups for checking around the bodies
+   of maybe-throwing functions.  h4 doesn't get any checks, because we
+   don't have noreturn checking enabled.  */
+
+#include "harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 6 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C
new file mode 100644
index 00000000000..32def637255
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C
@@ -0,0 +1,31 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -foptimize-sibling-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions.  These results depend on checking before
+   returning calls, which is only enabled when sibcall optimizations
+   are enabled, so change the optimization mode to -O1 for f and f2,
+   so that -foptimize-sibling-calls can take effect and enable
+   -fhardcfr-check-returning-calls, so that we get the same results.
+   There is a separate test for -O0.  */
+
+#if ! __OPTIMIZE__
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) f(int i);
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) f2(int i);
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) h3(void);
+#endif
+
+#include "harden-cfr-throw.C"
+
+/* f gets out-of-line checks before the unwrapped tail call and in the
+   else edge.  */
+/* f2 gets out-of-line checks before both unwrapped tail calls.  */
+/* h gets out-of-line checks before the implicit return and in the
+   cleanup block.  */
+/* h2 and h2b get out-of-line checks before the cleanup returning
+   call, and in the cleanup block.  */
+/* h3 gets an inline check before the __cxa_end_catch returning call.  */
+/* h4 gets an inline check in the cleanup block.  */
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 10 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C
new file mode 100644
index 00000000000..992fbdad381
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C
@@ -0,0 +1,65 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions.  */
+
+extern void g (void);
+extern void g2 (void);
+
+void f(int i) {
+  if (i)
+    g ();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void f2(int i) {
+  if (i)
+    g ();
+  else
+    g2 ();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h(void) {
+  try {
+    g ();
+  } catch (...) {
+    throw;
+  }
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+struct needs_cleanup {
+  ~needs_cleanup();
+};
+
+void h2(void) {
+  needs_cleanup y; /* No check in the cleanup handler.  */
+  g();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+extern void __attribute__ ((__nothrow__)) another_cleanup (void*);
+
+void h2b(void) {
+  int x __attribute__ ((cleanup (another_cleanup)));
+  g();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h3(void) {
+  try {
+    throw 1;
+  } catch (...) {
+  }
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h4(void) {
+  throw 1;
+  /* Inline check in the cleanup around the __cxa_throw noreturn call.  */
+}
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 12 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c b/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c
new file mode 100644
index 00000000000..8e4ee1fab08
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c
@@ -0,0 +1,15 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C makes for implicit nothrow in noreturn handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#include "../../c-c++-common/torture/harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c b/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c
new file mode 100644
index 00000000000..634d98f1ffc
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c
@@ -0,0 +1,40 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-hardcfr-check-exceptions -fdump-tree-hardcfr -ffat-lto-objects -Wno-return-type" } */
+
+/* In C only, check some additional cases (comparing with
+   c-c++-common/torture/harden-cfr-tail.c) of falling off the end of non-void
+   function.  C++ would issue an unreachable call in these cases.  */
+
+extern int g (int i);
+
+int f1(int i) {
+  /* Inline check before the returning call, that doesn't return anything.  */
+  g (i);
+  /* Implicit return without value, despite the return type; this combination
+     enables tail-calling of g, and is recognized as a returning call.  */
+}
+
+extern void g2 (int i);
+
+int f2(int i) {
+  /* Inline check before the returning call, that disregards its return
+     value.  */
+  g2 (i);
+  /* Implicit return without value, despite the return type; this combination
+     enables tail-calling of g2, and is recognized as a returning call.  */
+}
+
+int f3(int i) {
+  if (i)
+    /* Out-of-line check before the returning call.  */
+    return g (i);
+  /* Out-of-line check before implicit return.  */
+}
+
+/* Out-of-line checks in f3, before returning calls and before return.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking in all other functions.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
+/* Check before tail-call in all functions, but f3 is out-of-line.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/gnat.dg/hardcfr.adb b/gcc/testsuite/gnat.dg/hardcfr.adb
index b578a913d75..abe1605c029 100644
--- a/gcc/testsuite/gnat.dg/hardcfr.adb
+++ b/gcc/testsuite/gnat.dg/hardcfr.adb
@@ -1,5 +1,5 @@
 --  { dg-do run }
---  { dg-options "-fharden-control-flow-redundancy -fdump-tree-hardcfr --param=hardcfr-max-blocks=22 --param=hardcfr-max-inline-blocks=12 -O0" }
+--  { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions -fdump-tree-hardcfr --param=hardcfr-max-blocks=22 --param=hardcfr-max-inline-blocks=12 -O0" }
 
 procedure HardCFR is
    function F (I, J : Integer) return Integer is
diff --git a/libgcc/hardcfr.c b/libgcc/hardcfr.c
index 8ef29428111..55f6b995f2f 100644
--- a/libgcc/hardcfr.c
+++ b/libgcc/hardcfr.c
@@ -60,14 +60,25 @@ extern void __hardcfr_check (size_t blocks,
 			     vword const *visited,
 			     vword const *cfg);
 
+/* Compute the MASK for the bit representing BLOCK in WORDIDX's vword in a
+   visited blocks bit array.  */
+static inline void
+block2mask (size_t const block, vword *const mask, size_t *const wordidx)
+{
+  size_t wbits = __CHAR_BIT__ * sizeof (vword);
+  *wordidx = block / wbits;
+  *mask = (vword)1 << (block % wbits);
+}
 
 /* Check whether the bit corresponding to BLOCK is set in VISITED.  */
 static inline bool
 visited_p (size_t const block, vword const *const visited)
 {
-  size_t wbits = __CHAR_BIT__ * sizeof (vword);
-  vword w = visited[block / wbits];
-  return (w & ((vword)1 << (block % wbits))) != 0;
+  vword mask;
+  size_t wordidx;
+  block2mask (block, &mask, &wordidx);
+  vword w = visited[wordidx];
+  return (w & mask) != 0;
 }
 
 /* Read and consume a mask from **CFG_IT.  (Consume meaning advancing the
@@ -118,19 +129,19 @@ consume_seq (vword const **const cfg_it)
    we reach the terminator without finding any.  Consume the entire sequence
    otherwise, so that *CFG_IT points just past the terminator, which may be the
    beginning of the next sequence.  */
-static inline void
+static inline bool
 check_seq (vword const *const visited, vword const **const cfg_it)
 {
   vword mask;
   size_t wordidx;
 
   /* If the block was visited, check that at least one of the
-     preds was also visited.  */
+     preds/succs was also visited.  */
   do
     /* If we get to the end of the sequence without finding any
        match, something is amiss.  */
     if (!next_pair (cfg_it, &mask, &wordidx))
-      __builtin_trap ();
+      return false;
   /* Keep searching until we find a match, at which point the
      condition is satisfied.  */
   while (!test_mask (visited, mask, wordidx));
@@ -139,6 +150,94 @@ check_seq (vword const *const visited, vword const **const cfg_it)
      skipped the block, so as to position the iterator at the beginning of the
      next .  */
   consume_seq (cfg_it);
+
+  return true;
+}
+
+/* Print out the CFG with BLOCKS blocks, presumed to be associated with CALLER.
+   This is expected to be optimized out entirely, unless the verbose part of
+   __hardcfr_check_fail is enabled.  */
+static inline void
+__hardcfr_debug_cfg (size_t const blocks,
+		     void const *const caller,
+		     vword const *const cfg)
+{
+  __builtin_printf ("CFG at %p, for %p", cfg, caller);
+  vword const *cfg_it = cfg;
+  for (size_t i = 0; i < blocks; i++)
+    {
+      vword mask; size_t wordidx;
+      block2mask (i, &mask, &wordidx);
+      __builtin_printf ("\nblock %lu (%lu/0x%lx)\npreds: ",
+			(unsigned long)i,
+			(unsigned long)wordidx, (unsigned long)mask);
+      while (next_pair (&cfg_it, &mask, &wordidx))
+	__builtin_printf (" (%lu/0x%lx)",
+			  (unsigned long)wordidx, (unsigned long)mask);
+      __builtin_printf ("\nsuccs: ");
+      while (next_pair (&cfg_it, &mask, &wordidx))
+	__builtin_printf (" (%lu/0x%lx)",
+			  (unsigned long)wordidx, (unsigned long)mask);
+    }
+  __builtin_printf ("\n");
+}
+
+#ifndef ATTRIBUTE_UNUSED
+# define ATTRIBUTE_UNUSED __attribute__ ((__unused__))
+#endif
+
+/* This is called when an out-of-line hardcfr check fails.  All the arguments
+   are ignored, and it just traps, unless HARDCFR_VERBOSE_FAIL is enabled.  IF
+   it is, it prints the PART of the CFG, expected to have BLOCKS blocks, that
+   failed at CALLER's BLOCK, and the VISITED bitmap.  When the verbose mode is
+   enabled, it also forces __hardcfr_debug_cfg (above) to be compiled into an
+   out-of-line function, that could be called from a debugger.
+   */
+static inline void
+__hardcfr_check_fail (size_t const blocks ATTRIBUTE_UNUSED,
+		      vword const *const visited,
+		      vword const *const cfg ATTRIBUTE_UNUSED,
+		      size_t const block ATTRIBUTE_UNUSED,
+		      int const part ATTRIBUTE_UNUSED,
+		      void const *const caller ATTRIBUTE_UNUSED)
+{
+#if HARDCFR_VERBOSE_FAIL
+  static const char *parts[] = { "preds", "succs" };
+
+  vword mask; size_t wordidx;
+  block2mask (block, &mask, &wordidx);
+  __builtin_printf ("hardcfr fail at %p block %lu (%lu/0x%lx), expected %s:",
+		    caller, (unsigned long)block,
+		    (unsigned long)wordidx, (unsigned long)mask,
+		    parts[part]);
+
+  /* Skip data for previous blocks.  */
+  vword const *cfg_it = cfg;
+  for (size_t i = block; i--; )
+    {
+      consume_seq (&cfg_it);
+      consume_seq (&cfg_it);
+    }
+  for (size_t i = part; i--; )
+    consume_seq (&cfg_it);
+
+  while (next_pair (&cfg_it, &mask, &wordidx))
+    __builtin_printf (" (%lu/0x%lx)",
+		      (unsigned long)wordidx, (unsigned long)mask);
+
+  __builtin_printf ("\nvisited:");
+  block2mask (blocks, &mask, &wordidx);
+  for (size_t i = 0; i <= wordidx; i++)
+    __builtin_printf (" (%lu/0x%lx)",
+		      (unsigned long)i, (unsigned long)visited[i]);
+  __builtin_printf ("\n");
+
+  /* Reference __hardcfr_debug_cfg so that it's output out-of-line, so that it
+     can be called from a debugger.  */
+  if (!caller || caller == __hardcfr_debug_cfg)
+    return;
+#endif
+  __builtin_trap ();
 }
 
 /* Check that, for each of the BLOCKS basic blocks, if its bit is set in
@@ -168,9 +267,13 @@ __hardcfr_check (size_t const blocks,
       else
 	{
 	  /* Check predecessors.  */
-	  check_seq (visited, &cfg_it);
+	  if (!check_seq (visited, &cfg_it))
+	    __hardcfr_check_fail (blocks, visited, cfg, i, 0,
+				  __builtin_return_address (0));
 	  /* Check successors.  */
-	  check_seq (visited, &cfg_it);
+	  if (!check_seq (visited, &cfg_it))
+	    __hardcfr_check_fail (blocks, visited, cfg, i, 1,
+				  __builtin_return_address (0));
 	}
     }
 }

^ permalink raw reply	[flat|nested] 13+ messages in thread
* [gcc(refs/users/aoliva/heads/testme)] hardcfr: add optional checkpoints
@ 2022-10-06 10:31 Alexandre Oliva
  0 siblings, 0 replies; 13+ messages in thread
From: Alexandre Oliva @ 2022-10-06 10:31 UTC (permalink / raw)
  To: gcc-cvs

https://gcc.gnu.org/g:2e42fb0eaf037fafdf5793fea7014ed995f30192

commit 2e42fb0eaf037fafdf5793fea7014ed995f30192
Author: Alexandre Oliva <oliva@adacore.com>
Date:   Thu Oct 6 04:43:22 2022 -0300

    hardcfr: add optional checkpoints
    
    Previously, control flow redundancy only checked the visited bitmap
    against the control flow graph at return points and before mandatory
    tail calls, missing various other possibilities of exiting a
    subprogram, such as by raising or propagating exceptions, and calling
    noreturn functions.  The checks inserted before returns also prevented
    potential tail-call optimizations.
    
    This incremental change introduces options to control checking at each
    of these previously-missed checkpoints.  Unless disabled, a cleanup is
    introduced to check when an exceptions escapes a subprogram.  To avoid
    disrupting sibcall optimizations, when they are enabled, checks are
    introduced before calls whose results are immediately returned,
    whether or not they are ultimately optimized.  If enabled, checks are
    introduced before noreturn calls and exception raises, or only before
    nothrow noreturn calls.
    
    
    for  gcc/ChangeLog
    
            * common.opt (-fhardcfr-check-returning-calls): New.
            (-fhardcfr-check-exceptions): New.
            (-fhardcfr-check-noreturn-calls=*): New.
            (Enum hardcfr_check_noreturn_calls): New.
            * doc/invoke.texi: Document them.
            * flag-types.h (enum hardcfr_noret): New.
            * gimple-harden-control-flow.cc: Include more headers.
            (pass_data_harden_control_flow_redundancy): Move
            properties_finish to pass return.
            (pass_harden_control_flow_redundancy::gate): Move
            no-edge-to-exit checking to execute, after exception
            transformations.  Use symbolic NUM_FIXED_BLOCKS.  Test for CFG
            before testing block count.
            (check_returning_calls_p): New.
            (hardcfr_scan_block): New.
            (returnign_call_p): New.
            (chk_edges_t): New.
            (hardcfr_sibcall_search_block): New.
            (hardcfr_sibcall_search_preds): New.
            (rt_bb_visited::num2idx): Use symbolic NUM_FIXED_BLOCKS.
            (rt_bb_visited::rt_bb_visited): Likewise.  Take checkpoints
            count, test it to select out-of-line checks.
            (rt_bb_visited::insert_exit_check): Removed.
            (rt_bb_visited::insert_exit_check_in_blocks): New.
            (rt_bb_visited::insert_exit_check_on_edge): New.
            (rt_bb_visited::check): Take checkpoint edges and blocks.
            Move initialization insertion to the end.  Insert checks on
            edges, and before calls in select blocks.  Clobber the visited
            array to the check sequence.  Output actions to dump_file.
            (rt_bb_visited::visit): Take checkpoint and postcheck.  Assume
            an edge to exit if checkpoint, and skip testing if postcheck.
            (always_throwing_noreturn_call_p): New, dummy.
            (pass_harden_control_flow_redundancy::execute): Introduce
            cleanup block if requested and needed.  Scan blocks for
            noreturn and returning calls if checkpoints before them were
            requested.  Log actions to dump_file.  Visit blocks in index
            order.
    
    for  libgcc/ChangeLog
    
            * hardcfr.c (block2mask): Split out of...
            (visited_p): ... this.
            (check_seq): Move trapping out, return result instead.
            (__hardcfr_debug_cfg): New.
            (__hardcfr_check_fail): New, with optional verbose error
            printing before trapping.
            (__hardcfr_check): Call __hardcfr_check_fail when check_seq
            fails.
    
    for  gcc/testsuite/ChangeLog
    
            * c-c++-common/harden-cfr-noret-never-O0.c: New.
            * c-c++-common/torture/harden-cfr-noret-never.c: New.
            * c-c++-common/torture/harden-cfr-noret-noexcept.c: New.
            * c-c++-common/torture/harden-cfr-noret-nothrow.c: New.
            * c-c++-common/torture/harden-cfr-noret.c: New.
            * c-c++-common/torture/harden-cfr-notail.c: New.
            * c-c++-common/torture/harden-cfr-returning.c: New.
            * c-c++-common/torture/harden-cfr-tail.c: Extend.
            * c-c++-common/torture/harden-cfr-bret-always.c: New.
            * c-c++-common/torture/harden-cfr-bret-never.c: New.
            * c-c++-common/torture/harden-cfr-bret-noopt.c: New.
            * c-c++-common/torture/harden-cfr-bret-noret.c: New.
            * c-c++-common/torture/harden-cfr-bret-nothrow.c: New.
            * c-c++-common/torture/harden-cfr-bret-retcl.c: New.
            * c-c++-common/torture/harden-cfr-bret.c (g): New.
            * g++.dg/harden-cfr-throw-always-O0.C: New.
            * g++.dg/harden-cfr-throw-returning-O0.C: New.
            * g++.dg/torture/harden-cfr-noret-always-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-noret-never-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-noret-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-throw-always.C: New.
            * g++.dg/torture/harden-cfr-throw-nocleanup.C: New.
            * g++.dg/torture/harden-cfr-throw-returning.C: New.
            * g++.dg/torture/harden-cfr-throw.C: New.
            * gcc.dg/torture/harden-cfr-noret-no-nothrow.c: New.
            * gcc.dg/torture/harden-cfr-tail-ub.c: New.
            * gnat.dg/hardcfr.adb: Disable checking at exceptions.

Diff:
---
 gcc/common.opt                                     |  33 +
 gcc/doc/invoke.texi                                |  74 +-
 gcc/flag-types.h                                   |  10 +
 gcc/gimple-harden-control-flow.cc                  | 918 ++++++++++++++++++---
 .../c-c++-common/harden-cfr-noret-never-O0.c       |  12 +
 .../c-c++-common/torture/harden-cfr-bret-always.c  |  13 +
 .../c-c++-common/torture/harden-cfr-bret-never.c   |  13 +
 .../c-c++-common/torture/harden-cfr-bret-noopt.c   |  12 +
 .../c-c++-common/torture/harden-cfr-bret-noret.c   |  12 +
 .../c-c++-common/torture/harden-cfr-bret-nothrow.c |  13 +
 .../c-c++-common/torture/harden-cfr-bret-retcl.c   |  12 +
 .../c-c++-common/torture/harden-cfr-bret.c         |   8 +-
 .../c-c++-common/torture/harden-cfr-noret-never.c  |  18 +
 .../torture/harden-cfr-noret-noexcept.c            |  16 +
 .../torture/harden-cfr-noret-nothrow.c             |  13 +
 .../c-c++-common/torture/harden-cfr-noret.c        |  38 +
 .../c-c++-common/torture/harden-cfr-notail.c       |   8 +
 .../c-c++-common/torture/harden-cfr-returning.c    |  35 +
 .../c-c++-common/torture/harden-cfr-tail.c         |  55 +-
 gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C  |  11 +
 .../g++.dg/harden-cfr-throw-returning-O0.C         |  10 +
 .../torture/harden-cfr-noret-always-no-nothrow.C   |  16 +
 .../torture/harden-cfr-noret-never-no-nothrow.C    |  18 +
 .../g++.dg/torture/harden-cfr-noret-no-nothrow.C   |  23 +
 .../g++.dg/torture/harden-cfr-throw-always.C       |  20 +
 .../g++.dg/torture/harden-cfr-throw-nocleanup.C    |  11 +
 .../g++.dg/torture/harden-cfr-throw-returning.C    |  31 +
 gcc/testsuite/g++.dg/torture/harden-cfr-throw.C    |  65 ++
 .../gcc.dg/torture/harden-cfr-noret-no-nothrow.c   |  15 +
 gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c  |  40 +
 gcc/testsuite/gnat.dg/hardcfr.adb                  |   2 +-
 libgcc/hardcfr.c                                   | 119 ++-
 32 files changed, 1561 insertions(+), 133 deletions(-)

diff --git a/gcc/common.opt b/gcc/common.opt
index f093be93542..11c1ec95c8f 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -1795,6 +1795,39 @@ fharden-control-flow-redundancy
 Common Var(flag_harden_control_flow_redundancy) Optimization
 Harden control flow by recording and checking execution paths.
 
+fhardcfr-check-returning-calls
+Common Var(flag_harden_control_flow_redundancy_check_returning_calls) Init(-1) Optimization
+Check CFR execution paths also before calls followed by returns of their results.
+
+fhardcfr-check-exceptions
+Common Var(flag_harden_control_flow_redundancy_check_exceptions) Init(-1) Optimization
+Check CFR execution paths also when exiting a function through an exception.
+
+fhardcfr-check-noreturn-calls=
+Common Joined RejectNegative Enum(hardcfr_check_noreturn_calls) Var(flag_harden_control_flow_redundancy_check_noreturn) Init(HCFRNR_UNSPECIFIED) Optimization
+-fhardcfr-check-noreturn-calls=[always|nothrow|never]	Check CFR execution paths also before calling noreturn functions.
+
+Enum
+Name(hardcfr_check_noreturn_calls) Type(enum hardcfr_noret) UnknownError(unknown hardcfr noreturn checking level %qs)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(never) Value(HCFRNR_NEVER)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(nothrow) Value(HCFRNR_NOTHROW)
+
+; ??? There could be yet another option here, that checked before
+; noreturn calls, except for those known to always throw, if we had
+; means to distinguish noreturn functions known to always throw, such
+; as those used to (re)raise exceptions, from those that merely might
+; throw.  "not always" stands for "not always-throwing", but it also
+; contrasts with "always" below.
+; EnumValue
+; Enum(hardcfr_check_noreturn_calls) String(not-always) Value(HCFRNR_NOT_ALWAYS)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(always) Value(HCFRNR_ALWAYS)
+
 ; Nonzero means ignore `#ident' directives.  0 means handle them.
 ; Generate position-independent code for executables if possible
 ; On SVR4 targets, it also controls whether or not to emit a
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index eeb5b8ce210..ff2b5bce5cf 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -626,7 +626,9 @@ Objective-C and Objective-C++ Dialects}.
 -fsanitize-undefined-trap-on-error  -fbounds-check @gol
 -fcf-protection=@r{[}full@r{|}branch@r{|}return@r{|}none@r{|}check@r{]} @gol
 -fharden-compares -fharden-conditional-branches @gol
--fharden-control-flow-redundancy @gol
+-fharden-control-flow-redundancy  -fhardcfr-check-exceptions  @gol
+-fhardcfr-check-returning-calls  @gol
+-fhardcfr-check-noreturn-calls=@r{[}always@r{|}nothrow@r{|}never@r{]}  @gol
 -fstack-protector  -fstack-protector-all  -fstack-protector-strong @gol
 -fstack-protector-explicit  -fstack-check @gol
 -fstack-limit-register=@var{reg}  -fstack-limit-symbol=@var{sym} @gol
@@ -16601,11 +16603,75 @@ conditionals.
 @item -fharden-control-flow-redundancy
 @opindex fharden-control-flow-redundancy
 Emit extra code to set booleans when entering basic blocks, and to
-verify, at function exits, that they amount to an execution path that is
-consistent with the control flow graph, trapping otherwise.  Tuning
-options @option{--param hardcfr-max-blocks} and @option{--param
+verify and trap, at function exits, when the booleans do not form an
+execution path that is compatible with the control flow graph.
+
+Verification takes place before returns, before mandatory tail calls
+(see below) and, optionally, before escaping exceptions with
+@option{-fhardcfr-check-exceptions}, before returning calls with
+@option{-fhardcfr-check-returning-calls}, and before noreturn calls with
+@option{-fhardcfr-check-noreturn-calls}).  Tuning options
+@option{--param hardcfr-max-blocks} and @option{--param
 hardcfr-max-inline-blocks} are available.
 
+Tail call optimization takes place too late to affect control flow
+redundancy, but calls annotated as mandatory tail calls by language
+front-ends, and any calls marked early enough as potential tail calls
+would also have verification issued before the call, but these
+possibilities are merely theoretical, as these conditions can only be
+met when using custom compiler plugins.
+
+@item -fhardcfr-check-exceptions
+@opindex fhardcfr-check-exceptions
+@opindex fno-hardcfr-check-exceptions
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph at exception
+escape points, as if the function body was wrapped with a cleanup
+handler that performed the check and reraised.  This option is enabled
+by default; use @option{-fno-hardcfr-check-exceptions} to disable it.
+
+@item -fhardcfr-check-returning-calls
+@opindex fhardcfr-check-returning-calls
+@opindex fno-hardcfr-check-returning-calls
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph before any
+function call immediately followed by a return of its result, if any, so
+as to not prevent tail-call optimization, whether or not it is
+ultimately optimized to a tail call.
+
+This option is enabled by default whenever sibling call optimizations
+are enabled (see @option{-foptimize-sibling-calls}), but it can be
+enabled (or disabled, using its negated form) explicitly, regardless of
+the optimizations.
+
+@item -fhardcfr-check-noreturn-calls=@r{[}always@r{|}nothrow@r{|}never@r{]}
+@opindex fhardcfr-check-noreturn-calls
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph before
+@code{noreturn} calls, either all of them (@option{always}), those that
+may not return control to the caller through an exception either
+(@option{nothrow}), or none of them (@option{never}, the default).
+
+Checking before a @code{noreturn} function that may return control to
+the caller through an exception may cause checking to be performed more
+than once, if the exception is caught in the caller, whether by a
+handler or a cleanup.  When @option{-fhardcfr-check-exceptions} is also
+enabled, the compiler will avoid associating a @code{noreturn} call with
+the implicitly-added cleanup handler, since it would be redundant with
+the check performed before the call, but other handlers or cleanups in
+the function, if activated, will modify the recorded execution path and
+check it again when another checkpoint is hit.  The checkpoint may even
+be another @code{noreturn} call, so checking may end up performed
+multiple times.
+
+Various optimizers may cause calls to be marked as @code{noreturn}
+and/or @code{nothrow}, even in the absence of the corresponding
+attributes, which may affect the placement of checks before calls, as
+well as the addition of implicit cleanup handlers for them.  This
+unpredictability, and the fact that raising and reraising exceptions
+frequently amounts to implicitly calling @code{noreturn} functions, have
+made @option{never} the default setting for this option.
+
 @item -fstack-protector
 @opindex fstack-protector
 Emit extra code to check for buffer overflows, such as stack smashing
diff --git a/gcc/flag-types.h b/gcc/flag-types.h
index d2e751060ff..3fae7548cab 100644
--- a/gcc/flag-types.h
+++ b/gcc/flag-types.h
@@ -157,6 +157,16 @@ enum stack_reuse_level
   SR_ALL
 };
 
+/* Control Flow Redundancy hardening options for noreturn calls.  */
+enum hardcfr_noret
+{
+  HCFRNR_NEVER,
+  HCFRNR_NOTHROW,
+  HCFRNR_NOT_ALWAYS, /* Reserved for future use.  */
+  HCFRNR_ALWAYS,
+  HCFRNR_UNSPECIFIED = -1
+};
+
 /* The live patching level.  */
 enum live_patching_level
 {
diff --git a/gcc/gimple-harden-control-flow.cc b/gcc/gimple-harden-control-flow.cc
index 6b08846dbb1..cd5b3cf69b2 100644
--- a/gcc/gimple-harden-control-flow.cc
+++ b/gcc/gimple-harden-control-flow.cc
@@ -29,7 +29,12 @@ along with GCC; see the file COPYING3.  If not see
 #include "tree-pass.h"
 #include "ssa.h"
 #include "gimple-iterator.h"
+#include "gimple-pretty-print.h"
 #include "tree-cfg.h"
+#include "tree-cfgcleanup.h"
+#include "tree-eh.h"
+#include "except.h"
+#include "sbitmap.h"
 #include "basic-block.h"
 #include "cfghooks.h"
 #include "cfgloop.h"
@@ -60,9 +65,7 @@ const pass_data pass_data_harden_control_flow_redundancy = {
   0,	    // properties_provided
   0,	    // properties_destroyed
   TODO_cleanup_cfg, // properties_start
-  TODO_update_ssa
-  | TODO_cleanup_cfg
-  | TODO_verify_il, // properties_finish
+  0,        // properties_finish
 };
 
 class pass_harden_control_flow_redundancy : public gimple_opt_pass
@@ -79,16 +82,6 @@ public:
     if (!flag_harden_control_flow_redundancy)
       return false;
 
-    /* We don't verify when an exception escapes, propagated or raised
-       by the function itself, so we're only concerned with edges to
-       the exit block.  If there aren't any, the function doesn't
-       return normally, so there won't be any checking point, so
-       there's no point in running the pass.  Should we add
-       verification at exception escapes, we should at least look at
-       !flag_exceptions here.  */
-    if (EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (fun)->preds) == 0)
-      return false;
-
     /* Functions that return more than once, like setjmp and vfork
        (that also gets this flag set), will start recording a path
        after the first return, and then may take another path when
@@ -117,8 +110,9 @@ public:
 	return false;
       }
 
-    if (param_hardcfr_max_blocks > 0
-	&& n_basic_blocks_for_fn (fun) - 2 > param_hardcfr_max_blocks)
+    if (fun->cfg && param_hardcfr_max_blocks > 0
+	&& (n_basic_blocks_for_fn (fun) - NUM_FIXED_BLOCKS
+	    > param_hardcfr_max_blocks))
       {
 	warning_at (DECL_SOURCE_LOCATION (fun->decl), 0,
 		    "%qD has more than %u blocks, the requested"
@@ -134,6 +128,293 @@ public:
 
 }
 
+/* Return TRUE iff CFR checks should be inserted before returning
+   calls.  */
+
+static bool
+check_returning_calls_p ()
+{
+  return
+    flag_harden_control_flow_redundancy_check_returning_calls > 0
+    || (flag_harden_control_flow_redundancy_check_returning_calls < 0
+	/* Gates pass_tail_calls.  */
+	&& flag_optimize_sibling_calls
+	/* Gates pass_all_ptimizations.  */
+	&& optimize >= 1 && !optimize_debug);
+}
+
+/* Scan BB from the end, updating *RETPTR if given as return stmts and
+   copies are found.  Return a call or a stmt that cannot appear after
+   a tail call, or NULL if the top of the block is reached without
+   finding any.  */
+
+static gimple *
+hardcfr_scan_block (basic_block bb, tree **retptr)
+{
+  gimple_stmt_iterator gsi;
+  for (gsi = gsi_last_bb (bb); !gsi_end_p (gsi); gsi_prev (&gsi))
+    {
+      gimple *stmt = gsi_stmt (gsi);
+
+      /* Ignore labels, returns, nops, clobbers and debug stmts.  */
+      if (gimple_code (stmt) == GIMPLE_LABEL
+	  || gimple_code (stmt) == GIMPLE_NOP
+	  || gimple_code (stmt) == GIMPLE_PREDICT
+	  || gimple_clobber_p (stmt)
+	  || is_gimple_debug (stmt))
+	continue;
+
+      if (gimple_code (stmt) == GIMPLE_RETURN)
+	{
+	  greturn *gret = as_a <greturn *> (stmt);
+	  if (retptr)
+	    {
+	      gcc_checking_assert (!*retptr);
+	      *retptr = gimple_return_retval_ptr (gret);
+	    }
+	  continue;
+	}
+
+      /* Check for a call.  */
+      if (is_gimple_call (stmt))
+	return stmt;
+
+      /* Allow simple copies to the return value, updating the return
+	 value to be found in earlier assignments.  */
+      if (retptr && *retptr && gimple_assign_single_p (stmt)
+	  && **retptr == gimple_assign_lhs (stmt))
+	{
+	  *retptr = gimple_assign_rhs1_ptr (stmt);
+	  continue;
+	}
+
+      return stmt;
+    }
+
+  /* Any other kind of stmt will prevent a tail call.  */
+  return NULL;
+}
+
+/* Return TRUE iff CALL is to be preceded by a CFR checkpoint, i.e.,
+   if it's a returning call (one whose result is ultimately returned
+   without intervening non-copy statements) and we're checking
+   returning calls, a __builtin_return call (noreturn with a path to
+   the exit block), a must-tail call, or a tail call.  */
+
+static bool
+returning_call_p (gcall *call)
+{
+  if (!(gimple_call_noreturn_p (call)
+	|| gimple_call_must_tail_p (call)
+	|| gimple_call_tail_p (call)
+	|| check_returning_calls_p ()))
+    return false;
+
+  /* Quickly check that there's a path to exit compatible with a
+     returning call.  Detect infinite loops through the counter.  */
+  basic_block bb = gimple_bb (call);
+  auto_vec<basic_block, 10> path;
+  for (int i = n_basic_blocks_for_fn (cfun);
+       bb != EXIT_BLOCK_PTR_FOR_FN (cfun) && i--;
+       bb = single_succ (bb))
+    if (!single_succ_p (bb)
+	|| (single_succ_edge (bb)->flags & EDGE_EH) != 0)
+      return false;
+    else
+      path.safe_push (bb);
+
+  /* Check the stmts in the blocks and trace the return value.  */
+  tree *retptr = NULL;
+  for (;;)
+    {
+      gcc_checking_assert (!path.is_empty ());
+      basic_block bb = path.pop ();
+      gimple *stop = hardcfr_scan_block (bb, &retptr);
+      if (stop)
+	{
+	  if (stop != call)
+	    return false;
+	  gcc_checking_assert (path.is_empty ());
+	  break;
+	}
+
+      gphi *retphi = NULL;
+      if (retptr && *retptr && TREE_CODE (*retptr) == SSA_NAME
+	  && !SSA_NAME_IS_DEFAULT_DEF (*retptr)
+	  && SSA_NAME_DEF_STMT (*retptr)
+	  && is_a <gphi *> (SSA_NAME_DEF_STMT (*retptr))
+	  && gimple_bb (SSA_NAME_DEF_STMT (*retptr)) == bb)
+	{
+	  retphi = as_a <gphi *> (SSA_NAME_DEF_STMT (*retptr));
+	  gcc_checking_assert (gimple_phi_result (retphi) == *retptr);
+	}
+      else
+	continue;
+
+      gcc_checking_assert (!path.is_empty ());
+      edge e = single_succ_edge (path.last ());
+      int i = EDGE_COUNT (bb->preds);
+      while (i--)
+	if (EDGE_PRED (bb, i) == e)
+	  break;
+      gcc_checking_assert (i >= 0);
+      retptr = gimple_phi_arg_def_ptr (retphi, i);
+    }
+
+  return (gimple_call_noreturn_p (call)
+	  || gimple_call_must_tail_p (call)
+	  || gimple_call_tail_p (call)
+	  || (gimple_call_lhs (call) == (retptr ? *retptr : NULL)
+	      && check_returning_calls_p ()));
+}
+
+typedef auto_vec<edge, 10> chk_edges_t;
+
+/* Declare for mutual recursion.  */
+static bool hardcfr_sibcall_search_preds (basic_block bb,
+					  chk_edges_t &chk_edges,
+					  int &count_chkcall,
+					  auto_sbitmap &chkcall_blocks,
+					  int &count_postchk,
+					  auto_sbitmap &postchk_blocks,
+					  tree *retptr);
+
+/* Search backwards from the end of BB for a mandatory or potential
+   sibcall.  Schedule the block to be handled sort-of like noreturn if
+   so.  Recurse to preds, with updated RETPTR, if the block only
+   contains stmts that may follow such a call, scheduling checking at
+   edges and marking blocks as post-check as needed.  Return true iff,
+   at the end of the block, a check will have already been
+   performed.  */
+
+static bool
+hardcfr_sibcall_search_block (basic_block bb,
+			      chk_edges_t &chk_edges,
+			      int &count_chkcall,
+			      auto_sbitmap &chkcall_blocks,
+			      int &count_postchk,
+			      auto_sbitmap &postchk_blocks,
+			      tree *retptr)
+{
+  /* Conditionals and internal exceptions rule out tail calls.  */
+  if (!single_succ_p (bb)
+      || (single_succ_edge (bb)->flags & EDGE_EH) != 0)
+    return false;
+
+  gimple *stmt = hardcfr_scan_block (bb, &retptr);
+  if (!stmt)
+    return hardcfr_sibcall_search_preds (bb, chk_edges,
+					 count_chkcall, chkcall_blocks,
+					 count_postchk, postchk_blocks,
+					 retptr);
+
+  if (!is_a <gcall *> (stmt))
+    return false;
+
+  /* Avoid disrupting mandatory or early-marked tail calls,
+     inserting the check before them.  This works for
+     must-tail calls, but tail calling as an optimization is
+     detected too late for us.
+
+     Also check for noreturn calls here.  Noreturn calls won't
+     normally have edges to exit, so they won't be found here,
+     but __builtin_return does, and we must check before
+     it, so handle it like a tail call.  */
+  gcall *call = as_a <gcall *> (stmt);
+  if (!(gimple_call_noreturn_p (call)
+	|| gimple_call_must_tail_p (call)
+	|| gimple_call_tail_p (call)
+	|| (gimple_call_lhs (call) == (retptr ? *retptr : NULL)
+	    && check_returning_calls_p ())))
+    return false;
+
+  gcc_checking_assert (returning_call_p (call));
+
+  /* We found a call that is to be preceded by checking.  */
+  if (bitmap_set_bit (chkcall_blocks, bb->index))
+    ++count_chkcall;
+  else
+    gcc_unreachable ();
+  return true;
+}
+
+
+/* Search preds of BB for a mandatory or potential sibcall or
+   returning call, and arrange for the blocks containing them to have
+   a check inserted before the call, like noreturn calls.  If any
+   preds are found to perform checking, schedule checks at the edges
+   of those that don't, and mark BB as postcheck..  */
+
+static bool
+hardcfr_sibcall_search_preds (basic_block bb,
+			      chk_edges_t &chk_edges,
+			      int &count_chkcall,
+			      auto_sbitmap &chkcall_blocks,
+			      int &count_postchk,
+			      auto_sbitmap &postchk_blocks,
+			      tree *retptr)
+{
+  /* For the exit block, we wish to force a check at every
+     predecessor, so pretend we've already found a pred that had
+     checking, so that we schedule checking at every one of its pred
+     edges.  */
+  bool first = bb->index >= NUM_FIXED_BLOCKS;
+  bool postchecked = true;
+
+  gphi *retphi = NULL;
+  if (retptr && *retptr && TREE_CODE (*retptr) == SSA_NAME
+      && !SSA_NAME_IS_DEFAULT_DEF (*retptr)
+      && SSA_NAME_DEF_STMT (*retptr)
+      && is_a <gphi *> (SSA_NAME_DEF_STMT (*retptr))
+      && gimple_bb (SSA_NAME_DEF_STMT (*retptr)) == bb)
+    {
+      retphi = as_a <gphi *> (SSA_NAME_DEF_STMT (*retptr));
+      gcc_checking_assert (gimple_phi_result (retphi) == *retptr);
+    }
+
+  for (int i = EDGE_COUNT (bb->preds); i--; first = false)
+    {
+      edge e = EDGE_PRED (bb, i);
+
+      bool checked
+	= hardcfr_sibcall_search_block (e->src, chk_edges,
+					count_chkcall, chkcall_blocks,
+					count_postchk, postchk_blocks,
+					!retphi ? retptr
+					: gimple_phi_arg_def_ptr (retphi, i));
+
+      if (first)
+	{
+	  postchecked = checked;
+	  continue;
+	}
+
+      /* When we first find a checked block, force a check at every
+	 other incoming edge we've already visited, and those we
+	 visit afterwards that don't have their own check, so that
+	 when we reach BB, the check has already been performed.  */
+      if (!postchecked && checked)
+	{
+	  for (int j = EDGE_COUNT (bb->preds); --j > i; )
+	    chk_edges.safe_push (EDGE_PRED (bb, j));
+	  postchecked = true;
+	}
+      if (postchecked && !checked)
+	chk_edges.safe_push (EDGE_PRED (bb, i));
+    }
+
+  if (postchecked && bb->index >= NUM_FIXED_BLOCKS)
+    {
+      if (bitmap_set_bit (postchk_blocks, bb->index))
+	count_postchk++;
+      else
+	gcc_unreachable ();
+    }
+
+  return postchecked;
+}
+
+
 class rt_bb_visited
 {
   /* Use a sufficiently wide unsigned type to hold basic block numbers.  */
@@ -176,8 +457,8 @@ class rt_bb_visited
      neither ENTRY nor EXIT, but maybe one-past-the-end, to compute
      the visited array length.  */
   blknum num2idx (blknum n) {
-    gcc_checking_assert (n >= 2 && n <= nblocks);
-    return (n - 2);
+    gcc_checking_assert (n >= NUM_FIXED_BLOCKS && n <= nblocks);
+    return (n - NUM_FIXED_BLOCKS);
   }
   /* Return the block vindex for BB, that must not be ENTRY or
      EXIT.  */
@@ -249,8 +530,7 @@ class rt_bb_visited
   }
 
   /* Set the bit corresponding to BB in VISITED.  Add to SEQ any
-     required gimple statements, and return SEQ, possibly
-     modified.  */
+     required gimple stmts, and return SEQ, possibly modified.  */
   gimple_seq vset (basic_block bb, gimple_seq seq = NULL)
   {
     tree bit, setme = vword (bb, &bit);
@@ -270,7 +550,7 @@ class rt_bb_visited
 
 public:
   /* Prepare to add control flow redundancy testing to CFUN.  */
-  rt_bb_visited ()
+  rt_bb_visited (int checkpoints)
     : nblocks (n_basic_blocks_for_fn (cfun)),
       vword_type (NULL), ckseq (NULL), rtcfg (NULL)
   {
@@ -353,8 +633,8 @@ public:
 					 NULL, NULL);
     gimple_seq_add_stmt (&ckseq, detach);
 
-    if (nblocks - 2 > blknum (param_hardcfr_max_inline_blocks)
-	|| !single_pred_p (EXIT_BLOCK_PTR_FOR_FN (cfun)))
+    if (nblocks - NUM_FIXED_BLOCKS > blknum (param_hardcfr_max_inline_blocks)
+	|| checkpoints > 1)
       {
 	/* Make sure vword_bits is wide enough for the representation
 	   of nblocks in rtcfg.  Compare with vword_bits << vword_bits,
@@ -379,65 +659,33 @@ public:
     gimple_seq_add_stmt (&ckseq, ckfail_init);
   }
 
-  /* Insert SEQ on E, or close enough (e.g., before a noreturn or tail
-     call at the end of E->src).  */
-  void insert_exit_check (gimple_seq seq, edge e)
+  /* Insert SEQ before a resx or a call in INSBB.  */
+  void insert_exit_check_in_block (gimple_seq seq, basic_block insbb)
   {
-    basic_block insbb = e->src;
-
-    /* If the returning block ends with a noreturn call, insert
-       checking before it.  This is particularly important for
-       __builtin_return.  Other noreturn calls won't have an edge to
-       the exit block, so they won't even be considered as exit paths.
-
-       Insert-on-edge inserts before other return stmts, but not
-       before calls, and if a single-block function had the check
-       sequence inserted after a noreturn call, it would be useless,
-       but checking would still flag it as malformed if block 2 has a
-       fallthru edge to the exit block.
-
-       Also avoid disrupting tail calls, inserting the check before
-       them.  This works for must-tail calls, but tail calling as an
-       optimization is detected too late for us.  */
     gimple_stmt_iterator gsi = gsi_last_bb (insbb);
-    gimple *ret = gsi_stmt (gsi);
-    if (ret && is_a <greturn *> (ret))
-      {
+
+    while (!gsi_end_p (gsi))
+      if (is_a <gresx *> (gsi_stmt (gsi))
+	  || is_a <gcall *> (gsi_stmt (gsi)))
+	break;
+      else
 	gsi_prev (&gsi);
-	if (!gsi_end_p (gsi))
-	  ret = gsi_stmt (gsi);
-      }
-    if (ret && is_a <gcall *> (ret)
-	&& (gimple_call_noreturn_p (ret)
-	    || gimple_call_must_tail_p (as_a <gcall *> (ret))
-	    || gimple_call_tail_p (as_a <gcall *> (ret))))
-      gsi_insert_seq_before (&gsi, seq, GSI_SAME_STMT);
-    else
-      gsi_insert_seq_on_edge_immediate (e, seq);
+
+    gsi_insert_seq_before (&gsi, seq, GSI_SAME_STMT);
+  }
+
+  /* Insert SEQ on E.  */
+  void insert_exit_check_on_edge (gimple_seq seq, edge e)
+  {
+    gsi_insert_seq_on_edge_immediate (e, seq);
   }
 
-  /* Add checking code on every exit edge, and initialization code on
+  /* Add checking code to CHK_EDGES, and initialization code on
      the entry edge.  Before this point, the CFG has been undisturbed,
      and all the needed data has been collected and safely stowed.  */
-  void check ()
+  void check (chk_edges_t &chk_edges,
+	      int count_chkcall, auto_sbitmap const &chkcall_blocks)
   {
-    /* Insert initializers for visited at the entry.  */
-    gimple_seq iseq = NULL;
-
-    gcall *vinit = gimple_build_call (builtin_decl_explicit
-				      (BUILT_IN_MEMSET), 3,
-				      build1 (ADDR_EXPR,
-					      build_pointer_type
-					      (TREE_TYPE (visited)),
-					      visited),
-				      integer_zero_node,
-				      TYPE_SIZE_UNIT (TREE_TYPE (visited)));
-    gimple_seq_add_stmt (&iseq, vinit);
-
-    gsi_insert_seq_on_edge_immediate (single_succ_edge
-				      (ENTRY_BLOCK_PTR_FOR_FN (cfun)),
-				      iseq);
-
     /* If we're using out-of-line checking, create and statically
        initialize the CFG checking representation, generate the
        checker call for the checking sequence, and insert it in all
@@ -498,28 +746,116 @@ public:
 						     rtcfg));
 	gimple_seq_add_stmt (&ckseq, call_chk);
 
+	gimple *clobber = gimple_build_assign (visited,
+					       build_clobber
+					       (TREE_TYPE (visited)));
+	gimple_seq_add_stmt (&ckseq, clobber);
+
 	/* If we have multiple exit edges, insert (copies of)
 	   ckseq in all of them.  */
-	for (int i = EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (cfun)->preds);
-	     i--; )
+	for (int i = chk_edges.length (); i--; )
 	  {
 	    gimple_seq seq = ckseq;
 	    /* Copy the sequence, unless we're dealing with the
 	       last edge (we're counting down to zero).  */
-	    if (i)
+	    if (i || count_chkcall)
+	      seq = gimple_seq_copy (seq);
+
+	    edge e = chk_edges[i];
+
+	    if (dump_file)
+	      {
+		if (e->dest == EXIT_BLOCK_PTR_FOR_FN (cfun))
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to exit.\n",
+			   e->src->index);
+		else
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to postcheck block %i.\n",
+			   e->src->index, e->dest->index);
+	      }
+
+	    insert_exit_check_on_edge (seq, e);
+
+	    gcc_checking_assert (!bitmap_bit_p (chkcall_blocks, e->src->index));
+	  }
+
+	sbitmap_iterator it;
+	unsigned i;
+	EXECUTE_IF_SET_IN_BITMAP (chkcall_blocks, 0, i, it)
+	  {
+	    basic_block bb = BASIC_BLOCK_FOR_FN (cfun, i);
+
+	    gimple_seq seq = ckseq;
+	    gcc_checking_assert (count_chkcall > 0);
+	    if (--count_chkcall)
 	      seq = gimple_seq_copy (seq);
 
-	    insert_exit_check (seq,
-			       EDGE_PRED (EXIT_BLOCK_PTR_FOR_FN (cfun), i));
+	    if (dump_file)
+	      fprintf (dump_file,
+		       "Inserting out-of-line check before stmt in block %i.\n",
+		       bb->index);
+
+	    insert_exit_check_in_block (seq, bb);
 	  }
+
+	gcc_checking_assert (count_chkcall == 0);
       }
     else
       {
 	/* Inline checking requires a single exit edge.  */
-	gimple *last = gsi_stmt (gsi_last (ckseq));
+	gimple *last = gimple_build_assign (visited,
+					       build_clobber
+					       (TREE_TYPE (visited)));
+	gimple_seq_add_stmt (&ckseq, last);
 
-	insert_exit_check (ckseq,
-			   single_pred_edge (EXIT_BLOCK_PTR_FOR_FN (cfun)));
+	if (!count_chkcall)
+	  {
+	    edge e = single_pred_edge (EXIT_BLOCK_PTR_FOR_FN (cfun));
+
+	    if (dump_file)
+	      {
+		if (e->dest == EXIT_BLOCK_PTR_FOR_FN (cfun))
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to postcheck block %i.\n",
+			   e->src->index, e->dest->index);
+		else
+		  fprintf (dump_file,
+			   "Inserting inline check in"
+			   " block %i's edge to exit.\n",
+			   e->src->index);
+	      }
+
+	    insert_exit_check_on_edge (ckseq, e);
+	  }
+	else
+	  {
+	    gcc_checking_assert (count_chkcall == 1);
+
+	    sbitmap_iterator it;
+	    unsigned i;
+	    EXECUTE_IF_SET_IN_BITMAP (chkcall_blocks, 0, i, it)
+	      {
+		basic_block bb = BASIC_BLOCK_FOR_FN (cfun, i);
+
+		gimple_seq seq = ckseq;
+		gcc_checking_assert (count_chkcall > 0);
+		if (--count_chkcall)
+		  seq = gimple_seq_copy (seq);
+
+		if (dump_file)
+		  fprintf (dump_file,
+			   "Inserting inline check before stmt in block %i.\n",
+			   bb->index);
+
+		insert_exit_check_in_block (seq, bb);
+	      }
+
+	    gcc_checking_assert (count_chkcall == 0);
+	  }
 
 	/* The inserted ckseq computes CKFAIL at LAST.  Now we have to
 	   conditionally trap on it.  */
@@ -540,8 +876,7 @@ public:
 	  add_bb_to_loop (trp, current_loops->tree_root);
 
 	/* Insert a conditional branch to the trap block.  If the
-	   conditional wouldn't be the last statement, split the
-	   block.  */
+	   conditional wouldn't be the last stmt, split the block.  */
 	gimple_stmt_iterator gsi = gsi_for_stmt (last);
 	if (!gsi_one_before_end_p (gsi))
 	  split_block (gsi_bb (gsi), gsi_stmt (gsi));
@@ -564,6 +899,24 @@ public:
 	if (dom_info_available_p (CDI_DOMINATORS))
 	  set_immediate_dominator (CDI_DOMINATORS, trp, gimple_bb (last));
       }
+
+    /* Insert initializers for visited at the entry.  Do this after
+       other insertions, to avoid messing with block numbers.  */
+    gimple_seq iseq = NULL;
+
+    gcall *vinit = gimple_build_call (builtin_decl_explicit
+				      (BUILT_IN_MEMSET), 3,
+				      build1 (ADDR_EXPR,
+					      build_pointer_type
+					      (TREE_TYPE (visited)),
+					      visited),
+				      integer_zero_node,
+				      TYPE_SIZE_UNIT (TREE_TYPE (visited)));
+    gimple_seq_add_stmt (&iseq, vinit);
+
+    gsi_insert_seq_on_edge_immediate (single_succ_edge
+				      (ENTRY_BLOCK_PTR_FOR_FN (cfun)),
+				      iseq);
   }
 
   /* Push onto RTCFG a (mask, index) pair to test for IBB when BB is
@@ -607,7 +960,7 @@ public:
     return false;
   }
 
-  /* Add to CKSEQ statements to clear CKPART if OBB is visited.  */
+  /* Add to CKSEQ stmts to clear CKPART if OBB is visited.  */
   void
   build_block_check (basic_block obb)
   {
@@ -632,34 +985,48 @@ public:
 
   /* Add to BB code to set its bit in VISITED, and add to RTCFG or
      CKSEQ the data or code needed to check BB's predecessors and
-     successors.  Do NOT change the CFG.  */
-  void visit (basic_block bb)
+     successors.  If CHECKPOINT, assume the block is a checkpoint,
+     whether or not it has an edge to EXIT.  If POSTCHECK, assume the
+     block post-dominates checkpoints and therefore no bitmap setting
+     or checks are to be performed in or for it.  Do NOT change the
+     CFG.  */
+  void visit (basic_block bb, bool checkpoint, bool postcheck)
   {
     /* Set the bit in VISITED when entering the block.  */
     gimple_stmt_iterator gsi = gsi_after_labels (bb);
-    gsi_insert_seq_before (&gsi, vset (bb), GSI_SAME_STMT);
+    if (!postcheck)
+      gsi_insert_seq_before (&gsi, vset (bb), GSI_SAME_STMT);
 
     if (rtcfg)
       {
-	/* Build a list of (index, mask) terminated by (NULL, 0).
-	   Consolidate masks with the same index when they're
-	   adjacent.  First, predecessors.  Count backwards, because
-	   we're going to reverse the list.  The order shouldn't
-	   matter, but let's not make it surprising.  */
-	for (int i = EDGE_COUNT (bb->preds); i--; )
-	  if (push_rtcfg_pair (EDGE_PRED (bb, i)->src, bb,
-			       ENTRY_BLOCK_PTR_FOR_FN (cfun)))
-	    break;
+	if (!postcheck)
+	  {
+	    /* Build a list of (index, mask) terminated by (NULL, 0).
+	       Consolidate masks with the same index when they're
+	       adjacent.  First, predecessors.  Count backwards, because
+	       we're going to reverse the list.  The order shouldn't
+	       matter, but let's not make it surprising.  */
+	    for (int i = EDGE_COUNT (bb->preds); i--; )
+	      if (push_rtcfg_pair (EDGE_PRED (bb, i)->src, bb,
+				   ENTRY_BLOCK_PTR_FOR_FN (cfun)))
+		break;
+	  }
 	rtcfg = tree_cons (NULL_TREE, build_int_cst (vword_type, 0), rtcfg);
 
-	/* Then, successors.  */
-	for (int i = EDGE_COUNT (bb->succs); i--; )
-	  if (push_rtcfg_pair (EDGE_SUCC (bb, i)->dest, bb,
-			       EXIT_BLOCK_PTR_FOR_FN (cfun)))
-	    break;
+	if (!postcheck)
+	  {
+	    /* Then, successors.  */
+	    if (!checkpoint
+		|| !push_rtcfg_pair (EXIT_BLOCK_PTR_FOR_FN (cfun),
+				     bb, EXIT_BLOCK_PTR_FOR_FN (cfun)))
+	      for (int i = EDGE_COUNT (bb->succs); i--; )
+		if (push_rtcfg_pair (EDGE_SUCC (bb, i)->dest, bb,
+				     EXIT_BLOCK_PTR_FOR_FN (cfun)))
+		  break;
+	  }
 	rtcfg = tree_cons (NULL_TREE, build_int_cst (vword_type, 0), rtcfg);
       }
-    else
+    else if (!postcheck)
       {
 	/* Schedule test to fail if the block was reached but somehow none
 	   of its predecessors were.  */
@@ -677,6 +1044,8 @@ public:
 	gassign *blkruns = gimple_build_assign (ckpart, unshare_expr (bit));
 	gimple_seq_add_stmt (&ckseq, blkruns);
 
+	if (checkpoint)
+	  build_block_check (EXIT_BLOCK_PTR_FOR_FN (cfun));
 	for (int i = 0, e = EDGE_COUNT (bb->succs); i < e; i++)
 	  build_block_check (EDGE_SUCC (bb, i)->dest);
 
@@ -687,21 +1056,352 @@ public:
   }
 };
 
+/* It might be useful to avoid checking before noreturn calls that are
+   known to always finish by throwing an exception, rather than by
+   ending the program or looping forever.  Such functions would have
+   to be annotated somehow, with an attribute or flag.
+   Exception-raising functions, such as C++'s __cxa_throw,
+   __cxa_rethrow, and Ada's */
+static bool
+always_throwing_noreturn_call_p (gimple *)
+{
+  return false;
+}
+
 /* Control flow redundancy hardening: record the execution path, and
    verify at exit that an expect path was taken.  */
 
 unsigned int
-pass_harden_control_flow_redundancy::execute (function *)
+pass_harden_control_flow_redundancy::execute (function *fun)
 {
-  rt_bb_visited vstd;
-
+  bool const check_at_escaping_exceptions
+    = (flag_exceptions
+       && flag_harden_control_flow_redundancy_check_exceptions);
+  bool const check_before_noreturn_calls
+    = flag_harden_control_flow_redundancy_check_noreturn > HCFRNR_NEVER;
+  bool const check_before_nothrow_noreturn_calls
+    = (check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn >= HCFRNR_NOTHROW);
+  bool const check_before_throwing_noreturn_calls
+    = (flag_exceptions
+       && check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn > HCFRNR_NOTHROW);
+  bool const check_before_always_throwing_noreturn_calls
+    = (flag_exceptions
+       && check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn >= HCFRNR_ALWAYS);
   basic_block bb;
-  FOR_EACH_BB_FN (bb, cfun)
-    vstd.visit (bb);
+  basic_block bb_eh_cleanup = NULL;
+
+  if (check_at_escaping_exceptions)
+    {
+      int lp_eh_cleanup = -1;
+
+      /* Record the preexisting blocks, to avoid visiting newly-created
+	 blocks.  */
+      auto_sbitmap to_visit (last_basic_block_for_fn (fun));
+      bitmap_clear (to_visit);
+
+      FOR_EACH_BB_FN (bb, fun)
+	bitmap_set_bit (to_visit, bb->index);
+
+      /* Scan the blocks for stmts with escaping exceptions, that
+	 wouldn't be denoted in the CFG, and associate them with an
+	 empty cleanup handler around the whole function.  Walk
+	 backwards, so that even when we split the block, */
+      sbitmap_iterator it;
+      unsigned i;
+      EXECUTE_IF_SET_IN_BITMAP (to_visit, 0, i, it)
+	{
+	  bb = BASIC_BLOCK_FOR_FN (fun, i);
+
+	  for (gimple_stmt_iterator gsi = gsi_last_bb (bb);
+	       !gsi_end_p (gsi); gsi_prev (&gsi))
+	    {
+	      gimple *stmt = gsi_stmt (gsi);
+	      if (!stmt_could_throw_p (fun, stmt))
+		continue;
+
+	      /* If it must not throw, or if it already has a handler,
+		 we need not worry about it.  */
+	      if (lookup_stmt_eh_lp (stmt) != 0)
+		continue;
+
+	      /* Don't split blocks at, nor add EH edges to, tail
+		 calls, we will add verification before the call
+		 anyway.  */
+	      if (is_a <gcall *> (stmt)
+		  && (gimple_call_must_tail_p (as_a <gcall *> (stmt))
+		      || gimple_call_tail_p (as_a <gcall *> (stmt))
+		      || returning_call_p (as_a <gcall *> (stmt))))
+		continue;
+
+	      if (!gsi_one_before_end_p (gsi))
+		split_block (bb, stmt);
+	      /* A resx or noreturn call needs not be associated with
+		 the cleanup handler if we're going to add checking
+		 before it.  We only test cases that didn't require
+		 block splitting because noreturn calls would always
+		 be at the end of blocks, and we test for zero
+		 successors because if there is an edge, it's not
+		 noreturn, as any EH edges would have already been
+		 caught by the lookup_stmt_eh_lp test above.  */
+	      else if (check_before_noreturn_calls
+		       && EDGE_COUNT (bb->succs) == 0
+		       && (is_a <gresx *> (stmt)
+			   ? check_before_always_throwing_noreturn_calls
+			   : (!is_a <gcall *> (stmt)
+			      || !gimple_call_noreturn_p (stmt))
+			   ? (gcc_unreachable (), false)
+			   : (!flag_exceptions
+			      || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+			   ? check_before_nothrow_noreturn_calls
+			   : always_throwing_noreturn_call_p (stmt)
+			   ? check_before_always_throwing_noreturn_calls
+			   : check_before_throwing_noreturn_calls))
+		{
+		  if (dump_file)
+		    {
+		      fprintf (dump_file,
+			       "Bypassing cleanup for noreturn stmt"
+			       " in block %i:\n",
+			       bb->index);
+		      print_gimple_stmt (dump_file, stmt, 0);
+		    }
+		  continue;
+		}
+
+	      if (!bb_eh_cleanup)
+		{
+		  bb_eh_cleanup = create_empty_bb (bb);
+		  if (dom_info_available_p (CDI_DOMINATORS))
+		    set_immediate_dominator (CDI_DOMINATORS, bb_eh_cleanup, bb);
+		  if (current_loops)
+		    add_bb_to_loop (bb_eh_cleanup, current_loops->tree_root);
+
+		  /* Make the new block an EH cleanup for the call.  */
+		  eh_region new_r = gen_eh_region_cleanup (NULL);
+		  eh_landing_pad lp = gen_eh_landing_pad (new_r);
+		  tree label = gimple_block_label (bb_eh_cleanup);
+		  lp->post_landing_pad = label;
+		  EH_LANDING_PAD_NR (label) = lp_eh_cleanup = lp->index;
+
+		  /* Just propagate the exception.
+		     We will later insert the verifier call.  */
+		  gimple_stmt_iterator ehgsi;
+		  ehgsi = gsi_after_labels (bb_eh_cleanup);
+		  gresx *resx = gimple_build_resx (new_r->index);
+		  gsi_insert_before (&ehgsi, resx, GSI_SAME_STMT);
+
+		  if (dump_file)
+		    fprintf (dump_file,
+			     "Created cleanup block %i:\n",
+			     bb_eh_cleanup->index);
+		}
+	      else if (dom_info_available_p (CDI_DOMINATORS))
+		{
+		  basic_block immdom;
+		  immdom = get_immediate_dominator (CDI_DOMINATORS,
+						    bb_eh_cleanup);
+		  if (!dominated_by_p (CDI_DOMINATORS, bb, immdom))
+		    {
+		      immdom = nearest_common_dominator (CDI_DOMINATORS,
+							 immdom, bb);
+		      set_immediate_dominator (CDI_DOMINATORS,
+					       bb_eh_cleanup, immdom);
+		    }
+		}
+
+	      if (dump_file)
+		{
+		  fprintf (dump_file,
+			   "Associated cleanup block with stmt in block %i:\n",
+			   bb->index);
+		  print_gimple_stmt (dump_file, stmt, 0);
+		}
+
+	      add_stmt_to_eh_lp (stmt, lp_eh_cleanup);
+	      /* Finally, wire the EH cleanup block into the CFG.  */
+	      make_eh_edges (stmt);
+	    }
+	}
+
+      if (bb_eh_cleanup)
+	{
+	  /* A cfg_cleanup after bb_eh_cleanup makes for a more compact
+	     rtcfg, and it avoids bb numbering differences when we split
+	     blocks because of trailing debug insns only.  */
+	  cleanup_tree_cfg ();
+	  gcc_checking_assert (EDGE_COUNT (bb_eh_cleanup->succs) == 0);
+	}
+    }
+
+  /* These record blocks with calls that are to be preceded by
+     checkpoints, such as noreturn calls (if so chosen), must-tail
+     calls, potential early-marked tail calls, and returning calls (if
+     so chosen).  */
+  int count_chkcall = 0;
+  auto_sbitmap chkcall_blocks (last_basic_block_for_fn (fun));
+  bitmap_clear (chkcall_blocks);
+
+  /* We wish to add verification at blocks without successors, such as
+     noreturn calls (raising or not) and the reraise at the cleanup
+     block, but not other reraises: they will go through the cleanup
+     block.  */
+  if (check_before_noreturn_calls)
+    FOR_EACH_BB_FN (bb, fun)
+      {
+	gimple_stmt_iterator gsi = gsi_last_bb (bb);
+	if (gsi_end_p (gsi))
+	  continue;
+	gimple *stmt = gsi_stmt (gsi);
 
-  vstd.check ();
+	if (EDGE_COUNT (bb->succs) == 0)
+	  {
+	    /* A stmt at the end of a block without any successors is
+	       either a resx or a noreturn call without a local
+	       handler.  Check that it's one of the desired
+	       checkpoints.  */
+	    if (flag_exceptions && is_a <gresx *> (stmt)
+		? (check_before_always_throwing_noreturn_calls
+		   || bb == bb_eh_cleanup)
+		: (!is_a <gcall *> (stmt)
+		   || !gimple_call_noreturn_p (stmt))
+		? (/* Catch cases in which successors would be
+		      expected.  */
+		   gcc_unreachable (), false)
+		: (!flag_exceptions
+		   || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+		? check_before_nothrow_noreturn_calls
+		: always_throwing_noreturn_call_p (stmt)
+		? check_before_always_throwing_noreturn_calls
+		: check_before_throwing_noreturn_calls)
+	      {
+		if (dump_file)
+		  {
+		    fprintf (dump_file,
+			     "Scheduling check before stmt"
+			     " in succ-less block %i:\n",
+			     bb->index);
+		    print_gimple_stmt (dump_file, stmt, 0);
+		  }
+
+		if (bitmap_set_bit (chkcall_blocks, bb->index))
+		  count_chkcall++;
+		else
+		  gcc_unreachable ();
+	      }
+	    continue;
+	  }
 
-  return 0;
+	/* If there are no exceptions, it would seem like any noreturn
+	   call must have zero successor edges, but __builtin_return
+	   gets successor edges.  We don't want to handle it here, it
+	   will be dealt with in sibcall_search_preds.  Otherwise,
+	   check for blocks without non-EH successors, but skip those
+	   with resx stmts and edges (i.e., those other than that in
+	   bb_eh_cleanup), since those will go through bb_eh_cleanup,
+	   that will have been counted as noreturn above because it
+	   has no successors.  */
+	gcc_checking_assert (bb != bb_eh_cleanup
+			     || !check_at_escaping_exceptions);
+	if (flag_exceptions && is_a <gresx *> (stmt)
+	    ? check_before_always_throwing_noreturn_calls
+	    : (!is_a <gcall *> (stmt)
+	       || !gimple_call_noreturn_p (stmt))
+	    ? false
+	    : (!flag_exceptions
+	       || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+	    ? false /* rather than check_before_nothrow_noreturn_calls */
+	    : always_throwing_noreturn_call_p (stmt)
+	    ? check_before_always_throwing_noreturn_calls
+	    : check_before_throwing_noreturn_calls)
+	  {
+	    gcc_checking_assert (single_succ_p (bb)
+				 && (single_succ_edge (bb)->flags & EDGE_EH));
+
+	    if (dump_file)
+	      {
+		fprintf (dump_file,
+			 "Scheduling check before stmt"
+			 " in EH-succ block %i:\n",
+			 bb->index);
+		print_gimple_stmt (dump_file, stmt, 0);
+	      }
+
+	    if (bitmap_set_bit (chkcall_blocks, bb->index))
+	      count_chkcall++;
+	    else
+	      gcc_unreachable ();
+	  }
+      }
+  else if (bb_eh_cleanup)
+    {
+      if (bitmap_set_bit (chkcall_blocks, bb_eh_cleanup->index))
+	count_chkcall++;
+      else
+	gcc_unreachable ();
+    }
+
+  gcc_checking_assert (!bb_eh_cleanup
+		       || bitmap_bit_p (chkcall_blocks, bb_eh_cleanup->index));
+
+  /* If we don't have edges to exit nor noreturn calls (including the
+     cleanup reraise), then we may skip instrumentation: that would
+     amount to a function that ends with an infinite loop.  */
+  if (!count_chkcall
+      && EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (fun)->preds) == 0)
+    {
+      if (dump_file)
+	fprintf (dump_file,
+		 "Disabling CFR, no exit paths to check\n");
+
+      return 0;
+    }
+
+  /* Search for must-tail calls, early-marked potential tail calls,
+     and, if requested, returning calls.  As we introduce early
+     checks, */
+  int count_postchk = 0;
+  auto_sbitmap postchk_blocks (last_basic_block_for_fn (fun));
+  bitmap_clear (postchk_blocks);
+  chk_edges_t chk_edges;
+  hardcfr_sibcall_search_preds (EXIT_BLOCK_PTR_FOR_FN (fun), chk_edges,
+				count_chkcall, chkcall_blocks,
+				count_postchk, postchk_blocks,
+				NULL);
+
+  rt_bb_visited vstd (chk_edges.length () + count_chkcall);
+
+  auto_sbitmap combined_blocks (last_basic_block_for_fn (fun));
+  bitmap_copy (combined_blocks, chkcall_blocks);
+  int i;
+  edge *e;
+  FOR_EACH_VEC_ELT (chk_edges, i, e)
+    if (!bitmap_set_bit (combined_blocks, (*e)->src->index))
+      gcc_unreachable ();
+
+  /* Visit blocks in index order, because building rtcfg depends on
+     that.  Blocks must be compact, which the cleanup_cfg requirement
+     ensures.  This would also enable FOR_EACH_BB_FN to be used to
+     iterate in index order, but bb_eh_cleanup block splits and
+     insertions changes that.  */
+  gcc_checking_assert (n_basic_blocks_for_fn (fun)
+		       == last_basic_block_for_fn (fun));
+  for (int i = NUM_FIXED_BLOCKS; i < n_basic_blocks_for_fn (fun); i++)
+    {
+      bb = BASIC_BLOCK_FOR_FN (fun, i);
+      gcc_checking_assert (bb->index == i);
+      vstd.visit (bb, bitmap_bit_p (combined_blocks, i),
+		  bitmap_bit_p (postchk_blocks, i));
+    }
+
+  vstd.check (chk_edges, count_chkcall, chkcall_blocks);
+
+  return
+    TODO_update_ssa
+    | TODO_cleanup_cfg
+    | TODO_verify_il;
 }
 
 /* Instantiate a hardcfr pass.  */
diff --git a/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c b/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c
new file mode 100644
index 00000000000..a6992eb9f8e
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -O0 -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we don't insert checking before noreturn calls.  -O0 is tested
+   separately because h is not found to be noreturn without optimization.  */
+
+#include "torture/harden-cfr-noret.c"
+
+/* No out-of-line checks.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* Only one inline check at the end of f and of h2.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-always.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-always.c
new file mode 100644
index 00000000000..779896c60e8
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-always.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even enabling all checks before noreturn calls (leaving
+   returning calls enabled), we get checks before __builtin_return without
+   duplication (__builtin_return is both noreturn and a returning call).  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-never.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-never.c
new file mode 100644
index 00000000000..49ce17f5b93
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-never.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even enabling checks before never noreturn calls (leaving
+   returning calls enabled), we get checks before __builtin_return without
+   duplication (__builtin_return is both noreturn and a returning call).  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noopt.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noopt.c
new file mode 100644
index 00000000000..1512614791f
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noopt.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even disabling checks before both noreturn and returning
+   calls, we still get checks before __builtin_return.  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noret.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noret.c
new file mode 100644
index 00000000000..fd95bb7e3e3
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-noret.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even disabling checks before returning calls (leaving noreturn
+   calls enabled), we still get checks before __builtin_return.  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-nothrow.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-nothrow.c
new file mode 100644
index 00000000000..c5c361234c4
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-nothrow.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even enabling checks before nothrow noreturn calls (leaving
+   returning calls enabled), we get checks before __builtin_return without
+   duplication (__builtin_return is both noreturn and a returning call).  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-retcl.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-retcl.c
new file mode 100644
index 00000000000..137dfbb95d6
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret-retcl.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that, even disabling checks before noreturn calls (leaving returning
+   calls enabled), we still get checks before __builtin_return.  */
+
+#include "harden-cfr-bret.c"
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
+/* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c
index 70acdc95f25..b459ff6b864 100644
--- a/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-bret.c
@@ -7,5 +7,11 @@ int f(int i) {
   return i;
 }
 
-/* Out-of-line checking, before both builtin_return and return.  */
+int g(int i) {
+  __builtin_return (&i);
+}
+
+/* Out-of-line checking, before both builtin_return and return in f.  */
 /* { dg-final { scan-tree-dump-times "__hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking before builtin_return in g.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c
new file mode 100644
index 00000000000..8bd2d13ac18
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c
@@ -0,0 +1,18 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we don't insert checking before noreturn calls.  -O0 is tested
+   separately because h is not found to be noreturn without optimization, which
+   affects codegen for h2, so h2 is omitted here at -O0.  */
+
+#if !__OPTIMIZE__
+# define OMIT_H2
+#endif
+
+#include "harden-cfr-noret.c"
+
+
+/* No out-of-line checks.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* Only one inline check at the end of f.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c
new file mode 100644
index 00000000000..a804a6cfe59
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fno-exceptions -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that -fno-exceptions makes for implicit nothrow in noreturn
+   handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#include "harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c
new file mode 100644
index 00000000000..f390cfdbc59
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checking before nothrow noreturn calls.  */
+
+#include "harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c
new file mode 100644
index 00000000000..fdd803109a4
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c
@@ -0,0 +1,38 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checking before all noreturn calls.  */
+
+#ifndef ATTR_NOTHROW_OPT /* Overridden in harden-cfr-noret-noexcept.  */
+#define ATTR_NOTHROW_OPT __attribute__ ((__nothrow__))
+#endif
+
+extern void __attribute__ ((__noreturn__)) ATTR_NOTHROW_OPT g (void);
+
+void f(int i) {
+  if (i)
+    /* Out-of-line checks here...  */
+    g ();
+  /* ... and here.  */
+}
+
+void __attribute__ ((__noinline__, __noclone__))
+h(void) {
+  /* Inline check here.  */
+  g ();
+}
+
+#ifndef OMIT_H2 /* from harden-cfr-noret-never.  */
+void h2(void) {
+  /* Inline check either here, whether because of noreturn or tail call...  */
+  h ();
+  /* ... or here, if not optimizing.  */
+}
+#endif
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c
new file mode 100644
index 00000000000..6d11487bbba
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c
@@ -0,0 +1,8 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+#include "harden-cfr-tail.c"
+
+/* Inline checking after the calls, disabling tail calling.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 5 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 0 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c
new file mode 100644
index 00000000000..550b02ca088
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c
@@ -0,0 +1,35 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-exceptions -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checks before returning calls and alternate paths, even
+   at -O0, because of the explicit command-line flag.  */
+
+void g (void);
+void g2 (void);
+void g3 (void);
+
+void f (int i) {
+  if (!i)
+    /* Out-of-line checks here...  */
+    g ();
+  else if (i > 0)
+    /* here...  */
+    g2 ();
+  /* else */
+    /* and in the implicit else here.  */
+}
+
+void f2 (int i) {
+  if (!i)
+    /* Out-of-line check here...  */
+    g ();
+  else if (i > 0)
+    /* here...  */
+    g2 ();
+  else
+    /* and here.  */
+    g3 ();
+}
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 6 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 0 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
index 40d76c5c163..d5467eafa9f 100644
--- a/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
@@ -1,17 +1,52 @@
 /* { dg-do compile } */
-/* { dg-options "-fharden-control-flow-redundancy -fdump-tree-hardcfr -ffat-lto-objects" } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-hardcfr-check-exceptions -fdump-tree-hardcfr -ffat-lto-objects -Wno-return-type" } */
 
-/* We'd like to check that we insert checking so as to not disrupt tail calls,
-   but unfortunately mandatory tail calls are not available in C, and optimizing
-   calls as tail calls only takes place after hardcfr.  */
+/* Check that we insert CFR checking so as to not disrupt tail calls.
+   Mandatory tail calls are not available in C, and optimizing calls as tail
+   calls only takes place after hardcfr, so we insert checking before calls
+   followed by copies and return stmts with the same return value, that might
+   (or might not) end up optimized to tail calls.  */
 
-extern int g(int i);
+extern int g (int i);
 
-int f(int i) {
+int f1(int i) {
+  /* Inline check before the returning call.  */
   return g (i);
 }
 
-/* Inline checking before the tail call.  */
-/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
-/* Inline checking before the tail call.  */
-/* { dg-final { scan-tree-dump-times "\\\[tail\]" 1 "hardcfr" { xfail *-*-* } } } */
+extern void g2 (int i);
+
+void f2(int i) {
+  /* Inline check before the returning call, that ignores the returned value,
+     matching the value-less return.  */
+  g2 (i);
+  return;
+}
+
+void f3(int i) {
+  /* Inline check before the returning call.  */
+  g (i);
+}
+
+void f4(int i) {
+  if (i)
+    /* Out-of-line check before the returning call.  */
+    return g2 (i);
+  /* Out-of-line check before implicit return.  */
+}
+
+int f5(int i) {
+  /* Not regarded as a returning call, returning value other than callee's
+     returned value.  */
+  g (i);
+  /* Inline check after the non-returning call.  */
+  return i;
+}
+
+/* Out-of-line checks in f4, before returning calls and before return.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking in all other functions.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 4 "hardcfr" } } */
+/* Check before tail-call in all but f5, but f4 is out-of-line.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 3 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C b/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C
new file mode 100644
index 00000000000..17ea79f7cfb
--- /dev/null
+++ b/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C
@@ -0,0 +1,11 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects -O0" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions, and also checking before noreturn
+   calls.  h2 and h2b get an extra resx without ehcleanup.  */
+
+#include "torture/harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 16 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C b/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C
new file mode 100644
index 00000000000..f0338ccc361
--- /dev/null
+++ b/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C
@@ -0,0 +1,10 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -foptimize-sibling-calls -fdump-tree-hardcfr -O0" } */
+
+/* -fhardcfr-check-returning-calls gets implicitly disabled because,
+   -at O0, -foptimize-sibling-calls has no effect.  */
+
+#include "torture/harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 12 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C
new file mode 100644
index 00000000000..0d35920c7ee
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  */
+
+#include "harden-cfr-noret-no-nothrow.C"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Bypassing cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C
new file mode 100644
index 00000000000..b7d247ff43c
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C
@@ -0,0 +1,18 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  Expected results for =never and =nothrow are the same,
+   since the functions are not nothrow.  */
+
+#include "harden-cfr-noret-no-nothrow.C"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Associated cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C
new file mode 100644
index 00000000000..62c58cfd406
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C
@@ -0,0 +1,23 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#if ! __OPTIMIZE__
+void __attribute__ ((__noreturn__)) h (void);
+#endif
+
+#include "../../c-c++-common/torture/harden-cfr-noret.c"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Associated cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C
new file mode 100644
index 00000000000..0286f6e6d3f
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C
@@ -0,0 +1,20 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions, and also checking before noreturn
+   calls.  */
+
+#if ! __OPTIMIZE__
+/* Without optimization, functions with cleanups end up with an extra
+   resx that is not optimized out, so arrange to optimize them.  */
+void __attribute__ ((__optimize__ (1))) h2(void);
+void __attribute__ ((__optimize__ (1))) h2b(void);
+#endif
+
+#include "harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 14 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
+/* h, h2, h2b, and h4.  */
+/* { dg-final { scan-tree-dump-times "Bypassing" 4 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C
new file mode 100644
index 00000000000..885b0b236af
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C
@@ -0,0 +1,11 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions  -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we do not insert cleanups for checking around the bodies
+   of maybe-throwing functions.  h4 doesn't get any checks, because we
+   don't have noreturn checking enabled.  */
+
+#include "harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 6 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C
new file mode 100644
index 00000000000..32def637255
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C
@@ -0,0 +1,31 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -foptimize-sibling-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions.  These results depend on checking before
+   returning calls, which is only enabled when sibcall optimizations
+   are enabled, so change the optimization mode to -O1 for f and f2,
+   so that -foptimize-sibling-calls can take effect and enable
+   -fhardcfr-check-returning-calls, so that we get the same results.
+   There is a separate test for -O0.  */
+
+#if ! __OPTIMIZE__
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) f(int i);
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) f2(int i);
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) h3(void);
+#endif
+
+#include "harden-cfr-throw.C"
+
+/* f gets out-of-line checks before the unwrapped tail call and in the
+   else edge.  */
+/* f2 gets out-of-line checks before both unwrapped tail calls.  */
+/* h gets out-of-line checks before the implicit return and in the
+   cleanup block.  */
+/* h2 and h2b get out-of-line checks before the cleanup returning
+   call, and in the cleanup block.  */
+/* h3 gets an inline check before the __cxa_end_catch returning call.  */
+/* h4 gets an inline check in the cleanup block.  */
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 10 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C
new file mode 100644
index 00000000000..992fbdad381
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C
@@ -0,0 +1,65 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions.  */
+
+extern void g (void);
+extern void g2 (void);
+
+void f(int i) {
+  if (i)
+    g ();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void f2(int i) {
+  if (i)
+    g ();
+  else
+    g2 ();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h(void) {
+  try {
+    g ();
+  } catch (...) {
+    throw;
+  }
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+struct needs_cleanup {
+  ~needs_cleanup();
+};
+
+void h2(void) {
+  needs_cleanup y; /* No check in the cleanup handler.  */
+  g();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+extern void __attribute__ ((__nothrow__)) another_cleanup (void*);
+
+void h2b(void) {
+  int x __attribute__ ((cleanup (another_cleanup)));
+  g();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h3(void) {
+  try {
+    throw 1;
+  } catch (...) {
+  }
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h4(void) {
+  throw 1;
+  /* Inline check in the cleanup around the __cxa_throw noreturn call.  */
+}
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 12 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c b/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c
new file mode 100644
index 00000000000..8e4ee1fab08
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c
@@ -0,0 +1,15 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C makes for implicit nothrow in noreturn handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#include "../../c-c++-common/torture/harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c b/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c
new file mode 100644
index 00000000000..634d98f1ffc
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c
@@ -0,0 +1,40 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-hardcfr-check-exceptions -fdump-tree-hardcfr -ffat-lto-objects -Wno-return-type" } */
+
+/* In C only, check some additional cases (comparing with
+   c-c++-common/torture/harden-cfr-tail.c) of falling off the end of non-void
+   function.  C++ would issue an unreachable call in these cases.  */
+
+extern int g (int i);
+
+int f1(int i) {
+  /* Inline check before the returning call, that doesn't return anything.  */
+  g (i);
+  /* Implicit return without value, despite the return type; this combination
+     enables tail-calling of g, and is recognized as a returning call.  */
+}
+
+extern void g2 (int i);
+
+int f2(int i) {
+  /* Inline check before the returning call, that disregards its return
+     value.  */
+  g2 (i);
+  /* Implicit return without value, despite the return type; this combination
+     enables tail-calling of g2, and is recognized as a returning call.  */
+}
+
+int f3(int i) {
+  if (i)
+    /* Out-of-line check before the returning call.  */
+    return g (i);
+  /* Out-of-line check before implicit return.  */
+}
+
+/* Out-of-line checks in f3, before returning calls and before return.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking in all other functions.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
+/* Check before tail-call in all functions, but f3 is out-of-line.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/gnat.dg/hardcfr.adb b/gcc/testsuite/gnat.dg/hardcfr.adb
index b578a913d75..abe1605c029 100644
--- a/gcc/testsuite/gnat.dg/hardcfr.adb
+++ b/gcc/testsuite/gnat.dg/hardcfr.adb
@@ -1,5 +1,5 @@
 --  { dg-do run }
---  { dg-options "-fharden-control-flow-redundancy -fdump-tree-hardcfr --param=hardcfr-max-blocks=22 --param=hardcfr-max-inline-blocks=12 -O0" }
+--  { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions -fdump-tree-hardcfr --param=hardcfr-max-blocks=22 --param=hardcfr-max-inline-blocks=12 -O0" }
 
 procedure HardCFR is
    function F (I, J : Integer) return Integer is
diff --git a/libgcc/hardcfr.c b/libgcc/hardcfr.c
index 8ef29428111..55f6b995f2f 100644
--- a/libgcc/hardcfr.c
+++ b/libgcc/hardcfr.c
@@ -60,14 +60,25 @@ extern void __hardcfr_check (size_t blocks,
 			     vword const *visited,
 			     vword const *cfg);
 
+/* Compute the MASK for the bit representing BLOCK in WORDIDX's vword in a
+   visited blocks bit array.  */
+static inline void
+block2mask (size_t const block, vword *const mask, size_t *const wordidx)
+{
+  size_t wbits = __CHAR_BIT__ * sizeof (vword);
+  *wordidx = block / wbits;
+  *mask = (vword)1 << (block % wbits);
+}
 
 /* Check whether the bit corresponding to BLOCK is set in VISITED.  */
 static inline bool
 visited_p (size_t const block, vword const *const visited)
 {
-  size_t wbits = __CHAR_BIT__ * sizeof (vword);
-  vword w = visited[block / wbits];
-  return (w & ((vword)1 << (block % wbits))) != 0;
+  vword mask;
+  size_t wordidx;
+  block2mask (block, &mask, &wordidx);
+  vword w = visited[wordidx];
+  return (w & mask) != 0;
 }
 
 /* Read and consume a mask from **CFG_IT.  (Consume meaning advancing the
@@ -118,19 +129,19 @@ consume_seq (vword const **const cfg_it)
    we reach the terminator without finding any.  Consume the entire sequence
    otherwise, so that *CFG_IT points just past the terminator, which may be the
    beginning of the next sequence.  */
-static inline void
+static inline bool
 check_seq (vword const *const visited, vword const **const cfg_it)
 {
   vword mask;
   size_t wordidx;
 
   /* If the block was visited, check that at least one of the
-     preds was also visited.  */
+     preds/succs was also visited.  */
   do
     /* If we get to the end of the sequence without finding any
        match, something is amiss.  */
     if (!next_pair (cfg_it, &mask, &wordidx))
-      __builtin_trap ();
+      return false;
   /* Keep searching until we find a match, at which point the
      condition is satisfied.  */
   while (!test_mask (visited, mask, wordidx));
@@ -139,6 +150,94 @@ check_seq (vword const *const visited, vword const **const cfg_it)
      skipped the block, so as to position the iterator at the beginning of the
      next .  */
   consume_seq (cfg_it);
+
+  return true;
+}
+
+/* Print out the CFG with BLOCKS blocks, presumed to be associated with CALLER.
+   This is expected to be optimized out entirely, unless the verbose part of
+   __hardcfr_check_fail is enabled.  */
+static inline void
+__hardcfr_debug_cfg (size_t const blocks,
+		     void const *const caller,
+		     vword const *const cfg)
+{
+  __builtin_printf ("CFG at %p, for %p", cfg, caller);
+  vword const *cfg_it = cfg;
+  for (size_t i = 0; i < blocks; i++)
+    {
+      vword mask; size_t wordidx;
+      block2mask (i, &mask, &wordidx);
+      __builtin_printf ("\nblock %lu (%lu/0x%lx)\npreds: ",
+			(unsigned long)i,
+			(unsigned long)wordidx, (unsigned long)mask);
+      while (next_pair (&cfg_it, &mask, &wordidx))
+	__builtin_printf (" (%lu/0x%lx)",
+			  (unsigned long)wordidx, (unsigned long)mask);
+      __builtin_printf ("\nsuccs: ");
+      while (next_pair (&cfg_it, &mask, &wordidx))
+	__builtin_printf (" (%lu/0x%lx)",
+			  (unsigned long)wordidx, (unsigned long)mask);
+    }
+  __builtin_printf ("\n");
+}
+
+#ifndef ATTRIBUTE_UNUSED
+# define ATTRIBUTE_UNUSED __attribute__ ((__unused__))
+#endif
+
+/* This is called when an out-of-line hardcfr check fails.  All the arguments
+   are ignored, and it just traps, unless HARDCFR_VERBOSE_FAIL is enabled.  IF
+   it is, it prints the PART of the CFG, expected to have BLOCKS blocks, that
+   failed at CALLER's BLOCK, and the VISITED bitmap.  When the verbose mode is
+   enabled, it also forces __hardcfr_debug_cfg (above) to be compiled into an
+   out-of-line function, that could be called from a debugger.
+   */
+static inline void
+__hardcfr_check_fail (size_t const blocks ATTRIBUTE_UNUSED,
+		      vword const *const visited,
+		      vword const *const cfg ATTRIBUTE_UNUSED,
+		      size_t const block ATTRIBUTE_UNUSED,
+		      int const part ATTRIBUTE_UNUSED,
+		      void const *const caller ATTRIBUTE_UNUSED)
+{
+#if HARDCFR_VERBOSE_FAIL
+  static const char *parts[] = { "preds", "succs" };
+
+  vword mask; size_t wordidx;
+  block2mask (block, &mask, &wordidx);
+  __builtin_printf ("hardcfr fail at %p block %lu (%lu/0x%lx), expected %s:",
+		    caller, (unsigned long)block,
+		    (unsigned long)wordidx, (unsigned long)mask,
+		    parts[part]);
+
+  /* Skip data for previous blocks.  */
+  vword const *cfg_it = cfg;
+  for (size_t i = block; i--; )
+    {
+      consume_seq (&cfg_it);
+      consume_seq (&cfg_it);
+    }
+  for (size_t i = part; i--; )
+    consume_seq (&cfg_it);
+
+  while (next_pair (&cfg_it, &mask, &wordidx))
+    __builtin_printf (" (%lu/0x%lx)",
+		      (unsigned long)wordidx, (unsigned long)mask);
+
+  __builtin_printf ("\nvisited:");
+  block2mask (blocks, &mask, &wordidx);
+  for (size_t i = 0; i <= wordidx; i++)
+    __builtin_printf (" (%lu/0x%lx)",
+		      (unsigned long)i, (unsigned long)visited[i]);
+  __builtin_printf ("\n");
+
+  /* Reference __hardcfr_debug_cfg so that it's output out-of-line, so that it
+     can be called from a debugger.  */
+  if (!caller || caller == __hardcfr_debug_cfg)
+    return;
+#endif
+  __builtin_trap ();
 }
 
 /* Check that, for each of the BLOCKS basic blocks, if its bit is set in
@@ -168,9 +267,13 @@ __hardcfr_check (size_t const blocks,
       else
 	{
 	  /* Check predecessors.  */
-	  check_seq (visited, &cfg_it);
+	  if (!check_seq (visited, &cfg_it))
+	    __hardcfr_check_fail (blocks, visited, cfg, i, 0,
+				  __builtin_return_address (0));
 	  /* Check successors.  */
-	  check_seq (visited, &cfg_it);
+	  if (!check_seq (visited, &cfg_it))
+	    __hardcfr_check_fail (blocks, visited, cfg, i, 1,
+				  __builtin_return_address (0));
 	}
     }
 }

^ permalink raw reply	[flat|nested] 13+ messages in thread
* [gcc(refs/users/aoliva/heads/testme)] hardcfr: add optional checkpoints
@ 2022-10-01  4:50 Alexandre Oliva
  0 siblings, 0 replies; 13+ messages in thread
From: Alexandre Oliva @ 2022-10-01  4:50 UTC (permalink / raw)
  To: gcc-cvs

https://gcc.gnu.org/g:d76a74ab71898f393606254ff17a72352f05e775

commit d76a74ab71898f393606254ff17a72352f05e775
Author: Alexandre Oliva <oliva@adacore.com>
Date:   Fri Sep 30 23:20:31 2022 -0300

    hardcfr: add optional checkpoints
    
    Previously, control flow redundancy only checked the visited bitmap
    against the control flow graph at return points and before mandatory
    tail calls, missing various other possibilities of exiting a
    subprogram, such as by raising or propagating exceptions, and calling
    noreturn functions.  The checks inserted before returns also prevented
    potential tail-call optimizations.
    
    This incremental change introduces options to control checking at each
    of these previously-missed checkpoints.  Unless disabled, a cleanup is
    introduced to check when an exceptions escapes a subprogram.  To avoid
    disrupting sibcall optimizations, when they are enabled, checks are
    introduced before calls whose results are immediately returned,
    whether or not they are ultimately optimized.  If enabled, checks are
    introduced before noreturn calls and exception raises, or only before
    nothrow noreturn calls.
    
    Add examples of code transformations to the GNAT RM.
    
    
    for  gcc/ada/ChangeLog
    
            * doc/gnat_rm/security_hardening_features.rst: Document
            optional hardcfr checkpoints.
    
    for  gcc/ChangeLog
    
            * common.opt (-fhardcfr-check-returning-calls): New.
            (-fhardcfr-check-exceptions): New.
            (-fhardcfr-check-noreturn-calls=*): New.  (Enum
            hardcfr_check_noreturn_calls): New.  * doc/invoke.texi:
            Document them.  * flag-types.h (enum hardcfr_noret): New.  *
            gimple-harden-control-flow.cc: Include more headers.
            (pass_data_harden_control_flow_redundancy): Move
            properties_finish to pass return.
            (pass_harden_control_flow_redundancy::gate): Move
            no-edge-to-exit checking to execute, after exception
            transformations.  Use symbolic NUM_FIXED_BLOCKS.  Test for CFG
            before testing block count.  (check_returning_calls_p): New.
            (hardcfr_scan_block): New.  (returnign_call_p): New.
            (chk_edges_t): New.  (hardcfr_sibcall_search_block): New.
            (hardcfr_sibcall_search_preds): New.
            (rt_bb_visited::num2idx): Use symbolic NUM_FIXED_BLOCKS.
            (rt_bb_visited::rt_bb_visited): Likewise.  Take checkpoints
            count, test it to select out-of-line checks.
            (rt_bb_visited::insert_exit_check): Removed.
            (rt_bb_visited::insert_exit_check_in_blocks): New.
            (rt_bb_visited::insert_exit_check_on_edge): New.
            (rt_bb_visited::check): Take checkpoint edges and blocks.
            Move initialization insertion to the end.  Insert checks on
            edges, and before calls in select blocks.  Clobber the visited
            array to the check sequence.  Output actions to dump_file.
            (rt_bb_visited::visit): Take checkpoint and postcheck.  Assume
            an edge to exit if checkpoint, and skip testing if postcheck.
            (always_throwing_noreturn_call_p): New, dummy.
            (pass_harden_control_flow_redundancy::execute): Introduce
            cleanup block if requested and needed.  Scan blocks for
            noreturn and returning calls if checkpoints before them were
            requested.  Log actions to dump_file.  Visit blocks in index
            order.
    
    for  libgcc/ChangeLog
    
            * hardcfr.c (block2mask): Split out of...
            (visited_p): ... this.
            (check_seq): Move trapping out, return result instead.
            (__hardcfr_debug_cfg): New.
            (__hardcfr_check_fail): New, with optional verbose error
            printing before trapping.
            (__hardcfr_check): Call __hardcfr_check_fail when check_seq
            fails.
    
    for  gcc/testsuite/ChangeLog
    
            * c-c++-common/harden-cfr-noret-never-O0.c: New.
            * c-c++-common/torture/harden-cfr-noret-never.c: New.
            * c-c++-common/torture/harden-cfr-noret-noexcept.c: New.
            * c-c++-common/torture/harden-cfr-noret-nothrow.c: New.
            * c-c++-common/torture/harden-cfr-noret.c: New.
            * c-c++-common/torture/harden-cfr-notail.c: New.
            * c-c++-common/torture/harden-cfr-returning.c: New.
            * c-c++-common/torture/harden-cfr-tail.c: Extend.
            * g++.dg/harden-cfr-throw-always-O0.C: New.
            * g++.dg/harden-cfr-throw-returning-O0.C: New.
            * g++.dg/torture/harden-cfr-noret-always-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-noret-never-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-noret-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-throw-always.C: New.
            * g++.dg/torture/harden-cfr-throw-nocleanup.C: New.
            * g++.dg/torture/harden-cfr-throw-returning.C: New.
            * g++.dg/torture/harden-cfr-throw.C: New.
            * gcc.dg/torture/harden-cfr-noret-no-nothrow.c: New.
            * gcc.dg/torture/harden-cfr-tail-ub.c: New.
            * gnat.dg/hardcfr.adb: Disable checking at exceptions.

Diff:
---
 .../doc/gnat_rm/security_hardening_features.rst    | 126 ++-
 gcc/common.opt                                     |  33 +
 gcc/doc/invoke.texi                                |  74 +-
 gcc/flag-types.h                                   |  10 +
 gcc/gimple-harden-control-flow.cc                  | 916 ++++++++++++++++++---
 .../c-c++-common/harden-cfr-noret-never-O0.c       |  12 +
 .../c-c++-common/torture/harden-cfr-noret-never.c  |  18 +
 .../torture/harden-cfr-noret-noexcept.c            |  16 +
 .../torture/harden-cfr-noret-nothrow.c             |  13 +
 .../c-c++-common/torture/harden-cfr-noret.c        |  38 +
 .../c-c++-common/torture/harden-cfr-notail.c       |   8 +
 .../c-c++-common/torture/harden-cfr-returning.c    |  35 +
 .../c-c++-common/torture/harden-cfr-tail.c         |  55 +-
 gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C  |  11 +
 .../g++.dg/harden-cfr-throw-returning-O0.C         |  10 +
 .../torture/harden-cfr-noret-always-no-nothrow.C   |  16 +
 .../torture/harden-cfr-noret-never-no-nothrow.C    |  18 +
 .../g++.dg/torture/harden-cfr-noret-no-nothrow.C   |  23 +
 .../g++.dg/torture/harden-cfr-throw-always.C       |  20 +
 .../g++.dg/torture/harden-cfr-throw-nocleanup.C    |  11 +
 .../g++.dg/torture/harden-cfr-throw-returning.C    |  31 +
 gcc/testsuite/g++.dg/torture/harden-cfr-throw.C    |  65 ++
 .../gcc.dg/torture/harden-cfr-noret-no-nothrow.c   |  15 +
 gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c  |  40 +
 gcc/testsuite/gnat.dg/hardcfr.adb                  |   2 +-
 libgcc/hardcfr.c                                   | 119 ++-
 26 files changed, 1598 insertions(+), 137 deletions(-)

diff --git a/gcc/ada/doc/gnat_rm/security_hardening_features.rst b/gcc/ada/doc/gnat_rm/security_hardening_features.rst
index f5fdc8e46b4..4dfda486795 100644
--- a/gcc/ada/doc/gnat_rm/security_hardening_features.rst
+++ b/gcc/ada/doc/gnat_rm/security_hardening_features.rst
@@ -263,11 +263,127 @@ For each block that is marked as visited, the mechanism checks that at
 least one of its predecessors, and at least one of its successors, are
 also marked as visited.
 
-Verification is performed just before returning.  Subprogram
-executions that complete by raising or propagating an exception bypass
-verification-and-return points.  A subprogram that can only complete
-by raising or propagating an exception may have instrumentation
-disabled altogether.
+Verification is performed just before a subprogram returns.  The
+following fragment:
+
+.. code-block:: ada
+
+   if X then
+     Y := F (Z);
+     return;
+   end if;
+
+
+gets turned into:
+
+.. code-block:: ada
+
+   type Visited_Bitmap is array (1..N) of Boolean with Pack;
+   Visited : aliased Visited_Bitmap := (others => False);
+   --  Bitmap of visited blocks.  N is the basic block count.
+   [...]
+   --  Basic block #I
+   Visited(I) := True;
+   if X then
+     --  Basic block #J
+     Visited(J) := True;
+     Y := F (Z);
+     CFR.Check (N, Visited'Access, CFG'Access);
+     --  CFR is a hypothetical package whose Check procedure calls
+     --  libgcc's __hardcfr_check, that traps if the Visited bitmap
+     --  does not hold a valid path in CFG, the run-time
+     --  representation of the control flow graph in the enclosing
+     --  subprogram.
+     return;
+   end if;
+   --  Basic block #K
+   Visited(K) := True;
+
+
+Verification would also be performed before must-tail calls, and
+before early-marked potential tail calls, but these do not occur in
+practice, as potential tail-calls are only detected in late
+optimization phases, too late for this transformation to act on it.
+
+In order to avoid adding verification after potential tail calls,
+which would prevent tail-call optimization, we recognize returning
+calls, i.e., calls whose result, if any, is returned by the calling
+subprogram to its caller immediately after the call returns.
+Verification is performed before such calls, whether or not they are
+ultimately optimized to tail calls.  This behavior is enabled by
+default whenever sibcall optimization is enabled (see
+:switch:`-foptimize-sibling-calls`); it may be disabled with
+:switch:`-fno-hardcfr-check-returning-calls`, or enabled with
+:switch:`-fhardcfr-check-returning-calls`, regardless of the
+optimization, but the lack of other optimizations may prevent calls
+from being recognized as returning calls:
+
+.. code-block:: ada
+
+     --  CFR.Check here, with -fhardcfr-check-returning-calls.
+     P (X);
+     --  CFR.Check here, with -fno-hardcfr-check-returning-calls.
+     return;
+
+or:
+
+.. code-block:: ada
+
+     --  CFR.Check here, with -fhardcfr-check-returning-calls.
+     R := F (X);
+     --  CFR.Check here, with -fno-hardcfr-check-returning-calls.
+     return R;
+
+
+Any subprogram from which an exception may escape, i.e., that may
+raise or propagate an exception that isn't handled internally, is
+conceptually enclosed by a cleanup handler that performs verification,
+unless this is disabled with :switch:`-fno-hardcfr-check-exceptions`.
+With this feature enabled, a subprogram body containing:
+
+.. code-block:: ada
+
+     --  ...
+       Y := F (X);  -- May raise exceptions.
+     --  ...
+       raise E;  -- Not handled internally.
+     --  ...
+
+
+gets modified as follows:
+
+.. code-block:: ada
+
+   begin
+     --  ...
+       Y := F (X);  -- May raise exceptions.
+     --  ...
+       raise E;  -- Not handled internally.
+     --  ...
+   exception
+     when others =>
+       CFR.Check (N, Visited'Access, CFG'Access);
+       raise;
+   end;
+
+
+Verification may also be performed before No_Return calls, whether
+only nothrow ones, with
+:switch:`-fhardcfr-check-noreturn-calls=nothrow`, or all of them, with
+:switch:`-fhardcfr-check-noreturn-calls=always`.  The default is
+:switch:`-fhardcfr-check-noreturn-calls=never` for this feature, that
+disables checking before No_Return calls.
+
+When a No_Return call returns control to its caller through an
+exception, verification may have already been performed before the
+call, if :switch:`-fhardcfr-check-noreturn-calls=always` is in effect.
+The compiler arranges for already-checked No_Return calls without a
+preexisting handler to bypass the implicitly-added cleanup handler and
+thus the redundant check, but a local exception or cleanup handler, if
+present, will modify the set of visited blocks, and checking will take
+place again when the caller reaches the next verification point,
+whether it is a return or reraise statement after the exception is
+otherwise handled, or even another No_Return call.
 
 The instrumentation for hardening with control flow redundancy can be
 observed in dump files generated by the command-line option
diff --git a/gcc/common.opt b/gcc/common.opt
index f093be93542..11c1ec95c8f 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -1795,6 +1795,39 @@ fharden-control-flow-redundancy
 Common Var(flag_harden_control_flow_redundancy) Optimization
 Harden control flow by recording and checking execution paths.
 
+fhardcfr-check-returning-calls
+Common Var(flag_harden_control_flow_redundancy_check_returning_calls) Init(-1) Optimization
+Check CFR execution paths also before calls followed by returns of their results.
+
+fhardcfr-check-exceptions
+Common Var(flag_harden_control_flow_redundancy_check_exceptions) Init(-1) Optimization
+Check CFR execution paths also when exiting a function through an exception.
+
+fhardcfr-check-noreturn-calls=
+Common Joined RejectNegative Enum(hardcfr_check_noreturn_calls) Var(flag_harden_control_flow_redundancy_check_noreturn) Init(HCFRNR_UNSPECIFIED) Optimization
+-fhardcfr-check-noreturn-calls=[always|nothrow|never]	Check CFR execution paths also before calling noreturn functions.
+
+Enum
+Name(hardcfr_check_noreturn_calls) Type(enum hardcfr_noret) UnknownError(unknown hardcfr noreturn checking level %qs)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(never) Value(HCFRNR_NEVER)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(nothrow) Value(HCFRNR_NOTHROW)
+
+; ??? There could be yet another option here, that checked before
+; noreturn calls, except for those known to always throw, if we had
+; means to distinguish noreturn functions known to always throw, such
+; as those used to (re)raise exceptions, from those that merely might
+; throw.  "not always" stands for "not always-throwing", but it also
+; contrasts with "always" below.
+; EnumValue
+; Enum(hardcfr_check_noreturn_calls) String(not-always) Value(HCFRNR_NOT_ALWAYS)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(always) Value(HCFRNR_ALWAYS)
+
 ; Nonzero means ignore `#ident' directives.  0 means handle them.
 ; Generate position-independent code for executables if possible
 ; On SVR4 targets, it also controls whether or not to emit a
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index e6b36c8caad..e61dae228a0 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -626,7 +626,9 @@ Objective-C and Objective-C++ Dialects}.
 -fsanitize-undefined-trap-on-error  -fbounds-check @gol
 -fcf-protection=@r{[}full@r{|}branch@r{|}return@r{|}none@r{|}check@r{]} @gol
 -fharden-compares -fharden-conditional-branches @gol
--fharden-control-flow-redundancy @gol
+-fharden-control-flow-redundancy  -fhardcfr-check-exceptions  @gol
+-fhardcfr-check-returning-calls  @gol
+-fhardcfr-check-noreturn-calls=@r{[}always@r{|}nothrow@r{|}never@r{]}  @gol
 -fstack-protector  -fstack-protector-all  -fstack-protector-strong @gol
 -fstack-protector-explicit  -fstack-check @gol
 -fstack-limit-register=@var{reg}  -fstack-limit-symbol=@var{sym} @gol
@@ -16597,11 +16599,75 @@ conditionals.
 @item -fharden-control-flow-redundancy
 @opindex fharden-control-flow-redundancy
 Emit extra code to set booleans when entering basic blocks, and to
-verify, at function exits, that they amount to an execution path that is
-consistent with the control flow graph, trapping otherwise.  Tuning
-options @option{--param hardcfr-max-blocks} and @option{--param
+verify and trap, at function exits, when the booleans do not form an
+execution path that is compatible with the control flow graph.
+
+Verification takes place before returns, before mandatory tail calls
+(see below) and, optionally, before escaping exceptions with
+@option{-fhardcfr-check-exceptions}, before returning calls with
+@option{-fhardcfr-check-returning-calls}, and before noreturn calls with
+@option{-fhardcfr-check-noreturn-calls}).  Tuning options
+@option{--param hardcfr-max-blocks} and @option{--param
 hardcfr-max-inline-blocks} are available.
 
+Tail call optimization takes place too late to affect control flow
+redundancy, but calls annotated as mandatory tail calls by language
+front-ends, and any calls marked early enough as potential tail calls
+would also have verification issued before the call, but these
+possibilities are merely theoretical, as these conditions can only be
+met when using custom compiler plugins.
+
+@item -fhardcfr-check-exceptions
+@opindex fhardcfr-check-exceptions
+@opindex fno-hardcfr-check-exceptions
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph at exception
+escape points, as if the function body was wrapped with a cleanup
+handler that performed the check and reraised.  This option is enabled
+by default; use @option{-fno-hardcfr-check-exceptions} to disable it.
+
+@item -fhardcfr-check-returning-calls
+@opindex fhardcfr-check-returning-calls
+@opindex fno-hardcfr-check-returning-calls
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph before any
+function call immediately followed by a return of its result, if any, so
+as to not prevent tail-call optimization, whether or not it is
+ultimately optimized to a tail call.
+
+This option is enabled by default whenever sibling call optimizations
+are enabled (see @option{-foptimize-sibling-calls}), but it can be
+enabled (or disabled, using its negated form) explicitly, regardless of
+the optimizations.
+
+@item -fhardcfr-check-noreturn-calls=@r{[}always@r{|}nothrow@r{|}never@r{]}
+@opindex fhardcfr-check-noreturn-calls
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph before
+@code{noreturn} calls, either all of them (@option{always}), those that
+may not return control to the caller through an exception either
+(@option{nothrow}), or none of them (@option{never}, the default).
+
+Checking before a @code{noreturn} function that may return control to
+the caller through an exception may cause checking to be performed more
+than once, if the exception is caught in the caller, whether by a
+handler or a cleanup.  When @option{-fhardcfr-check-exceptions} is also
+enabled, the compiler will avoid associating a @code{noreturn} call with
+the implicitly-added cleanup handler, since it would be redundant with
+the check performed before the call, but other handlers or cleanups in
+the function, if activated, will modify the recorded execution path and
+check it again when another checkpoint is hit.  The checkpoint may even
+be another @code{noreturn} call, so checking may end up performed
+multiple times.
+
+Various optimizers may cause calls to be marked as @code{noreturn}
+and/or @code{nothrow}, even in the absence of the corresponding
+attributes, which may affect the placement of checks before calls, as
+well as the addition of implicit cleanup handlers for them.  This
+unpredictability, and the fact that raising and reraising exceptions
+frequently amounts to implicitly calling @code{noreturn} functions, have
+made @option{never} the default setting for this option.
+
 @item -fstack-protector
 @opindex fstack-protector
 Emit extra code to check for buffer overflows, such as stack smashing
diff --git a/gcc/flag-types.h b/gcc/flag-types.h
index d2e751060ff..3fae7548cab 100644
--- a/gcc/flag-types.h
+++ b/gcc/flag-types.h
@@ -157,6 +157,16 @@ enum stack_reuse_level
   SR_ALL
 };
 
+/* Control Flow Redundancy hardening options for noreturn calls.  */
+enum hardcfr_noret
+{
+  HCFRNR_NEVER,
+  HCFRNR_NOTHROW,
+  HCFRNR_NOT_ALWAYS, /* Reserved for future use.  */
+  HCFRNR_ALWAYS,
+  HCFRNR_UNSPECIFIED = -1
+};
+
 /* The live patching level.  */
 enum live_patching_level
 {
diff --git a/gcc/gimple-harden-control-flow.cc b/gcc/gimple-harden-control-flow.cc
index 6b08846dbb1..1c93bf622e8 100644
--- a/gcc/gimple-harden-control-flow.cc
+++ b/gcc/gimple-harden-control-flow.cc
@@ -29,7 +29,12 @@ along with GCC; see the file COPYING3.  If not see
 #include "tree-pass.h"
 #include "ssa.h"
 #include "gimple-iterator.h"
+#include "gimple-pretty-print.h"
 #include "tree-cfg.h"
+#include "tree-cfgcleanup.h"
+#include "tree-eh.h"
+#include "except.h"
+#include "sbitmap.h"
 #include "basic-block.h"
 #include "cfghooks.h"
 #include "cfgloop.h"
@@ -60,9 +65,7 @@ const pass_data pass_data_harden_control_flow_redundancy = {
   0,	    // properties_provided
   0,	    // properties_destroyed
   TODO_cleanup_cfg, // properties_start
-  TODO_update_ssa
-  | TODO_cleanup_cfg
-  | TODO_verify_il, // properties_finish
+  0,        // properties_finish
 };
 
 class pass_harden_control_flow_redundancy : public gimple_opt_pass
@@ -79,16 +82,6 @@ public:
     if (!flag_harden_control_flow_redundancy)
       return false;
 
-    /* We don't verify when an exception escapes, propagated or raised
-       by the function itself, so we're only concerned with edges to
-       the exit block.  If there aren't any, the function doesn't
-       return normally, so there won't be any checking point, so
-       there's no point in running the pass.  Should we add
-       verification at exception escapes, we should at least look at
-       !flag_exceptions here.  */
-    if (EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (fun)->preds) == 0)
-      return false;
-
     /* Functions that return more than once, like setjmp and vfork
        (that also gets this flag set), will start recording a path
        after the first return, and then may take another path when
@@ -117,8 +110,9 @@ public:
 	return false;
       }
 
-    if (param_hardcfr_max_blocks > 0
-	&& n_basic_blocks_for_fn (fun) - 2 > param_hardcfr_max_blocks)
+    if (fun->cfg && param_hardcfr_max_blocks > 0
+	&& (n_basic_blocks_for_fn (fun) - NUM_FIXED_BLOCKS
+	    > param_hardcfr_max_blocks))
       {
 	warning_at (DECL_SOURCE_LOCATION (fun->decl), 0,
 		    "%qD has more than %u blocks, the requested"
@@ -134,6 +128,293 @@ public:
 
 }
 
+/* Return TRUE iff CFR checks should be inserted before returning
+   calls.  */
+
+static bool
+check_returning_calls_p ()
+{
+  return
+    flag_harden_control_flow_redundancy_check_returning_calls > 0
+    || (flag_harden_control_flow_redundancy_check_returning_calls < 0
+	/* Gates pass_tail_calls.  */
+	&& flag_optimize_sibling_calls
+	/* Gates pass_all_ptimizations.  */
+	&& optimize >= 1 && !optimize_debug);
+}
+
+/* Scan BB from the end, updating *RETPTR if given as return stmts and
+   copies are found.  Return a call or a stmt that cannot appear after
+   a tail call, or NULL if the top of the block is reached without
+   finding any.  */
+
+static gimple *
+hardcfr_scan_block (basic_block bb, tree **retptr)
+{
+  gimple_stmt_iterator gsi;
+  for (gsi = gsi_last_bb (bb); !gsi_end_p (gsi); gsi_prev (&gsi))
+    {
+      gimple *stmt = gsi_stmt (gsi);
+
+      /* Ignore labels, returns, nops, clobbers and debug stmts.  */
+      if (gimple_code (stmt) == GIMPLE_LABEL
+	  || gimple_code (stmt) == GIMPLE_NOP
+	  || gimple_code (stmt) == GIMPLE_PREDICT
+	  || gimple_clobber_p (stmt)
+	  || is_gimple_debug (stmt))
+	continue;
+
+      if (gimple_code (stmt) == GIMPLE_RETURN)
+	{
+	  greturn *gret = as_a <greturn *> (stmt);
+	  if (retptr)
+	    {
+	      gcc_checking_assert (!*retptr);
+	      *retptr = gimple_return_retval_ptr (gret);
+	    }
+	  continue;
+	}
+
+      /* Check for a call.  */
+      if (is_gimple_call (stmt))
+	return stmt;
+
+      /* Allow simple copies to the return value, updating the return
+	 value to be found in earlier assignments.  */
+      if (retptr && *retptr && gimple_assign_single_p (stmt)
+	  && **retptr == gimple_assign_lhs (stmt))
+	{
+	  *retptr = gimple_assign_rhs1_ptr (stmt);
+	  continue;
+	}
+
+      return stmt;
+    }
+
+  /* Any other kind of stmt will prevent a tail call.  */
+  return NULL;
+}
+
+/* Return TRUE iff CALL is to be preceded by a CFR checkpoint, i.e.,
+   if it's a returning call (one whose result is ultimately returned
+   without intervening non-copy statements) and we're checking
+   returning calls, a __builtin_return call (noreturn with a path to
+   the exit block), a must-tail call, or a tail call.  */
+
+static bool
+returning_call_p (gcall *call)
+{
+  if (!(gimple_call_noreturn_p (call)
+	|| gimple_call_must_tail_p (call)
+	|| gimple_call_tail_p (call)
+	|| check_returning_calls_p ()))
+    return false;
+
+  /* Quickly check that there's a path to exit compatible with a
+     returning call.  Detect infinite loops through the counter.  */
+  basic_block bb = gimple_bb (call);
+  auto_vec<basic_block, 10> path;
+  for (int i = n_basic_blocks_for_fn (cfun);
+       bb != EXIT_BLOCK_PTR_FOR_FN (cfun) && i--;
+       bb = single_succ (bb))
+    if (!single_succ_p (bb)
+	|| (single_succ_edge (bb)->flags & EDGE_EH) != 0)
+      return false;
+    else
+      path.safe_push (bb);
+
+  /* Check the stmts in the blocks and trace the return value.  */
+  tree *retptr = NULL;
+  for (;;)
+    {
+      gcc_checking_assert (!path.is_empty ());
+      basic_block bb = path.pop ();
+      gimple *stop = hardcfr_scan_block (bb, &retptr);
+      if (stop)
+	{
+	  if (stop != call)
+	    return false;
+	  gcc_checking_assert (path.is_empty ());
+	  break;
+	}
+
+      gphi *retphi = NULL;
+      if (retptr && *retptr && TREE_CODE (*retptr) == SSA_NAME
+	  && !SSA_NAME_IS_DEFAULT_DEF (*retptr)
+	  && SSA_NAME_DEF_STMT (*retptr)
+	  && is_a <gphi *> (SSA_NAME_DEF_STMT (*retptr))
+	  && gimple_bb (SSA_NAME_DEF_STMT (*retptr)) == bb)
+	{
+	  retphi = as_a <gphi *> (SSA_NAME_DEF_STMT (*retptr));
+	  gcc_checking_assert (gimple_phi_result (retphi) == *retptr);
+	}
+      else
+	continue;
+
+      gcc_checking_assert (!path.is_empty ());
+      edge e = single_succ_edge (path.last ());
+      int i = EDGE_COUNT (bb->preds);
+      while (i--)
+	if (EDGE_PRED (bb, i) == e)
+	  break;
+      gcc_checking_assert (i >= 0);
+      retptr = gimple_phi_arg_def_ptr (retphi, i);
+    }
+
+  return (gimple_call_noreturn_p (call)
+	  || gimple_call_must_tail_p (call)
+	  || gimple_call_tail_p (call)
+	  || (gimple_call_lhs (call) == (retptr ? *retptr : NULL)
+	      && check_returning_calls_p ()));
+}
+
+typedef auto_vec<edge, 10> chk_edges_t;
+
+/* Declare for mutual recursion.  */
+static bool hardcfr_sibcall_search_preds (basic_block bb,
+					  chk_edges_t &chk_edges,
+					  int &count_chkcall,
+					  auto_sbitmap &chkcall_blocks,
+					  int &count_postchk,
+					  auto_sbitmap &postchk_blocks,
+					  tree *retptr);
+
+/* Search backwards from the end of BB for a mandatory or potential
+   sibcall.  Schedule the block to be handled sort-of like noreturn if
+   so.  Recurse to preds, with updated RETPTR, if the block only
+   contains stmts that may follow such a call, scheduling checking at
+   edges and marking blocks as post-check as needed.  Return true iff,
+   at the end of the block, a check will have already been
+   performed.  */
+
+static bool
+hardcfr_sibcall_search_block (basic_block bb,
+			      chk_edges_t &chk_edges,
+			      int &count_chkcall,
+			      auto_sbitmap &chkcall_blocks,
+			      int &count_postchk,
+			      auto_sbitmap &postchk_blocks,
+			      tree *retptr)
+{
+  /* Conditionals and internal exceptions rule out tail calls.  */
+  if (!single_succ_p (bb)
+      || (single_succ_edge (bb)->flags & EDGE_EH) != 0)
+    return false;
+
+  gimple *stmt = hardcfr_scan_block (bb, &retptr);
+  if (!stmt)
+    return hardcfr_sibcall_search_preds (bb, chk_edges,
+					 count_chkcall, chkcall_blocks,
+					 count_postchk, postchk_blocks,
+					 retptr);
+
+  if (!is_a <gcall *> (stmt))
+    return false;
+
+  /* Avoid disrupting mandatory or early-marked tail calls,
+     inserting the check before them.  This works for
+     must-tail calls, but tail calling as an optimization is
+     detected too late for us.
+
+     Also check for noreturn calls here.  Noreturn calls won't
+     normally have edges to exit, so they won't be found here,
+     but __builtin_return does, and we must check before
+     it, so handle it like a tail call.  */
+  gcall *call = as_a <gcall *> (stmt);
+  if (!(gimple_call_noreturn_p (call)
+	|| gimple_call_must_tail_p (call)
+	|| gimple_call_tail_p (call)
+	|| (gimple_call_lhs (call) == (retptr ? *retptr : NULL)
+	    && check_returning_calls_p ())))
+    return false;
+
+  gcc_checking_assert (returning_call_p (call));
+
+  /* We found a call that is to be preceded by checking.  */
+  if (bitmap_set_bit (chkcall_blocks, bb->index))
+    ++count_chkcall;
+  else
+    gcc_unreachable ();
+  return true;
+}
+
+
+/* Search preds of BB for a mandatory or potential sibcall or
+   returning call, and arrange for the blocks containing them to have
+   a check inserted before the call, like noreturn calls.  If any
+   preds are found to perform checking, schedule checks at the edges
+   of those that don't, and mark BB as postcheck..  */
+
+static bool
+hardcfr_sibcall_search_preds (basic_block bb,
+			      chk_edges_t &chk_edges,
+			      int &count_chkcall,
+			      auto_sbitmap &chkcall_blocks,
+			      int &count_postchk,
+			      auto_sbitmap &postchk_blocks,
+			      tree *retptr)
+{
+  /* For the exit block, we wish to force a check at every
+     predecessor, so pretend we've already found a pred that had
+     checking, so that we schedule checking at every one of its pred
+     edges.  */
+  bool first = bb->index >= NUM_FIXED_BLOCKS;
+  bool postchecked = true;
+
+  gphi *retphi = NULL;
+  if (retptr && *retptr && TREE_CODE (*retptr) == SSA_NAME
+      && !SSA_NAME_IS_DEFAULT_DEF (*retptr)
+      && SSA_NAME_DEF_STMT (*retptr)
+      && is_a <gphi *> (SSA_NAME_DEF_STMT (*retptr))
+      && gimple_bb (SSA_NAME_DEF_STMT (*retptr)) == bb)
+    {
+      retphi = as_a <gphi *> (SSA_NAME_DEF_STMT (*retptr));
+      gcc_checking_assert (gimple_phi_result (retphi) == *retptr);
+    }
+
+  for (int i = EDGE_COUNT (bb->preds); i--; first = false)
+    {
+      edge e = EDGE_PRED (bb, i);
+
+      bool checked
+	= hardcfr_sibcall_search_block (e->src, chk_edges,
+					count_chkcall, chkcall_blocks,
+					count_postchk, postchk_blocks,
+					!retphi ? retptr
+					: gimple_phi_arg_def_ptr (retphi, i));
+
+      if (first)
+	{
+	  postchecked = checked;
+	  continue;
+	}
+
+      /* When we first find a checked block, force a check at every
+	 other incoming edge we've already visited, and those we
+	 visit afterwards that don't have their own check, so that
+	 when we reach BB, the check has already been performed.  */
+      if (!postchecked && checked)
+	{
+	  for (int j = EDGE_COUNT (bb->preds); --j > i; )
+	    chk_edges.safe_push (EDGE_PRED (bb, j));
+	  postchecked = true;
+	}
+      if (postchecked && !checked)
+	chk_edges.safe_push (EDGE_PRED (bb, i));
+    }
+
+  if (postchecked && bb->index >= NUM_FIXED_BLOCKS)
+    {
+      if (bitmap_set_bit (postchk_blocks, bb->index))
+	count_postchk++;
+      else
+	gcc_unreachable ();
+    }
+
+  return postchecked;
+}
+
+
 class rt_bb_visited
 {
   /* Use a sufficiently wide unsigned type to hold basic block numbers.  */
@@ -176,8 +457,8 @@ class rt_bb_visited
      neither ENTRY nor EXIT, but maybe one-past-the-end, to compute
      the visited array length.  */
   blknum num2idx (blknum n) {
-    gcc_checking_assert (n >= 2 && n <= nblocks);
-    return (n - 2);
+    gcc_checking_assert (n >= NUM_FIXED_BLOCKS && n <= nblocks);
+    return (n - NUM_FIXED_BLOCKS);
   }
   /* Return the block vindex for BB, that must not be ENTRY or
      EXIT.  */
@@ -249,8 +530,7 @@ class rt_bb_visited
   }
 
   /* Set the bit corresponding to BB in VISITED.  Add to SEQ any
-     required gimple statements, and return SEQ, possibly
-     modified.  */
+     required gimple stmts, and return SEQ, possibly modified.  */
   gimple_seq vset (basic_block bb, gimple_seq seq = NULL)
   {
     tree bit, setme = vword (bb, &bit);
@@ -270,7 +550,7 @@ class rt_bb_visited
 
 public:
   /* Prepare to add control flow redundancy testing to CFUN.  */
-  rt_bb_visited ()
+  rt_bb_visited (int checkpoints)
     : nblocks (n_basic_blocks_for_fn (cfun)),
       vword_type (NULL), ckseq (NULL), rtcfg (NULL)
   {
@@ -353,8 +633,8 @@ public:
 					 NULL, NULL);
     gimple_seq_add_stmt (&ckseq, detach);
 
-    if (nblocks - 2 > blknum (param_hardcfr_max_inline_blocks)
-	|| !single_pred_p (EXIT_BLOCK_PTR_FOR_FN (cfun)))
+    if (nblocks - NUM_FIXED_BLOCKS > blknum (param_hardcfr_max_inline_blocks)
+	|| checkpoints > 1)
       {
 	/* Make sure vword_bits is wide enough for the representation
 	   of nblocks in rtcfg.  Compare with vword_bits << vword_bits,
@@ -379,65 +659,33 @@ public:
     gimple_seq_add_stmt (&ckseq, ckfail_init);
   }
 
-  /* Insert SEQ on E, or close enough (e.g., before a noreturn or tail
-     call at the end of E->src).  */
-  void insert_exit_check (gimple_seq seq, edge e)
+  /* Insert SEQ before a resx or a call in INSBB.  */
+  void insert_exit_check_in_block (gimple_seq seq, basic_block insbb)
   {
-    basic_block insbb = e->src;
-
-    /* If the returning block ends with a noreturn call, insert
-       checking before it.  This is particularly important for
-       __builtin_return.  Other noreturn calls won't have an edge to
-       the exit block, so they won't even be considered as exit paths.
-
-       Insert-on-edge inserts before other return stmts, but not
-       before calls, and if a single-block function had the check
-       sequence inserted after a noreturn call, it would be useless,
-       but checking would still flag it as malformed if block 2 has a
-       fallthru edge to the exit block.
-
-       Also avoid disrupting tail calls, inserting the check before
-       them.  This works for must-tail calls, but tail calling as an
-       optimization is detected too late for us.  */
     gimple_stmt_iterator gsi = gsi_last_bb (insbb);
-    gimple *ret = gsi_stmt (gsi);
-    if (ret && is_a <greturn *> (ret))
-      {
+
+    while (!gsi_end_p (gsi))
+      if (is_a <gresx *> (gsi_stmt (gsi))
+	  || is_a <gcall *> (gsi_stmt (gsi)))
+	break;
+      else
 	gsi_prev (&gsi);
-	if (!gsi_end_p (gsi))
-	  ret = gsi_stmt (gsi);
-      }
-    if (ret && is_a <gcall *> (ret)
-	&& (gimple_call_noreturn_p (ret)
-	    || gimple_call_must_tail_p (as_a <gcall *> (ret))
-	    || gimple_call_tail_p (as_a <gcall *> (ret))))
-      gsi_insert_seq_before (&gsi, seq, GSI_SAME_STMT);
-    else
-      gsi_insert_seq_on_edge_immediate (e, seq);
+
+    gsi_insert_seq_before (&gsi, seq, GSI_SAME_STMT);
+  }
+
+  /* Insert SEQ on E.  */
+  void insert_exit_check_on_edge (gimple_seq seq, edge e)
+  {
+    gsi_insert_seq_on_edge_immediate (e, seq);
   }
 
-  /* Add checking code on every exit edge, and initialization code on
+  /* Add checking code to CHK_EDGES, and initialization code on
      the entry edge.  Before this point, the CFG has been undisturbed,
      and all the needed data has been collected and safely stowed.  */
-  void check ()
+  void check (chk_edges_t &chk_edges,
+	      int count_chkcall, auto_sbitmap const &chkcall_blocks)
   {
-    /* Insert initializers for visited at the entry.  */
-    gimple_seq iseq = NULL;
-
-    gcall *vinit = gimple_build_call (builtin_decl_explicit
-				      (BUILT_IN_MEMSET), 3,
-				      build1 (ADDR_EXPR,
-					      build_pointer_type
-					      (TREE_TYPE (visited)),
-					      visited),
-				      integer_zero_node,
-				      TYPE_SIZE_UNIT (TREE_TYPE (visited)));
-    gimple_seq_add_stmt (&iseq, vinit);
-
-    gsi_insert_seq_on_edge_immediate (single_succ_edge
-				      (ENTRY_BLOCK_PTR_FOR_FN (cfun)),
-				      iseq);
-
     /* If we're using out-of-line checking, create and statically
        initialize the CFG checking representation, generate the
        checker call for the checking sequence, and insert it in all
@@ -498,28 +746,116 @@ public:
 						     rtcfg));
 	gimple_seq_add_stmt (&ckseq, call_chk);
 
+	gimple *clobber = gimple_build_assign (visited,
+					       build_clobber
+					       (TREE_TYPE (visited)));
+	gimple_seq_add_stmt (&ckseq, clobber);
+
 	/* If we have multiple exit edges, insert (copies of)
 	   ckseq in all of them.  */
-	for (int i = EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (cfun)->preds);
-	     i--; )
+	for (int i = chk_edges.length (); i--; )
 	  {
 	    gimple_seq seq = ckseq;
 	    /* Copy the sequence, unless we're dealing with the
 	       last edge (we're counting down to zero).  */
-	    if (i)
+	    if (i || count_chkcall)
+	      seq = gimple_seq_copy (seq);
+
+	    edge e = chk_edges[i];
+
+	    if (dump_file)
+	      {
+		if (e->dest == EXIT_BLOCK_PTR_FOR_FN (cfun))
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to exit.\n",
+			   e->src->index);
+		else
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to postcheck block %i.\n",
+			   e->src->index, e->dest->index);
+	      }
+
+	    insert_exit_check_on_edge (seq, e);
+
+	    gcc_checking_assert (!bitmap_bit_p (chkcall_blocks, e->src->index));
+	  }
+
+	sbitmap_iterator it;
+	unsigned i;
+	EXECUTE_IF_SET_IN_BITMAP (chkcall_blocks, 0, i, it)
+	  {
+	    basic_block bb = BASIC_BLOCK_FOR_FN (cfun, i);
+
+	    gimple_seq seq = ckseq;
+	    gcc_checking_assert (count_chkcall > 0);
+	    if (--count_chkcall)
 	      seq = gimple_seq_copy (seq);
 
-	    insert_exit_check (seq,
-			       EDGE_PRED (EXIT_BLOCK_PTR_FOR_FN (cfun), i));
+	    if (dump_file)
+	      fprintf (dump_file,
+		       "Inserting out-of-line check before stmt in block %i.\n",
+		       bb->index);
+
+	    insert_exit_check_in_block (seq, bb);
 	  }
+
+	gcc_checking_assert (count_chkcall == 0);
       }
     else
       {
 	/* Inline checking requires a single exit edge.  */
-	gimple *last = gsi_stmt (gsi_last (ckseq));
+	gimple *last = gimple_build_assign (visited,
+					       build_clobber
+					       (TREE_TYPE (visited)));
+	gimple_seq_add_stmt (&ckseq, last);
 
-	insert_exit_check (ckseq,
-			   single_pred_edge (EXIT_BLOCK_PTR_FOR_FN (cfun)));
+	if (!count_chkcall)
+	  {
+	    edge e = single_pred_edge (EXIT_BLOCK_PTR_FOR_FN (cfun));
+
+	    if (dump_file)
+	      {
+		if (e->dest == EXIT_BLOCK_PTR_FOR_FN (cfun))
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to postcheck block %i.\n",
+			   e->src->index, e->dest->index);
+		else
+		  fprintf (dump_file,
+			   "Inserting inline check in"
+			   " block %i's edge to exit.\n",
+			   e->src->index);
+	      }
+
+	    insert_exit_check_on_edge (ckseq, e);
+	  }
+	else
+	  {
+	    gcc_checking_assert (count_chkcall == 1);
+
+	    sbitmap_iterator it;
+	    unsigned i;
+	    EXECUTE_IF_SET_IN_BITMAP (chkcall_blocks, 0, i, it)
+	      {
+		basic_block bb = BASIC_BLOCK_FOR_FN (cfun, i);
+
+		gimple_seq seq = ckseq;
+		gcc_checking_assert (count_chkcall > 0);
+		if (--count_chkcall)
+		  seq = gimple_seq_copy (seq);
+
+		if (dump_file)
+		  fprintf (dump_file,
+			   "Inserting inline check before stmt in block %i.\n",
+			   bb->index);
+
+		insert_exit_check_in_block (seq, bb);
+	      }
+
+	    gcc_checking_assert (count_chkcall == 0);
+	  }
 
 	/* The inserted ckseq computes CKFAIL at LAST.  Now we have to
 	   conditionally trap on it.  */
@@ -540,8 +876,7 @@ public:
 	  add_bb_to_loop (trp, current_loops->tree_root);
 
 	/* Insert a conditional branch to the trap block.  If the
-	   conditional wouldn't be the last statement, split the
-	   block.  */
+	   conditional wouldn't be the last stmt, split the block.  */
 	gimple_stmt_iterator gsi = gsi_for_stmt (last);
 	if (!gsi_one_before_end_p (gsi))
 	  split_block (gsi_bb (gsi), gsi_stmt (gsi));
@@ -564,6 +899,24 @@ public:
 	if (dom_info_available_p (CDI_DOMINATORS))
 	  set_immediate_dominator (CDI_DOMINATORS, trp, gimple_bb (last));
       }
+
+    /* Insert initializers for visited at the entry.  Do this after
+       other insertions, to avoid messing with block numbers.  */
+    gimple_seq iseq = NULL;
+
+    gcall *vinit = gimple_build_call (builtin_decl_explicit
+				      (BUILT_IN_MEMSET), 3,
+				      build1 (ADDR_EXPR,
+					      build_pointer_type
+					      (TREE_TYPE (visited)),
+					      visited),
+				      integer_zero_node,
+				      TYPE_SIZE_UNIT (TREE_TYPE (visited)));
+    gimple_seq_add_stmt (&iseq, vinit);
+
+    gsi_insert_seq_on_edge_immediate (single_succ_edge
+				      (ENTRY_BLOCK_PTR_FOR_FN (cfun)),
+				      iseq);
   }
 
   /* Push onto RTCFG a (mask, index) pair to test for IBB when BB is
@@ -607,7 +960,7 @@ public:
     return false;
   }
 
-  /* Add to CKSEQ statements to clear CKPART if OBB is visited.  */
+  /* Add to CKSEQ stmts to clear CKPART if OBB is visited.  */
   void
   build_block_check (basic_block obb)
   {
@@ -632,34 +985,48 @@ public:
 
   /* Add to BB code to set its bit in VISITED, and add to RTCFG or
      CKSEQ the data or code needed to check BB's predecessors and
-     successors.  Do NOT change the CFG.  */
-  void visit (basic_block bb)
+     successors.  If CHECKPOINT, assume the block is a checkpoint,
+     whether or not it has an edge to EXIT.  If POSTCHECK, assume the
+     block post-dominates checkpoints and therefore no bitmap setting
+     or checks are to be performed in or for it.  Do NOT change the
+     CFG.  */
+  void visit (basic_block bb, bool checkpoint, bool postcheck)
   {
     /* Set the bit in VISITED when entering the block.  */
     gimple_stmt_iterator gsi = gsi_after_labels (bb);
-    gsi_insert_seq_before (&gsi, vset (bb), GSI_SAME_STMT);
+    if (!postcheck)
+      gsi_insert_seq_before (&gsi, vset (bb), GSI_SAME_STMT);
 
     if (rtcfg)
       {
-	/* Build a list of (index, mask) terminated by (NULL, 0).
-	   Consolidate masks with the same index when they're
-	   adjacent.  First, predecessors.  Count backwards, because
-	   we're going to reverse the list.  The order shouldn't
-	   matter, but let's not make it surprising.  */
-	for (int i = EDGE_COUNT (bb->preds); i--; )
-	  if (push_rtcfg_pair (EDGE_PRED (bb, i)->src, bb,
-			       ENTRY_BLOCK_PTR_FOR_FN (cfun)))
-	    break;
+	if (!postcheck)
+	  {
+	    /* Build a list of (index, mask) terminated by (NULL, 0).
+	       Consolidate masks with the same index when they're
+	       adjacent.  First, predecessors.  Count backwards, because
+	       we're going to reverse the list.  The order shouldn't
+	       matter, but let's not make it surprising.  */
+	    for (int i = EDGE_COUNT (bb->preds); i--; )
+	      if (push_rtcfg_pair (EDGE_PRED (bb, i)->src, bb,
+				   ENTRY_BLOCK_PTR_FOR_FN (cfun)))
+		break;
+	  }
 	rtcfg = tree_cons (NULL_TREE, build_int_cst (vword_type, 0), rtcfg);
 
-	/* Then, successors.  */
-	for (int i = EDGE_COUNT (bb->succs); i--; )
-	  if (push_rtcfg_pair (EDGE_SUCC (bb, i)->dest, bb,
-			       EXIT_BLOCK_PTR_FOR_FN (cfun)))
-	    break;
+	if (!postcheck)
+	  {
+	    /* Then, successors.  */
+	    if (!checkpoint
+		|| !push_rtcfg_pair (EXIT_BLOCK_PTR_FOR_FN (cfun),
+				     bb, EXIT_BLOCK_PTR_FOR_FN (cfun)))
+	      for (int i = EDGE_COUNT (bb->succs); i--; )
+		if (push_rtcfg_pair (EDGE_SUCC (bb, i)->dest, bb,
+				     EXIT_BLOCK_PTR_FOR_FN (cfun)))
+		  break;
+	  }
 	rtcfg = tree_cons (NULL_TREE, build_int_cst (vword_type, 0), rtcfg);
       }
-    else
+    else if (!postcheck)
       {
 	/* Schedule test to fail if the block was reached but somehow none
 	   of its predecessors were.  */
@@ -677,6 +1044,8 @@ public:
 	gassign *blkruns = gimple_build_assign (ckpart, unshare_expr (bit));
 	gimple_seq_add_stmt (&ckseq, blkruns);
 
+	if (checkpoint)
+	  build_block_check (EXIT_BLOCK_PTR_FOR_FN (cfun));
 	for (int i = 0, e = EDGE_COUNT (bb->succs); i < e; i++)
 	  build_block_check (EDGE_SUCC (bb, i)->dest);
 
@@ -687,21 +1056,350 @@ public:
   }
 };
 
+/* It might be useful to avoid checking before noreturn calls that are
+   known to always finish by throwing an exception, rather than by
+   ending the program or looping forever.  Such functions would have
+   to be annotated somehow, with an attribute or flag.
+   Exception-raising functions, such as C++'s __cxa_throw,
+   __cxa_rethrow, and Ada's */
+static bool
+always_throwing_noreturn_call_p (gimple *)
+{
+  return false;
+}
+
 /* Control flow redundancy hardening: record the execution path, and
    verify at exit that an expect path was taken.  */
 
 unsigned int
-pass_harden_control_flow_redundancy::execute (function *)
+pass_harden_control_flow_redundancy::execute (function *fun)
 {
-  rt_bb_visited vstd;
-
+  bool const check_at_escaping_exceptions
+    = (flag_exceptions
+       && flag_harden_control_flow_redundancy_check_exceptions);
+  bool const check_before_noreturn_calls
+    = flag_harden_control_flow_redundancy_check_noreturn > HCFRNR_NEVER;
+  bool const check_before_nothrow_noreturn_calls
+    = (check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn >= HCFRNR_NOTHROW);
+  bool const check_before_throwing_noreturn_calls
+    = (flag_exceptions
+       && check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn > HCFRNR_NOTHROW);
+  bool const check_before_always_throwing_noreturn_calls
+    = (flag_exceptions
+       && check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn >= HCFRNR_ALWAYS);
   basic_block bb;
-  FOR_EACH_BB_FN (bb, cfun)
-    vstd.visit (bb);
+  basic_block bb_eh_cleanup = NULL;
+
+  if (check_at_escaping_exceptions)
+    {
+      int lp_eh_cleanup = -1;
+
+      /* Record the preexisting blocks, to avoid visiting newly-created
+	 blocks.  */
+      auto_sbitmap to_visit (last_basic_block_for_fn (fun));
+      bitmap_clear (to_visit);
+
+      FOR_EACH_BB_FN (bb, fun)
+	bitmap_set_bit (to_visit, bb->index);
+
+      /* Scan the blocks for stmts with escaping exceptions, that
+	 wouldn't be denoted in the CFG, and associate them with an
+	 empty cleanup handler around the whole function.  Walk
+	 backwards, so that even when we split the block, */
+      sbitmap_iterator it;
+      unsigned i;
+      EXECUTE_IF_SET_IN_BITMAP (to_visit, 0, i, it)
+	{
+	  bb = BASIC_BLOCK_FOR_FN (fun, i);
+
+	  for (gimple_stmt_iterator gsi = gsi_last_bb (bb);
+	       !gsi_end_p (gsi); gsi_prev (&gsi))
+	    {
+	      gimple *stmt = gsi_stmt (gsi);
+	      if (!stmt_could_throw_p (fun, stmt))
+		continue;
+
+	      /* If it must not throw, or if it already has a handler,
+		 we need not worry about it.  */
+	      if (lookup_stmt_eh_lp (stmt) != 0)
+		continue;
+
+	      /* Don't split blocks at, nor add EH edges to, tail
+		 calls, we will add verification before the call
+		 anyway.  */
+	      if (is_a <gcall *> (stmt)
+		  && (gimple_call_must_tail_p (as_a <gcall *> (stmt))
+		      || gimple_call_tail_p (as_a <gcall *> (stmt))
+		      || returning_call_p (as_a <gcall *> (stmt))))
+		continue;
+
+	      if (!gsi_one_before_end_p (gsi))
+		split_block (bb, stmt);
+	      /* A resx or noreturn call needs not be associated with
+		 the cleanup handler if we're going to add checking
+		 before it.  We only test cases that didn't require
+		 block splitting because noreturn calls would always
+		 be at the end of blocks, and we test for zero
+		 successors because if there is an edge, it's not
+		 noreturn, as any EH edges would have already been
+		 caught by the lookup_stmt_eh_lp test above.  */
+	      else if (check_before_noreturn_calls
+		       && EDGE_COUNT (bb->succs) == 0
+		       && (is_a <gresx *> (stmt)
+			   ? check_before_always_throwing_noreturn_calls
+			   : (!is_a <gcall *> (stmt)
+			      || !gimple_call_noreturn_p (stmt))
+			   ? (gcc_unreachable (), false)
+			   : (!flag_exceptions
+			      || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+			   ? check_before_nothrow_noreturn_calls
+			   : always_throwing_noreturn_call_p (stmt)
+			   ? check_before_always_throwing_noreturn_calls
+			   : check_before_throwing_noreturn_calls))
+		{
+		  if (dump_file)
+		    {
+		      fprintf (dump_file,
+			       "Bypassing cleanup for noreturn stmt"
+			       " in block %i:\n",
+			       bb->index);
+		      print_gimple_stmt (dump_file, stmt, 0);
+		    }
+		  continue;
+		}
+
+	      if (!bb_eh_cleanup)
+		{
+		  bb_eh_cleanup = create_empty_bb (bb);
+		  if (dom_info_available_p (CDI_DOMINATORS))
+		    set_immediate_dominator (CDI_DOMINATORS, bb_eh_cleanup, bb);
+		  if (current_loops)
+		    add_bb_to_loop (bb_eh_cleanup, current_loops->tree_root);
+
+		  /* Make the new block an EH cleanup for the call.  */
+		  eh_region new_r = gen_eh_region_cleanup (NULL);
+		  eh_landing_pad lp = gen_eh_landing_pad (new_r);
+		  tree label = gimple_block_label (bb_eh_cleanup);
+		  lp->post_landing_pad = label;
+		  EH_LANDING_PAD_NR (label) = lp_eh_cleanup = lp->index;
+
+		  /* Just propagate the exception.
+		     We will later insert the verifier call.  */
+		  gimple_stmt_iterator ehgsi;
+		  ehgsi = gsi_after_labels (bb_eh_cleanup);
+		  gresx *resx = gimple_build_resx (new_r->index);
+		  gsi_insert_before (&ehgsi, resx, GSI_SAME_STMT);
+
+		  if (dump_file)
+		    fprintf (dump_file,
+			     "Created cleanup block %i:\n",
+			     bb_eh_cleanup->index);
+		}
+	      else if (dom_info_available_p (CDI_DOMINATORS))
+		{
+		  basic_block immdom;
+		  immdom = get_immediate_dominator (CDI_DOMINATORS,
+						    bb_eh_cleanup);
+		  if (!dominated_by_p (CDI_DOMINATORS, bb, immdom))
+		    {
+		      immdom = nearest_common_dominator (CDI_DOMINATORS,
+							 immdom, bb);
+		      set_immediate_dominator (CDI_DOMINATORS,
+					       bb_eh_cleanup, immdom);
+		    }
+		}
+
+	      if (dump_file)
+		{
+		  fprintf (dump_file,
+			   "Associated cleanup block with stmt in block %i:\n",
+			   bb->index);
+		  print_gimple_stmt (dump_file, stmt, 0);
+		}
+
+	      add_stmt_to_eh_lp (stmt, lp_eh_cleanup);
+	      /* Finally, wire the EH cleanup block into the CFG.  */
+	      make_eh_edges (stmt);
+	    }
+	}
+
+      if (bb_eh_cleanup)
+	{
+	  /* A cfg_cleanup after bb_eh_cleanup makes for a more compact
+	     rtcfg, and it avoids bb numbering differences when we split
+	     blocks because of trailing debug insns only.  */
+	  cleanup_tree_cfg ();
+	  gcc_checking_assert (EDGE_COUNT (bb_eh_cleanup->succs) == 0);
+	}
+    }
+
+  /* These record blocks with calls that are to be preceded by
+     checkpoints, such as noreturn calls (if so chosen), must-tail
+     calls, potential early-marked tail calls, and returning calls (if
+     so chosen).  */
+  int count_chkcall = 0;
+  auto_sbitmap chkcall_blocks (last_basic_block_for_fn (fun));
+  bitmap_clear (chkcall_blocks);
+
+  /* We wish to add verification at blocks without successors, such as
+     noreturn calls (raising or not) and the reraise at the cleanup
+     block, but not other reraises: they will go through the cleanup
+     block.  */
+  if (check_before_noreturn_calls)
+    FOR_EACH_BB_FN (bb, fun)
+      {
+	gimple_stmt_iterator gsi = gsi_last_bb (bb);
+	if (gsi_end_p (gsi))
+	  continue;
+	gimple *stmt = gsi_stmt (gsi);
 
-  vstd.check ();
+	if (EDGE_COUNT (bb->succs) == 0)
+	  {
+	    /* A stmt at the end of a block without any successors is
+	       either a resx or a noreturn call without a local
+	       handler.  Check that it's one of the desired
+	       checkpoints.  */
+	    if (flag_exceptions && is_a <gresx *> (stmt)
+		? (check_before_always_throwing_noreturn_calls
+		   || bb == bb_eh_cleanup)
+		: (!is_a <gcall *> (stmt)
+		   || !gimple_call_noreturn_p (stmt))
+		? (/* Catch cases in which successors would be
+		      expected.  */
+		   gcc_unreachable (), false)
+		: (!flag_exceptions
+		   || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+		? check_before_nothrow_noreturn_calls
+		: always_throwing_noreturn_call_p (stmt)
+		? check_before_always_throwing_noreturn_calls
+		: check_before_throwing_noreturn_calls)
+	      {
+		if (dump_file)
+		  {
+		    fprintf (dump_file,
+			     "Scheduling check before stmt"
+			     " in succ-less block %i:\n",
+			     bb->index);
+		    print_gimple_stmt (dump_file, stmt, 0);
+		  }
+
+		if (bitmap_set_bit (chkcall_blocks, bb->index))
+		  count_chkcall++;
+		else
+		  gcc_unreachable ();
+	      }
+	    continue;
+	  }
 
-  return 0;
+	/* If there are no exceptions, then any noreturn call must have
+	   zero successor edges.  Otherwise, check for blocks without
+	   non-EH successors, but skip those with resx stmts and edges
+	   (i.e., those other than that in bb_eh_cleanup), since those
+	   will go through bb_eh_cleanup, that will have been counted as
+	   noreturn above because it has no successors.  */
+	gcc_checking_assert (bb != bb_eh_cleanup
+			     || !check_at_escaping_exceptions);
+	if (flag_exceptions && is_a <gresx *> (stmt)
+	    ? check_before_always_throwing_noreturn_calls
+	    : (!is_a <gcall *> (stmt)
+	       || !gimple_call_noreturn_p (stmt))
+	    ? false
+	    : (!flag_exceptions
+	       || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+	    ? (/* Catch cases that should not have successors.  */
+	       gcc_unreachable (), check_before_nothrow_noreturn_calls)
+	    : always_throwing_noreturn_call_p (stmt)
+	    ? check_before_always_throwing_noreturn_calls
+	    : check_before_throwing_noreturn_calls)
+	  {
+	    gcc_checking_assert (single_succ_p (bb)
+				 && (single_succ_edge (bb)->flags & EDGE_EH));
+
+	    if (dump_file)
+	      {
+		fprintf (dump_file,
+			 "Scheduling check before stmt"
+			 " in EH-succ block %i:\n",
+			 bb->index);
+		print_gimple_stmt (dump_file, stmt, 0);
+	      }
+
+	    if (bitmap_set_bit (chkcall_blocks, bb->index))
+	      count_chkcall++;
+	    else
+	      gcc_unreachable ();
+	  }
+      }
+  else if (bb_eh_cleanup)
+    {
+      if (bitmap_set_bit (chkcall_blocks, bb_eh_cleanup->index))
+	count_chkcall++;
+      else
+	gcc_unreachable ();
+    }
+
+  gcc_checking_assert (!bb_eh_cleanup
+		       || bitmap_bit_p (chkcall_blocks, bb_eh_cleanup->index));
+
+  /* If we don't have edges to exit nor noreturn calls (including the
+     cleanup reraise), then we may skip instrumentation: that would
+     amount to a function that ends with an infinite loop.  */
+  if (!count_chkcall
+      && EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (fun)->preds) == 0)
+    {
+      if (dump_file)
+	fprintf (dump_file,
+		 "Disabling CFR, no exit paths to check\n");
+
+      return 0;
+    }
+
+  /* Search for must-tail calls, early-marked potential tail calls,
+     and, if requested, returning calls.  As we introduce early
+     checks, */
+  int count_postchk = 0;
+  auto_sbitmap postchk_blocks (last_basic_block_for_fn (fun));
+  bitmap_clear (postchk_blocks);
+  chk_edges_t chk_edges;
+  hardcfr_sibcall_search_preds (EXIT_BLOCK_PTR_FOR_FN (fun), chk_edges,
+				count_chkcall, chkcall_blocks,
+				count_postchk, postchk_blocks,
+				NULL);
+
+  rt_bb_visited vstd (chk_edges.length () + count_chkcall);
+
+  auto_sbitmap combined_blocks (last_basic_block_for_fn (fun));
+  bitmap_copy (combined_blocks, chkcall_blocks);
+  int i;
+  edge *e;
+  FOR_EACH_VEC_ELT (chk_edges, i, e)
+    if (!bitmap_set_bit (combined_blocks, (*e)->src->index))
+      gcc_unreachable ();
+
+  /* Visit blocks in index order, because building rtcfg depends on
+     that.  Blocks must be compact, which the cleanup_cfg requirement
+     ensures.  This would also enable FOR_EACH_BB_FN to be used to
+     iterate in index order, but bb_eh_cleanup block splits and
+     insertions changes that.  */
+  gcc_checking_assert (n_basic_blocks_for_fn (fun)
+		       == last_basic_block_for_fn (fun));
+  for (int i = NUM_FIXED_BLOCKS; i < n_basic_blocks_for_fn (fun); i++)
+    {
+      bb = BASIC_BLOCK_FOR_FN (fun, i);
+      gcc_checking_assert (bb->index == i);
+      vstd.visit (bb, bitmap_bit_p (combined_blocks, i),
+		  bitmap_bit_p (postchk_blocks, i));
+    }
+
+  vstd.check (chk_edges, count_chkcall, chkcall_blocks);
+
+  return
+    TODO_update_ssa
+    | TODO_cleanup_cfg
+    | TODO_verify_il;
 }
 
 /* Instantiate a hardcfr pass.  */
diff --git a/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c b/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c
new file mode 100644
index 00000000000..a6992eb9f8e
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -O0 -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we don't insert checking before noreturn calls.  -O0 is tested
+   separately because h is not found to be noreturn without optimization.  */
+
+#include "torture/harden-cfr-noret.c"
+
+/* No out-of-line checks.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* Only one inline check at the end of f and of h2.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c
new file mode 100644
index 00000000000..8bd2d13ac18
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c
@@ -0,0 +1,18 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we don't insert checking before noreturn calls.  -O0 is tested
+   separately because h is not found to be noreturn without optimization, which
+   affects codegen for h2, so h2 is omitted here at -O0.  */
+
+#if !__OPTIMIZE__
+# define OMIT_H2
+#endif
+
+#include "harden-cfr-noret.c"
+
+
+/* No out-of-line checks.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* Only one inline check at the end of f.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c
new file mode 100644
index 00000000000..a804a6cfe59
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fno-exceptions -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that -fno-exceptions makes for implicit nothrow in noreturn
+   handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#include "harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c
new file mode 100644
index 00000000000..f390cfdbc59
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checking before nothrow noreturn calls.  */
+
+#include "harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c
new file mode 100644
index 00000000000..fdd803109a4
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c
@@ -0,0 +1,38 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checking before all noreturn calls.  */
+
+#ifndef ATTR_NOTHROW_OPT /* Overridden in harden-cfr-noret-noexcept.  */
+#define ATTR_NOTHROW_OPT __attribute__ ((__nothrow__))
+#endif
+
+extern void __attribute__ ((__noreturn__)) ATTR_NOTHROW_OPT g (void);
+
+void f(int i) {
+  if (i)
+    /* Out-of-line checks here...  */
+    g ();
+  /* ... and here.  */
+}
+
+void __attribute__ ((__noinline__, __noclone__))
+h(void) {
+  /* Inline check here.  */
+  g ();
+}
+
+#ifndef OMIT_H2 /* from harden-cfr-noret-never.  */
+void h2(void) {
+  /* Inline check either here, whether because of noreturn or tail call...  */
+  h ();
+  /* ... or here, if not optimizing.  */
+}
+#endif
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c
new file mode 100644
index 00000000000..6d11487bbba
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c
@@ -0,0 +1,8 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+#include "harden-cfr-tail.c"
+
+/* Inline checking after the calls, disabling tail calling.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 5 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 0 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c
new file mode 100644
index 00000000000..550b02ca088
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c
@@ -0,0 +1,35 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-exceptions -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checks before returning calls and alternate paths, even
+   at -O0, because of the explicit command-line flag.  */
+
+void g (void);
+void g2 (void);
+void g3 (void);
+
+void f (int i) {
+  if (!i)
+    /* Out-of-line checks here...  */
+    g ();
+  else if (i > 0)
+    /* here...  */
+    g2 ();
+  /* else */
+    /* and in the implicit else here.  */
+}
+
+void f2 (int i) {
+  if (!i)
+    /* Out-of-line check here...  */
+    g ();
+  else if (i > 0)
+    /* here...  */
+    g2 ();
+  else
+    /* and here.  */
+    g3 ();
+}
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 6 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 0 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
index 40d76c5c163..d5467eafa9f 100644
--- a/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
@@ -1,17 +1,52 @@
 /* { dg-do compile } */
-/* { dg-options "-fharden-control-flow-redundancy -fdump-tree-hardcfr -ffat-lto-objects" } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-hardcfr-check-exceptions -fdump-tree-hardcfr -ffat-lto-objects -Wno-return-type" } */
 
-/* We'd like to check that we insert checking so as to not disrupt tail calls,
-   but unfortunately mandatory tail calls are not available in C, and optimizing
-   calls as tail calls only takes place after hardcfr.  */
+/* Check that we insert CFR checking so as to not disrupt tail calls.
+   Mandatory tail calls are not available in C, and optimizing calls as tail
+   calls only takes place after hardcfr, so we insert checking before calls
+   followed by copies and return stmts with the same return value, that might
+   (or might not) end up optimized to tail calls.  */
 
-extern int g(int i);
+extern int g (int i);
 
-int f(int i) {
+int f1(int i) {
+  /* Inline check before the returning call.  */
   return g (i);
 }
 
-/* Inline checking before the tail call.  */
-/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
-/* Inline checking before the tail call.  */
-/* { dg-final { scan-tree-dump-times "\\\[tail\]" 1 "hardcfr" { xfail *-*-* } } } */
+extern void g2 (int i);
+
+void f2(int i) {
+  /* Inline check before the returning call, that ignores the returned value,
+     matching the value-less return.  */
+  g2 (i);
+  return;
+}
+
+void f3(int i) {
+  /* Inline check before the returning call.  */
+  g (i);
+}
+
+void f4(int i) {
+  if (i)
+    /* Out-of-line check before the returning call.  */
+    return g2 (i);
+  /* Out-of-line check before implicit return.  */
+}
+
+int f5(int i) {
+  /* Not regarded as a returning call, returning value other than callee's
+     returned value.  */
+  g (i);
+  /* Inline check after the non-returning call.  */
+  return i;
+}
+
+/* Out-of-line checks in f4, before returning calls and before return.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking in all other functions.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 4 "hardcfr" } } */
+/* Check before tail-call in all but f5, but f4 is out-of-line.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 3 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C b/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C
new file mode 100644
index 00000000000..17ea79f7cfb
--- /dev/null
+++ b/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C
@@ -0,0 +1,11 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects -O0" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions, and also checking before noreturn
+   calls.  h2 and h2b get an extra resx without ehcleanup.  */
+
+#include "torture/harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 16 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C b/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C
new file mode 100644
index 00000000000..f0338ccc361
--- /dev/null
+++ b/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C
@@ -0,0 +1,10 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -foptimize-sibling-calls -fdump-tree-hardcfr -O0" } */
+
+/* -fhardcfr-check-returning-calls gets implicitly disabled because,
+   -at O0, -foptimize-sibling-calls has no effect.  */
+
+#include "torture/harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 12 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C
new file mode 100644
index 00000000000..0d35920c7ee
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  */
+
+#include "harden-cfr-noret-no-nothrow.C"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Bypassing cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C
new file mode 100644
index 00000000000..b7d247ff43c
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C
@@ -0,0 +1,18 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  Expected results for =never and =nothrow are the same,
+   since the functions are not nothrow.  */
+
+#include "harden-cfr-noret-no-nothrow.C"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Associated cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C
new file mode 100644
index 00000000000..62c58cfd406
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C
@@ -0,0 +1,23 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#if ! __OPTIMIZE__
+void __attribute__ ((__noreturn__)) h (void);
+#endif
+
+#include "../../c-c++-common/torture/harden-cfr-noret.c"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Associated cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C
new file mode 100644
index 00000000000..0286f6e6d3f
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C
@@ -0,0 +1,20 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions, and also checking before noreturn
+   calls.  */
+
+#if ! __OPTIMIZE__
+/* Without optimization, functions with cleanups end up with an extra
+   resx that is not optimized out, so arrange to optimize them.  */
+void __attribute__ ((__optimize__ (1))) h2(void);
+void __attribute__ ((__optimize__ (1))) h2b(void);
+#endif
+
+#include "harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 14 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
+/* h, h2, h2b, and h4.  */
+/* { dg-final { scan-tree-dump-times "Bypassing" 4 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C
new file mode 100644
index 00000000000..885b0b236af
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C
@@ -0,0 +1,11 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions  -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we do not insert cleanups for checking around the bodies
+   of maybe-throwing functions.  h4 doesn't get any checks, because we
+   don't have noreturn checking enabled.  */
+
+#include "harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 6 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C
new file mode 100644
index 00000000000..32def637255
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C
@@ -0,0 +1,31 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -foptimize-sibling-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions.  These results depend on checking before
+   returning calls, which is only enabled when sibcall optimizations
+   are enabled, so change the optimization mode to -O1 for f and f2,
+   so that -foptimize-sibling-calls can take effect and enable
+   -fhardcfr-check-returning-calls, so that we get the same results.
+   There is a separate test for -O0.  */
+
+#if ! __OPTIMIZE__
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) f(int i);
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) f2(int i);
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) h3(void);
+#endif
+
+#include "harden-cfr-throw.C"
+
+/* f gets out-of-line checks before the unwrapped tail call and in the
+   else edge.  */
+/* f2 gets out-of-line checks before both unwrapped tail calls.  */
+/* h gets out-of-line checks before the implicit return and in the
+   cleanup block.  */
+/* h2 and h2b get out-of-line checks before the cleanup returning
+   call, and in the cleanup block.  */
+/* h3 gets an inline check before the __cxa_end_catch returning call.  */
+/* h4 gets an inline check in the cleanup block.  */
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 10 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C
new file mode 100644
index 00000000000..992fbdad381
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C
@@ -0,0 +1,65 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions.  */
+
+extern void g (void);
+extern void g2 (void);
+
+void f(int i) {
+  if (i)
+    g ();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void f2(int i) {
+  if (i)
+    g ();
+  else
+    g2 ();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h(void) {
+  try {
+    g ();
+  } catch (...) {
+    throw;
+  }
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+struct needs_cleanup {
+  ~needs_cleanup();
+};
+
+void h2(void) {
+  needs_cleanup y; /* No check in the cleanup handler.  */
+  g();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+extern void __attribute__ ((__nothrow__)) another_cleanup (void*);
+
+void h2b(void) {
+  int x __attribute__ ((cleanup (another_cleanup)));
+  g();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h3(void) {
+  try {
+    throw 1;
+  } catch (...) {
+  }
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h4(void) {
+  throw 1;
+  /* Inline check in the cleanup around the __cxa_throw noreturn call.  */
+}
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 12 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c b/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c
new file mode 100644
index 00000000000..8e4ee1fab08
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c
@@ -0,0 +1,15 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C makes for implicit nothrow in noreturn handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#include "../../c-c++-common/torture/harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c b/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c
new file mode 100644
index 00000000000..634d98f1ffc
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c
@@ -0,0 +1,40 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-hardcfr-check-exceptions -fdump-tree-hardcfr -ffat-lto-objects -Wno-return-type" } */
+
+/* In C only, check some additional cases (comparing with
+   c-c++-common/torture/harden-cfr-tail.c) of falling off the end of non-void
+   function.  C++ would issue an unreachable call in these cases.  */
+
+extern int g (int i);
+
+int f1(int i) {
+  /* Inline check before the returning call, that doesn't return anything.  */
+  g (i);
+  /* Implicit return without value, despite the return type; this combination
+     enables tail-calling of g, and is recognized as a returning call.  */
+}
+
+extern void g2 (int i);
+
+int f2(int i) {
+  /* Inline check before the returning call, that disregards its return
+     value.  */
+  g2 (i);
+  /* Implicit return without value, despite the return type; this combination
+     enables tail-calling of g2, and is recognized as a returning call.  */
+}
+
+int f3(int i) {
+  if (i)
+    /* Out-of-line check before the returning call.  */
+    return g (i);
+  /* Out-of-line check before implicit return.  */
+}
+
+/* Out-of-line checks in f3, before returning calls and before return.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking in all other functions.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
+/* Check before tail-call in all functions, but f3 is out-of-line.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/gnat.dg/hardcfr.adb b/gcc/testsuite/gnat.dg/hardcfr.adb
index b578a913d75..abe1605c029 100644
--- a/gcc/testsuite/gnat.dg/hardcfr.adb
+++ b/gcc/testsuite/gnat.dg/hardcfr.adb
@@ -1,5 +1,5 @@
 --  { dg-do run }
---  { dg-options "-fharden-control-flow-redundancy -fdump-tree-hardcfr --param=hardcfr-max-blocks=22 --param=hardcfr-max-inline-blocks=12 -O0" }
+--  { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions -fdump-tree-hardcfr --param=hardcfr-max-blocks=22 --param=hardcfr-max-inline-blocks=12 -O0" }
 
 procedure HardCFR is
    function F (I, J : Integer) return Integer is
diff --git a/libgcc/hardcfr.c b/libgcc/hardcfr.c
index 8ef29428111..55f6b995f2f 100644
--- a/libgcc/hardcfr.c
+++ b/libgcc/hardcfr.c
@@ -60,14 +60,25 @@ extern void __hardcfr_check (size_t blocks,
 			     vword const *visited,
 			     vword const *cfg);
 
+/* Compute the MASK for the bit representing BLOCK in WORDIDX's vword in a
+   visited blocks bit array.  */
+static inline void
+block2mask (size_t const block, vword *const mask, size_t *const wordidx)
+{
+  size_t wbits = __CHAR_BIT__ * sizeof (vword);
+  *wordidx = block / wbits;
+  *mask = (vword)1 << (block % wbits);
+}
 
 /* Check whether the bit corresponding to BLOCK is set in VISITED.  */
 static inline bool
 visited_p (size_t const block, vword const *const visited)
 {
-  size_t wbits = __CHAR_BIT__ * sizeof (vword);
-  vword w = visited[block / wbits];
-  return (w & ((vword)1 << (block % wbits))) != 0;
+  vword mask;
+  size_t wordidx;
+  block2mask (block, &mask, &wordidx);
+  vword w = visited[wordidx];
+  return (w & mask) != 0;
 }
 
 /* Read and consume a mask from **CFG_IT.  (Consume meaning advancing the
@@ -118,19 +129,19 @@ consume_seq (vword const **const cfg_it)
    we reach the terminator without finding any.  Consume the entire sequence
    otherwise, so that *CFG_IT points just past the terminator, which may be the
    beginning of the next sequence.  */
-static inline void
+static inline bool
 check_seq (vword const *const visited, vword const **const cfg_it)
 {
   vword mask;
   size_t wordidx;
 
   /* If the block was visited, check that at least one of the
-     preds was also visited.  */
+     preds/succs was also visited.  */
   do
     /* If we get to the end of the sequence without finding any
        match, something is amiss.  */
     if (!next_pair (cfg_it, &mask, &wordidx))
-      __builtin_trap ();
+      return false;
   /* Keep searching until we find a match, at which point the
      condition is satisfied.  */
   while (!test_mask (visited, mask, wordidx));
@@ -139,6 +150,94 @@ check_seq (vword const *const visited, vword const **const cfg_it)
      skipped the block, so as to position the iterator at the beginning of the
      next .  */
   consume_seq (cfg_it);
+
+  return true;
+}
+
+/* Print out the CFG with BLOCKS blocks, presumed to be associated with CALLER.
+   This is expected to be optimized out entirely, unless the verbose part of
+   __hardcfr_check_fail is enabled.  */
+static inline void
+__hardcfr_debug_cfg (size_t const blocks,
+		     void const *const caller,
+		     vword const *const cfg)
+{
+  __builtin_printf ("CFG at %p, for %p", cfg, caller);
+  vword const *cfg_it = cfg;
+  for (size_t i = 0; i < blocks; i++)
+    {
+      vword mask; size_t wordidx;
+      block2mask (i, &mask, &wordidx);
+      __builtin_printf ("\nblock %lu (%lu/0x%lx)\npreds: ",
+			(unsigned long)i,
+			(unsigned long)wordidx, (unsigned long)mask);
+      while (next_pair (&cfg_it, &mask, &wordidx))
+	__builtin_printf (" (%lu/0x%lx)",
+			  (unsigned long)wordidx, (unsigned long)mask);
+      __builtin_printf ("\nsuccs: ");
+      while (next_pair (&cfg_it, &mask, &wordidx))
+	__builtin_printf (" (%lu/0x%lx)",
+			  (unsigned long)wordidx, (unsigned long)mask);
+    }
+  __builtin_printf ("\n");
+}
+
+#ifndef ATTRIBUTE_UNUSED
+# define ATTRIBUTE_UNUSED __attribute__ ((__unused__))
+#endif
+
+/* This is called when an out-of-line hardcfr check fails.  All the arguments
+   are ignored, and it just traps, unless HARDCFR_VERBOSE_FAIL is enabled.  IF
+   it is, it prints the PART of the CFG, expected to have BLOCKS blocks, that
+   failed at CALLER's BLOCK, and the VISITED bitmap.  When the verbose mode is
+   enabled, it also forces __hardcfr_debug_cfg (above) to be compiled into an
+   out-of-line function, that could be called from a debugger.
+   */
+static inline void
+__hardcfr_check_fail (size_t const blocks ATTRIBUTE_UNUSED,
+		      vword const *const visited,
+		      vword const *const cfg ATTRIBUTE_UNUSED,
+		      size_t const block ATTRIBUTE_UNUSED,
+		      int const part ATTRIBUTE_UNUSED,
+		      void const *const caller ATTRIBUTE_UNUSED)
+{
+#if HARDCFR_VERBOSE_FAIL
+  static const char *parts[] = { "preds", "succs" };
+
+  vword mask; size_t wordidx;
+  block2mask (block, &mask, &wordidx);
+  __builtin_printf ("hardcfr fail at %p block %lu (%lu/0x%lx), expected %s:",
+		    caller, (unsigned long)block,
+		    (unsigned long)wordidx, (unsigned long)mask,
+		    parts[part]);
+
+  /* Skip data for previous blocks.  */
+  vword const *cfg_it = cfg;
+  for (size_t i = block; i--; )
+    {
+      consume_seq (&cfg_it);
+      consume_seq (&cfg_it);
+    }
+  for (size_t i = part; i--; )
+    consume_seq (&cfg_it);
+
+  while (next_pair (&cfg_it, &mask, &wordidx))
+    __builtin_printf (" (%lu/0x%lx)",
+		      (unsigned long)wordidx, (unsigned long)mask);
+
+  __builtin_printf ("\nvisited:");
+  block2mask (blocks, &mask, &wordidx);
+  for (size_t i = 0; i <= wordidx; i++)
+    __builtin_printf (" (%lu/0x%lx)",
+		      (unsigned long)i, (unsigned long)visited[i]);
+  __builtin_printf ("\n");
+
+  /* Reference __hardcfr_debug_cfg so that it's output out-of-line, so that it
+     can be called from a debugger.  */
+  if (!caller || caller == __hardcfr_debug_cfg)
+    return;
+#endif
+  __builtin_trap ();
 }
 
 /* Check that, for each of the BLOCKS basic blocks, if its bit is set in
@@ -168,9 +267,13 @@ __hardcfr_check (size_t const blocks,
       else
 	{
 	  /* Check predecessors.  */
-	  check_seq (visited, &cfg_it);
+	  if (!check_seq (visited, &cfg_it))
+	    __hardcfr_check_fail (blocks, visited, cfg, i, 0,
+				  __builtin_return_address (0));
 	  /* Check successors.  */
-	  check_seq (visited, &cfg_it);
+	  if (!check_seq (visited, &cfg_it))
+	    __hardcfr_check_fail (blocks, visited, cfg, i, 1,
+				  __builtin_return_address (0));
 	}
     }
 }

^ permalink raw reply	[flat|nested] 13+ messages in thread
* [gcc(refs/users/aoliva/heads/testme)] hardcfr: add optional checkpoints
@ 2022-09-09  7:08 Alexandre Oliva
  0 siblings, 0 replies; 13+ messages in thread
From: Alexandre Oliva @ 2022-09-09  7:08 UTC (permalink / raw)
  To: gcc-cvs

https://gcc.gnu.org/g:b9db93e1ebce7eb838355723eb872c1c84327f16

commit b9db93e1ebce7eb838355723eb872c1c84327f16
Author: Alexandre Oliva <oliva@adacore.com>
Date:   Fri Sep 9 01:44:31 2022 -0300

    hardcfr: add optional checkpoints
    
    Previously, control flow redundancy only checked the visited bitmap
    against the control flow graph at return points and before mandatory
    tail calls, missing various other possibilities of exiting a
    subprogram, such as by raising or propagating exceptions, and calling
    noreturn functions.  The checks inserted before returns also prevented
    potential tail-call optimizations.
    
    This incremental change introduces options to control checking at each
    of these previously-missed checkpoints.  Unless disabled, a cleanup is
    introduced to check when an exceptions escapes a subprogram.  To avoid
    disrupting sibcall optimizations, when they are enabled, checks are
    introduced before calls whose results are immediately returned,
    whether or not they are ultimately optimized.  If enabled, checks are
    introduced before noreturn calls and exception raises, or only before
    nothrow noreturn calls.
    
    Add examples of code transformations to the GNAT RM.
    
    
    for  gcc/ada/ChangeLog
    
            * doc/gnat_rm/security_hardening_features.rst: Document
            optional hardcfr checkpoints.
    
    for  gcc/ChangeLog
    
            * common.opt (-fhardcfr-check-returning-calls): New.
            (-fhardcfr-check-exceptions): New.
            (-fhardcfr-check-noreturn-calls=*): New.  (Enum
            hardcfr_check_noreturn_calls): New.  * doc/invoke.texi:
            Document them.  * flag-types.h (enum hardcfr_noret): New.  *
            gimple-harden-control-flow.cc: Include more headers.
            (pass_data_harden_control_flow_redundancy): Move
            properties_finish to pass return.
            (pass_harden_control_flow_redundancy::gate): Move
            no-edge-to-exit checking to execute, after exception
            transformations.  Use symbolic NUM_FIXED_BLOCKS.  Test for CFG
            before testing block count.  (check_returning_calls_p): New.
            (hardcfr_scan_block): New.  (returnign_call_p): New.
            (chk_edges_t): New.  (hardcfr_sibcall_search_block): New.
            (hardcfr_sibcall_search_preds): New.
            (rt_bb_visited::num2idx): Use symbolic NUM_FIXED_BLOCKS.
            (rt_bb_visited::rt_bb_visited): Likewise.  Take checkpoints
            count, test it to select out-of-line checks.
            (rt_bb_visited::insert_exit_check): Removed.
            (rt_bb_visited::insert_exit_check_in_blocks): New.
            (rt_bb_visited::insert_exit_check_on_edge): New.
            (rt_bb_visited::check): Take checkpoint edges and blocks.
            Move initialization insertion to the end.  Insert checks on
            edges, and before calls in select blocks.  Clobber the visited
            array to the check sequence.  Output actions to dump_file.
            (rt_bb_visited::visit): Take checkpoint and postcheck.  Assume
            an edge to exit if checkpoint, and skip testing if postcheck.
            (always_throwing_noreturn_call_p): New, dummy.
            (pass_harden_control_flow_redundancy::execute): Introduce
            cleanup block if requested and needed.  Scan blocks for
            noreturn and returning calls if checkpoints before them were
            requested.  Log actions to dump_file.  Visit blocks in index
            order.
    
    for  libgcc/ChangeLog
    
            * hardcfr.c (block2mask): Split out of...
            (visited_p): ... this.
            (check_seq): Move trapping out, return result instead.
            (__hardcfr_debug_cfg): New.
            (__hardcfr_check_fail): New, with optional verbose error
            printing before trapping.
            (__hardcfr_check): Call __hardcfr_check_fail when check_seq
            fails.
    
    for  gcc/testsuite/ChangeLog
    
            * c-c++-common/harden-cfr-noret-never-O0.c: New.
            * c-c++-common/torture/harden-cfr-noret-never.c: New.
            * c-c++-common/torture/harden-cfr-noret-noexcept.c: New.
            * c-c++-common/torture/harden-cfr-noret-nothrow.c: New.
            * c-c++-common/torture/harden-cfr-noret.c: New.
            * c-c++-common/torture/harden-cfr-notail.c: New.
            * c-c++-common/torture/harden-cfr-returning.c: New.
            * c-c++-common/torture/harden-cfr-tail.c: Extend.
            * g++.dg/harden-cfr-throw-always-O0.C: New.
            * g++.dg/harden-cfr-throw-returning-O0.C: New.
            * g++.dg/torture/harden-cfr-noret-always-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-noret-never-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-noret-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-throw-always.C: New.
            * g++.dg/torture/harden-cfr-throw-nocleanup.C: New.
            * g++.dg/torture/harden-cfr-throw-returning.C: New.
            * g++.dg/torture/harden-cfr-throw.C: New.
            * gcc.dg/torture/harden-cfr-noret-no-nothrow.c: New.
            * gcc.dg/torture/harden-cfr-tail-ub.c: New.
            * gnat.dg/hardcfr.adb: Disable checking at exceptions.

Diff:
---
 .../doc/gnat_rm/security_hardening_features.rst    | 126 ++-
 gcc/common.opt                                     |  33 +
 gcc/doc/invoke.texi                                |  74 +-
 gcc/flag-types.h                                   |  10 +
 gcc/gimple-harden-control-flow.cc                  | 916 ++++++++++++++++++---
 .../c-c++-common/harden-cfr-noret-never-O0.c       |  12 +
 .../c-c++-common/torture/harden-cfr-noret-never.c  |  18 +
 .../torture/harden-cfr-noret-noexcept.c            |  16 +
 .../torture/harden-cfr-noret-nothrow.c             |  13 +
 .../c-c++-common/torture/harden-cfr-noret.c        |  38 +
 .../c-c++-common/torture/harden-cfr-notail.c       |   8 +
 .../c-c++-common/torture/harden-cfr-returning.c    |  35 +
 .../c-c++-common/torture/harden-cfr-tail.c         |  55 +-
 gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C  |  11 +
 .../g++.dg/harden-cfr-throw-returning-O0.C         |  10 +
 .../torture/harden-cfr-noret-always-no-nothrow.C   |  16 +
 .../torture/harden-cfr-noret-never-no-nothrow.C    |  18 +
 .../g++.dg/torture/harden-cfr-noret-no-nothrow.C   |  23 +
 .../g++.dg/torture/harden-cfr-throw-always.C       |  20 +
 .../g++.dg/torture/harden-cfr-throw-nocleanup.C    |  11 +
 .../g++.dg/torture/harden-cfr-throw-returning.C    |  31 +
 gcc/testsuite/g++.dg/torture/harden-cfr-throw.C    |  65 ++
 .../gcc.dg/torture/harden-cfr-noret-no-nothrow.c   |  15 +
 gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c  |  40 +
 gcc/testsuite/gnat.dg/hardcfr.adb                  |   2 +-
 libgcc/hardcfr.c                                   | 119 ++-
 26 files changed, 1598 insertions(+), 137 deletions(-)

diff --git a/gcc/ada/doc/gnat_rm/security_hardening_features.rst b/gcc/ada/doc/gnat_rm/security_hardening_features.rst
index f5fdc8e46b4..4dfda486795 100644
--- a/gcc/ada/doc/gnat_rm/security_hardening_features.rst
+++ b/gcc/ada/doc/gnat_rm/security_hardening_features.rst
@@ -263,11 +263,127 @@ For each block that is marked as visited, the mechanism checks that at
 least one of its predecessors, and at least one of its successors, are
 also marked as visited.
 
-Verification is performed just before returning.  Subprogram
-executions that complete by raising or propagating an exception bypass
-verification-and-return points.  A subprogram that can only complete
-by raising or propagating an exception may have instrumentation
-disabled altogether.
+Verification is performed just before a subprogram returns.  The
+following fragment:
+
+.. code-block:: ada
+
+   if X then
+     Y := F (Z);
+     return;
+   end if;
+
+
+gets turned into:
+
+.. code-block:: ada
+
+   type Visited_Bitmap is array (1..N) of Boolean with Pack;
+   Visited : aliased Visited_Bitmap := (others => False);
+   --  Bitmap of visited blocks.  N is the basic block count.
+   [...]
+   --  Basic block #I
+   Visited(I) := True;
+   if X then
+     --  Basic block #J
+     Visited(J) := True;
+     Y := F (Z);
+     CFR.Check (N, Visited'Access, CFG'Access);
+     --  CFR is a hypothetical package whose Check procedure calls
+     --  libgcc's __hardcfr_check, that traps if the Visited bitmap
+     --  does not hold a valid path in CFG, the run-time
+     --  representation of the control flow graph in the enclosing
+     --  subprogram.
+     return;
+   end if;
+   --  Basic block #K
+   Visited(K) := True;
+
+
+Verification would also be performed before must-tail calls, and
+before early-marked potential tail calls, but these do not occur in
+practice, as potential tail-calls are only detected in late
+optimization phases, too late for this transformation to act on it.
+
+In order to avoid adding verification after potential tail calls,
+which would prevent tail-call optimization, we recognize returning
+calls, i.e., calls whose result, if any, is returned by the calling
+subprogram to its caller immediately after the call returns.
+Verification is performed before such calls, whether or not they are
+ultimately optimized to tail calls.  This behavior is enabled by
+default whenever sibcall optimization is enabled (see
+:switch:`-foptimize-sibling-calls`); it may be disabled with
+:switch:`-fno-hardcfr-check-returning-calls`, or enabled with
+:switch:`-fhardcfr-check-returning-calls`, regardless of the
+optimization, but the lack of other optimizations may prevent calls
+from being recognized as returning calls:
+
+.. code-block:: ada
+
+     --  CFR.Check here, with -fhardcfr-check-returning-calls.
+     P (X);
+     --  CFR.Check here, with -fno-hardcfr-check-returning-calls.
+     return;
+
+or:
+
+.. code-block:: ada
+
+     --  CFR.Check here, with -fhardcfr-check-returning-calls.
+     R := F (X);
+     --  CFR.Check here, with -fno-hardcfr-check-returning-calls.
+     return R;
+
+
+Any subprogram from which an exception may escape, i.e., that may
+raise or propagate an exception that isn't handled internally, is
+conceptually enclosed by a cleanup handler that performs verification,
+unless this is disabled with :switch:`-fno-hardcfr-check-exceptions`.
+With this feature enabled, a subprogram body containing:
+
+.. code-block:: ada
+
+     --  ...
+       Y := F (X);  -- May raise exceptions.
+     --  ...
+       raise E;  -- Not handled internally.
+     --  ...
+
+
+gets modified as follows:
+
+.. code-block:: ada
+
+   begin
+     --  ...
+       Y := F (X);  -- May raise exceptions.
+     --  ...
+       raise E;  -- Not handled internally.
+     --  ...
+   exception
+     when others =>
+       CFR.Check (N, Visited'Access, CFG'Access);
+       raise;
+   end;
+
+
+Verification may also be performed before No_Return calls, whether
+only nothrow ones, with
+:switch:`-fhardcfr-check-noreturn-calls=nothrow`, or all of them, with
+:switch:`-fhardcfr-check-noreturn-calls=always`.  The default is
+:switch:`-fhardcfr-check-noreturn-calls=never` for this feature, that
+disables checking before No_Return calls.
+
+When a No_Return call returns control to its caller through an
+exception, verification may have already been performed before the
+call, if :switch:`-fhardcfr-check-noreturn-calls=always` is in effect.
+The compiler arranges for already-checked No_Return calls without a
+preexisting handler to bypass the implicitly-added cleanup handler and
+thus the redundant check, but a local exception or cleanup handler, if
+present, will modify the set of visited blocks, and checking will take
+place again when the caller reaches the next verification point,
+whether it is a return or reraise statement after the exception is
+otherwise handled, or even another No_Return call.
 
 The instrumentation for hardening with control flow redundancy can be
 observed in dump files generated by the command-line option
diff --git a/gcc/common.opt b/gcc/common.opt
index 661dcf2f485..983cb4db7c6 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -1795,6 +1795,39 @@ fharden-control-flow-redundancy
 Common Var(flag_harden_control_flow_redundancy) Optimization
 Harden control flow by recording and checking execution paths.
 
+fhardcfr-check-returning-calls
+Common Var(flag_harden_control_flow_redundancy_check_returning_calls) Init(-1) Optimization
+Check CFR execution paths also before calls followed by returns of their results.
+
+fhardcfr-check-exceptions
+Common Var(flag_harden_control_flow_redundancy_check_exceptions) Init(-1) Optimization
+Check CFR execution paths also when exiting a function through an exception.
+
+fhardcfr-check-noreturn-calls=
+Common Joined RejectNegative Enum(hardcfr_check_noreturn_calls) Var(flag_harden_control_flow_redundancy_check_noreturn) Init(HCFRNR_UNSPECIFIED) Optimization
+-fhardcfr-check-noreturn-calls=[always|nothrow|never]	Check CFR execution paths also before calling noreturn functions.
+
+Enum
+Name(hardcfr_check_noreturn_calls) Type(enum hardcfr_noret) UnknownError(unknown hardcfr noreturn checking level %qs)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(never) Value(HCFRNR_NEVER)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(nothrow) Value(HCFRNR_NOTHROW)
+
+; ??? There could be yet another option here, that checked before
+; noreturn calls, except for those known to always throw, if we had
+; means to distinguish noreturn functions known to always throw, such
+; as those used to (re)raise exceptions, from those that merely might
+; throw.  "not always" stands for "not always-throwing", but it also
+; contrasts with "always" below.
+; EnumValue
+; Enum(hardcfr_check_noreturn_calls) String(not-always) Value(HCFRNR_NOT_ALWAYS)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(always) Value(HCFRNR_ALWAYS)
+
 ; Nonzero means ignore `#ident' directives.  0 means handle them.
 ; Generate position-independent code for executables if possible
 ; On SVR4 targets, it also controls whether or not to emit a
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index 0a7f32d93ea..f838d11bdb9 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -625,7 +625,9 @@ Objective-C and Objective-C++ Dialects}.
 -fsanitize-undefined-trap-on-error  -fbounds-check @gol
 -fcf-protection=@r{[}full@r{|}branch@r{|}return@r{|}none@r{|}check@r{]} @gol
 -fharden-compares -fharden-conditional-branches @gol
--fharden-control-flow-redundancy @gol
+-fharden-control-flow-redundancy  -fhardcfr-check-exceptions  @gol
+-fhardcfr-check-returning-calls  @gol
+-fhardcfr-check-noreturn-calls=@r{[}always@r{|}nothrow@r{|}never@r{]}  @gol
 -fstack-protector  -fstack-protector-all  -fstack-protector-strong @gol
 -fstack-protector-explicit  -fstack-check @gol
 -fstack-limit-register=@var{reg}  -fstack-limit-symbol=@var{sym} @gol
@@ -16574,11 +16576,75 @@ conditionals.
 @item -fharden-control-flow-redundancy
 @opindex fharden-control-flow-redundancy
 Emit extra code to set booleans when entering basic blocks, and to
-verify, at function exits, that they amount to an execution path that is
-consistent with the control flow graph, trapping otherwise.  Tuning
-options @option{--param hardcfr-max-blocks} and @option{--param
+verify and trap, at function exits, when the booleans do not form an
+execution path that is compatible with the control flow graph.
+
+Verification takes place before returns, before mandatory tail calls
+(see below) and, optionally, before escaping exceptions with
+@option{-fhardcfr-check-exceptions}, before returning calls with
+@option{-fhardcfr-check-returning-calls}, and before noreturn calls with
+@option{-fhardcfr-check-noreturn-calls}).  Tuning options
+@option{--param hardcfr-max-blocks} and @option{--param
 hardcfr-max-inline-blocks} are available.
 
+Tail call optimization takes place too late to affect control flow
+redundancy, but calls annotated as mandatory tail calls by language
+front-ends, and any calls marked early enough as potential tail calls
+would also have verification issued before the call, but these
+possibilities are merely theoretical, as these conditions can only be
+met when using custom compiler plugins.
+
+@item -fhardcfr-check-exceptions
+@opindex fhardcfr-check-exceptions
+@opindex fno-hardcfr-check-exceptions
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph at exception
+escape points, as if the function body was wrapped with a cleanup
+handler that performed the check and reraised.  This option is enabled
+by default; use @option{-fno-hardcfr-check-exceptions} to disable it.
+
+@item -fhardcfr-check-returning-calls
+@opindex fhardcfr-check-returning-calls
+@opindex fno-hardcfr-check-returning-calls
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph before any
+function call immediately followed by a return of its result, if any, so
+as to not prevent tail-call optimization, whether or not it is
+ultimately optimized to a tail call.
+
+This option is enabled by default whenever sibling call optimizations
+are enabled (see @option{-foptimize-sibling-calls}), but it can be
+enabled (or disabled, using its negated form) explicitly, regardless of
+the optimizations.
+
+@item -fhardcfr-check-noreturn-calls=@r{[}always@r{|}nothrow@r{|}never@r{]}
+@opindex fhardcfr-check-noreturn-calls
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph before
+@code{noreturn} calls, either all of them (@option{always}), those that
+may not return control to the caller through an exception either
+(@option{nothrow}), or none of them (@option{never}, the default).
+
+Checking before a @code{noreturn} function that may return control to
+the caller through an exception may cause checking to be performed more
+than once, if the exception is caught in the caller, whether by a
+handler or a cleanup.  When @option{-fhardcfr-check-exceptions} is also
+enabled, the compiler will avoid associating a @code{noreturn} call with
+the implicitly-added cleanup handler, since it would be redundant with
+the check performed before the call, but other handlers or cleanups in
+the function, if activated, will modify the recorded execution path and
+check it again when another checkpoint is hit.  The checkpoint may even
+be another @code{noreturn} call, so checking may end up performed
+multiple times.
+
+Various optimizers may cause calls to be marked as @code{noreturn}
+and/or @code{nothrow}, even in the absence of the corresponding
+attributes, which may affect the placement of checks before calls, as
+well as the addition of implicit cleanup handlers for them.  This
+unpredictability, and the fact that raising and reraising exceptions
+frequently amounts to implicitly calling @code{noreturn} functions, have
+made @option{never} the default setting for this option.
+
 @item -fstack-protector
 @opindex fstack-protector
 Emit extra code to check for buffer overflows, such as stack smashing
diff --git a/gcc/flag-types.h b/gcc/flag-types.h
index d2e751060ff..3fae7548cab 100644
--- a/gcc/flag-types.h
+++ b/gcc/flag-types.h
@@ -157,6 +157,16 @@ enum stack_reuse_level
   SR_ALL
 };
 
+/* Control Flow Redundancy hardening options for noreturn calls.  */
+enum hardcfr_noret
+{
+  HCFRNR_NEVER,
+  HCFRNR_NOTHROW,
+  HCFRNR_NOT_ALWAYS, /* Reserved for future use.  */
+  HCFRNR_ALWAYS,
+  HCFRNR_UNSPECIFIED = -1
+};
+
 /* The live patching level.  */
 enum live_patching_level
 {
diff --git a/gcc/gimple-harden-control-flow.cc b/gcc/gimple-harden-control-flow.cc
index 6b08846dbb1..1c93bf622e8 100644
--- a/gcc/gimple-harden-control-flow.cc
+++ b/gcc/gimple-harden-control-flow.cc
@@ -29,7 +29,12 @@ along with GCC; see the file COPYING3.  If not see
 #include "tree-pass.h"
 #include "ssa.h"
 #include "gimple-iterator.h"
+#include "gimple-pretty-print.h"
 #include "tree-cfg.h"
+#include "tree-cfgcleanup.h"
+#include "tree-eh.h"
+#include "except.h"
+#include "sbitmap.h"
 #include "basic-block.h"
 #include "cfghooks.h"
 #include "cfgloop.h"
@@ -60,9 +65,7 @@ const pass_data pass_data_harden_control_flow_redundancy = {
   0,	    // properties_provided
   0,	    // properties_destroyed
   TODO_cleanup_cfg, // properties_start
-  TODO_update_ssa
-  | TODO_cleanup_cfg
-  | TODO_verify_il, // properties_finish
+  0,        // properties_finish
 };
 
 class pass_harden_control_flow_redundancy : public gimple_opt_pass
@@ -79,16 +82,6 @@ public:
     if (!flag_harden_control_flow_redundancy)
       return false;
 
-    /* We don't verify when an exception escapes, propagated or raised
-       by the function itself, so we're only concerned with edges to
-       the exit block.  If there aren't any, the function doesn't
-       return normally, so there won't be any checking point, so
-       there's no point in running the pass.  Should we add
-       verification at exception escapes, we should at least look at
-       !flag_exceptions here.  */
-    if (EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (fun)->preds) == 0)
-      return false;
-
     /* Functions that return more than once, like setjmp and vfork
        (that also gets this flag set), will start recording a path
        after the first return, and then may take another path when
@@ -117,8 +110,9 @@ public:
 	return false;
       }
 
-    if (param_hardcfr_max_blocks > 0
-	&& n_basic_blocks_for_fn (fun) - 2 > param_hardcfr_max_blocks)
+    if (fun->cfg && param_hardcfr_max_blocks > 0
+	&& (n_basic_blocks_for_fn (fun) - NUM_FIXED_BLOCKS
+	    > param_hardcfr_max_blocks))
       {
 	warning_at (DECL_SOURCE_LOCATION (fun->decl), 0,
 		    "%qD has more than %u blocks, the requested"
@@ -134,6 +128,293 @@ public:
 
 }
 
+/* Return TRUE iff CFR checks should be inserted before returning
+   calls.  */
+
+static bool
+check_returning_calls_p ()
+{
+  return
+    flag_harden_control_flow_redundancy_check_returning_calls > 0
+    || (flag_harden_control_flow_redundancy_check_returning_calls < 0
+	/* Gates pass_tail_calls.  */
+	&& flag_optimize_sibling_calls
+	/* Gates pass_all_ptimizations.  */
+	&& optimize >= 1 && !optimize_debug);
+}
+
+/* Scan BB from the end, updating *RETPTR if given as return stmts and
+   copies are found.  Return a call or a stmt that cannot appear after
+   a tail call, or NULL if the top of the block is reached without
+   finding any.  */
+
+static gimple *
+hardcfr_scan_block (basic_block bb, tree **retptr)
+{
+  gimple_stmt_iterator gsi;
+  for (gsi = gsi_last_bb (bb); !gsi_end_p (gsi); gsi_prev (&gsi))
+    {
+      gimple *stmt = gsi_stmt (gsi);
+
+      /* Ignore labels, returns, nops, clobbers and debug stmts.  */
+      if (gimple_code (stmt) == GIMPLE_LABEL
+	  || gimple_code (stmt) == GIMPLE_NOP
+	  || gimple_code (stmt) == GIMPLE_PREDICT
+	  || gimple_clobber_p (stmt)
+	  || is_gimple_debug (stmt))
+	continue;
+
+      if (gimple_code (stmt) == GIMPLE_RETURN)
+	{
+	  greturn *gret = as_a <greturn *> (stmt);
+	  if (retptr)
+	    {
+	      gcc_checking_assert (!*retptr);
+	      *retptr = gimple_return_retval_ptr (gret);
+	    }
+	  continue;
+	}
+
+      /* Check for a call.  */
+      if (is_gimple_call (stmt))
+	return stmt;
+
+      /* Allow simple copies to the return value, updating the return
+	 value to be found in earlier assignments.  */
+      if (retptr && *retptr && gimple_assign_single_p (stmt)
+	  && **retptr == gimple_assign_lhs (stmt))
+	{
+	  *retptr = gimple_assign_rhs1_ptr (stmt);
+	  continue;
+	}
+
+      return stmt;
+    }
+
+  /* Any other kind of stmt will prevent a tail call.  */
+  return NULL;
+}
+
+/* Return TRUE iff CALL is to be preceded by a CFR checkpoint, i.e.,
+   if it's a returning call (one whose result is ultimately returned
+   without intervening non-copy statements) and we're checking
+   returning calls, a __builtin_return call (noreturn with a path to
+   the exit block), a must-tail call, or a tail call.  */
+
+static bool
+returning_call_p (gcall *call)
+{
+  if (!(gimple_call_noreturn_p (call)
+	|| gimple_call_must_tail_p (call)
+	|| gimple_call_tail_p (call)
+	|| check_returning_calls_p ()))
+    return false;
+
+  /* Quickly check that there's a path to exit compatible with a
+     returning call.  Detect infinite loops through the counter.  */
+  basic_block bb = gimple_bb (call);
+  auto_vec<basic_block, 10> path;
+  for (int i = n_basic_blocks_for_fn (cfun);
+       bb != EXIT_BLOCK_PTR_FOR_FN (cfun) && i--;
+       bb = single_succ (bb))
+    if (!single_succ_p (bb)
+	|| (single_succ_edge (bb)->flags & EDGE_EH) != 0)
+      return false;
+    else
+      path.safe_push (bb);
+
+  /* Check the stmts in the blocks and trace the return value.  */
+  tree *retptr = NULL;
+  for (;;)
+    {
+      gcc_checking_assert (!path.is_empty ());
+      basic_block bb = path.pop ();
+      gimple *stop = hardcfr_scan_block (bb, &retptr);
+      if (stop)
+	{
+	  if (stop != call)
+	    return false;
+	  gcc_checking_assert (path.is_empty ());
+	  break;
+	}
+
+      gphi *retphi = NULL;
+      if (retptr && *retptr && TREE_CODE (*retptr) == SSA_NAME
+	  && !SSA_NAME_IS_DEFAULT_DEF (*retptr)
+	  && SSA_NAME_DEF_STMT (*retptr)
+	  && is_a <gphi *> (SSA_NAME_DEF_STMT (*retptr))
+	  && gimple_bb (SSA_NAME_DEF_STMT (*retptr)) == bb)
+	{
+	  retphi = as_a <gphi *> (SSA_NAME_DEF_STMT (*retptr));
+	  gcc_checking_assert (gimple_phi_result (retphi) == *retptr);
+	}
+      else
+	continue;
+
+      gcc_checking_assert (!path.is_empty ());
+      edge e = single_succ_edge (path.last ());
+      int i = EDGE_COUNT (bb->preds);
+      while (i--)
+	if (EDGE_PRED (bb, i) == e)
+	  break;
+      gcc_checking_assert (i >= 0);
+      retptr = gimple_phi_arg_def_ptr (retphi, i);
+    }
+
+  return (gimple_call_noreturn_p (call)
+	  || gimple_call_must_tail_p (call)
+	  || gimple_call_tail_p (call)
+	  || (gimple_call_lhs (call) == (retptr ? *retptr : NULL)
+	      && check_returning_calls_p ()));
+}
+
+typedef auto_vec<edge, 10> chk_edges_t;
+
+/* Declare for mutual recursion.  */
+static bool hardcfr_sibcall_search_preds (basic_block bb,
+					  chk_edges_t &chk_edges,
+					  int &count_chkcall,
+					  auto_sbitmap &chkcall_blocks,
+					  int &count_postchk,
+					  auto_sbitmap &postchk_blocks,
+					  tree *retptr);
+
+/* Search backwards from the end of BB for a mandatory or potential
+   sibcall.  Schedule the block to be handled sort-of like noreturn if
+   so.  Recurse to preds, with updated RETPTR, if the block only
+   contains stmts that may follow such a call, scheduling checking at
+   edges and marking blocks as post-check as needed.  Return true iff,
+   at the end of the block, a check will have already been
+   performed.  */
+
+static bool
+hardcfr_sibcall_search_block (basic_block bb,
+			      chk_edges_t &chk_edges,
+			      int &count_chkcall,
+			      auto_sbitmap &chkcall_blocks,
+			      int &count_postchk,
+			      auto_sbitmap &postchk_blocks,
+			      tree *retptr)
+{
+  /* Conditionals and internal exceptions rule out tail calls.  */
+  if (!single_succ_p (bb)
+      || (single_succ_edge (bb)->flags & EDGE_EH) != 0)
+    return false;
+
+  gimple *stmt = hardcfr_scan_block (bb, &retptr);
+  if (!stmt)
+    return hardcfr_sibcall_search_preds (bb, chk_edges,
+					 count_chkcall, chkcall_blocks,
+					 count_postchk, postchk_blocks,
+					 retptr);
+
+  if (!is_a <gcall *> (stmt))
+    return false;
+
+  /* Avoid disrupting mandatory or early-marked tail calls,
+     inserting the check before them.  This works for
+     must-tail calls, but tail calling as an optimization is
+     detected too late for us.
+
+     Also check for noreturn calls here.  Noreturn calls won't
+     normally have edges to exit, so they won't be found here,
+     but __builtin_return does, and we must check before
+     it, so handle it like a tail call.  */
+  gcall *call = as_a <gcall *> (stmt);
+  if (!(gimple_call_noreturn_p (call)
+	|| gimple_call_must_tail_p (call)
+	|| gimple_call_tail_p (call)
+	|| (gimple_call_lhs (call) == (retptr ? *retptr : NULL)
+	    && check_returning_calls_p ())))
+    return false;
+
+  gcc_checking_assert (returning_call_p (call));
+
+  /* We found a call that is to be preceded by checking.  */
+  if (bitmap_set_bit (chkcall_blocks, bb->index))
+    ++count_chkcall;
+  else
+    gcc_unreachable ();
+  return true;
+}
+
+
+/* Search preds of BB for a mandatory or potential sibcall or
+   returning call, and arrange for the blocks containing them to have
+   a check inserted before the call, like noreturn calls.  If any
+   preds are found to perform checking, schedule checks at the edges
+   of those that don't, and mark BB as postcheck..  */
+
+static bool
+hardcfr_sibcall_search_preds (basic_block bb,
+			      chk_edges_t &chk_edges,
+			      int &count_chkcall,
+			      auto_sbitmap &chkcall_blocks,
+			      int &count_postchk,
+			      auto_sbitmap &postchk_blocks,
+			      tree *retptr)
+{
+  /* For the exit block, we wish to force a check at every
+     predecessor, so pretend we've already found a pred that had
+     checking, so that we schedule checking at every one of its pred
+     edges.  */
+  bool first = bb->index >= NUM_FIXED_BLOCKS;
+  bool postchecked = true;
+
+  gphi *retphi = NULL;
+  if (retptr && *retptr && TREE_CODE (*retptr) == SSA_NAME
+      && !SSA_NAME_IS_DEFAULT_DEF (*retptr)
+      && SSA_NAME_DEF_STMT (*retptr)
+      && is_a <gphi *> (SSA_NAME_DEF_STMT (*retptr))
+      && gimple_bb (SSA_NAME_DEF_STMT (*retptr)) == bb)
+    {
+      retphi = as_a <gphi *> (SSA_NAME_DEF_STMT (*retptr));
+      gcc_checking_assert (gimple_phi_result (retphi) == *retptr);
+    }
+
+  for (int i = EDGE_COUNT (bb->preds); i--; first = false)
+    {
+      edge e = EDGE_PRED (bb, i);
+
+      bool checked
+	= hardcfr_sibcall_search_block (e->src, chk_edges,
+					count_chkcall, chkcall_blocks,
+					count_postchk, postchk_blocks,
+					!retphi ? retptr
+					: gimple_phi_arg_def_ptr (retphi, i));
+
+      if (first)
+	{
+	  postchecked = checked;
+	  continue;
+	}
+
+      /* When we first find a checked block, force a check at every
+	 other incoming edge we've already visited, and those we
+	 visit afterwards that don't have their own check, so that
+	 when we reach BB, the check has already been performed.  */
+      if (!postchecked && checked)
+	{
+	  for (int j = EDGE_COUNT (bb->preds); --j > i; )
+	    chk_edges.safe_push (EDGE_PRED (bb, j));
+	  postchecked = true;
+	}
+      if (postchecked && !checked)
+	chk_edges.safe_push (EDGE_PRED (bb, i));
+    }
+
+  if (postchecked && bb->index >= NUM_FIXED_BLOCKS)
+    {
+      if (bitmap_set_bit (postchk_blocks, bb->index))
+	count_postchk++;
+      else
+	gcc_unreachable ();
+    }
+
+  return postchecked;
+}
+
+
 class rt_bb_visited
 {
   /* Use a sufficiently wide unsigned type to hold basic block numbers.  */
@@ -176,8 +457,8 @@ class rt_bb_visited
      neither ENTRY nor EXIT, but maybe one-past-the-end, to compute
      the visited array length.  */
   blknum num2idx (blknum n) {
-    gcc_checking_assert (n >= 2 && n <= nblocks);
-    return (n - 2);
+    gcc_checking_assert (n >= NUM_FIXED_BLOCKS && n <= nblocks);
+    return (n - NUM_FIXED_BLOCKS);
   }
   /* Return the block vindex for BB, that must not be ENTRY or
      EXIT.  */
@@ -249,8 +530,7 @@ class rt_bb_visited
   }
 
   /* Set the bit corresponding to BB in VISITED.  Add to SEQ any
-     required gimple statements, and return SEQ, possibly
-     modified.  */
+     required gimple stmts, and return SEQ, possibly modified.  */
   gimple_seq vset (basic_block bb, gimple_seq seq = NULL)
   {
     tree bit, setme = vword (bb, &bit);
@@ -270,7 +550,7 @@ class rt_bb_visited
 
 public:
   /* Prepare to add control flow redundancy testing to CFUN.  */
-  rt_bb_visited ()
+  rt_bb_visited (int checkpoints)
     : nblocks (n_basic_blocks_for_fn (cfun)),
       vword_type (NULL), ckseq (NULL), rtcfg (NULL)
   {
@@ -353,8 +633,8 @@ public:
 					 NULL, NULL);
     gimple_seq_add_stmt (&ckseq, detach);
 
-    if (nblocks - 2 > blknum (param_hardcfr_max_inline_blocks)
-	|| !single_pred_p (EXIT_BLOCK_PTR_FOR_FN (cfun)))
+    if (nblocks - NUM_FIXED_BLOCKS > blknum (param_hardcfr_max_inline_blocks)
+	|| checkpoints > 1)
       {
 	/* Make sure vword_bits is wide enough for the representation
 	   of nblocks in rtcfg.  Compare with vword_bits << vword_bits,
@@ -379,65 +659,33 @@ public:
     gimple_seq_add_stmt (&ckseq, ckfail_init);
   }
 
-  /* Insert SEQ on E, or close enough (e.g., before a noreturn or tail
-     call at the end of E->src).  */
-  void insert_exit_check (gimple_seq seq, edge e)
+  /* Insert SEQ before a resx or a call in INSBB.  */
+  void insert_exit_check_in_block (gimple_seq seq, basic_block insbb)
   {
-    basic_block insbb = e->src;
-
-    /* If the returning block ends with a noreturn call, insert
-       checking before it.  This is particularly important for
-       __builtin_return.  Other noreturn calls won't have an edge to
-       the exit block, so they won't even be considered as exit paths.
-
-       Insert-on-edge inserts before other return stmts, but not
-       before calls, and if a single-block function had the check
-       sequence inserted after a noreturn call, it would be useless,
-       but checking would still flag it as malformed if block 2 has a
-       fallthru edge to the exit block.
-
-       Also avoid disrupting tail calls, inserting the check before
-       them.  This works for must-tail calls, but tail calling as an
-       optimization is detected too late for us.  */
     gimple_stmt_iterator gsi = gsi_last_bb (insbb);
-    gimple *ret = gsi_stmt (gsi);
-    if (ret && is_a <greturn *> (ret))
-      {
+
+    while (!gsi_end_p (gsi))
+      if (is_a <gresx *> (gsi_stmt (gsi))
+	  || is_a <gcall *> (gsi_stmt (gsi)))
+	break;
+      else
 	gsi_prev (&gsi);
-	if (!gsi_end_p (gsi))
-	  ret = gsi_stmt (gsi);
-      }
-    if (ret && is_a <gcall *> (ret)
-	&& (gimple_call_noreturn_p (ret)
-	    || gimple_call_must_tail_p (as_a <gcall *> (ret))
-	    || gimple_call_tail_p (as_a <gcall *> (ret))))
-      gsi_insert_seq_before (&gsi, seq, GSI_SAME_STMT);
-    else
-      gsi_insert_seq_on_edge_immediate (e, seq);
+
+    gsi_insert_seq_before (&gsi, seq, GSI_SAME_STMT);
+  }
+
+  /* Insert SEQ on E.  */
+  void insert_exit_check_on_edge (gimple_seq seq, edge e)
+  {
+    gsi_insert_seq_on_edge_immediate (e, seq);
   }
 
-  /* Add checking code on every exit edge, and initialization code on
+  /* Add checking code to CHK_EDGES, and initialization code on
      the entry edge.  Before this point, the CFG has been undisturbed,
      and all the needed data has been collected and safely stowed.  */
-  void check ()
+  void check (chk_edges_t &chk_edges,
+	      int count_chkcall, auto_sbitmap const &chkcall_blocks)
   {
-    /* Insert initializers for visited at the entry.  */
-    gimple_seq iseq = NULL;
-
-    gcall *vinit = gimple_build_call (builtin_decl_explicit
-				      (BUILT_IN_MEMSET), 3,
-				      build1 (ADDR_EXPR,
-					      build_pointer_type
-					      (TREE_TYPE (visited)),
-					      visited),
-				      integer_zero_node,
-				      TYPE_SIZE_UNIT (TREE_TYPE (visited)));
-    gimple_seq_add_stmt (&iseq, vinit);
-
-    gsi_insert_seq_on_edge_immediate (single_succ_edge
-				      (ENTRY_BLOCK_PTR_FOR_FN (cfun)),
-				      iseq);
-
     /* If we're using out-of-line checking, create and statically
        initialize the CFG checking representation, generate the
        checker call for the checking sequence, and insert it in all
@@ -498,28 +746,116 @@ public:
 						     rtcfg));
 	gimple_seq_add_stmt (&ckseq, call_chk);
 
+	gimple *clobber = gimple_build_assign (visited,
+					       build_clobber
+					       (TREE_TYPE (visited)));
+	gimple_seq_add_stmt (&ckseq, clobber);
+
 	/* If we have multiple exit edges, insert (copies of)
 	   ckseq in all of them.  */
-	for (int i = EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (cfun)->preds);
-	     i--; )
+	for (int i = chk_edges.length (); i--; )
 	  {
 	    gimple_seq seq = ckseq;
 	    /* Copy the sequence, unless we're dealing with the
 	       last edge (we're counting down to zero).  */
-	    if (i)
+	    if (i || count_chkcall)
+	      seq = gimple_seq_copy (seq);
+
+	    edge e = chk_edges[i];
+
+	    if (dump_file)
+	      {
+		if (e->dest == EXIT_BLOCK_PTR_FOR_FN (cfun))
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to exit.\n",
+			   e->src->index);
+		else
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to postcheck block %i.\n",
+			   e->src->index, e->dest->index);
+	      }
+
+	    insert_exit_check_on_edge (seq, e);
+
+	    gcc_checking_assert (!bitmap_bit_p (chkcall_blocks, e->src->index));
+	  }
+
+	sbitmap_iterator it;
+	unsigned i;
+	EXECUTE_IF_SET_IN_BITMAP (chkcall_blocks, 0, i, it)
+	  {
+	    basic_block bb = BASIC_BLOCK_FOR_FN (cfun, i);
+
+	    gimple_seq seq = ckseq;
+	    gcc_checking_assert (count_chkcall > 0);
+	    if (--count_chkcall)
 	      seq = gimple_seq_copy (seq);
 
-	    insert_exit_check (seq,
-			       EDGE_PRED (EXIT_BLOCK_PTR_FOR_FN (cfun), i));
+	    if (dump_file)
+	      fprintf (dump_file,
+		       "Inserting out-of-line check before stmt in block %i.\n",
+		       bb->index);
+
+	    insert_exit_check_in_block (seq, bb);
 	  }
+
+	gcc_checking_assert (count_chkcall == 0);
       }
     else
       {
 	/* Inline checking requires a single exit edge.  */
-	gimple *last = gsi_stmt (gsi_last (ckseq));
+	gimple *last = gimple_build_assign (visited,
+					       build_clobber
+					       (TREE_TYPE (visited)));
+	gimple_seq_add_stmt (&ckseq, last);
 
-	insert_exit_check (ckseq,
-			   single_pred_edge (EXIT_BLOCK_PTR_FOR_FN (cfun)));
+	if (!count_chkcall)
+	  {
+	    edge e = single_pred_edge (EXIT_BLOCK_PTR_FOR_FN (cfun));
+
+	    if (dump_file)
+	      {
+		if (e->dest == EXIT_BLOCK_PTR_FOR_FN (cfun))
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to postcheck block %i.\n",
+			   e->src->index, e->dest->index);
+		else
+		  fprintf (dump_file,
+			   "Inserting inline check in"
+			   " block %i's edge to exit.\n",
+			   e->src->index);
+	      }
+
+	    insert_exit_check_on_edge (ckseq, e);
+	  }
+	else
+	  {
+	    gcc_checking_assert (count_chkcall == 1);
+
+	    sbitmap_iterator it;
+	    unsigned i;
+	    EXECUTE_IF_SET_IN_BITMAP (chkcall_blocks, 0, i, it)
+	      {
+		basic_block bb = BASIC_BLOCK_FOR_FN (cfun, i);
+
+		gimple_seq seq = ckseq;
+		gcc_checking_assert (count_chkcall > 0);
+		if (--count_chkcall)
+		  seq = gimple_seq_copy (seq);
+
+		if (dump_file)
+		  fprintf (dump_file,
+			   "Inserting inline check before stmt in block %i.\n",
+			   bb->index);
+
+		insert_exit_check_in_block (seq, bb);
+	      }
+
+	    gcc_checking_assert (count_chkcall == 0);
+	  }
 
 	/* The inserted ckseq computes CKFAIL at LAST.  Now we have to
 	   conditionally trap on it.  */
@@ -540,8 +876,7 @@ public:
 	  add_bb_to_loop (trp, current_loops->tree_root);
 
 	/* Insert a conditional branch to the trap block.  If the
-	   conditional wouldn't be the last statement, split the
-	   block.  */
+	   conditional wouldn't be the last stmt, split the block.  */
 	gimple_stmt_iterator gsi = gsi_for_stmt (last);
 	if (!gsi_one_before_end_p (gsi))
 	  split_block (gsi_bb (gsi), gsi_stmt (gsi));
@@ -564,6 +899,24 @@ public:
 	if (dom_info_available_p (CDI_DOMINATORS))
 	  set_immediate_dominator (CDI_DOMINATORS, trp, gimple_bb (last));
       }
+
+    /* Insert initializers for visited at the entry.  Do this after
+       other insertions, to avoid messing with block numbers.  */
+    gimple_seq iseq = NULL;
+
+    gcall *vinit = gimple_build_call (builtin_decl_explicit
+				      (BUILT_IN_MEMSET), 3,
+				      build1 (ADDR_EXPR,
+					      build_pointer_type
+					      (TREE_TYPE (visited)),
+					      visited),
+				      integer_zero_node,
+				      TYPE_SIZE_UNIT (TREE_TYPE (visited)));
+    gimple_seq_add_stmt (&iseq, vinit);
+
+    gsi_insert_seq_on_edge_immediate (single_succ_edge
+				      (ENTRY_BLOCK_PTR_FOR_FN (cfun)),
+				      iseq);
   }
 
   /* Push onto RTCFG a (mask, index) pair to test for IBB when BB is
@@ -607,7 +960,7 @@ public:
     return false;
   }
 
-  /* Add to CKSEQ statements to clear CKPART if OBB is visited.  */
+  /* Add to CKSEQ stmts to clear CKPART if OBB is visited.  */
   void
   build_block_check (basic_block obb)
   {
@@ -632,34 +985,48 @@ public:
 
   /* Add to BB code to set its bit in VISITED, and add to RTCFG or
      CKSEQ the data or code needed to check BB's predecessors and
-     successors.  Do NOT change the CFG.  */
-  void visit (basic_block bb)
+     successors.  If CHECKPOINT, assume the block is a checkpoint,
+     whether or not it has an edge to EXIT.  If POSTCHECK, assume the
+     block post-dominates checkpoints and therefore no bitmap setting
+     or checks are to be performed in or for it.  Do NOT change the
+     CFG.  */
+  void visit (basic_block bb, bool checkpoint, bool postcheck)
   {
     /* Set the bit in VISITED when entering the block.  */
     gimple_stmt_iterator gsi = gsi_after_labels (bb);
-    gsi_insert_seq_before (&gsi, vset (bb), GSI_SAME_STMT);
+    if (!postcheck)
+      gsi_insert_seq_before (&gsi, vset (bb), GSI_SAME_STMT);
 
     if (rtcfg)
       {
-	/* Build a list of (index, mask) terminated by (NULL, 0).
-	   Consolidate masks with the same index when they're
-	   adjacent.  First, predecessors.  Count backwards, because
-	   we're going to reverse the list.  The order shouldn't
-	   matter, but let's not make it surprising.  */
-	for (int i = EDGE_COUNT (bb->preds); i--; )
-	  if (push_rtcfg_pair (EDGE_PRED (bb, i)->src, bb,
-			       ENTRY_BLOCK_PTR_FOR_FN (cfun)))
-	    break;
+	if (!postcheck)
+	  {
+	    /* Build a list of (index, mask) terminated by (NULL, 0).
+	       Consolidate masks with the same index when they're
+	       adjacent.  First, predecessors.  Count backwards, because
+	       we're going to reverse the list.  The order shouldn't
+	       matter, but let's not make it surprising.  */
+	    for (int i = EDGE_COUNT (bb->preds); i--; )
+	      if (push_rtcfg_pair (EDGE_PRED (bb, i)->src, bb,
+				   ENTRY_BLOCK_PTR_FOR_FN (cfun)))
+		break;
+	  }
 	rtcfg = tree_cons (NULL_TREE, build_int_cst (vword_type, 0), rtcfg);
 
-	/* Then, successors.  */
-	for (int i = EDGE_COUNT (bb->succs); i--; )
-	  if (push_rtcfg_pair (EDGE_SUCC (bb, i)->dest, bb,
-			       EXIT_BLOCK_PTR_FOR_FN (cfun)))
-	    break;
+	if (!postcheck)
+	  {
+	    /* Then, successors.  */
+	    if (!checkpoint
+		|| !push_rtcfg_pair (EXIT_BLOCK_PTR_FOR_FN (cfun),
+				     bb, EXIT_BLOCK_PTR_FOR_FN (cfun)))
+	      for (int i = EDGE_COUNT (bb->succs); i--; )
+		if (push_rtcfg_pair (EDGE_SUCC (bb, i)->dest, bb,
+				     EXIT_BLOCK_PTR_FOR_FN (cfun)))
+		  break;
+	  }
 	rtcfg = tree_cons (NULL_TREE, build_int_cst (vword_type, 0), rtcfg);
       }
-    else
+    else if (!postcheck)
       {
 	/* Schedule test to fail if the block was reached but somehow none
 	   of its predecessors were.  */
@@ -677,6 +1044,8 @@ public:
 	gassign *blkruns = gimple_build_assign (ckpart, unshare_expr (bit));
 	gimple_seq_add_stmt (&ckseq, blkruns);
 
+	if (checkpoint)
+	  build_block_check (EXIT_BLOCK_PTR_FOR_FN (cfun));
 	for (int i = 0, e = EDGE_COUNT (bb->succs); i < e; i++)
 	  build_block_check (EDGE_SUCC (bb, i)->dest);
 
@@ -687,21 +1056,350 @@ public:
   }
 };
 
+/* It might be useful to avoid checking before noreturn calls that are
+   known to always finish by throwing an exception, rather than by
+   ending the program or looping forever.  Such functions would have
+   to be annotated somehow, with an attribute or flag.
+   Exception-raising functions, such as C++'s __cxa_throw,
+   __cxa_rethrow, and Ada's */
+static bool
+always_throwing_noreturn_call_p (gimple *)
+{
+  return false;
+}
+
 /* Control flow redundancy hardening: record the execution path, and
    verify at exit that an expect path was taken.  */
 
 unsigned int
-pass_harden_control_flow_redundancy::execute (function *)
+pass_harden_control_flow_redundancy::execute (function *fun)
 {
-  rt_bb_visited vstd;
-
+  bool const check_at_escaping_exceptions
+    = (flag_exceptions
+       && flag_harden_control_flow_redundancy_check_exceptions);
+  bool const check_before_noreturn_calls
+    = flag_harden_control_flow_redundancy_check_noreturn > HCFRNR_NEVER;
+  bool const check_before_nothrow_noreturn_calls
+    = (check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn >= HCFRNR_NOTHROW);
+  bool const check_before_throwing_noreturn_calls
+    = (flag_exceptions
+       && check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn > HCFRNR_NOTHROW);
+  bool const check_before_always_throwing_noreturn_calls
+    = (flag_exceptions
+       && check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn >= HCFRNR_ALWAYS);
   basic_block bb;
-  FOR_EACH_BB_FN (bb, cfun)
-    vstd.visit (bb);
+  basic_block bb_eh_cleanup = NULL;
+
+  if (check_at_escaping_exceptions)
+    {
+      int lp_eh_cleanup = -1;
+
+      /* Record the preexisting blocks, to avoid visiting newly-created
+	 blocks.  */
+      auto_sbitmap to_visit (last_basic_block_for_fn (fun));
+      bitmap_clear (to_visit);
+
+      FOR_EACH_BB_FN (bb, fun)
+	bitmap_set_bit (to_visit, bb->index);
+
+      /* Scan the blocks for stmts with escaping exceptions, that
+	 wouldn't be denoted in the CFG, and associate them with an
+	 empty cleanup handler around the whole function.  Walk
+	 backwards, so that even when we split the block, */
+      sbitmap_iterator it;
+      unsigned i;
+      EXECUTE_IF_SET_IN_BITMAP (to_visit, 0, i, it)
+	{
+	  bb = BASIC_BLOCK_FOR_FN (fun, i);
+
+	  for (gimple_stmt_iterator gsi = gsi_last_bb (bb);
+	       !gsi_end_p (gsi); gsi_prev (&gsi))
+	    {
+	      gimple *stmt = gsi_stmt (gsi);
+	      if (!stmt_could_throw_p (fun, stmt))
+		continue;
+
+	      /* If it must not throw, or if it already has a handler,
+		 we need not worry about it.  */
+	      if (lookup_stmt_eh_lp (stmt) != 0)
+		continue;
+
+	      /* Don't split blocks at, nor add EH edges to, tail
+		 calls, we will add verification before the call
+		 anyway.  */
+	      if (is_a <gcall *> (stmt)
+		  && (gimple_call_must_tail_p (as_a <gcall *> (stmt))
+		      || gimple_call_tail_p (as_a <gcall *> (stmt))
+		      || returning_call_p (as_a <gcall *> (stmt))))
+		continue;
+
+	      if (!gsi_one_before_end_p (gsi))
+		split_block (bb, stmt);
+	      /* A resx or noreturn call needs not be associated with
+		 the cleanup handler if we're going to add checking
+		 before it.  We only test cases that didn't require
+		 block splitting because noreturn calls would always
+		 be at the end of blocks, and we test for zero
+		 successors because if there is an edge, it's not
+		 noreturn, as any EH edges would have already been
+		 caught by the lookup_stmt_eh_lp test above.  */
+	      else if (check_before_noreturn_calls
+		       && EDGE_COUNT (bb->succs) == 0
+		       && (is_a <gresx *> (stmt)
+			   ? check_before_always_throwing_noreturn_calls
+			   : (!is_a <gcall *> (stmt)
+			      || !gimple_call_noreturn_p (stmt))
+			   ? (gcc_unreachable (), false)
+			   : (!flag_exceptions
+			      || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+			   ? check_before_nothrow_noreturn_calls
+			   : always_throwing_noreturn_call_p (stmt)
+			   ? check_before_always_throwing_noreturn_calls
+			   : check_before_throwing_noreturn_calls))
+		{
+		  if (dump_file)
+		    {
+		      fprintf (dump_file,
+			       "Bypassing cleanup for noreturn stmt"
+			       " in block %i:\n",
+			       bb->index);
+		      print_gimple_stmt (dump_file, stmt, 0);
+		    }
+		  continue;
+		}
+
+	      if (!bb_eh_cleanup)
+		{
+		  bb_eh_cleanup = create_empty_bb (bb);
+		  if (dom_info_available_p (CDI_DOMINATORS))
+		    set_immediate_dominator (CDI_DOMINATORS, bb_eh_cleanup, bb);
+		  if (current_loops)
+		    add_bb_to_loop (bb_eh_cleanup, current_loops->tree_root);
+
+		  /* Make the new block an EH cleanup for the call.  */
+		  eh_region new_r = gen_eh_region_cleanup (NULL);
+		  eh_landing_pad lp = gen_eh_landing_pad (new_r);
+		  tree label = gimple_block_label (bb_eh_cleanup);
+		  lp->post_landing_pad = label;
+		  EH_LANDING_PAD_NR (label) = lp_eh_cleanup = lp->index;
+
+		  /* Just propagate the exception.
+		     We will later insert the verifier call.  */
+		  gimple_stmt_iterator ehgsi;
+		  ehgsi = gsi_after_labels (bb_eh_cleanup);
+		  gresx *resx = gimple_build_resx (new_r->index);
+		  gsi_insert_before (&ehgsi, resx, GSI_SAME_STMT);
+
+		  if (dump_file)
+		    fprintf (dump_file,
+			     "Created cleanup block %i:\n",
+			     bb_eh_cleanup->index);
+		}
+	      else if (dom_info_available_p (CDI_DOMINATORS))
+		{
+		  basic_block immdom;
+		  immdom = get_immediate_dominator (CDI_DOMINATORS,
+						    bb_eh_cleanup);
+		  if (!dominated_by_p (CDI_DOMINATORS, bb, immdom))
+		    {
+		      immdom = nearest_common_dominator (CDI_DOMINATORS,
+							 immdom, bb);
+		      set_immediate_dominator (CDI_DOMINATORS,
+					       bb_eh_cleanup, immdom);
+		    }
+		}
+
+	      if (dump_file)
+		{
+		  fprintf (dump_file,
+			   "Associated cleanup block with stmt in block %i:\n",
+			   bb->index);
+		  print_gimple_stmt (dump_file, stmt, 0);
+		}
+
+	      add_stmt_to_eh_lp (stmt, lp_eh_cleanup);
+	      /* Finally, wire the EH cleanup block into the CFG.  */
+	      make_eh_edges (stmt);
+	    }
+	}
+
+      if (bb_eh_cleanup)
+	{
+	  /* A cfg_cleanup after bb_eh_cleanup makes for a more compact
+	     rtcfg, and it avoids bb numbering differences when we split
+	     blocks because of trailing debug insns only.  */
+	  cleanup_tree_cfg ();
+	  gcc_checking_assert (EDGE_COUNT (bb_eh_cleanup->succs) == 0);
+	}
+    }
+
+  /* These record blocks with calls that are to be preceded by
+     checkpoints, such as noreturn calls (if so chosen), must-tail
+     calls, potential early-marked tail calls, and returning calls (if
+     so chosen).  */
+  int count_chkcall = 0;
+  auto_sbitmap chkcall_blocks (last_basic_block_for_fn (fun));
+  bitmap_clear (chkcall_blocks);
+
+  /* We wish to add verification at blocks without successors, such as
+     noreturn calls (raising or not) and the reraise at the cleanup
+     block, but not other reraises: they will go through the cleanup
+     block.  */
+  if (check_before_noreturn_calls)
+    FOR_EACH_BB_FN (bb, fun)
+      {
+	gimple_stmt_iterator gsi = gsi_last_bb (bb);
+	if (gsi_end_p (gsi))
+	  continue;
+	gimple *stmt = gsi_stmt (gsi);
 
-  vstd.check ();
+	if (EDGE_COUNT (bb->succs) == 0)
+	  {
+	    /* A stmt at the end of a block without any successors is
+	       either a resx or a noreturn call without a local
+	       handler.  Check that it's one of the desired
+	       checkpoints.  */
+	    if (flag_exceptions && is_a <gresx *> (stmt)
+		? (check_before_always_throwing_noreturn_calls
+		   || bb == bb_eh_cleanup)
+		: (!is_a <gcall *> (stmt)
+		   || !gimple_call_noreturn_p (stmt))
+		? (/* Catch cases in which successors would be
+		      expected.  */
+		   gcc_unreachable (), false)
+		: (!flag_exceptions
+		   || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+		? check_before_nothrow_noreturn_calls
+		: always_throwing_noreturn_call_p (stmt)
+		? check_before_always_throwing_noreturn_calls
+		: check_before_throwing_noreturn_calls)
+	      {
+		if (dump_file)
+		  {
+		    fprintf (dump_file,
+			     "Scheduling check before stmt"
+			     " in succ-less block %i:\n",
+			     bb->index);
+		    print_gimple_stmt (dump_file, stmt, 0);
+		  }
+
+		if (bitmap_set_bit (chkcall_blocks, bb->index))
+		  count_chkcall++;
+		else
+		  gcc_unreachable ();
+	      }
+	    continue;
+	  }
 
-  return 0;
+	/* If there are no exceptions, then any noreturn call must have
+	   zero successor edges.  Otherwise, check for blocks without
+	   non-EH successors, but skip those with resx stmts and edges
+	   (i.e., those other than that in bb_eh_cleanup), since those
+	   will go through bb_eh_cleanup, that will have been counted as
+	   noreturn above because it has no successors.  */
+	gcc_checking_assert (bb != bb_eh_cleanup
+			     || !check_at_escaping_exceptions);
+	if (flag_exceptions && is_a <gresx *> (stmt)
+	    ? check_before_always_throwing_noreturn_calls
+	    : (!is_a <gcall *> (stmt)
+	       || !gimple_call_noreturn_p (stmt))
+	    ? false
+	    : (!flag_exceptions
+	       || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+	    ? (/* Catch cases that should not have successors.  */
+	       gcc_unreachable (), check_before_nothrow_noreturn_calls)
+	    : always_throwing_noreturn_call_p (stmt)
+	    ? check_before_always_throwing_noreturn_calls
+	    : check_before_throwing_noreturn_calls)
+	  {
+	    gcc_checking_assert (single_succ_p (bb)
+				 && (single_succ_edge (bb)->flags & EDGE_EH));
+
+	    if (dump_file)
+	      {
+		fprintf (dump_file,
+			 "Scheduling check before stmt"
+			 " in EH-succ block %i:\n",
+			 bb->index);
+		print_gimple_stmt (dump_file, stmt, 0);
+	      }
+
+	    if (bitmap_set_bit (chkcall_blocks, bb->index))
+	      count_chkcall++;
+	    else
+	      gcc_unreachable ();
+	  }
+      }
+  else if (bb_eh_cleanup)
+    {
+      if (bitmap_set_bit (chkcall_blocks, bb_eh_cleanup->index))
+	count_chkcall++;
+      else
+	gcc_unreachable ();
+    }
+
+  gcc_checking_assert (!bb_eh_cleanup
+		       || bitmap_bit_p (chkcall_blocks, bb_eh_cleanup->index));
+
+  /* If we don't have edges to exit nor noreturn calls (including the
+     cleanup reraise), then we may skip instrumentation: that would
+     amount to a function that ends with an infinite loop.  */
+  if (!count_chkcall
+      && EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (fun)->preds) == 0)
+    {
+      if (dump_file)
+	fprintf (dump_file,
+		 "Disabling CFR, no exit paths to check\n");
+
+      return 0;
+    }
+
+  /* Search for must-tail calls, early-marked potential tail calls,
+     and, if requested, returning calls.  As we introduce early
+     checks, */
+  int count_postchk = 0;
+  auto_sbitmap postchk_blocks (last_basic_block_for_fn (fun));
+  bitmap_clear (postchk_blocks);
+  chk_edges_t chk_edges;
+  hardcfr_sibcall_search_preds (EXIT_BLOCK_PTR_FOR_FN (fun), chk_edges,
+				count_chkcall, chkcall_blocks,
+				count_postchk, postchk_blocks,
+				NULL);
+
+  rt_bb_visited vstd (chk_edges.length () + count_chkcall);
+
+  auto_sbitmap combined_blocks (last_basic_block_for_fn (fun));
+  bitmap_copy (combined_blocks, chkcall_blocks);
+  int i;
+  edge *e;
+  FOR_EACH_VEC_ELT (chk_edges, i, e)
+    if (!bitmap_set_bit (combined_blocks, (*e)->src->index))
+      gcc_unreachable ();
+
+  /* Visit blocks in index order, because building rtcfg depends on
+     that.  Blocks must be compact, which the cleanup_cfg requirement
+     ensures.  This would also enable FOR_EACH_BB_FN to be used to
+     iterate in index order, but bb_eh_cleanup block splits and
+     insertions changes that.  */
+  gcc_checking_assert (n_basic_blocks_for_fn (fun)
+		       == last_basic_block_for_fn (fun));
+  for (int i = NUM_FIXED_BLOCKS; i < n_basic_blocks_for_fn (fun); i++)
+    {
+      bb = BASIC_BLOCK_FOR_FN (fun, i);
+      gcc_checking_assert (bb->index == i);
+      vstd.visit (bb, bitmap_bit_p (combined_blocks, i),
+		  bitmap_bit_p (postchk_blocks, i));
+    }
+
+  vstd.check (chk_edges, count_chkcall, chkcall_blocks);
+
+  return
+    TODO_update_ssa
+    | TODO_cleanup_cfg
+    | TODO_verify_il;
 }
 
 /* Instantiate a hardcfr pass.  */
diff --git a/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c b/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c
new file mode 100644
index 00000000000..a6992eb9f8e
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -O0 -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we don't insert checking before noreturn calls.  -O0 is tested
+   separately because h is not found to be noreturn without optimization.  */
+
+#include "torture/harden-cfr-noret.c"
+
+/* No out-of-line checks.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* Only one inline check at the end of f and of h2.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c
new file mode 100644
index 00000000000..8bd2d13ac18
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c
@@ -0,0 +1,18 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we don't insert checking before noreturn calls.  -O0 is tested
+   separately because h is not found to be noreturn without optimization, which
+   affects codegen for h2, so h2 is omitted here at -O0.  */
+
+#if !__OPTIMIZE__
+# define OMIT_H2
+#endif
+
+#include "harden-cfr-noret.c"
+
+
+/* No out-of-line checks.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* Only one inline check at the end of f.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c
new file mode 100644
index 00000000000..a804a6cfe59
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fno-exceptions -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that -fno-exceptions makes for implicit nothrow in noreturn
+   handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#include "harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c
new file mode 100644
index 00000000000..f390cfdbc59
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checking before nothrow noreturn calls.  */
+
+#include "harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c
new file mode 100644
index 00000000000..fdd803109a4
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c
@@ -0,0 +1,38 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checking before all noreturn calls.  */
+
+#ifndef ATTR_NOTHROW_OPT /* Overridden in harden-cfr-noret-noexcept.  */
+#define ATTR_NOTHROW_OPT __attribute__ ((__nothrow__))
+#endif
+
+extern void __attribute__ ((__noreturn__)) ATTR_NOTHROW_OPT g (void);
+
+void f(int i) {
+  if (i)
+    /* Out-of-line checks here...  */
+    g ();
+  /* ... and here.  */
+}
+
+void __attribute__ ((__noinline__, __noclone__))
+h(void) {
+  /* Inline check here.  */
+  g ();
+}
+
+#ifndef OMIT_H2 /* from harden-cfr-noret-never.  */
+void h2(void) {
+  /* Inline check either here, whether because of noreturn or tail call...  */
+  h ();
+  /* ... or here, if not optimizing.  */
+}
+#endif
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c
new file mode 100644
index 00000000000..6d11487bbba
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c
@@ -0,0 +1,8 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+#include "harden-cfr-tail.c"
+
+/* Inline checking after the calls, disabling tail calling.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 5 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 0 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c
new file mode 100644
index 00000000000..550b02ca088
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c
@@ -0,0 +1,35 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-exceptions -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checks before returning calls and alternate paths, even
+   at -O0, because of the explicit command-line flag.  */
+
+void g (void);
+void g2 (void);
+void g3 (void);
+
+void f (int i) {
+  if (!i)
+    /* Out-of-line checks here...  */
+    g ();
+  else if (i > 0)
+    /* here...  */
+    g2 ();
+  /* else */
+    /* and in the implicit else here.  */
+}
+
+void f2 (int i) {
+  if (!i)
+    /* Out-of-line check here...  */
+    g ();
+  else if (i > 0)
+    /* here...  */
+    g2 ();
+  else
+    /* and here.  */
+    g3 ();
+}
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 6 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 0 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
index 40d76c5c163..d5467eafa9f 100644
--- a/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
@@ -1,17 +1,52 @@
 /* { dg-do compile } */
-/* { dg-options "-fharden-control-flow-redundancy -fdump-tree-hardcfr -ffat-lto-objects" } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-hardcfr-check-exceptions -fdump-tree-hardcfr -ffat-lto-objects -Wno-return-type" } */
 
-/* We'd like to check that we insert checking so as to not disrupt tail calls,
-   but unfortunately mandatory tail calls are not available in C, and optimizing
-   calls as tail calls only takes place after hardcfr.  */
+/* Check that we insert CFR checking so as to not disrupt tail calls.
+   Mandatory tail calls are not available in C, and optimizing calls as tail
+   calls only takes place after hardcfr, so we insert checking before calls
+   followed by copies and return stmts with the same return value, that might
+   (or might not) end up optimized to tail calls.  */
 
-extern int g(int i);
+extern int g (int i);
 
-int f(int i) {
+int f1(int i) {
+  /* Inline check before the returning call.  */
   return g (i);
 }
 
-/* Inline checking before the tail call.  */
-/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
-/* Inline checking before the tail call.  */
-/* { dg-final { scan-tree-dump-times "\\\[tail\]" 1 "hardcfr" { xfail *-*-* } } } */
+extern void g2 (int i);
+
+void f2(int i) {
+  /* Inline check before the returning call, that ignores the returned value,
+     matching the value-less return.  */
+  g2 (i);
+  return;
+}
+
+void f3(int i) {
+  /* Inline check before the returning call.  */
+  g (i);
+}
+
+void f4(int i) {
+  if (i)
+    /* Out-of-line check before the returning call.  */
+    return g2 (i);
+  /* Out-of-line check before implicit return.  */
+}
+
+int f5(int i) {
+  /* Not regarded as a returning call, returning value other than callee's
+     returned value.  */
+  g (i);
+  /* Inline check after the non-returning call.  */
+  return i;
+}
+
+/* Out-of-line checks in f4, before returning calls and before return.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking in all other functions.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 4 "hardcfr" } } */
+/* Check before tail-call in all but f5, but f4 is out-of-line.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 3 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C b/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C
new file mode 100644
index 00000000000..17ea79f7cfb
--- /dev/null
+++ b/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C
@@ -0,0 +1,11 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects -O0" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions, and also checking before noreturn
+   calls.  h2 and h2b get an extra resx without ehcleanup.  */
+
+#include "torture/harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 16 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C b/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C
new file mode 100644
index 00000000000..f0338ccc361
--- /dev/null
+++ b/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C
@@ -0,0 +1,10 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -foptimize-sibling-calls -fdump-tree-hardcfr -O0" } */
+
+/* -fhardcfr-check-returning-calls gets implicitly disabled because,
+   -at O0, -foptimize-sibling-calls has no effect.  */
+
+#include "torture/harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 12 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C
new file mode 100644
index 00000000000..0d35920c7ee
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  */
+
+#include "harden-cfr-noret-no-nothrow.C"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Bypassing cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C
new file mode 100644
index 00000000000..b7d247ff43c
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C
@@ -0,0 +1,18 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  Expected results for =never and =nothrow are the same,
+   since the functions are not nothrow.  */
+
+#include "harden-cfr-noret-no-nothrow.C"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Associated cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C
new file mode 100644
index 00000000000..62c58cfd406
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C
@@ -0,0 +1,23 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#if ! __OPTIMIZE__
+void __attribute__ ((__noreturn__)) h (void);
+#endif
+
+#include "../../c-c++-common/torture/harden-cfr-noret.c"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Associated cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C
new file mode 100644
index 00000000000..0286f6e6d3f
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C
@@ -0,0 +1,20 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions, and also checking before noreturn
+   calls.  */
+
+#if ! __OPTIMIZE__
+/* Without optimization, functions with cleanups end up with an extra
+   resx that is not optimized out, so arrange to optimize them.  */
+void __attribute__ ((__optimize__ (1))) h2(void);
+void __attribute__ ((__optimize__ (1))) h2b(void);
+#endif
+
+#include "harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 14 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
+/* h, h2, h2b, and h4.  */
+/* { dg-final { scan-tree-dump-times "Bypassing" 4 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C
new file mode 100644
index 00000000000..885b0b236af
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C
@@ -0,0 +1,11 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions  -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we do not insert cleanups for checking around the bodies
+   of maybe-throwing functions.  h4 doesn't get any checks, because we
+   don't have noreturn checking enabled.  */
+
+#include "harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 6 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C
new file mode 100644
index 00000000000..32def637255
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C
@@ -0,0 +1,31 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -foptimize-sibling-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions.  These results depend on checking before
+   returning calls, which is only enabled when sibcall optimizations
+   are enabled, so change the optimization mode to -O1 for f and f2,
+   so that -foptimize-sibling-calls can take effect and enable
+   -fhardcfr-check-returning-calls, so that we get the same results.
+   There is a separate test for -O0.  */
+
+#if ! __OPTIMIZE__
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) f(int i);
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) f2(int i);
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) h3(void);
+#endif
+
+#include "harden-cfr-throw.C"
+
+/* f gets out-of-line checks before the unwrapped tail call and in the
+   else edge.  */
+/* f2 gets out-of-line checks before both unwrapped tail calls.  */
+/* h gets out-of-line checks before the implicit return and in the
+   cleanup block.  */
+/* h2 and h2b get out-of-line checks before the cleanup returning
+   call, and in the cleanup block.  */
+/* h3 gets an inline check before the __cxa_end_catch returning call.  */
+/* h4 gets an inline check in the cleanup block.  */
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 10 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C
new file mode 100644
index 00000000000..992fbdad381
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C
@@ -0,0 +1,65 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions.  */
+
+extern void g (void);
+extern void g2 (void);
+
+void f(int i) {
+  if (i)
+    g ();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void f2(int i) {
+  if (i)
+    g ();
+  else
+    g2 ();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h(void) {
+  try {
+    g ();
+  } catch (...) {
+    throw;
+  }
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+struct needs_cleanup {
+  ~needs_cleanup();
+};
+
+void h2(void) {
+  needs_cleanup y; /* No check in the cleanup handler.  */
+  g();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+extern void __attribute__ ((__nothrow__)) another_cleanup (void*);
+
+void h2b(void) {
+  int x __attribute__ ((cleanup (another_cleanup)));
+  g();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h3(void) {
+  try {
+    throw 1;
+  } catch (...) {
+  }
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h4(void) {
+  throw 1;
+  /* Inline check in the cleanup around the __cxa_throw noreturn call.  */
+}
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 12 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c b/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c
new file mode 100644
index 00000000000..8e4ee1fab08
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c
@@ -0,0 +1,15 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C makes for implicit nothrow in noreturn handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#include "../../c-c++-common/torture/harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c b/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c
new file mode 100644
index 00000000000..634d98f1ffc
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c
@@ -0,0 +1,40 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-hardcfr-check-exceptions -fdump-tree-hardcfr -ffat-lto-objects -Wno-return-type" } */
+
+/* In C only, check some additional cases (comparing with
+   c-c++-common/torture/harden-cfr-tail.c) of falling off the end of non-void
+   function.  C++ would issue an unreachable call in these cases.  */
+
+extern int g (int i);
+
+int f1(int i) {
+  /* Inline check before the returning call, that doesn't return anything.  */
+  g (i);
+  /* Implicit return without value, despite the return type; this combination
+     enables tail-calling of g, and is recognized as a returning call.  */
+}
+
+extern void g2 (int i);
+
+int f2(int i) {
+  /* Inline check before the returning call, that disregards its return
+     value.  */
+  g2 (i);
+  /* Implicit return without value, despite the return type; this combination
+     enables tail-calling of g2, and is recognized as a returning call.  */
+}
+
+int f3(int i) {
+  if (i)
+    /* Out-of-line check before the returning call.  */
+    return g (i);
+  /* Out-of-line check before implicit return.  */
+}
+
+/* Out-of-line checks in f3, before returning calls and before return.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking in all other functions.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
+/* Check before tail-call in all functions, but f3 is out-of-line.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/gnat.dg/hardcfr.adb b/gcc/testsuite/gnat.dg/hardcfr.adb
index b578a913d75..abe1605c029 100644
--- a/gcc/testsuite/gnat.dg/hardcfr.adb
+++ b/gcc/testsuite/gnat.dg/hardcfr.adb
@@ -1,5 +1,5 @@
 --  { dg-do run }
---  { dg-options "-fharden-control-flow-redundancy -fdump-tree-hardcfr --param=hardcfr-max-blocks=22 --param=hardcfr-max-inline-blocks=12 -O0" }
+--  { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions -fdump-tree-hardcfr --param=hardcfr-max-blocks=22 --param=hardcfr-max-inline-blocks=12 -O0" }
 
 procedure HardCFR is
    function F (I, J : Integer) return Integer is
diff --git a/libgcc/hardcfr.c b/libgcc/hardcfr.c
index 8ef29428111..55f6b995f2f 100644
--- a/libgcc/hardcfr.c
+++ b/libgcc/hardcfr.c
@@ -60,14 +60,25 @@ extern void __hardcfr_check (size_t blocks,
 			     vword const *visited,
 			     vword const *cfg);
 
+/* Compute the MASK for the bit representing BLOCK in WORDIDX's vword in a
+   visited blocks bit array.  */
+static inline void
+block2mask (size_t const block, vword *const mask, size_t *const wordidx)
+{
+  size_t wbits = __CHAR_BIT__ * sizeof (vword);
+  *wordidx = block / wbits;
+  *mask = (vword)1 << (block % wbits);
+}
 
 /* Check whether the bit corresponding to BLOCK is set in VISITED.  */
 static inline bool
 visited_p (size_t const block, vword const *const visited)
 {
-  size_t wbits = __CHAR_BIT__ * sizeof (vword);
-  vword w = visited[block / wbits];
-  return (w & ((vword)1 << (block % wbits))) != 0;
+  vword mask;
+  size_t wordidx;
+  block2mask (block, &mask, &wordidx);
+  vword w = visited[wordidx];
+  return (w & mask) != 0;
 }
 
 /* Read and consume a mask from **CFG_IT.  (Consume meaning advancing the
@@ -118,19 +129,19 @@ consume_seq (vword const **const cfg_it)
    we reach the terminator without finding any.  Consume the entire sequence
    otherwise, so that *CFG_IT points just past the terminator, which may be the
    beginning of the next sequence.  */
-static inline void
+static inline bool
 check_seq (vword const *const visited, vword const **const cfg_it)
 {
   vword mask;
   size_t wordidx;
 
   /* If the block was visited, check that at least one of the
-     preds was also visited.  */
+     preds/succs was also visited.  */
   do
     /* If we get to the end of the sequence without finding any
        match, something is amiss.  */
     if (!next_pair (cfg_it, &mask, &wordidx))
-      __builtin_trap ();
+      return false;
   /* Keep searching until we find a match, at which point the
      condition is satisfied.  */
   while (!test_mask (visited, mask, wordidx));
@@ -139,6 +150,94 @@ check_seq (vword const *const visited, vword const **const cfg_it)
      skipped the block, so as to position the iterator at the beginning of the
      next .  */
   consume_seq (cfg_it);
+
+  return true;
+}
+
+/* Print out the CFG with BLOCKS blocks, presumed to be associated with CALLER.
+   This is expected to be optimized out entirely, unless the verbose part of
+   __hardcfr_check_fail is enabled.  */
+static inline void
+__hardcfr_debug_cfg (size_t const blocks,
+		     void const *const caller,
+		     vword const *const cfg)
+{
+  __builtin_printf ("CFG at %p, for %p", cfg, caller);
+  vword const *cfg_it = cfg;
+  for (size_t i = 0; i < blocks; i++)
+    {
+      vword mask; size_t wordidx;
+      block2mask (i, &mask, &wordidx);
+      __builtin_printf ("\nblock %lu (%lu/0x%lx)\npreds: ",
+			(unsigned long)i,
+			(unsigned long)wordidx, (unsigned long)mask);
+      while (next_pair (&cfg_it, &mask, &wordidx))
+	__builtin_printf (" (%lu/0x%lx)",
+			  (unsigned long)wordidx, (unsigned long)mask);
+      __builtin_printf ("\nsuccs: ");
+      while (next_pair (&cfg_it, &mask, &wordidx))
+	__builtin_printf (" (%lu/0x%lx)",
+			  (unsigned long)wordidx, (unsigned long)mask);
+    }
+  __builtin_printf ("\n");
+}
+
+#ifndef ATTRIBUTE_UNUSED
+# define ATTRIBUTE_UNUSED __attribute__ ((__unused__))
+#endif
+
+/* This is called when an out-of-line hardcfr check fails.  All the arguments
+   are ignored, and it just traps, unless HARDCFR_VERBOSE_FAIL is enabled.  IF
+   it is, it prints the PART of the CFG, expected to have BLOCKS blocks, that
+   failed at CALLER's BLOCK, and the VISITED bitmap.  When the verbose mode is
+   enabled, it also forces __hardcfr_debug_cfg (above) to be compiled into an
+   out-of-line function, that could be called from a debugger.
+   */
+static inline void
+__hardcfr_check_fail (size_t const blocks ATTRIBUTE_UNUSED,
+		      vword const *const visited,
+		      vword const *const cfg ATTRIBUTE_UNUSED,
+		      size_t const block ATTRIBUTE_UNUSED,
+		      int const part ATTRIBUTE_UNUSED,
+		      void const *const caller ATTRIBUTE_UNUSED)
+{
+#if HARDCFR_VERBOSE_FAIL
+  static const char *parts[] = { "preds", "succs" };
+
+  vword mask; size_t wordidx;
+  block2mask (block, &mask, &wordidx);
+  __builtin_printf ("hardcfr fail at %p block %lu (%lu/0x%lx), expected %s:",
+		    caller, (unsigned long)block,
+		    (unsigned long)wordidx, (unsigned long)mask,
+		    parts[part]);
+
+  /* Skip data for previous blocks.  */
+  vword const *cfg_it = cfg;
+  for (size_t i = block; i--; )
+    {
+      consume_seq (&cfg_it);
+      consume_seq (&cfg_it);
+    }
+  for (size_t i = part; i--; )
+    consume_seq (&cfg_it);
+
+  while (next_pair (&cfg_it, &mask, &wordidx))
+    __builtin_printf (" (%lu/0x%lx)",
+		      (unsigned long)wordidx, (unsigned long)mask);
+
+  __builtin_printf ("\nvisited:");
+  block2mask (blocks, &mask, &wordidx);
+  for (size_t i = 0; i <= wordidx; i++)
+    __builtin_printf (" (%lu/0x%lx)",
+		      (unsigned long)i, (unsigned long)visited[i]);
+  __builtin_printf ("\n");
+
+  /* Reference __hardcfr_debug_cfg so that it's output out-of-line, so that it
+     can be called from a debugger.  */
+  if (!caller || caller == __hardcfr_debug_cfg)
+    return;
+#endif
+  __builtin_trap ();
 }
 
 /* Check that, for each of the BLOCKS basic blocks, if its bit is set in
@@ -168,9 +267,13 @@ __hardcfr_check (size_t const blocks,
       else
 	{
 	  /* Check predecessors.  */
-	  check_seq (visited, &cfg_it);
+	  if (!check_seq (visited, &cfg_it))
+	    __hardcfr_check_fail (blocks, visited, cfg, i, 0,
+				  __builtin_return_address (0));
 	  /* Check successors.  */
-	  check_seq (visited, &cfg_it);
+	  if (!check_seq (visited, &cfg_it))
+	    __hardcfr_check_fail (blocks, visited, cfg, i, 1,
+				  __builtin_return_address (0));
 	}
     }
 }

^ permalink raw reply	[flat|nested] 13+ messages in thread
* [gcc(refs/users/aoliva/heads/testme)] hardcfr: add optional checkpoints
@ 2022-09-07 23:51 Alexandre Oliva
  0 siblings, 0 replies; 13+ messages in thread
From: Alexandre Oliva @ 2022-09-07 23:51 UTC (permalink / raw)
  To: gcc-cvs

https://gcc.gnu.org/g:1d4013d83411d89bbd2a4aa95e84ce0cf3e1560f

commit 1d4013d83411d89bbd2a4aa95e84ce0cf3e1560f
Author: Alexandre Oliva <oliva@adacore.com>
Date:   Fri Sep 2 20:33:12 2022 -0300

    hardcfr: add optional checkpoints
    
    Previously, control flow redundancy only checked the visited bitmap
    against the control flow graph at return points and before mandatory
    tail calls, missing various other possibilities of exiting a
    subprogram, such as by raising or propagating exceptions, and calling
    noreturn functions.  The checks inserted before returns also prevented
    potential tail-call optimizations.
    
    This incremental change introduces options to control checking at each
    of these previously-missed checkpoints.  Unless disabled, a cleanup is
    introduced to check when an exceptions escapes a subprogram.  To avoid
    disrupting sibcall optimizations, when they are enabled, checks are
    introduced before calls whose results are immediately returned,
    whether or not they are ultimately optimized.  If enabled, checks are
    introduced before noreturn calls and exception raises, or only before
    nothrow noreturn calls.
    
    Add examples of code transformations to the GNAT RM.
    
    
    for  gcc/ada/ChangeLog
    
            * doc/gnat_rm/security_hardening_features.rst: Document
            optional hardcfr checkpoints.
    
    for  gcc/ChangeLog
    
            * common.opt (-fhardcfr-check-returning-calls): New.
            (-fhardcfr-check-exceptions): New.
            (-fhardcfr-check-noreturn-calls=*): New.  (Enum
            hardcfr_check_noreturn_calls): New.  * doc/invoke.texi:
            Document them.  * flag-types.h (enum hardcfr_noret): New.  *
            gimple-harden-control-flow.cc: Include more headers.
            (pass_data_harden_control_flow_redundancy): Move
            properties_finish to pass return.
            (pass_harden_control_flow_redundancy::gate): Move
            no-edge-to-exit checking to execute, after exception
            transformations.  Use symbolic NUM_FIXED_BLOCKS.  Test for CFG
            before testing block count.  (check_returning_calls_p): New.
            (hardcfr_scan_block): New.  (returnign_call_p): New.
            (chk_edges_t): New.  (hardcfr_sibcall_search_block): New.
            (hardcfr_sibcall_search_preds): New.
            (rt_bb_visited::num2idx): Use symbolic NUM_FIXED_BLOCKS.
            (rt_bb_visited::rt_bb_visited): Likewise.  Take checkpoints
            count, test it to select out-of-line checks.
            (rt_bb_visited::insert_exit_check): Removed.
            (rt_bb_visited::insert_exit_check_in_blocks): New.
            (rt_bb_visited::insert_exit_check_on_edge): New.
            (rt_bb_visited::check): Take checkpoint edges and blocks.
            Move initialization insertion to the end.  Insert checks on
            edges, and before calls in select blocks.  Clobber the visited
            array to the check sequence.  Output actions to dump_file.
            (rt_bb_visited::visit): Take checkpoint and postcheck.  Assume
            an edge to exit if checkpoint, and skip testing if postcheck.
            (always_throwing_noreturn_call_p): New, dummy.
            (pass_harden_control_flow_redundancy::execute): Introduce
            cleanup block if requested and needed.  Scan blocks for
            noreturn and returning calls if checkpoints before them were
            requested.  Log actions to dump_file.  Visit blocks in index
            order.
    
    for  libgcc/ChangeLog
    
            * hardcfr.c (block2mask): Split out of...
            (visited_p): ... this.
            (check_seq): Move trapping out, return result instead.
            (__hardcfr_debug_cfg): New.
            (__hardcfr_check_fail): New, with optional verbose error
            printing before trapping.
            (__hardcfr_check): Call __hardcfr_check_fail when check_seq
            fails.
    
    for  gcc/testsuite/ChangeLog
    
            * c-c++-common/harden-cfr-noret-never-O0.c: New.
            * c-c++-common/torture/harden-cfr-noret-never.c: New.
            * c-c++-common/torture/harden-cfr-noret-noexcept.c: New.
            * c-c++-common/torture/harden-cfr-noret-nothrow.c: New.
            * c-c++-common/torture/harden-cfr-noret.c: New.
            * c-c++-common/torture/harden-cfr-notail.c: New.
            * c-c++-common/torture/harden-cfr-tail.c: Extend.
            * g++.dg/harden-cfr-throw-always-O0.C: New.
            * g++.dg/harden-cfr-throw-returning-O0.C: New.
            * g++.dg/torture/harden-cfr-noret-always-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-noret-never-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-noret-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-throw-always.C: New.
            * g++.dg/torture/harden-cfr-throw-nocleanup.C: New.
            * g++.dg/torture/harden-cfr-throw-returning.C: New.
            * g++.dg/torture/harden-cfr-throw.C: New.
            * gcc.dg/torture/harden-cfr-noret-no-nothrow.c: New.
            * gcc.dg/torture/harden-cfr-tail-ub.c: New.
            * gnat.dg/hardcfr.adb: Disable checking at exceptions.

Diff:
---
 .../doc/gnat_rm/security_hardening_features.rst    | 126 ++-
 gcc/common.opt                                     |  33 +
 gcc/doc/invoke.texi                                |  74 +-
 gcc/flag-types.h                                   |  10 +
 gcc/gimple-harden-control-flow.cc                  | 916 ++++++++++++++++++---
 .../c-c++-common/harden-cfr-noret-never-O0.c       |  12 +
 .../c-c++-common/torture/harden-cfr-noret-never.c  |  18 +
 .../torture/harden-cfr-noret-noexcept.c            |  16 +
 .../torture/harden-cfr-noret-nothrow.c             |  13 +
 .../c-c++-common/torture/harden-cfr-noret.c        |  38 +
 .../c-c++-common/torture/harden-cfr-notail.c       |   8 +
 .../c-c++-common/torture/harden-cfr-returning.c    |  35 +
 .../c-c++-common/torture/harden-cfr-tail.c         |  55 +-
 gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C  |  11 +
 .../g++.dg/harden-cfr-throw-returning-O0.C         |  10 +
 .../torture/harden-cfr-noret-always-no-nothrow.C   |  16 +
 .../torture/harden-cfr-noret-never-no-nothrow.C    |  18 +
 .../g++.dg/torture/harden-cfr-noret-no-nothrow.C   |  23 +
 .../g++.dg/torture/harden-cfr-throw-always.C       |  20 +
 .../g++.dg/torture/harden-cfr-throw-nocleanup.C    |  11 +
 .../g++.dg/torture/harden-cfr-throw-returning.C    |  31 +
 gcc/testsuite/g++.dg/torture/harden-cfr-throw.C    |  65 ++
 .../gcc.dg/torture/harden-cfr-noret-no-nothrow.c   |  15 +
 gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c  |  40 +
 gcc/testsuite/gnat.dg/hardcfr.adb                  |   2 +-
 libgcc/hardcfr.c                                   | 119 ++-
 26 files changed, 1598 insertions(+), 137 deletions(-)

diff --git a/gcc/ada/doc/gnat_rm/security_hardening_features.rst b/gcc/ada/doc/gnat_rm/security_hardening_features.rst
index f5fdc8e46b4..4dfda486795 100644
--- a/gcc/ada/doc/gnat_rm/security_hardening_features.rst
+++ b/gcc/ada/doc/gnat_rm/security_hardening_features.rst
@@ -263,11 +263,127 @@ For each block that is marked as visited, the mechanism checks that at
 least one of its predecessors, and at least one of its successors, are
 also marked as visited.
 
-Verification is performed just before returning.  Subprogram
-executions that complete by raising or propagating an exception bypass
-verification-and-return points.  A subprogram that can only complete
-by raising or propagating an exception may have instrumentation
-disabled altogether.
+Verification is performed just before a subprogram returns.  The
+following fragment:
+
+.. code-block:: ada
+
+   if X then
+     Y := F (Z);
+     return;
+   end if;
+
+
+gets turned into:
+
+.. code-block:: ada
+
+   type Visited_Bitmap is array (1..N) of Boolean with Pack;
+   Visited : aliased Visited_Bitmap := (others => False);
+   --  Bitmap of visited blocks.  N is the basic block count.
+   [...]
+   --  Basic block #I
+   Visited(I) := True;
+   if X then
+     --  Basic block #J
+     Visited(J) := True;
+     Y := F (Z);
+     CFR.Check (N, Visited'Access, CFG'Access);
+     --  CFR is a hypothetical package whose Check procedure calls
+     --  libgcc's __hardcfr_check, that traps if the Visited bitmap
+     --  does not hold a valid path in CFG, the run-time
+     --  representation of the control flow graph in the enclosing
+     --  subprogram.
+     return;
+   end if;
+   --  Basic block #K
+   Visited(K) := True;
+
+
+Verification would also be performed before must-tail calls, and
+before early-marked potential tail calls, but these do not occur in
+practice, as potential tail-calls are only detected in late
+optimization phases, too late for this transformation to act on it.
+
+In order to avoid adding verification after potential tail calls,
+which would prevent tail-call optimization, we recognize returning
+calls, i.e., calls whose result, if any, is returned by the calling
+subprogram to its caller immediately after the call returns.
+Verification is performed before such calls, whether or not they are
+ultimately optimized to tail calls.  This behavior is enabled by
+default whenever sibcall optimization is enabled (see
+:switch:`-foptimize-sibling-calls`); it may be disabled with
+:switch:`-fno-hardcfr-check-returning-calls`, or enabled with
+:switch:`-fhardcfr-check-returning-calls`, regardless of the
+optimization, but the lack of other optimizations may prevent calls
+from being recognized as returning calls:
+
+.. code-block:: ada
+
+     --  CFR.Check here, with -fhardcfr-check-returning-calls.
+     P (X);
+     --  CFR.Check here, with -fno-hardcfr-check-returning-calls.
+     return;
+
+or:
+
+.. code-block:: ada
+
+     --  CFR.Check here, with -fhardcfr-check-returning-calls.
+     R := F (X);
+     --  CFR.Check here, with -fno-hardcfr-check-returning-calls.
+     return R;
+
+
+Any subprogram from which an exception may escape, i.e., that may
+raise or propagate an exception that isn't handled internally, is
+conceptually enclosed by a cleanup handler that performs verification,
+unless this is disabled with :switch:`-fno-hardcfr-check-exceptions`.
+With this feature enabled, a subprogram body containing:
+
+.. code-block:: ada
+
+     --  ...
+       Y := F (X);  -- May raise exceptions.
+     --  ...
+       raise E;  -- Not handled internally.
+     --  ...
+
+
+gets modified as follows:
+
+.. code-block:: ada
+
+   begin
+     --  ...
+       Y := F (X);  -- May raise exceptions.
+     --  ...
+       raise E;  -- Not handled internally.
+     --  ...
+   exception
+     when others =>
+       CFR.Check (N, Visited'Access, CFG'Access);
+       raise;
+   end;
+
+
+Verification may also be performed before No_Return calls, whether
+only nothrow ones, with
+:switch:`-fhardcfr-check-noreturn-calls=nothrow`, or all of them, with
+:switch:`-fhardcfr-check-noreturn-calls=always`.  The default is
+:switch:`-fhardcfr-check-noreturn-calls=never` for this feature, that
+disables checking before No_Return calls.
+
+When a No_Return call returns control to its caller through an
+exception, verification may have already been performed before the
+call, if :switch:`-fhardcfr-check-noreturn-calls=always` is in effect.
+The compiler arranges for already-checked No_Return calls without a
+preexisting handler to bypass the implicitly-added cleanup handler and
+thus the redundant check, but a local exception or cleanup handler, if
+present, will modify the set of visited blocks, and checking will take
+place again when the caller reaches the next verification point,
+whether it is a return or reraise statement after the exception is
+otherwise handled, or even another No_Return call.
 
 The instrumentation for hardening with control flow redundancy can be
 observed in dump files generated by the command-line option
diff --git a/gcc/common.opt b/gcc/common.opt
index 661dcf2f485..983cb4db7c6 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -1795,6 +1795,39 @@ fharden-control-flow-redundancy
 Common Var(flag_harden_control_flow_redundancy) Optimization
 Harden control flow by recording and checking execution paths.
 
+fhardcfr-check-returning-calls
+Common Var(flag_harden_control_flow_redundancy_check_returning_calls) Init(-1) Optimization
+Check CFR execution paths also before calls followed by returns of their results.
+
+fhardcfr-check-exceptions
+Common Var(flag_harden_control_flow_redundancy_check_exceptions) Init(-1) Optimization
+Check CFR execution paths also when exiting a function through an exception.
+
+fhardcfr-check-noreturn-calls=
+Common Joined RejectNegative Enum(hardcfr_check_noreturn_calls) Var(flag_harden_control_flow_redundancy_check_noreturn) Init(HCFRNR_UNSPECIFIED) Optimization
+-fhardcfr-check-noreturn-calls=[always|nothrow|never]	Check CFR execution paths also before calling noreturn functions.
+
+Enum
+Name(hardcfr_check_noreturn_calls) Type(enum hardcfr_noret) UnknownError(unknown hardcfr noreturn checking level %qs)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(never) Value(HCFRNR_NEVER)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(nothrow) Value(HCFRNR_NOTHROW)
+
+; ??? There could be yet another option here, that checked before
+; noreturn calls, except for those known to always throw, if we had
+; means to distinguish noreturn functions known to always throw, such
+; as those used to (re)raise exceptions, from those that merely might
+; throw.  "not always" stands for "not always-throwing", but it also
+; contrasts with "always" below.
+; EnumValue
+; Enum(hardcfr_check_noreturn_calls) String(not-always) Value(HCFRNR_NOT_ALWAYS)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(always) Value(HCFRNR_ALWAYS)
+
 ; Nonzero means ignore `#ident' directives.  0 means handle them.
 ; Generate position-independent code for executables if possible
 ; On SVR4 targets, it also controls whether or not to emit a
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index 3bec4b42437..ae324b9256b 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -625,7 +625,9 @@ Objective-C and Objective-C++ Dialects}.
 -fsanitize-undefined-trap-on-error  -fbounds-check @gol
 -fcf-protection=@r{[}full@r{|}branch@r{|}return@r{|}none@r{|}check@r{]} @gol
 -fharden-compares -fharden-conditional-branches @gol
--fharden-control-flow-redundancy @gol
+-fharden-control-flow-redundancy  -fhardcfr-check-exceptions  @gol
+-fhardcfr-check-returning-calls  @gol
+-fhardcfr-check-noreturn-calls=@r{[}always@r{|}nothrow@r{|}never@r{]}  @gol
 -fstack-protector  -fstack-protector-all  -fstack-protector-strong @gol
 -fstack-protector-explicit  -fstack-check @gol
 -fstack-limit-register=@var{reg}  -fstack-limit-symbol=@var{sym} @gol
@@ -16565,11 +16567,75 @@ conditionals.
 @item -fharden-control-flow-redundancy
 @opindex fharden-control-flow-redundancy
 Emit extra code to set booleans when entering basic blocks, and to
-verify, at function exits, that they amount to an execution path that is
-consistent with the control flow graph, trapping otherwise.  Tuning
-options @option{--param hardcfr-max-blocks} and @option{--param
+verify and trap, at function exits, when the booleans do not form an
+execution path that is compatible with the control flow graph.
+
+Verification takes place before returns, before mandatory tail calls
+(see below) and, optionally, before escaping exceptions with
+@option{-fhardcfr-check-exceptions}, before returning calls with
+@option{-fhardcfr-check-returning-calls}, and before noreturn calls with
+@option{-fhardcfr-check-noreturn-calls}).  Tuning options
+@option{--param hardcfr-max-blocks} and @option{--param
 hardcfr-max-inline-blocks} are available.
 
+Tail call optimization takes place too late to affect control flow
+redundancy, but calls annotated as mandatory tail calls by language
+front-ends, and any calls marked early enough as potential tail calls
+would also have verification issued before the call, but these
+possibilities are merely theoretical, as these conditions can only be
+met when using custom compiler plugins.
+
+@item -fhardcfr-check-exceptions
+@opindex fhardcfr-check-exceptions
+@opindex fno-hardcfr-check-exceptions
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph at exception
+escape points, as if the function body was wrapped with a cleanup
+handler that performed the check and reraised.  This option is enabled
+by default; use @option{-fno-hardcfr-check-exceptions} to disable it.
+
+@item -fhardcfr-check-returning-calls
+@opindex fhardcfr-check-returning-calls
+@opindex fno-hardcfr-check-returning-calls
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph before any
+function call immediately followed by a return of its result, if any, so
+as to not prevent tail-call optimization, whether or not it is
+ultimately optimized to a tail call.
+
+This option is enabled by default whenever sibling call optimizations
+are enabled (see @option{-foptimize-sibling-calls}), but it can be
+enabled (or disabled, using its negated form) explicitly, regardless of
+the optimizations.
+
+@item -fhardcfr-check-noreturn-calls=@r{[}always@r{|}nothrow@r{|}never@r{]}
+@opindex fhardcfr-check-noreturn-calls
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph before
+@code{noreturn} calls, either all of them (@option{always}), those that
+may not return control to the caller through an exception either
+(@option{nothrow}), or none of them (@option{never}, the default).
+
+Checking before a @code{noreturn} function that may return control to
+the caller through an exception may cause checking to be performed more
+than once, if the exception is caught in the caller, whether by a
+handler or a cleanup.  When @option{-fhardcfr-check-exceptions} is also
+enabled, the compiler will avoid associating a @code{noreturn} call with
+the implicitly-added cleanup handler, since it would be redundant with
+the check performed before the call, but other handlers or cleanups in
+the function, if activated, will modify the recorded execution path and
+check it again when another checkpoint is hit.  The checkpoint may even
+be another @code{noreturn} call, so checking may end up performed
+multiple times.
+
+Various optimizers may cause calls to be marked as @code{noreturn}
+and/or @code{nothrow}, even in the absence of the corresponding
+attributes, which may affect the placement of checks before calls, as
+well as the addition of implicit cleanup handlers for them.  This
+unpredictability, and the fact that raising and reraising exceptions
+frequently amounts to implicitly calling @code{noreturn} functions, have
+made @option{never} the default setting for this option.
+
 @item -fstack-protector
 @opindex fstack-protector
 Emit extra code to check for buffer overflows, such as stack smashing
diff --git a/gcc/flag-types.h b/gcc/flag-types.h
index d2e751060ff..3fae7548cab 100644
--- a/gcc/flag-types.h
+++ b/gcc/flag-types.h
@@ -157,6 +157,16 @@ enum stack_reuse_level
   SR_ALL
 };
 
+/* Control Flow Redundancy hardening options for noreturn calls.  */
+enum hardcfr_noret
+{
+  HCFRNR_NEVER,
+  HCFRNR_NOTHROW,
+  HCFRNR_NOT_ALWAYS, /* Reserved for future use.  */
+  HCFRNR_ALWAYS,
+  HCFRNR_UNSPECIFIED = -1
+};
+
 /* The live patching level.  */
 enum live_patching_level
 {
diff --git a/gcc/gimple-harden-control-flow.cc b/gcc/gimple-harden-control-flow.cc
index 6b08846dbb1..1c93bf622e8 100644
--- a/gcc/gimple-harden-control-flow.cc
+++ b/gcc/gimple-harden-control-flow.cc
@@ -29,7 +29,12 @@ along with GCC; see the file COPYING3.  If not see
 #include "tree-pass.h"
 #include "ssa.h"
 #include "gimple-iterator.h"
+#include "gimple-pretty-print.h"
 #include "tree-cfg.h"
+#include "tree-cfgcleanup.h"
+#include "tree-eh.h"
+#include "except.h"
+#include "sbitmap.h"
 #include "basic-block.h"
 #include "cfghooks.h"
 #include "cfgloop.h"
@@ -60,9 +65,7 @@ const pass_data pass_data_harden_control_flow_redundancy = {
   0,	    // properties_provided
   0,	    // properties_destroyed
   TODO_cleanup_cfg, // properties_start
-  TODO_update_ssa
-  | TODO_cleanup_cfg
-  | TODO_verify_il, // properties_finish
+  0,        // properties_finish
 };
 
 class pass_harden_control_flow_redundancy : public gimple_opt_pass
@@ -79,16 +82,6 @@ public:
     if (!flag_harden_control_flow_redundancy)
       return false;
 
-    /* We don't verify when an exception escapes, propagated or raised
-       by the function itself, so we're only concerned with edges to
-       the exit block.  If there aren't any, the function doesn't
-       return normally, so there won't be any checking point, so
-       there's no point in running the pass.  Should we add
-       verification at exception escapes, we should at least look at
-       !flag_exceptions here.  */
-    if (EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (fun)->preds) == 0)
-      return false;
-
     /* Functions that return more than once, like setjmp and vfork
        (that also gets this flag set), will start recording a path
        after the first return, and then may take another path when
@@ -117,8 +110,9 @@ public:
 	return false;
       }
 
-    if (param_hardcfr_max_blocks > 0
-	&& n_basic_blocks_for_fn (fun) - 2 > param_hardcfr_max_blocks)
+    if (fun->cfg && param_hardcfr_max_blocks > 0
+	&& (n_basic_blocks_for_fn (fun) - NUM_FIXED_BLOCKS
+	    > param_hardcfr_max_blocks))
       {
 	warning_at (DECL_SOURCE_LOCATION (fun->decl), 0,
 		    "%qD has more than %u blocks, the requested"
@@ -134,6 +128,293 @@ public:
 
 }
 
+/* Return TRUE iff CFR checks should be inserted before returning
+   calls.  */
+
+static bool
+check_returning_calls_p ()
+{
+  return
+    flag_harden_control_flow_redundancy_check_returning_calls > 0
+    || (flag_harden_control_flow_redundancy_check_returning_calls < 0
+	/* Gates pass_tail_calls.  */
+	&& flag_optimize_sibling_calls
+	/* Gates pass_all_ptimizations.  */
+	&& optimize >= 1 && !optimize_debug);
+}
+
+/* Scan BB from the end, updating *RETPTR if given as return stmts and
+   copies are found.  Return a call or a stmt that cannot appear after
+   a tail call, or NULL if the top of the block is reached without
+   finding any.  */
+
+static gimple *
+hardcfr_scan_block (basic_block bb, tree **retptr)
+{
+  gimple_stmt_iterator gsi;
+  for (gsi = gsi_last_bb (bb); !gsi_end_p (gsi); gsi_prev (&gsi))
+    {
+      gimple *stmt = gsi_stmt (gsi);
+
+      /* Ignore labels, returns, nops, clobbers and debug stmts.  */
+      if (gimple_code (stmt) == GIMPLE_LABEL
+	  || gimple_code (stmt) == GIMPLE_NOP
+	  || gimple_code (stmt) == GIMPLE_PREDICT
+	  || gimple_clobber_p (stmt)
+	  || is_gimple_debug (stmt))
+	continue;
+
+      if (gimple_code (stmt) == GIMPLE_RETURN)
+	{
+	  greturn *gret = as_a <greturn *> (stmt);
+	  if (retptr)
+	    {
+	      gcc_checking_assert (!*retptr);
+	      *retptr = gimple_return_retval_ptr (gret);
+	    }
+	  continue;
+	}
+
+      /* Check for a call.  */
+      if (is_gimple_call (stmt))
+	return stmt;
+
+      /* Allow simple copies to the return value, updating the return
+	 value to be found in earlier assignments.  */
+      if (retptr && *retptr && gimple_assign_single_p (stmt)
+	  && **retptr == gimple_assign_lhs (stmt))
+	{
+	  *retptr = gimple_assign_rhs1_ptr (stmt);
+	  continue;
+	}
+
+      return stmt;
+    }
+
+  /* Any other kind of stmt will prevent a tail call.  */
+  return NULL;
+}
+
+/* Return TRUE iff CALL is to be preceded by a CFR checkpoint, i.e.,
+   if it's a returning call (one whose result is ultimately returned
+   without intervening non-copy statements) and we're checking
+   returning calls, a __builtin_return call (noreturn with a path to
+   the exit block), a must-tail call, or a tail call.  */
+
+static bool
+returning_call_p (gcall *call)
+{
+  if (!(gimple_call_noreturn_p (call)
+	|| gimple_call_must_tail_p (call)
+	|| gimple_call_tail_p (call)
+	|| check_returning_calls_p ()))
+    return false;
+
+  /* Quickly check that there's a path to exit compatible with a
+     returning call.  Detect infinite loops through the counter.  */
+  basic_block bb = gimple_bb (call);
+  auto_vec<basic_block, 10> path;
+  for (int i = n_basic_blocks_for_fn (cfun);
+       bb != EXIT_BLOCK_PTR_FOR_FN (cfun) && i--;
+       bb = single_succ (bb))
+    if (!single_succ_p (bb)
+	|| (single_succ_edge (bb)->flags & EDGE_EH) != 0)
+      return false;
+    else
+      path.safe_push (bb);
+
+  /* Check the stmts in the blocks and trace the return value.  */
+  tree *retptr = NULL;
+  for (;;)
+    {
+      gcc_checking_assert (!path.is_empty ());
+      basic_block bb = path.pop ();
+      gimple *stop = hardcfr_scan_block (bb, &retptr);
+      if (stop)
+	{
+	  if (stop != call)
+	    return false;
+	  gcc_checking_assert (path.is_empty ());
+	  break;
+	}
+
+      gphi *retphi = NULL;
+      if (retptr && *retptr && TREE_CODE (*retptr) == SSA_NAME
+	  && !SSA_NAME_IS_DEFAULT_DEF (*retptr)
+	  && SSA_NAME_DEF_STMT (*retptr)
+	  && is_a <gphi *> (SSA_NAME_DEF_STMT (*retptr))
+	  && gimple_bb (SSA_NAME_DEF_STMT (*retptr)) == bb)
+	{
+	  retphi = as_a <gphi *> (SSA_NAME_DEF_STMT (*retptr));
+	  gcc_checking_assert (gimple_phi_result (retphi) == *retptr);
+	}
+      else
+	continue;
+
+      gcc_checking_assert (!path.is_empty ());
+      edge e = single_succ_edge (path.last ());
+      int i = EDGE_COUNT (bb->preds);
+      while (i--)
+	if (EDGE_PRED (bb, i) == e)
+	  break;
+      gcc_checking_assert (i >= 0);
+      retptr = gimple_phi_arg_def_ptr (retphi, i);
+    }
+
+  return (gimple_call_noreturn_p (call)
+	  || gimple_call_must_tail_p (call)
+	  || gimple_call_tail_p (call)
+	  || (gimple_call_lhs (call) == (retptr ? *retptr : NULL)
+	      && check_returning_calls_p ()));
+}
+
+typedef auto_vec<edge, 10> chk_edges_t;
+
+/* Declare for mutual recursion.  */
+static bool hardcfr_sibcall_search_preds (basic_block bb,
+					  chk_edges_t &chk_edges,
+					  int &count_chkcall,
+					  auto_sbitmap &chkcall_blocks,
+					  int &count_postchk,
+					  auto_sbitmap &postchk_blocks,
+					  tree *retptr);
+
+/* Search backwards from the end of BB for a mandatory or potential
+   sibcall.  Schedule the block to be handled sort-of like noreturn if
+   so.  Recurse to preds, with updated RETPTR, if the block only
+   contains stmts that may follow such a call, scheduling checking at
+   edges and marking blocks as post-check as needed.  Return true iff,
+   at the end of the block, a check will have already been
+   performed.  */
+
+static bool
+hardcfr_sibcall_search_block (basic_block bb,
+			      chk_edges_t &chk_edges,
+			      int &count_chkcall,
+			      auto_sbitmap &chkcall_blocks,
+			      int &count_postchk,
+			      auto_sbitmap &postchk_blocks,
+			      tree *retptr)
+{
+  /* Conditionals and internal exceptions rule out tail calls.  */
+  if (!single_succ_p (bb)
+      || (single_succ_edge (bb)->flags & EDGE_EH) != 0)
+    return false;
+
+  gimple *stmt = hardcfr_scan_block (bb, &retptr);
+  if (!stmt)
+    return hardcfr_sibcall_search_preds (bb, chk_edges,
+					 count_chkcall, chkcall_blocks,
+					 count_postchk, postchk_blocks,
+					 retptr);
+
+  if (!is_a <gcall *> (stmt))
+    return false;
+
+  /* Avoid disrupting mandatory or early-marked tail calls,
+     inserting the check before them.  This works for
+     must-tail calls, but tail calling as an optimization is
+     detected too late for us.
+
+     Also check for noreturn calls here.  Noreturn calls won't
+     normally have edges to exit, so they won't be found here,
+     but __builtin_return does, and we must check before
+     it, so handle it like a tail call.  */
+  gcall *call = as_a <gcall *> (stmt);
+  if (!(gimple_call_noreturn_p (call)
+	|| gimple_call_must_tail_p (call)
+	|| gimple_call_tail_p (call)
+	|| (gimple_call_lhs (call) == (retptr ? *retptr : NULL)
+	    && check_returning_calls_p ())))
+    return false;
+
+  gcc_checking_assert (returning_call_p (call));
+
+  /* We found a call that is to be preceded by checking.  */
+  if (bitmap_set_bit (chkcall_blocks, bb->index))
+    ++count_chkcall;
+  else
+    gcc_unreachable ();
+  return true;
+}
+
+
+/* Search preds of BB for a mandatory or potential sibcall or
+   returning call, and arrange for the blocks containing them to have
+   a check inserted before the call, like noreturn calls.  If any
+   preds are found to perform checking, schedule checks at the edges
+   of those that don't, and mark BB as postcheck..  */
+
+static bool
+hardcfr_sibcall_search_preds (basic_block bb,
+			      chk_edges_t &chk_edges,
+			      int &count_chkcall,
+			      auto_sbitmap &chkcall_blocks,
+			      int &count_postchk,
+			      auto_sbitmap &postchk_blocks,
+			      tree *retptr)
+{
+  /* For the exit block, we wish to force a check at every
+     predecessor, so pretend we've already found a pred that had
+     checking, so that we schedule checking at every one of its pred
+     edges.  */
+  bool first = bb->index >= NUM_FIXED_BLOCKS;
+  bool postchecked = true;
+
+  gphi *retphi = NULL;
+  if (retptr && *retptr && TREE_CODE (*retptr) == SSA_NAME
+      && !SSA_NAME_IS_DEFAULT_DEF (*retptr)
+      && SSA_NAME_DEF_STMT (*retptr)
+      && is_a <gphi *> (SSA_NAME_DEF_STMT (*retptr))
+      && gimple_bb (SSA_NAME_DEF_STMT (*retptr)) == bb)
+    {
+      retphi = as_a <gphi *> (SSA_NAME_DEF_STMT (*retptr));
+      gcc_checking_assert (gimple_phi_result (retphi) == *retptr);
+    }
+
+  for (int i = EDGE_COUNT (bb->preds); i--; first = false)
+    {
+      edge e = EDGE_PRED (bb, i);
+
+      bool checked
+	= hardcfr_sibcall_search_block (e->src, chk_edges,
+					count_chkcall, chkcall_blocks,
+					count_postchk, postchk_blocks,
+					!retphi ? retptr
+					: gimple_phi_arg_def_ptr (retphi, i));
+
+      if (first)
+	{
+	  postchecked = checked;
+	  continue;
+	}
+
+      /* When we first find a checked block, force a check at every
+	 other incoming edge we've already visited, and those we
+	 visit afterwards that don't have their own check, so that
+	 when we reach BB, the check has already been performed.  */
+      if (!postchecked && checked)
+	{
+	  for (int j = EDGE_COUNT (bb->preds); --j > i; )
+	    chk_edges.safe_push (EDGE_PRED (bb, j));
+	  postchecked = true;
+	}
+      if (postchecked && !checked)
+	chk_edges.safe_push (EDGE_PRED (bb, i));
+    }
+
+  if (postchecked && bb->index >= NUM_FIXED_BLOCKS)
+    {
+      if (bitmap_set_bit (postchk_blocks, bb->index))
+	count_postchk++;
+      else
+	gcc_unreachable ();
+    }
+
+  return postchecked;
+}
+
+
 class rt_bb_visited
 {
   /* Use a sufficiently wide unsigned type to hold basic block numbers.  */
@@ -176,8 +457,8 @@ class rt_bb_visited
      neither ENTRY nor EXIT, but maybe one-past-the-end, to compute
      the visited array length.  */
   blknum num2idx (blknum n) {
-    gcc_checking_assert (n >= 2 && n <= nblocks);
-    return (n - 2);
+    gcc_checking_assert (n >= NUM_FIXED_BLOCKS && n <= nblocks);
+    return (n - NUM_FIXED_BLOCKS);
   }
   /* Return the block vindex for BB, that must not be ENTRY or
      EXIT.  */
@@ -249,8 +530,7 @@ class rt_bb_visited
   }
 
   /* Set the bit corresponding to BB in VISITED.  Add to SEQ any
-     required gimple statements, and return SEQ, possibly
-     modified.  */
+     required gimple stmts, and return SEQ, possibly modified.  */
   gimple_seq vset (basic_block bb, gimple_seq seq = NULL)
   {
     tree bit, setme = vword (bb, &bit);
@@ -270,7 +550,7 @@ class rt_bb_visited
 
 public:
   /* Prepare to add control flow redundancy testing to CFUN.  */
-  rt_bb_visited ()
+  rt_bb_visited (int checkpoints)
     : nblocks (n_basic_blocks_for_fn (cfun)),
       vword_type (NULL), ckseq (NULL), rtcfg (NULL)
   {
@@ -353,8 +633,8 @@ public:
 					 NULL, NULL);
     gimple_seq_add_stmt (&ckseq, detach);
 
-    if (nblocks - 2 > blknum (param_hardcfr_max_inline_blocks)
-	|| !single_pred_p (EXIT_BLOCK_PTR_FOR_FN (cfun)))
+    if (nblocks - NUM_FIXED_BLOCKS > blknum (param_hardcfr_max_inline_blocks)
+	|| checkpoints > 1)
       {
 	/* Make sure vword_bits is wide enough for the representation
 	   of nblocks in rtcfg.  Compare with vword_bits << vword_bits,
@@ -379,65 +659,33 @@ public:
     gimple_seq_add_stmt (&ckseq, ckfail_init);
   }
 
-  /* Insert SEQ on E, or close enough (e.g., before a noreturn or tail
-     call at the end of E->src).  */
-  void insert_exit_check (gimple_seq seq, edge e)
+  /* Insert SEQ before a resx or a call in INSBB.  */
+  void insert_exit_check_in_block (gimple_seq seq, basic_block insbb)
   {
-    basic_block insbb = e->src;
-
-    /* If the returning block ends with a noreturn call, insert
-       checking before it.  This is particularly important for
-       __builtin_return.  Other noreturn calls won't have an edge to
-       the exit block, so they won't even be considered as exit paths.
-
-       Insert-on-edge inserts before other return stmts, but not
-       before calls, and if a single-block function had the check
-       sequence inserted after a noreturn call, it would be useless,
-       but checking would still flag it as malformed if block 2 has a
-       fallthru edge to the exit block.
-
-       Also avoid disrupting tail calls, inserting the check before
-       them.  This works for must-tail calls, but tail calling as an
-       optimization is detected too late for us.  */
     gimple_stmt_iterator gsi = gsi_last_bb (insbb);
-    gimple *ret = gsi_stmt (gsi);
-    if (ret && is_a <greturn *> (ret))
-      {
+
+    while (!gsi_end_p (gsi))
+      if (is_a <gresx *> (gsi_stmt (gsi))
+	  || is_a <gcall *> (gsi_stmt (gsi)))
+	break;
+      else
 	gsi_prev (&gsi);
-	if (!gsi_end_p (gsi))
-	  ret = gsi_stmt (gsi);
-      }
-    if (ret && is_a <gcall *> (ret)
-	&& (gimple_call_noreturn_p (ret)
-	    || gimple_call_must_tail_p (as_a <gcall *> (ret))
-	    || gimple_call_tail_p (as_a <gcall *> (ret))))
-      gsi_insert_seq_before (&gsi, seq, GSI_SAME_STMT);
-    else
-      gsi_insert_seq_on_edge_immediate (e, seq);
+
+    gsi_insert_seq_before (&gsi, seq, GSI_SAME_STMT);
+  }
+
+  /* Insert SEQ on E.  */
+  void insert_exit_check_on_edge (gimple_seq seq, edge e)
+  {
+    gsi_insert_seq_on_edge_immediate (e, seq);
   }
 
-  /* Add checking code on every exit edge, and initialization code on
+  /* Add checking code to CHK_EDGES, and initialization code on
      the entry edge.  Before this point, the CFG has been undisturbed,
      and all the needed data has been collected and safely stowed.  */
-  void check ()
+  void check (chk_edges_t &chk_edges,
+	      int count_chkcall, auto_sbitmap const &chkcall_blocks)
   {
-    /* Insert initializers for visited at the entry.  */
-    gimple_seq iseq = NULL;
-
-    gcall *vinit = gimple_build_call (builtin_decl_explicit
-				      (BUILT_IN_MEMSET), 3,
-				      build1 (ADDR_EXPR,
-					      build_pointer_type
-					      (TREE_TYPE (visited)),
-					      visited),
-				      integer_zero_node,
-				      TYPE_SIZE_UNIT (TREE_TYPE (visited)));
-    gimple_seq_add_stmt (&iseq, vinit);
-
-    gsi_insert_seq_on_edge_immediate (single_succ_edge
-				      (ENTRY_BLOCK_PTR_FOR_FN (cfun)),
-				      iseq);
-
     /* If we're using out-of-line checking, create and statically
        initialize the CFG checking representation, generate the
        checker call for the checking sequence, and insert it in all
@@ -498,28 +746,116 @@ public:
 						     rtcfg));
 	gimple_seq_add_stmt (&ckseq, call_chk);
 
+	gimple *clobber = gimple_build_assign (visited,
+					       build_clobber
+					       (TREE_TYPE (visited)));
+	gimple_seq_add_stmt (&ckseq, clobber);
+
 	/* If we have multiple exit edges, insert (copies of)
 	   ckseq in all of them.  */
-	for (int i = EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (cfun)->preds);
-	     i--; )
+	for (int i = chk_edges.length (); i--; )
 	  {
 	    gimple_seq seq = ckseq;
 	    /* Copy the sequence, unless we're dealing with the
 	       last edge (we're counting down to zero).  */
-	    if (i)
+	    if (i || count_chkcall)
+	      seq = gimple_seq_copy (seq);
+
+	    edge e = chk_edges[i];
+
+	    if (dump_file)
+	      {
+		if (e->dest == EXIT_BLOCK_PTR_FOR_FN (cfun))
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to exit.\n",
+			   e->src->index);
+		else
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to postcheck block %i.\n",
+			   e->src->index, e->dest->index);
+	      }
+
+	    insert_exit_check_on_edge (seq, e);
+
+	    gcc_checking_assert (!bitmap_bit_p (chkcall_blocks, e->src->index));
+	  }
+
+	sbitmap_iterator it;
+	unsigned i;
+	EXECUTE_IF_SET_IN_BITMAP (chkcall_blocks, 0, i, it)
+	  {
+	    basic_block bb = BASIC_BLOCK_FOR_FN (cfun, i);
+
+	    gimple_seq seq = ckseq;
+	    gcc_checking_assert (count_chkcall > 0);
+	    if (--count_chkcall)
 	      seq = gimple_seq_copy (seq);
 
-	    insert_exit_check (seq,
-			       EDGE_PRED (EXIT_BLOCK_PTR_FOR_FN (cfun), i));
+	    if (dump_file)
+	      fprintf (dump_file,
+		       "Inserting out-of-line check before stmt in block %i.\n",
+		       bb->index);
+
+	    insert_exit_check_in_block (seq, bb);
 	  }
+
+	gcc_checking_assert (count_chkcall == 0);
       }
     else
       {
 	/* Inline checking requires a single exit edge.  */
-	gimple *last = gsi_stmt (gsi_last (ckseq));
+	gimple *last = gimple_build_assign (visited,
+					       build_clobber
+					       (TREE_TYPE (visited)));
+	gimple_seq_add_stmt (&ckseq, last);
 
-	insert_exit_check (ckseq,
-			   single_pred_edge (EXIT_BLOCK_PTR_FOR_FN (cfun)));
+	if (!count_chkcall)
+	  {
+	    edge e = single_pred_edge (EXIT_BLOCK_PTR_FOR_FN (cfun));
+
+	    if (dump_file)
+	      {
+		if (e->dest == EXIT_BLOCK_PTR_FOR_FN (cfun))
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to postcheck block %i.\n",
+			   e->src->index, e->dest->index);
+		else
+		  fprintf (dump_file,
+			   "Inserting inline check in"
+			   " block %i's edge to exit.\n",
+			   e->src->index);
+	      }
+
+	    insert_exit_check_on_edge (ckseq, e);
+	  }
+	else
+	  {
+	    gcc_checking_assert (count_chkcall == 1);
+
+	    sbitmap_iterator it;
+	    unsigned i;
+	    EXECUTE_IF_SET_IN_BITMAP (chkcall_blocks, 0, i, it)
+	      {
+		basic_block bb = BASIC_BLOCK_FOR_FN (cfun, i);
+
+		gimple_seq seq = ckseq;
+		gcc_checking_assert (count_chkcall > 0);
+		if (--count_chkcall)
+		  seq = gimple_seq_copy (seq);
+
+		if (dump_file)
+		  fprintf (dump_file,
+			   "Inserting inline check before stmt in block %i.\n",
+			   bb->index);
+
+		insert_exit_check_in_block (seq, bb);
+	      }
+
+	    gcc_checking_assert (count_chkcall == 0);
+	  }
 
 	/* The inserted ckseq computes CKFAIL at LAST.  Now we have to
 	   conditionally trap on it.  */
@@ -540,8 +876,7 @@ public:
 	  add_bb_to_loop (trp, current_loops->tree_root);
 
 	/* Insert a conditional branch to the trap block.  If the
-	   conditional wouldn't be the last statement, split the
-	   block.  */
+	   conditional wouldn't be the last stmt, split the block.  */
 	gimple_stmt_iterator gsi = gsi_for_stmt (last);
 	if (!gsi_one_before_end_p (gsi))
 	  split_block (gsi_bb (gsi), gsi_stmt (gsi));
@@ -564,6 +899,24 @@ public:
 	if (dom_info_available_p (CDI_DOMINATORS))
 	  set_immediate_dominator (CDI_DOMINATORS, trp, gimple_bb (last));
       }
+
+    /* Insert initializers for visited at the entry.  Do this after
+       other insertions, to avoid messing with block numbers.  */
+    gimple_seq iseq = NULL;
+
+    gcall *vinit = gimple_build_call (builtin_decl_explicit
+				      (BUILT_IN_MEMSET), 3,
+				      build1 (ADDR_EXPR,
+					      build_pointer_type
+					      (TREE_TYPE (visited)),
+					      visited),
+				      integer_zero_node,
+				      TYPE_SIZE_UNIT (TREE_TYPE (visited)));
+    gimple_seq_add_stmt (&iseq, vinit);
+
+    gsi_insert_seq_on_edge_immediate (single_succ_edge
+				      (ENTRY_BLOCK_PTR_FOR_FN (cfun)),
+				      iseq);
   }
 
   /* Push onto RTCFG a (mask, index) pair to test for IBB when BB is
@@ -607,7 +960,7 @@ public:
     return false;
   }
 
-  /* Add to CKSEQ statements to clear CKPART if OBB is visited.  */
+  /* Add to CKSEQ stmts to clear CKPART if OBB is visited.  */
   void
   build_block_check (basic_block obb)
   {
@@ -632,34 +985,48 @@ public:
 
   /* Add to BB code to set its bit in VISITED, and add to RTCFG or
      CKSEQ the data or code needed to check BB's predecessors and
-     successors.  Do NOT change the CFG.  */
-  void visit (basic_block bb)
+     successors.  If CHECKPOINT, assume the block is a checkpoint,
+     whether or not it has an edge to EXIT.  If POSTCHECK, assume the
+     block post-dominates checkpoints and therefore no bitmap setting
+     or checks are to be performed in or for it.  Do NOT change the
+     CFG.  */
+  void visit (basic_block bb, bool checkpoint, bool postcheck)
   {
     /* Set the bit in VISITED when entering the block.  */
     gimple_stmt_iterator gsi = gsi_after_labels (bb);
-    gsi_insert_seq_before (&gsi, vset (bb), GSI_SAME_STMT);
+    if (!postcheck)
+      gsi_insert_seq_before (&gsi, vset (bb), GSI_SAME_STMT);
 
     if (rtcfg)
       {
-	/* Build a list of (index, mask) terminated by (NULL, 0).
-	   Consolidate masks with the same index when they're
-	   adjacent.  First, predecessors.  Count backwards, because
-	   we're going to reverse the list.  The order shouldn't
-	   matter, but let's not make it surprising.  */
-	for (int i = EDGE_COUNT (bb->preds); i--; )
-	  if (push_rtcfg_pair (EDGE_PRED (bb, i)->src, bb,
-			       ENTRY_BLOCK_PTR_FOR_FN (cfun)))
-	    break;
+	if (!postcheck)
+	  {
+	    /* Build a list of (index, mask) terminated by (NULL, 0).
+	       Consolidate masks with the same index when they're
+	       adjacent.  First, predecessors.  Count backwards, because
+	       we're going to reverse the list.  The order shouldn't
+	       matter, but let's not make it surprising.  */
+	    for (int i = EDGE_COUNT (bb->preds); i--; )
+	      if (push_rtcfg_pair (EDGE_PRED (bb, i)->src, bb,
+				   ENTRY_BLOCK_PTR_FOR_FN (cfun)))
+		break;
+	  }
 	rtcfg = tree_cons (NULL_TREE, build_int_cst (vword_type, 0), rtcfg);
 
-	/* Then, successors.  */
-	for (int i = EDGE_COUNT (bb->succs); i--; )
-	  if (push_rtcfg_pair (EDGE_SUCC (bb, i)->dest, bb,
-			       EXIT_BLOCK_PTR_FOR_FN (cfun)))
-	    break;
+	if (!postcheck)
+	  {
+	    /* Then, successors.  */
+	    if (!checkpoint
+		|| !push_rtcfg_pair (EXIT_BLOCK_PTR_FOR_FN (cfun),
+				     bb, EXIT_BLOCK_PTR_FOR_FN (cfun)))
+	      for (int i = EDGE_COUNT (bb->succs); i--; )
+		if (push_rtcfg_pair (EDGE_SUCC (bb, i)->dest, bb,
+				     EXIT_BLOCK_PTR_FOR_FN (cfun)))
+		  break;
+	  }
 	rtcfg = tree_cons (NULL_TREE, build_int_cst (vword_type, 0), rtcfg);
       }
-    else
+    else if (!postcheck)
       {
 	/* Schedule test to fail if the block was reached but somehow none
 	   of its predecessors were.  */
@@ -677,6 +1044,8 @@ public:
 	gassign *blkruns = gimple_build_assign (ckpart, unshare_expr (bit));
 	gimple_seq_add_stmt (&ckseq, blkruns);
 
+	if (checkpoint)
+	  build_block_check (EXIT_BLOCK_PTR_FOR_FN (cfun));
 	for (int i = 0, e = EDGE_COUNT (bb->succs); i < e; i++)
 	  build_block_check (EDGE_SUCC (bb, i)->dest);
 
@@ -687,21 +1056,350 @@ public:
   }
 };
 
+/* It might be useful to avoid checking before noreturn calls that are
+   known to always finish by throwing an exception, rather than by
+   ending the program or looping forever.  Such functions would have
+   to be annotated somehow, with an attribute or flag.
+   Exception-raising functions, such as C++'s __cxa_throw,
+   __cxa_rethrow, and Ada's */
+static bool
+always_throwing_noreturn_call_p (gimple *)
+{
+  return false;
+}
+
 /* Control flow redundancy hardening: record the execution path, and
    verify at exit that an expect path was taken.  */
 
 unsigned int
-pass_harden_control_flow_redundancy::execute (function *)
+pass_harden_control_flow_redundancy::execute (function *fun)
 {
-  rt_bb_visited vstd;
-
+  bool const check_at_escaping_exceptions
+    = (flag_exceptions
+       && flag_harden_control_flow_redundancy_check_exceptions);
+  bool const check_before_noreturn_calls
+    = flag_harden_control_flow_redundancy_check_noreturn > HCFRNR_NEVER;
+  bool const check_before_nothrow_noreturn_calls
+    = (check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn >= HCFRNR_NOTHROW);
+  bool const check_before_throwing_noreturn_calls
+    = (flag_exceptions
+       && check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn > HCFRNR_NOTHROW);
+  bool const check_before_always_throwing_noreturn_calls
+    = (flag_exceptions
+       && check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn >= HCFRNR_ALWAYS);
   basic_block bb;
-  FOR_EACH_BB_FN (bb, cfun)
-    vstd.visit (bb);
+  basic_block bb_eh_cleanup = NULL;
+
+  if (check_at_escaping_exceptions)
+    {
+      int lp_eh_cleanup = -1;
+
+      /* Record the preexisting blocks, to avoid visiting newly-created
+	 blocks.  */
+      auto_sbitmap to_visit (last_basic_block_for_fn (fun));
+      bitmap_clear (to_visit);
+
+      FOR_EACH_BB_FN (bb, fun)
+	bitmap_set_bit (to_visit, bb->index);
+
+      /* Scan the blocks for stmts with escaping exceptions, that
+	 wouldn't be denoted in the CFG, and associate them with an
+	 empty cleanup handler around the whole function.  Walk
+	 backwards, so that even when we split the block, */
+      sbitmap_iterator it;
+      unsigned i;
+      EXECUTE_IF_SET_IN_BITMAP (to_visit, 0, i, it)
+	{
+	  bb = BASIC_BLOCK_FOR_FN (fun, i);
+
+	  for (gimple_stmt_iterator gsi = gsi_last_bb (bb);
+	       !gsi_end_p (gsi); gsi_prev (&gsi))
+	    {
+	      gimple *stmt = gsi_stmt (gsi);
+	      if (!stmt_could_throw_p (fun, stmt))
+		continue;
+
+	      /* If it must not throw, or if it already has a handler,
+		 we need not worry about it.  */
+	      if (lookup_stmt_eh_lp (stmt) != 0)
+		continue;
+
+	      /* Don't split blocks at, nor add EH edges to, tail
+		 calls, we will add verification before the call
+		 anyway.  */
+	      if (is_a <gcall *> (stmt)
+		  && (gimple_call_must_tail_p (as_a <gcall *> (stmt))
+		      || gimple_call_tail_p (as_a <gcall *> (stmt))
+		      || returning_call_p (as_a <gcall *> (stmt))))
+		continue;
+
+	      if (!gsi_one_before_end_p (gsi))
+		split_block (bb, stmt);
+	      /* A resx or noreturn call needs not be associated with
+		 the cleanup handler if we're going to add checking
+		 before it.  We only test cases that didn't require
+		 block splitting because noreturn calls would always
+		 be at the end of blocks, and we test for zero
+		 successors because if there is an edge, it's not
+		 noreturn, as any EH edges would have already been
+		 caught by the lookup_stmt_eh_lp test above.  */
+	      else if (check_before_noreturn_calls
+		       && EDGE_COUNT (bb->succs) == 0
+		       && (is_a <gresx *> (stmt)
+			   ? check_before_always_throwing_noreturn_calls
+			   : (!is_a <gcall *> (stmt)
+			      || !gimple_call_noreturn_p (stmt))
+			   ? (gcc_unreachable (), false)
+			   : (!flag_exceptions
+			      || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+			   ? check_before_nothrow_noreturn_calls
+			   : always_throwing_noreturn_call_p (stmt)
+			   ? check_before_always_throwing_noreturn_calls
+			   : check_before_throwing_noreturn_calls))
+		{
+		  if (dump_file)
+		    {
+		      fprintf (dump_file,
+			       "Bypassing cleanup for noreturn stmt"
+			       " in block %i:\n",
+			       bb->index);
+		      print_gimple_stmt (dump_file, stmt, 0);
+		    }
+		  continue;
+		}
+
+	      if (!bb_eh_cleanup)
+		{
+		  bb_eh_cleanup = create_empty_bb (bb);
+		  if (dom_info_available_p (CDI_DOMINATORS))
+		    set_immediate_dominator (CDI_DOMINATORS, bb_eh_cleanup, bb);
+		  if (current_loops)
+		    add_bb_to_loop (bb_eh_cleanup, current_loops->tree_root);
+
+		  /* Make the new block an EH cleanup for the call.  */
+		  eh_region new_r = gen_eh_region_cleanup (NULL);
+		  eh_landing_pad lp = gen_eh_landing_pad (new_r);
+		  tree label = gimple_block_label (bb_eh_cleanup);
+		  lp->post_landing_pad = label;
+		  EH_LANDING_PAD_NR (label) = lp_eh_cleanup = lp->index;
+
+		  /* Just propagate the exception.
+		     We will later insert the verifier call.  */
+		  gimple_stmt_iterator ehgsi;
+		  ehgsi = gsi_after_labels (bb_eh_cleanup);
+		  gresx *resx = gimple_build_resx (new_r->index);
+		  gsi_insert_before (&ehgsi, resx, GSI_SAME_STMT);
+
+		  if (dump_file)
+		    fprintf (dump_file,
+			     "Created cleanup block %i:\n",
+			     bb_eh_cleanup->index);
+		}
+	      else if (dom_info_available_p (CDI_DOMINATORS))
+		{
+		  basic_block immdom;
+		  immdom = get_immediate_dominator (CDI_DOMINATORS,
+						    bb_eh_cleanup);
+		  if (!dominated_by_p (CDI_DOMINATORS, bb, immdom))
+		    {
+		      immdom = nearest_common_dominator (CDI_DOMINATORS,
+							 immdom, bb);
+		      set_immediate_dominator (CDI_DOMINATORS,
+					       bb_eh_cleanup, immdom);
+		    }
+		}
+
+	      if (dump_file)
+		{
+		  fprintf (dump_file,
+			   "Associated cleanup block with stmt in block %i:\n",
+			   bb->index);
+		  print_gimple_stmt (dump_file, stmt, 0);
+		}
+
+	      add_stmt_to_eh_lp (stmt, lp_eh_cleanup);
+	      /* Finally, wire the EH cleanup block into the CFG.  */
+	      make_eh_edges (stmt);
+	    }
+	}
+
+      if (bb_eh_cleanup)
+	{
+	  /* A cfg_cleanup after bb_eh_cleanup makes for a more compact
+	     rtcfg, and it avoids bb numbering differences when we split
+	     blocks because of trailing debug insns only.  */
+	  cleanup_tree_cfg ();
+	  gcc_checking_assert (EDGE_COUNT (bb_eh_cleanup->succs) == 0);
+	}
+    }
+
+  /* These record blocks with calls that are to be preceded by
+     checkpoints, such as noreturn calls (if so chosen), must-tail
+     calls, potential early-marked tail calls, and returning calls (if
+     so chosen).  */
+  int count_chkcall = 0;
+  auto_sbitmap chkcall_blocks (last_basic_block_for_fn (fun));
+  bitmap_clear (chkcall_blocks);
+
+  /* We wish to add verification at blocks without successors, such as
+     noreturn calls (raising or not) and the reraise at the cleanup
+     block, but not other reraises: they will go through the cleanup
+     block.  */
+  if (check_before_noreturn_calls)
+    FOR_EACH_BB_FN (bb, fun)
+      {
+	gimple_stmt_iterator gsi = gsi_last_bb (bb);
+	if (gsi_end_p (gsi))
+	  continue;
+	gimple *stmt = gsi_stmt (gsi);
 
-  vstd.check ();
+	if (EDGE_COUNT (bb->succs) == 0)
+	  {
+	    /* A stmt at the end of a block without any successors is
+	       either a resx or a noreturn call without a local
+	       handler.  Check that it's one of the desired
+	       checkpoints.  */
+	    if (flag_exceptions && is_a <gresx *> (stmt)
+		? (check_before_always_throwing_noreturn_calls
+		   || bb == bb_eh_cleanup)
+		: (!is_a <gcall *> (stmt)
+		   || !gimple_call_noreturn_p (stmt))
+		? (/* Catch cases in which successors would be
+		      expected.  */
+		   gcc_unreachable (), false)
+		: (!flag_exceptions
+		   || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+		? check_before_nothrow_noreturn_calls
+		: always_throwing_noreturn_call_p (stmt)
+		? check_before_always_throwing_noreturn_calls
+		: check_before_throwing_noreturn_calls)
+	      {
+		if (dump_file)
+		  {
+		    fprintf (dump_file,
+			     "Scheduling check before stmt"
+			     " in succ-less block %i:\n",
+			     bb->index);
+		    print_gimple_stmt (dump_file, stmt, 0);
+		  }
+
+		if (bitmap_set_bit (chkcall_blocks, bb->index))
+		  count_chkcall++;
+		else
+		  gcc_unreachable ();
+	      }
+	    continue;
+	  }
 
-  return 0;
+	/* If there are no exceptions, then any noreturn call must have
+	   zero successor edges.  Otherwise, check for blocks without
+	   non-EH successors, but skip those with resx stmts and edges
+	   (i.e., those other than that in bb_eh_cleanup), since those
+	   will go through bb_eh_cleanup, that will have been counted as
+	   noreturn above because it has no successors.  */
+	gcc_checking_assert (bb != bb_eh_cleanup
+			     || !check_at_escaping_exceptions);
+	if (flag_exceptions && is_a <gresx *> (stmt)
+	    ? check_before_always_throwing_noreturn_calls
+	    : (!is_a <gcall *> (stmt)
+	       || !gimple_call_noreturn_p (stmt))
+	    ? false
+	    : (!flag_exceptions
+	       || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+	    ? (/* Catch cases that should not have successors.  */
+	       gcc_unreachable (), check_before_nothrow_noreturn_calls)
+	    : always_throwing_noreturn_call_p (stmt)
+	    ? check_before_always_throwing_noreturn_calls
+	    : check_before_throwing_noreturn_calls)
+	  {
+	    gcc_checking_assert (single_succ_p (bb)
+				 && (single_succ_edge (bb)->flags & EDGE_EH));
+
+	    if (dump_file)
+	      {
+		fprintf (dump_file,
+			 "Scheduling check before stmt"
+			 " in EH-succ block %i:\n",
+			 bb->index);
+		print_gimple_stmt (dump_file, stmt, 0);
+	      }
+
+	    if (bitmap_set_bit (chkcall_blocks, bb->index))
+	      count_chkcall++;
+	    else
+	      gcc_unreachable ();
+	  }
+      }
+  else if (bb_eh_cleanup)
+    {
+      if (bitmap_set_bit (chkcall_blocks, bb_eh_cleanup->index))
+	count_chkcall++;
+      else
+	gcc_unreachable ();
+    }
+
+  gcc_checking_assert (!bb_eh_cleanup
+		       || bitmap_bit_p (chkcall_blocks, bb_eh_cleanup->index));
+
+  /* If we don't have edges to exit nor noreturn calls (including the
+     cleanup reraise), then we may skip instrumentation: that would
+     amount to a function that ends with an infinite loop.  */
+  if (!count_chkcall
+      && EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (fun)->preds) == 0)
+    {
+      if (dump_file)
+	fprintf (dump_file,
+		 "Disabling CFR, no exit paths to check\n");
+
+      return 0;
+    }
+
+  /* Search for must-tail calls, early-marked potential tail calls,
+     and, if requested, returning calls.  As we introduce early
+     checks, */
+  int count_postchk = 0;
+  auto_sbitmap postchk_blocks (last_basic_block_for_fn (fun));
+  bitmap_clear (postchk_blocks);
+  chk_edges_t chk_edges;
+  hardcfr_sibcall_search_preds (EXIT_BLOCK_PTR_FOR_FN (fun), chk_edges,
+				count_chkcall, chkcall_blocks,
+				count_postchk, postchk_blocks,
+				NULL);
+
+  rt_bb_visited vstd (chk_edges.length () + count_chkcall);
+
+  auto_sbitmap combined_blocks (last_basic_block_for_fn (fun));
+  bitmap_copy (combined_blocks, chkcall_blocks);
+  int i;
+  edge *e;
+  FOR_EACH_VEC_ELT (chk_edges, i, e)
+    if (!bitmap_set_bit (combined_blocks, (*e)->src->index))
+      gcc_unreachable ();
+
+  /* Visit blocks in index order, because building rtcfg depends on
+     that.  Blocks must be compact, which the cleanup_cfg requirement
+     ensures.  This would also enable FOR_EACH_BB_FN to be used to
+     iterate in index order, but bb_eh_cleanup block splits and
+     insertions changes that.  */
+  gcc_checking_assert (n_basic_blocks_for_fn (fun)
+		       == last_basic_block_for_fn (fun));
+  for (int i = NUM_FIXED_BLOCKS; i < n_basic_blocks_for_fn (fun); i++)
+    {
+      bb = BASIC_BLOCK_FOR_FN (fun, i);
+      gcc_checking_assert (bb->index == i);
+      vstd.visit (bb, bitmap_bit_p (combined_blocks, i),
+		  bitmap_bit_p (postchk_blocks, i));
+    }
+
+  vstd.check (chk_edges, count_chkcall, chkcall_blocks);
+
+  return
+    TODO_update_ssa
+    | TODO_cleanup_cfg
+    | TODO_verify_il;
 }
 
 /* Instantiate a hardcfr pass.  */
diff --git a/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c b/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c
new file mode 100644
index 00000000000..a6992eb9f8e
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -O0 -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we don't insert checking before noreturn calls.  -O0 is tested
+   separately because h is not found to be noreturn without optimization.  */
+
+#include "torture/harden-cfr-noret.c"
+
+/* No out-of-line checks.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* Only one inline check at the end of f and of h2.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c
new file mode 100644
index 00000000000..8bd2d13ac18
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c
@@ -0,0 +1,18 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we don't insert checking before noreturn calls.  -O0 is tested
+   separately because h is not found to be noreturn without optimization, which
+   affects codegen for h2, so h2 is omitted here at -O0.  */
+
+#if !__OPTIMIZE__
+# define OMIT_H2
+#endif
+
+#include "harden-cfr-noret.c"
+
+
+/* No out-of-line checks.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* Only one inline check at the end of f.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c
new file mode 100644
index 00000000000..a804a6cfe59
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fno-exceptions -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that -fno-exceptions makes for implicit nothrow in noreturn
+   handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#include "harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c
new file mode 100644
index 00000000000..f390cfdbc59
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checking before nothrow noreturn calls.  */
+
+#include "harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c
new file mode 100644
index 00000000000..fdd803109a4
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c
@@ -0,0 +1,38 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checking before all noreturn calls.  */
+
+#ifndef ATTR_NOTHROW_OPT /* Overridden in harden-cfr-noret-noexcept.  */
+#define ATTR_NOTHROW_OPT __attribute__ ((__nothrow__))
+#endif
+
+extern void __attribute__ ((__noreturn__)) ATTR_NOTHROW_OPT g (void);
+
+void f(int i) {
+  if (i)
+    /* Out-of-line checks here...  */
+    g ();
+  /* ... and here.  */
+}
+
+void __attribute__ ((__noinline__, __noclone__))
+h(void) {
+  /* Inline check here.  */
+  g ();
+}
+
+#ifndef OMIT_H2 /* from harden-cfr-noret-never.  */
+void h2(void) {
+  /* Inline check either here, whether because of noreturn or tail call...  */
+  h ();
+  /* ... or here, if not optimizing.  */
+}
+#endif
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c
new file mode 100644
index 00000000000..6d11487bbba
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c
@@ -0,0 +1,8 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+#include "harden-cfr-tail.c"
+
+/* Inline checking after the calls, disabling tail calling.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 5 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 0 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c
new file mode 100644
index 00000000000..550b02ca088
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-returning.c
@@ -0,0 +1,35 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-exceptions -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checks before returning calls and alternate paths, even
+   at -O0, because of the explicit command-line flag.  */
+
+void g (void);
+void g2 (void);
+void g3 (void);
+
+void f (int i) {
+  if (!i)
+    /* Out-of-line checks here...  */
+    g ();
+  else if (i > 0)
+    /* here...  */
+    g2 ();
+  /* else */
+    /* and in the implicit else here.  */
+}
+
+void f2 (int i) {
+  if (!i)
+    /* Out-of-line check here...  */
+    g ();
+  else if (i > 0)
+    /* here...  */
+    g2 ();
+  else
+    /* and here.  */
+    g3 ();
+}
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 6 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 0 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
index 40d76c5c163..d5467eafa9f 100644
--- a/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
@@ -1,17 +1,52 @@
 /* { dg-do compile } */
-/* { dg-options "-fharden-control-flow-redundancy -fdump-tree-hardcfr -ffat-lto-objects" } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-hardcfr-check-exceptions -fdump-tree-hardcfr -ffat-lto-objects -Wno-return-type" } */
 
-/* We'd like to check that we insert checking so as to not disrupt tail calls,
-   but unfortunately mandatory tail calls are not available in C, and optimizing
-   calls as tail calls only takes place after hardcfr.  */
+/* Check that we insert CFR checking so as to not disrupt tail calls.
+   Mandatory tail calls are not available in C, and optimizing calls as tail
+   calls only takes place after hardcfr, so we insert checking before calls
+   followed by copies and return stmts with the same return value, that might
+   (or might not) end up optimized to tail calls.  */
 
-extern int g(int i);
+extern int g (int i);
 
-int f(int i) {
+int f1(int i) {
+  /* Inline check before the returning call.  */
   return g (i);
 }
 
-/* Inline checking before the tail call.  */
-/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
-/* Inline checking before the tail call.  */
-/* { dg-final { scan-tree-dump-times "\\\[tail\]" 1 "hardcfr" { xfail *-*-* } } } */
+extern void g2 (int i);
+
+void f2(int i) {
+  /* Inline check before the returning call, that ignores the returned value,
+     matching the value-less return.  */
+  g2 (i);
+  return;
+}
+
+void f3(int i) {
+  /* Inline check before the returning call.  */
+  g (i);
+}
+
+void f4(int i) {
+  if (i)
+    /* Out-of-line check before the returning call.  */
+    return g2 (i);
+  /* Out-of-line check before implicit return.  */
+}
+
+int f5(int i) {
+  /* Not regarded as a returning call, returning value other than callee's
+     returned value.  */
+  g (i);
+  /* Inline check after the non-returning call.  */
+  return i;
+}
+
+/* Out-of-line checks in f4, before returning calls and before return.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking in all other functions.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 4 "hardcfr" } } */
+/* Check before tail-call in all but f5, but f4 is out-of-line.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 3 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C b/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C
new file mode 100644
index 00000000000..17ea79f7cfb
--- /dev/null
+++ b/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C
@@ -0,0 +1,11 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects -O0" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions, and also checking before noreturn
+   calls.  h2 and h2b get an extra resx without ehcleanup.  */
+
+#include "torture/harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 16 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C b/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C
new file mode 100644
index 00000000000..f0338ccc361
--- /dev/null
+++ b/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C
@@ -0,0 +1,10 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -foptimize-sibling-calls -fdump-tree-hardcfr -O0" } */
+
+/* -fhardcfr-check-returning-calls gets implicitly disabled because,
+   -at O0, -foptimize-sibling-calls has no effect.  */
+
+#include "torture/harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 12 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C
new file mode 100644
index 00000000000..0d35920c7ee
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  */
+
+#include "harden-cfr-noret-no-nothrow.C"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Bypassing cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C
new file mode 100644
index 00000000000..b7d247ff43c
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C
@@ -0,0 +1,18 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  Expected results for =never and =nothrow are the same,
+   since the functions are not nothrow.  */
+
+#include "harden-cfr-noret-no-nothrow.C"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Associated cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C
new file mode 100644
index 00000000000..62c58cfd406
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C
@@ -0,0 +1,23 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#if ! __OPTIMIZE__
+void __attribute__ ((__noreturn__)) h (void);
+#endif
+
+#include "../../c-c++-common/torture/harden-cfr-noret.c"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Associated cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C
new file mode 100644
index 00000000000..0286f6e6d3f
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C
@@ -0,0 +1,20 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions, and also checking before noreturn
+   calls.  */
+
+#if ! __OPTIMIZE__
+/* Without optimization, functions with cleanups end up with an extra
+   resx that is not optimized out, so arrange to optimize them.  */
+void __attribute__ ((__optimize__ (1))) h2(void);
+void __attribute__ ((__optimize__ (1))) h2b(void);
+#endif
+
+#include "harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 14 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
+/* h, h2, h2b, and h4.  */
+/* { dg-final { scan-tree-dump-times "Bypassing" 4 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C
new file mode 100644
index 00000000000..885b0b236af
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C
@@ -0,0 +1,11 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions  -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we do not insert cleanups for checking around the bodies
+   of maybe-throwing functions.  h4 doesn't get any checks, because we
+   don't have noreturn checking enabled.  */
+
+#include "harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 6 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C
new file mode 100644
index 00000000000..32def637255
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C
@@ -0,0 +1,31 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -foptimize-sibling-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions.  These results depend on checking before
+   returning calls, which is only enabled when sibcall optimizations
+   are enabled, so change the optimization mode to -O1 for f and f2,
+   so that -foptimize-sibling-calls can take effect and enable
+   -fhardcfr-check-returning-calls, so that we get the same results.
+   There is a separate test for -O0.  */
+
+#if ! __OPTIMIZE__
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) f(int i);
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) f2(int i);
+void __attribute__ ((__optimize__ (1, "-foptimize-sibling-calls"))) h3(void);
+#endif
+
+#include "harden-cfr-throw.C"
+
+/* f gets out-of-line checks before the unwrapped tail call and in the
+   else edge.  */
+/* f2 gets out-of-line checks before both unwrapped tail calls.  */
+/* h gets out-of-line checks before the implicit return and in the
+   cleanup block.  */
+/* h2 and h2b get out-of-line checks before the cleanup returning
+   call, and in the cleanup block.  */
+/* h3 gets an inline check before the __cxa_end_catch returning call.  */
+/* h4 gets an inline check in the cleanup block.  */
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 10 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C
new file mode 100644
index 00000000000..992fbdad381
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C
@@ -0,0 +1,65 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions.  */
+
+extern void g (void);
+extern void g2 (void);
+
+void f(int i) {
+  if (i)
+    g ();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void f2(int i) {
+  if (i)
+    g ();
+  else
+    g2 ();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h(void) {
+  try {
+    g ();
+  } catch (...) {
+    throw;
+  }
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+struct needs_cleanup {
+  ~needs_cleanup();
+};
+
+void h2(void) {
+  needs_cleanup y; /* No check in the cleanup handler.  */
+  g();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+extern void __attribute__ ((__nothrow__)) another_cleanup (void*);
+
+void h2b(void) {
+  int x __attribute__ ((cleanup (another_cleanup)));
+  g();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h3(void) {
+  try {
+    throw 1;
+  } catch (...) {
+  }
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h4(void) {
+  throw 1;
+  /* Inline check in the cleanup around the __cxa_throw noreturn call.  */
+}
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 12 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c b/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c
new file mode 100644
index 00000000000..8e4ee1fab08
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c
@@ -0,0 +1,15 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C makes for implicit nothrow in noreturn handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#include "../../c-c++-common/torture/harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c b/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c
new file mode 100644
index 00000000000..634d98f1ffc
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c
@@ -0,0 +1,40 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-hardcfr-check-exceptions -fdump-tree-hardcfr -ffat-lto-objects -Wno-return-type" } */
+
+/* In C only, check some additional cases (comparing with
+   c-c++-common/torture/harden-cfr-tail.c) of falling off the end of non-void
+   function.  C++ would issue an unreachable call in these cases.  */
+
+extern int g (int i);
+
+int f1(int i) {
+  /* Inline check before the returning call, that doesn't return anything.  */
+  g (i);
+  /* Implicit return without value, despite the return type; this combination
+     enables tail-calling of g, and is recognized as a returning call.  */
+}
+
+extern void g2 (int i);
+
+int f2(int i) {
+  /* Inline check before the returning call, that disregards its return
+     value.  */
+  g2 (i);
+  /* Implicit return without value, despite the return type; this combination
+     enables tail-calling of g2, and is recognized as a returning call.  */
+}
+
+int f3(int i) {
+  if (i)
+    /* Out-of-line check before the returning call.  */
+    return g (i);
+  /* Out-of-line check before implicit return.  */
+}
+
+/* Out-of-line checks in f3, before returning calls and before return.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking in all other functions.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
+/* Check before tail-call in all functions, but f3 is out-of-line.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/gnat.dg/hardcfr.adb b/gcc/testsuite/gnat.dg/hardcfr.adb
index b578a913d75..abe1605c029 100644
--- a/gcc/testsuite/gnat.dg/hardcfr.adb
+++ b/gcc/testsuite/gnat.dg/hardcfr.adb
@@ -1,5 +1,5 @@
 --  { dg-do run }
---  { dg-options "-fharden-control-flow-redundancy -fdump-tree-hardcfr --param=hardcfr-max-blocks=22 --param=hardcfr-max-inline-blocks=12 -O0" }
+--  { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions -fdump-tree-hardcfr --param=hardcfr-max-blocks=22 --param=hardcfr-max-inline-blocks=12 -O0" }
 
 procedure HardCFR is
    function F (I, J : Integer) return Integer is
diff --git a/libgcc/hardcfr.c b/libgcc/hardcfr.c
index 8ef29428111..55f6b995f2f 100644
--- a/libgcc/hardcfr.c
+++ b/libgcc/hardcfr.c
@@ -60,14 +60,25 @@ extern void __hardcfr_check (size_t blocks,
 			     vword const *visited,
 			     vword const *cfg);
 
+/* Compute the MASK for the bit representing BLOCK in WORDIDX's vword in a
+   visited blocks bit array.  */
+static inline void
+block2mask (size_t const block, vword *const mask, size_t *const wordidx)
+{
+  size_t wbits = __CHAR_BIT__ * sizeof (vword);
+  *wordidx = block / wbits;
+  *mask = (vword)1 << (block % wbits);
+}
 
 /* Check whether the bit corresponding to BLOCK is set in VISITED.  */
 static inline bool
 visited_p (size_t const block, vword const *const visited)
 {
-  size_t wbits = __CHAR_BIT__ * sizeof (vword);
-  vword w = visited[block / wbits];
-  return (w & ((vword)1 << (block % wbits))) != 0;
+  vword mask;
+  size_t wordidx;
+  block2mask (block, &mask, &wordidx);
+  vword w = visited[wordidx];
+  return (w & mask) != 0;
 }
 
 /* Read and consume a mask from **CFG_IT.  (Consume meaning advancing the
@@ -118,19 +129,19 @@ consume_seq (vword const **const cfg_it)
    we reach the terminator without finding any.  Consume the entire sequence
    otherwise, so that *CFG_IT points just past the terminator, which may be the
    beginning of the next sequence.  */
-static inline void
+static inline bool
 check_seq (vword const *const visited, vword const **const cfg_it)
 {
   vword mask;
   size_t wordidx;
 
   /* If the block was visited, check that at least one of the
-     preds was also visited.  */
+     preds/succs was also visited.  */
   do
     /* If we get to the end of the sequence without finding any
        match, something is amiss.  */
     if (!next_pair (cfg_it, &mask, &wordidx))
-      __builtin_trap ();
+      return false;
   /* Keep searching until we find a match, at which point the
      condition is satisfied.  */
   while (!test_mask (visited, mask, wordidx));
@@ -139,6 +150,94 @@ check_seq (vword const *const visited, vword const **const cfg_it)
      skipped the block, so as to position the iterator at the beginning of the
      next .  */
   consume_seq (cfg_it);
+
+  return true;
+}
+
+/* Print out the CFG with BLOCKS blocks, presumed to be associated with CALLER.
+   This is expected to be optimized out entirely, unless the verbose part of
+   __hardcfr_check_fail is enabled.  */
+static inline void
+__hardcfr_debug_cfg (size_t const blocks,
+		     void const *const caller,
+		     vword const *const cfg)
+{
+  __builtin_printf ("CFG at %p, for %p", cfg, caller);
+  vword const *cfg_it = cfg;
+  for (size_t i = 0; i < blocks; i++)
+    {
+      vword mask; size_t wordidx;
+      block2mask (i, &mask, &wordidx);
+      __builtin_printf ("\nblock %lu (%lu/0x%lx)\npreds: ",
+			(unsigned long)i,
+			(unsigned long)wordidx, (unsigned long)mask);
+      while (next_pair (&cfg_it, &mask, &wordidx))
+	__builtin_printf (" (%lu/0x%lx)",
+			  (unsigned long)wordidx, (unsigned long)mask);
+      __builtin_printf ("\nsuccs: ");
+      while (next_pair (&cfg_it, &mask, &wordidx))
+	__builtin_printf (" (%lu/0x%lx)",
+			  (unsigned long)wordidx, (unsigned long)mask);
+    }
+  __builtin_printf ("\n");
+}
+
+#ifndef ATTRIBUTE_UNUSED
+# define ATTRIBUTE_UNUSED __attribute__ ((__unused__))
+#endif
+
+/* This is called when an out-of-line hardcfr check fails.  All the arguments
+   are ignored, and it just traps, unless HARDCFR_VERBOSE_FAIL is enabled.  IF
+   it is, it prints the PART of the CFG, expected to have BLOCKS blocks, that
+   failed at CALLER's BLOCK, and the VISITED bitmap.  When the verbose mode is
+   enabled, it also forces __hardcfr_debug_cfg (above) to be compiled into an
+   out-of-line function, that could be called from a debugger.
+   */
+static inline void
+__hardcfr_check_fail (size_t const blocks ATTRIBUTE_UNUSED,
+		      vword const *const visited,
+		      vword const *const cfg ATTRIBUTE_UNUSED,
+		      size_t const block ATTRIBUTE_UNUSED,
+		      int const part ATTRIBUTE_UNUSED,
+		      void const *const caller ATTRIBUTE_UNUSED)
+{
+#if HARDCFR_VERBOSE_FAIL
+  static const char *parts[] = { "preds", "succs" };
+
+  vword mask; size_t wordidx;
+  block2mask (block, &mask, &wordidx);
+  __builtin_printf ("hardcfr fail at %p block %lu (%lu/0x%lx), expected %s:",
+		    caller, (unsigned long)block,
+		    (unsigned long)wordidx, (unsigned long)mask,
+		    parts[part]);
+
+  /* Skip data for previous blocks.  */
+  vword const *cfg_it = cfg;
+  for (size_t i = block; i--; )
+    {
+      consume_seq (&cfg_it);
+      consume_seq (&cfg_it);
+    }
+  for (size_t i = part; i--; )
+    consume_seq (&cfg_it);
+
+  while (next_pair (&cfg_it, &mask, &wordidx))
+    __builtin_printf (" (%lu/0x%lx)",
+		      (unsigned long)wordidx, (unsigned long)mask);
+
+  __builtin_printf ("\nvisited:");
+  block2mask (blocks, &mask, &wordidx);
+  for (size_t i = 0; i <= wordidx; i++)
+    __builtin_printf (" (%lu/0x%lx)",
+		      (unsigned long)i, (unsigned long)visited[i]);
+  __builtin_printf ("\n");
+
+  /* Reference __hardcfr_debug_cfg so that it's output out-of-line, so that it
+     can be called from a debugger.  */
+  if (!caller || caller == __hardcfr_debug_cfg)
+    return;
+#endif
+  __builtin_trap ();
 }
 
 /* Check that, for each of the BLOCKS basic blocks, if its bit is set in
@@ -168,9 +267,13 @@ __hardcfr_check (size_t const blocks,
       else
 	{
 	  /* Check predecessors.  */
-	  check_seq (visited, &cfg_it);
+	  if (!check_seq (visited, &cfg_it))
+	    __hardcfr_check_fail (blocks, visited, cfg, i, 0,
+				  __builtin_return_address (0));
 	  /* Check successors.  */
-	  check_seq (visited, &cfg_it);
+	  if (!check_seq (visited, &cfg_it))
+	    __hardcfr_check_fail (blocks, visited, cfg, i, 1,
+				  __builtin_return_address (0));
 	}
     }
 }

^ permalink raw reply	[flat|nested] 13+ messages in thread
* [gcc(refs/users/aoliva/heads/testme)] hardcfr: add optional checkpoints
@ 2022-09-06  1:28 Alexandre Oliva
  0 siblings, 0 replies; 13+ messages in thread
From: Alexandre Oliva @ 2022-09-06  1:28 UTC (permalink / raw)
  To: gcc-cvs

https://gcc.gnu.org/g:a24f0b8b80664c56851227679ca1e5c2e3777779

commit a24f0b8b80664c56851227679ca1e5c2e3777779
Author: Alexandre Oliva <oliva@adacore.com>
Date:   Fri Sep 2 20:33:12 2022 -0300

    hardcfr: add optional checkpoints
    
    Previously, control flow redundancy only checked the visited bitmap
    against the control flow graph at return points and before mandatory
    tail calls, missing various other possibilities of exiting a
    subprogram, such as by raising or propagating exceptions, and calling
    noreturn functions.  The checks inserted before returns also prevented
    potential tail-call optimizations.
    
    This incremental change introduces options to control checking at each
    of these previously-missed checkpoints.  Unless disabled, a cleanup is
    introduced to check when an exceptions escapes a subprogram.  To avoid
    disrupting sibcall optimizations, when they are enabled, checks are
    introduced before calls whose results are immediately returned,
    whether or not they are ultimately optimized.  If enabled, checks are
    introduced before noreturn calls and exception raises, or only before
    nothrow noreturn calls.
    
    Add examples of code transformations to the GNAT RM.
    
    
    for  gcc/ada/ChangeLog
    
            * doc/gnat_rm/security_hardening_features.rst: Document
            optional hardcfr checkpoints.
    
    for  gcc/ChangeLog
    
            * common.opt (-fhardcfr-check-returning-calls): New.
            (-fhardcfr-check-exceptions): New.
            (-fhardcfr-check-noreturn-calls=*): New.  (Enum
            hardcfr_check_noreturn_calls): New.  * doc/invoke.texi:
            Document them.  * flag-types.h (enum hardcfr_noret): New.  *
            gimple-harden-control-flow.cc: Include more headers.
            (pass_data_harden_control_flow_redundancy): Move
            properties_finish to pass return.
            (pass_harden_control_flow_redundancy::gate): Move
            no-edge-to-exit checking to execute, after exception
            transformations.  Use symbolic NUM_FIXED_BLOCKS.  Test for CFG
            before testing block count.  (check_returning_calls_p): New.
            (hardcfr_scan_block): New.  (returnign_call_p): New.
            (chk_edges_t): New.  (hardcfr_sibcall_search_block): New.
            (hardcfr_sibcall_search_preds): New.
            (rt_bb_visited::num2idx): Use symbolic NUM_FIXED_BLOCKS.
            (rt_bb_visited::rt_bb_visited): Likewise.  Take checkpoints
            count, test it to select out-of-line checks.
            (rt_bb_visited::insert_exit_check): Removed.
            (rt_bb_visited::insert_exit_check_in_blocks): New.
            (rt_bb_visited::insert_exit_check_on_edge): New.
            (rt_bb_visited::check): Take checkpoint edges and blocks.
            Move initialization insertion to the end.  Insert checks on
            edges, and before calls in select blocks.  Clobber the visited
            array to the check sequence.  Output actions to dump_file.
            (rt_bb_visited::visit): Take checkpoint and postcheck.  Assume
            an edge to exit if checkpoint, and skip testing if postcheck.
            (always_throwing_noreturn_call_p): New, dummy.
            (pass_harden_control_flow_redundancy::execute): Introduce
            cleanup block if requested and needed.  Scan blocks for
            noreturn and returning calls if checkpoints before them were
            requested.  Log actions to dump_file.  Visit blocks in index
            order.
    
    for  libgcc/ChangeLog
    
            * hardcfr.c (block2mask): Split out of...
            (visited_p): ... this.
            (check_seq): Move trapping out, return result instead.
            (__hardcfr_debug_cfg): New.
            (__hardcfr_check_fail): New, with optional verbose error
            printing before trapping.
            (__hardcfr_check): Call __hardcfr_check_fail when check_seq
            fails.
    
    for  gcc/testsuite/ChangeLog
    
            * c-c++-common/harden-cfr-noret-never-O0.c: New.
            * c-c++-common/torture/harden-cfr-noret-never.c: New.
            * c-c++-common/torture/harden-cfr-noret-noexcept.c: New.
            * c-c++-common/torture/harden-cfr-noret-nothrow.c: New.
            * c-c++-common/torture/harden-cfr-noret.c: New.
            * c-c++-common/torture/harden-cfr-notail.c: New.
            * c-c++-common/torture/harden-cfr-tail.c: Extend.
            * g++.dg/harden-cfr-throw-always-O0.C: New.
            * g++.dg/harden-cfr-throw-returning-O0.C: New.
            * g++.dg/torture/harden-cfr-noret-always-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-noret-never-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-noret-no-nothrow.C: New.
            * g++.dg/torture/harden-cfr-throw-always.C: New.
            * g++.dg/torture/harden-cfr-throw-nocleanup.C: New.
            * g++.dg/torture/harden-cfr-throw-returning.C: New.
            * g++.dg/torture/harden-cfr-throw.C: New.
            * gcc.dg/torture/harden-cfr-noret-no-nothrow.c: New.
            * gcc.dg/torture/harden-cfr-tail-ub.c: New.
            * gnat.dg/hardcfr.adb: Disable checking at exceptions.

Diff:
---
 .../doc/gnat_rm/security_hardening_features.rst    | 126 ++-
 gcc/common.opt                                     |  33 +
 gcc/doc/invoke.texi                                |  74 +-
 gcc/flag-types.h                                   |  10 +
 gcc/gimple-harden-control-flow.cc                  | 916 ++++++++++++++++++---
 .../c-c++-common/harden-cfr-noret-never-O0.c       |  12 +
 .../c-c++-common/torture/harden-cfr-noret-never.c  |  18 +
 .../torture/harden-cfr-noret-noexcept.c            |  16 +
 .../torture/harden-cfr-noret-nothrow.c             |  13 +
 .../c-c++-common/torture/harden-cfr-noret.c        |  38 +
 .../c-c++-common/torture/harden-cfr-notail.c       |   8 +
 .../c-c++-common/torture/harden-cfr-tail.c         |  55 +-
 gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C  |  11 +
 .../g++.dg/harden-cfr-throw-returning-O0.C         |  10 +
 .../torture/harden-cfr-noret-always-no-nothrow.C   |  16 +
 .../torture/harden-cfr-noret-never-no-nothrow.C    |  18 +
 .../g++.dg/torture/harden-cfr-noret-no-nothrow.C   |  23 +
 .../g++.dg/torture/harden-cfr-throw-always.C       |  20 +
 .../g++.dg/torture/harden-cfr-throw-nocleanup.C    |  11 +
 .../g++.dg/torture/harden-cfr-throw-returning.C    |  31 +
 gcc/testsuite/g++.dg/torture/harden-cfr-throw.C    |  65 ++
 .../gcc.dg/torture/harden-cfr-noret-no-nothrow.c   |  15 +
 gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c  |  40 +
 gcc/testsuite/gnat.dg/hardcfr.adb                  |   2 +-
 libgcc/hardcfr.c                                   | 119 ++-
 25 files changed, 1563 insertions(+), 137 deletions(-)

diff --git a/gcc/ada/doc/gnat_rm/security_hardening_features.rst b/gcc/ada/doc/gnat_rm/security_hardening_features.rst
index f5fdc8e46b4..4dfda486795 100644
--- a/gcc/ada/doc/gnat_rm/security_hardening_features.rst
+++ b/gcc/ada/doc/gnat_rm/security_hardening_features.rst
@@ -263,11 +263,127 @@ For each block that is marked as visited, the mechanism checks that at
 least one of its predecessors, and at least one of its successors, are
 also marked as visited.
 
-Verification is performed just before returning.  Subprogram
-executions that complete by raising or propagating an exception bypass
-verification-and-return points.  A subprogram that can only complete
-by raising or propagating an exception may have instrumentation
-disabled altogether.
+Verification is performed just before a subprogram returns.  The
+following fragment:
+
+.. code-block:: ada
+
+   if X then
+     Y := F (Z);
+     return;
+   end if;
+
+
+gets turned into:
+
+.. code-block:: ada
+
+   type Visited_Bitmap is array (1..N) of Boolean with Pack;
+   Visited : aliased Visited_Bitmap := (others => False);
+   --  Bitmap of visited blocks.  N is the basic block count.
+   [...]
+   --  Basic block #I
+   Visited(I) := True;
+   if X then
+     --  Basic block #J
+     Visited(J) := True;
+     Y := F (Z);
+     CFR.Check (N, Visited'Access, CFG'Access);
+     --  CFR is a hypothetical package whose Check procedure calls
+     --  libgcc's __hardcfr_check, that traps if the Visited bitmap
+     --  does not hold a valid path in CFG, the run-time
+     --  representation of the control flow graph in the enclosing
+     --  subprogram.
+     return;
+   end if;
+   --  Basic block #K
+   Visited(K) := True;
+
+
+Verification would also be performed before must-tail calls, and
+before early-marked potential tail calls, but these do not occur in
+practice, as potential tail-calls are only detected in late
+optimization phases, too late for this transformation to act on it.
+
+In order to avoid adding verification after potential tail calls,
+which would prevent tail-call optimization, we recognize returning
+calls, i.e., calls whose result, if any, is returned by the calling
+subprogram to its caller immediately after the call returns.
+Verification is performed before such calls, whether or not they are
+ultimately optimized to tail calls.  This behavior is enabled by
+default whenever sibcall optimization is enabled (see
+:switch:`-foptimize-sibling-calls`); it may be disabled with
+:switch:`-fno-hardcfr-check-returning-calls`, or enabled with
+:switch:`-fhardcfr-check-returning-calls`, regardless of the
+optimization, but the lack of other optimizations may prevent calls
+from being recognized as returning calls:
+
+.. code-block:: ada
+
+     --  CFR.Check here, with -fhardcfr-check-returning-calls.
+     P (X);
+     --  CFR.Check here, with -fno-hardcfr-check-returning-calls.
+     return;
+
+or:
+
+.. code-block:: ada
+
+     --  CFR.Check here, with -fhardcfr-check-returning-calls.
+     R := F (X);
+     --  CFR.Check here, with -fno-hardcfr-check-returning-calls.
+     return R;
+
+
+Any subprogram from which an exception may escape, i.e., that may
+raise or propagate an exception that isn't handled internally, is
+conceptually enclosed by a cleanup handler that performs verification,
+unless this is disabled with :switch:`-fno-hardcfr-check-exceptions`.
+With this feature enabled, a subprogram body containing:
+
+.. code-block:: ada
+
+     --  ...
+       Y := F (X);  -- May raise exceptions.
+     --  ...
+       raise E;  -- Not handled internally.
+     --  ...
+
+
+gets modified as follows:
+
+.. code-block:: ada
+
+   begin
+     --  ...
+       Y := F (X);  -- May raise exceptions.
+     --  ...
+       raise E;  -- Not handled internally.
+     --  ...
+   exception
+     when others =>
+       CFR.Check (N, Visited'Access, CFG'Access);
+       raise;
+   end;
+
+
+Verification may also be performed before No_Return calls, whether
+only nothrow ones, with
+:switch:`-fhardcfr-check-noreturn-calls=nothrow`, or all of them, with
+:switch:`-fhardcfr-check-noreturn-calls=always`.  The default is
+:switch:`-fhardcfr-check-noreturn-calls=never` for this feature, that
+disables checking before No_Return calls.
+
+When a No_Return call returns control to its caller through an
+exception, verification may have already been performed before the
+call, if :switch:`-fhardcfr-check-noreturn-calls=always` is in effect.
+The compiler arranges for already-checked No_Return calls without a
+preexisting handler to bypass the implicitly-added cleanup handler and
+thus the redundant check, but a local exception or cleanup handler, if
+present, will modify the set of visited blocks, and checking will take
+place again when the caller reaches the next verification point,
+whether it is a return or reraise statement after the exception is
+otherwise handled, or even another No_Return call.
 
 The instrumentation for hardening with control flow redundancy can be
 observed in dump files generated by the command-line option
diff --git a/gcc/common.opt b/gcc/common.opt
index 661dcf2f485..983cb4db7c6 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -1795,6 +1795,39 @@ fharden-control-flow-redundancy
 Common Var(flag_harden_control_flow_redundancy) Optimization
 Harden control flow by recording and checking execution paths.
 
+fhardcfr-check-returning-calls
+Common Var(flag_harden_control_flow_redundancy_check_returning_calls) Init(-1) Optimization
+Check CFR execution paths also before calls followed by returns of their results.
+
+fhardcfr-check-exceptions
+Common Var(flag_harden_control_flow_redundancy_check_exceptions) Init(-1) Optimization
+Check CFR execution paths also when exiting a function through an exception.
+
+fhardcfr-check-noreturn-calls=
+Common Joined RejectNegative Enum(hardcfr_check_noreturn_calls) Var(flag_harden_control_flow_redundancy_check_noreturn) Init(HCFRNR_UNSPECIFIED) Optimization
+-fhardcfr-check-noreturn-calls=[always|nothrow|never]	Check CFR execution paths also before calling noreturn functions.
+
+Enum
+Name(hardcfr_check_noreturn_calls) Type(enum hardcfr_noret) UnknownError(unknown hardcfr noreturn checking level %qs)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(never) Value(HCFRNR_NEVER)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(nothrow) Value(HCFRNR_NOTHROW)
+
+; ??? There could be yet another option here, that checked before
+; noreturn calls, except for those known to always throw, if we had
+; means to distinguish noreturn functions known to always throw, such
+; as those used to (re)raise exceptions, from those that merely might
+; throw.  "not always" stands for "not always-throwing", but it also
+; contrasts with "always" below.
+; EnumValue
+; Enum(hardcfr_check_noreturn_calls) String(not-always) Value(HCFRNR_NOT_ALWAYS)
+
+EnumValue
+Enum(hardcfr_check_noreturn_calls) String(always) Value(HCFRNR_ALWAYS)
+
 ; Nonzero means ignore `#ident' directives.  0 means handle them.
 ; Generate position-independent code for executables if possible
 ; On SVR4 targets, it also controls whether or not to emit a
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index 3bec4b42437..ae324b9256b 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -625,7 +625,9 @@ Objective-C and Objective-C++ Dialects}.
 -fsanitize-undefined-trap-on-error  -fbounds-check @gol
 -fcf-protection=@r{[}full@r{|}branch@r{|}return@r{|}none@r{|}check@r{]} @gol
 -fharden-compares -fharden-conditional-branches @gol
--fharden-control-flow-redundancy @gol
+-fharden-control-flow-redundancy  -fhardcfr-check-exceptions  @gol
+-fhardcfr-check-returning-calls  @gol
+-fhardcfr-check-noreturn-calls=@r{[}always@r{|}nothrow@r{|}never@r{]}  @gol
 -fstack-protector  -fstack-protector-all  -fstack-protector-strong @gol
 -fstack-protector-explicit  -fstack-check @gol
 -fstack-limit-register=@var{reg}  -fstack-limit-symbol=@var{sym} @gol
@@ -16565,11 +16567,75 @@ conditionals.
 @item -fharden-control-flow-redundancy
 @opindex fharden-control-flow-redundancy
 Emit extra code to set booleans when entering basic blocks, and to
-verify, at function exits, that they amount to an execution path that is
-consistent with the control flow graph, trapping otherwise.  Tuning
-options @option{--param hardcfr-max-blocks} and @option{--param
+verify and trap, at function exits, when the booleans do not form an
+execution path that is compatible with the control flow graph.
+
+Verification takes place before returns, before mandatory tail calls
+(see below) and, optionally, before escaping exceptions with
+@option{-fhardcfr-check-exceptions}, before returning calls with
+@option{-fhardcfr-check-returning-calls}, and before noreturn calls with
+@option{-fhardcfr-check-noreturn-calls}).  Tuning options
+@option{--param hardcfr-max-blocks} and @option{--param
 hardcfr-max-inline-blocks} are available.
 
+Tail call optimization takes place too late to affect control flow
+redundancy, but calls annotated as mandatory tail calls by language
+front-ends, and any calls marked early enough as potential tail calls
+would also have verification issued before the call, but these
+possibilities are merely theoretical, as these conditions can only be
+met when using custom compiler plugins.
+
+@item -fhardcfr-check-exceptions
+@opindex fhardcfr-check-exceptions
+@opindex fno-hardcfr-check-exceptions
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph at exception
+escape points, as if the function body was wrapped with a cleanup
+handler that performed the check and reraised.  This option is enabled
+by default; use @option{-fno-hardcfr-check-exceptions} to disable it.
+
+@item -fhardcfr-check-returning-calls
+@opindex fhardcfr-check-returning-calls
+@opindex fno-hardcfr-check-returning-calls
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph before any
+function call immediately followed by a return of its result, if any, so
+as to not prevent tail-call optimization, whether or not it is
+ultimately optimized to a tail call.
+
+This option is enabled by default whenever sibling call optimizations
+are enabled (see @option{-foptimize-sibling-calls}), but it can be
+enabled (or disabled, using its negated form) explicitly, regardless of
+the optimizations.
+
+@item -fhardcfr-check-noreturn-calls=@r{[}always@r{|}nothrow@r{|}never@r{]}
+@opindex fhardcfr-check-noreturn-calls
+When @option{-fharden-control-flow-redundancy} is active, check the
+recorded execution path against the control flow graph before
+@code{noreturn} calls, either all of them (@option{always}), those that
+may not return control to the caller through an exception either
+(@option{nothrow}), or none of them (@option{never}, the default).
+
+Checking before a @code{noreturn} function that may return control to
+the caller through an exception may cause checking to be performed more
+than once, if the exception is caught in the caller, whether by a
+handler or a cleanup.  When @option{-fhardcfr-check-exceptions} is also
+enabled, the compiler will avoid associating a @code{noreturn} call with
+the implicitly-added cleanup handler, since it would be redundant with
+the check performed before the call, but other handlers or cleanups in
+the function, if activated, will modify the recorded execution path and
+check it again when another checkpoint is hit.  The checkpoint may even
+be another @code{noreturn} call, so checking may end up performed
+multiple times.
+
+Various optimizers may cause calls to be marked as @code{noreturn}
+and/or @code{nothrow}, even in the absence of the corresponding
+attributes, which may affect the placement of checks before calls, as
+well as the addition of implicit cleanup handlers for them.  This
+unpredictability, and the fact that raising and reraising exceptions
+frequently amounts to implicitly calling @code{noreturn} functions, have
+made @option{never} the default setting for this option.
+
 @item -fstack-protector
 @opindex fstack-protector
 Emit extra code to check for buffer overflows, such as stack smashing
diff --git a/gcc/flag-types.h b/gcc/flag-types.h
index d2e751060ff..3fae7548cab 100644
--- a/gcc/flag-types.h
+++ b/gcc/flag-types.h
@@ -157,6 +157,16 @@ enum stack_reuse_level
   SR_ALL
 };
 
+/* Control Flow Redundancy hardening options for noreturn calls.  */
+enum hardcfr_noret
+{
+  HCFRNR_NEVER,
+  HCFRNR_NOTHROW,
+  HCFRNR_NOT_ALWAYS, /* Reserved for future use.  */
+  HCFRNR_ALWAYS,
+  HCFRNR_UNSPECIFIED = -1
+};
+
 /* The live patching level.  */
 enum live_patching_level
 {
diff --git a/gcc/gimple-harden-control-flow.cc b/gcc/gimple-harden-control-flow.cc
index 6b08846dbb1..1c93bf622e8 100644
--- a/gcc/gimple-harden-control-flow.cc
+++ b/gcc/gimple-harden-control-flow.cc
@@ -29,7 +29,12 @@ along with GCC; see the file COPYING3.  If not see
 #include "tree-pass.h"
 #include "ssa.h"
 #include "gimple-iterator.h"
+#include "gimple-pretty-print.h"
 #include "tree-cfg.h"
+#include "tree-cfgcleanup.h"
+#include "tree-eh.h"
+#include "except.h"
+#include "sbitmap.h"
 #include "basic-block.h"
 #include "cfghooks.h"
 #include "cfgloop.h"
@@ -60,9 +65,7 @@ const pass_data pass_data_harden_control_flow_redundancy = {
   0,	    // properties_provided
   0,	    // properties_destroyed
   TODO_cleanup_cfg, // properties_start
-  TODO_update_ssa
-  | TODO_cleanup_cfg
-  | TODO_verify_il, // properties_finish
+  0,        // properties_finish
 };
 
 class pass_harden_control_flow_redundancy : public gimple_opt_pass
@@ -79,16 +82,6 @@ public:
     if (!flag_harden_control_flow_redundancy)
       return false;
 
-    /* We don't verify when an exception escapes, propagated or raised
-       by the function itself, so we're only concerned with edges to
-       the exit block.  If there aren't any, the function doesn't
-       return normally, so there won't be any checking point, so
-       there's no point in running the pass.  Should we add
-       verification at exception escapes, we should at least look at
-       !flag_exceptions here.  */
-    if (EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (fun)->preds) == 0)
-      return false;
-
     /* Functions that return more than once, like setjmp and vfork
        (that also gets this flag set), will start recording a path
        after the first return, and then may take another path when
@@ -117,8 +110,9 @@ public:
 	return false;
       }
 
-    if (param_hardcfr_max_blocks > 0
-	&& n_basic_blocks_for_fn (fun) - 2 > param_hardcfr_max_blocks)
+    if (fun->cfg && param_hardcfr_max_blocks > 0
+	&& (n_basic_blocks_for_fn (fun) - NUM_FIXED_BLOCKS
+	    > param_hardcfr_max_blocks))
       {
 	warning_at (DECL_SOURCE_LOCATION (fun->decl), 0,
 		    "%qD has more than %u blocks, the requested"
@@ -134,6 +128,293 @@ public:
 
 }
 
+/* Return TRUE iff CFR checks should be inserted before returning
+   calls.  */
+
+static bool
+check_returning_calls_p ()
+{
+  return
+    flag_harden_control_flow_redundancy_check_returning_calls > 0
+    || (flag_harden_control_flow_redundancy_check_returning_calls < 0
+	/* Gates pass_tail_calls.  */
+	&& flag_optimize_sibling_calls
+	/* Gates pass_all_ptimizations.  */
+	&& optimize >= 1 && !optimize_debug);
+}
+
+/* Scan BB from the end, updating *RETPTR if given as return stmts and
+   copies are found.  Return a call or a stmt that cannot appear after
+   a tail call, or NULL if the top of the block is reached without
+   finding any.  */
+
+static gimple *
+hardcfr_scan_block (basic_block bb, tree **retptr)
+{
+  gimple_stmt_iterator gsi;
+  for (gsi = gsi_last_bb (bb); !gsi_end_p (gsi); gsi_prev (&gsi))
+    {
+      gimple *stmt = gsi_stmt (gsi);
+
+      /* Ignore labels, returns, nops, clobbers and debug stmts.  */
+      if (gimple_code (stmt) == GIMPLE_LABEL
+	  || gimple_code (stmt) == GIMPLE_NOP
+	  || gimple_code (stmt) == GIMPLE_PREDICT
+	  || gimple_clobber_p (stmt)
+	  || is_gimple_debug (stmt))
+	continue;
+
+      if (gimple_code (stmt) == GIMPLE_RETURN)
+	{
+	  greturn *gret = as_a <greturn *> (stmt);
+	  if (retptr)
+	    {
+	      gcc_checking_assert (!*retptr);
+	      *retptr = gimple_return_retval_ptr (gret);
+	    }
+	  continue;
+	}
+
+      /* Check for a call.  */
+      if (is_gimple_call (stmt))
+	return stmt;
+
+      /* Allow simple copies to the return value, updating the return
+	 value to be found in earlier assignments.  */
+      if (retptr && *retptr && gimple_assign_single_p (stmt)
+	  && **retptr == gimple_assign_lhs (stmt))
+	{
+	  *retptr = gimple_assign_rhs1_ptr (stmt);
+	  continue;
+	}
+
+      return stmt;
+    }
+
+  /* Any other kind of stmt will prevent a tail call.  */
+  return NULL;
+}
+
+/* Return TRUE iff CALL is to be preceded by a CFR checkpoint, i.e.,
+   if it's a returning call (one whose result is ultimately returned
+   without intervening non-copy statements) and we're checking
+   returning calls, a __builtin_return call (noreturn with a path to
+   the exit block), a must-tail call, or a tail call.  */
+
+static bool
+returning_call_p (gcall *call)
+{
+  if (!(gimple_call_noreturn_p (call)
+	|| gimple_call_must_tail_p (call)
+	|| gimple_call_tail_p (call)
+	|| check_returning_calls_p ()))
+    return false;
+
+  /* Quickly check that there's a path to exit compatible with a
+     returning call.  Detect infinite loops through the counter.  */
+  basic_block bb = gimple_bb (call);
+  auto_vec<basic_block, 10> path;
+  for (int i = n_basic_blocks_for_fn (cfun);
+       bb != EXIT_BLOCK_PTR_FOR_FN (cfun) && i--;
+       bb = single_succ (bb))
+    if (!single_succ_p (bb)
+	|| (single_succ_edge (bb)->flags & EDGE_EH) != 0)
+      return false;
+    else
+      path.safe_push (bb);
+
+  /* Check the stmts in the blocks and trace the return value.  */
+  tree *retptr = NULL;
+  for (;;)
+    {
+      gcc_checking_assert (!path.is_empty ());
+      basic_block bb = path.pop ();
+      gimple *stop = hardcfr_scan_block (bb, &retptr);
+      if (stop)
+	{
+	  if (stop != call)
+	    return false;
+	  gcc_checking_assert (path.is_empty ());
+	  break;
+	}
+
+      gphi *retphi = NULL;
+      if (retptr && *retptr && TREE_CODE (*retptr) == SSA_NAME
+	  && !SSA_NAME_IS_DEFAULT_DEF (*retptr)
+	  && SSA_NAME_DEF_STMT (*retptr)
+	  && is_a <gphi *> (SSA_NAME_DEF_STMT (*retptr))
+	  && gimple_bb (SSA_NAME_DEF_STMT (*retptr)) == bb)
+	{
+	  retphi = as_a <gphi *> (SSA_NAME_DEF_STMT (*retptr));
+	  gcc_checking_assert (gimple_phi_result (retphi) == *retptr);
+	}
+      else
+	continue;
+
+      gcc_checking_assert (!path.is_empty ());
+      edge e = single_succ_edge (path.last ());
+      int i = EDGE_COUNT (bb->preds);
+      while (i--)
+	if (EDGE_PRED (bb, i) == e)
+	  break;
+      gcc_checking_assert (i >= 0);
+      retptr = gimple_phi_arg_def_ptr (retphi, i);
+    }
+
+  return (gimple_call_noreturn_p (call)
+	  || gimple_call_must_tail_p (call)
+	  || gimple_call_tail_p (call)
+	  || (gimple_call_lhs (call) == (retptr ? *retptr : NULL)
+	      && check_returning_calls_p ()));
+}
+
+typedef auto_vec<edge, 10> chk_edges_t;
+
+/* Declare for mutual recursion.  */
+static bool hardcfr_sibcall_search_preds (basic_block bb,
+					  chk_edges_t &chk_edges,
+					  int &count_chkcall,
+					  auto_sbitmap &chkcall_blocks,
+					  int &count_postchk,
+					  auto_sbitmap &postchk_blocks,
+					  tree *retptr);
+
+/* Search backwards from the end of BB for a mandatory or potential
+   sibcall.  Schedule the block to be handled sort-of like noreturn if
+   so.  Recurse to preds, with updated RETPTR, if the block only
+   contains stmts that may follow such a call, scheduling checking at
+   edges and marking blocks as post-check as needed.  Return true iff,
+   at the end of the block, a check will have already been
+   performed.  */
+
+static bool
+hardcfr_sibcall_search_block (basic_block bb,
+			      chk_edges_t &chk_edges,
+			      int &count_chkcall,
+			      auto_sbitmap &chkcall_blocks,
+			      int &count_postchk,
+			      auto_sbitmap &postchk_blocks,
+			      tree *retptr)
+{
+  /* Conditionals and internal exceptions rule out tail calls.  */
+  if (!single_succ_p (bb)
+      || (single_succ_edge (bb)->flags & EDGE_EH) != 0)
+    return false;
+
+  gimple *stmt = hardcfr_scan_block (bb, &retptr);
+  if (!stmt)
+    return hardcfr_sibcall_search_preds (bb, chk_edges,
+					 count_chkcall, chkcall_blocks,
+					 count_postchk, postchk_blocks,
+					 retptr);
+
+  if (!is_a <gcall *> (stmt))
+    return false;
+
+  /* Avoid disrupting mandatory or early-marked tail calls,
+     inserting the check before them.  This works for
+     must-tail calls, but tail calling as an optimization is
+     detected too late for us.
+
+     Also check for noreturn calls here.  Noreturn calls won't
+     normally have edges to exit, so they won't be found here,
+     but __builtin_return does, and we must check before
+     it, so handle it like a tail call.  */
+  gcall *call = as_a <gcall *> (stmt);
+  if (!(gimple_call_noreturn_p (call)
+	|| gimple_call_must_tail_p (call)
+	|| gimple_call_tail_p (call)
+	|| (gimple_call_lhs (call) == (retptr ? *retptr : NULL)
+	    && check_returning_calls_p ())))
+    return false;
+
+  gcc_checking_assert (returning_call_p (call));
+
+  /* We found a call that is to be preceded by checking.  */
+  if (bitmap_set_bit (chkcall_blocks, bb->index))
+    ++count_chkcall;
+  else
+    gcc_unreachable ();
+  return true;
+}
+
+
+/* Search preds of BB for a mandatory or potential sibcall or
+   returning call, and arrange for the blocks containing them to have
+   a check inserted before the call, like noreturn calls.  If any
+   preds are found to perform checking, schedule checks at the edges
+   of those that don't, and mark BB as postcheck..  */
+
+static bool
+hardcfr_sibcall_search_preds (basic_block bb,
+			      chk_edges_t &chk_edges,
+			      int &count_chkcall,
+			      auto_sbitmap &chkcall_blocks,
+			      int &count_postchk,
+			      auto_sbitmap &postchk_blocks,
+			      tree *retptr)
+{
+  /* For the exit block, we wish to force a check at every
+     predecessor, so pretend we've already found a pred that had
+     checking, so that we schedule checking at every one of its pred
+     edges.  */
+  bool first = bb->index >= NUM_FIXED_BLOCKS;
+  bool postchecked = true;
+
+  gphi *retphi = NULL;
+  if (retptr && *retptr && TREE_CODE (*retptr) == SSA_NAME
+      && !SSA_NAME_IS_DEFAULT_DEF (*retptr)
+      && SSA_NAME_DEF_STMT (*retptr)
+      && is_a <gphi *> (SSA_NAME_DEF_STMT (*retptr))
+      && gimple_bb (SSA_NAME_DEF_STMT (*retptr)) == bb)
+    {
+      retphi = as_a <gphi *> (SSA_NAME_DEF_STMT (*retptr));
+      gcc_checking_assert (gimple_phi_result (retphi) == *retptr);
+    }
+
+  for (int i = EDGE_COUNT (bb->preds); i--; first = false)
+    {
+      edge e = EDGE_PRED (bb, i);
+
+      bool checked
+	= hardcfr_sibcall_search_block (e->src, chk_edges,
+					count_chkcall, chkcall_blocks,
+					count_postchk, postchk_blocks,
+					!retphi ? retptr
+					: gimple_phi_arg_def_ptr (retphi, i));
+
+      if (first)
+	{
+	  postchecked = checked;
+	  continue;
+	}
+
+      /* When we first find a checked block, force a check at every
+	 other incoming edge we've already visited, and those we
+	 visit afterwards that don't have their own check, so that
+	 when we reach BB, the check has already been performed.  */
+      if (!postchecked && checked)
+	{
+	  for (int j = EDGE_COUNT (bb->preds); --j > i; )
+	    chk_edges.safe_push (EDGE_PRED (bb, j));
+	  postchecked = true;
+	}
+      if (postchecked && !checked)
+	chk_edges.safe_push (EDGE_PRED (bb, i));
+    }
+
+  if (postchecked && bb->index >= NUM_FIXED_BLOCKS)
+    {
+      if (bitmap_set_bit (postchk_blocks, bb->index))
+	count_postchk++;
+      else
+	gcc_unreachable ();
+    }
+
+  return postchecked;
+}
+
+
 class rt_bb_visited
 {
   /* Use a sufficiently wide unsigned type to hold basic block numbers.  */
@@ -176,8 +457,8 @@ class rt_bb_visited
      neither ENTRY nor EXIT, but maybe one-past-the-end, to compute
      the visited array length.  */
   blknum num2idx (blknum n) {
-    gcc_checking_assert (n >= 2 && n <= nblocks);
-    return (n - 2);
+    gcc_checking_assert (n >= NUM_FIXED_BLOCKS && n <= nblocks);
+    return (n - NUM_FIXED_BLOCKS);
   }
   /* Return the block vindex for BB, that must not be ENTRY or
      EXIT.  */
@@ -249,8 +530,7 @@ class rt_bb_visited
   }
 
   /* Set the bit corresponding to BB in VISITED.  Add to SEQ any
-     required gimple statements, and return SEQ, possibly
-     modified.  */
+     required gimple stmts, and return SEQ, possibly modified.  */
   gimple_seq vset (basic_block bb, gimple_seq seq = NULL)
   {
     tree bit, setme = vword (bb, &bit);
@@ -270,7 +550,7 @@ class rt_bb_visited
 
 public:
   /* Prepare to add control flow redundancy testing to CFUN.  */
-  rt_bb_visited ()
+  rt_bb_visited (int checkpoints)
     : nblocks (n_basic_blocks_for_fn (cfun)),
       vword_type (NULL), ckseq (NULL), rtcfg (NULL)
   {
@@ -353,8 +633,8 @@ public:
 					 NULL, NULL);
     gimple_seq_add_stmt (&ckseq, detach);
 
-    if (nblocks - 2 > blknum (param_hardcfr_max_inline_blocks)
-	|| !single_pred_p (EXIT_BLOCK_PTR_FOR_FN (cfun)))
+    if (nblocks - NUM_FIXED_BLOCKS > blknum (param_hardcfr_max_inline_blocks)
+	|| checkpoints > 1)
       {
 	/* Make sure vword_bits is wide enough for the representation
 	   of nblocks in rtcfg.  Compare with vword_bits << vword_bits,
@@ -379,65 +659,33 @@ public:
     gimple_seq_add_stmt (&ckseq, ckfail_init);
   }
 
-  /* Insert SEQ on E, or close enough (e.g., before a noreturn or tail
-     call at the end of E->src).  */
-  void insert_exit_check (gimple_seq seq, edge e)
+  /* Insert SEQ before a resx or a call in INSBB.  */
+  void insert_exit_check_in_block (gimple_seq seq, basic_block insbb)
   {
-    basic_block insbb = e->src;
-
-    /* If the returning block ends with a noreturn call, insert
-       checking before it.  This is particularly important for
-       __builtin_return.  Other noreturn calls won't have an edge to
-       the exit block, so they won't even be considered as exit paths.
-
-       Insert-on-edge inserts before other return stmts, but not
-       before calls, and if a single-block function had the check
-       sequence inserted after a noreturn call, it would be useless,
-       but checking would still flag it as malformed if block 2 has a
-       fallthru edge to the exit block.
-
-       Also avoid disrupting tail calls, inserting the check before
-       them.  This works for must-tail calls, but tail calling as an
-       optimization is detected too late for us.  */
     gimple_stmt_iterator gsi = gsi_last_bb (insbb);
-    gimple *ret = gsi_stmt (gsi);
-    if (ret && is_a <greturn *> (ret))
-      {
+
+    while (!gsi_end_p (gsi))
+      if (is_a <gresx *> (gsi_stmt (gsi))
+	  || is_a <gcall *> (gsi_stmt (gsi)))
+	break;
+      else
 	gsi_prev (&gsi);
-	if (!gsi_end_p (gsi))
-	  ret = gsi_stmt (gsi);
-      }
-    if (ret && is_a <gcall *> (ret)
-	&& (gimple_call_noreturn_p (ret)
-	    || gimple_call_must_tail_p (as_a <gcall *> (ret))
-	    || gimple_call_tail_p (as_a <gcall *> (ret))))
-      gsi_insert_seq_before (&gsi, seq, GSI_SAME_STMT);
-    else
-      gsi_insert_seq_on_edge_immediate (e, seq);
+
+    gsi_insert_seq_before (&gsi, seq, GSI_SAME_STMT);
+  }
+
+  /* Insert SEQ on E.  */
+  void insert_exit_check_on_edge (gimple_seq seq, edge e)
+  {
+    gsi_insert_seq_on_edge_immediate (e, seq);
   }
 
-  /* Add checking code on every exit edge, and initialization code on
+  /* Add checking code to CHK_EDGES, and initialization code on
      the entry edge.  Before this point, the CFG has been undisturbed,
      and all the needed data has been collected and safely stowed.  */
-  void check ()
+  void check (chk_edges_t &chk_edges,
+	      int count_chkcall, auto_sbitmap const &chkcall_blocks)
   {
-    /* Insert initializers for visited at the entry.  */
-    gimple_seq iseq = NULL;
-
-    gcall *vinit = gimple_build_call (builtin_decl_explicit
-				      (BUILT_IN_MEMSET), 3,
-				      build1 (ADDR_EXPR,
-					      build_pointer_type
-					      (TREE_TYPE (visited)),
-					      visited),
-				      integer_zero_node,
-				      TYPE_SIZE_UNIT (TREE_TYPE (visited)));
-    gimple_seq_add_stmt (&iseq, vinit);
-
-    gsi_insert_seq_on_edge_immediate (single_succ_edge
-				      (ENTRY_BLOCK_PTR_FOR_FN (cfun)),
-				      iseq);
-
     /* If we're using out-of-line checking, create and statically
        initialize the CFG checking representation, generate the
        checker call for the checking sequence, and insert it in all
@@ -498,28 +746,116 @@ public:
 						     rtcfg));
 	gimple_seq_add_stmt (&ckseq, call_chk);
 
+	gimple *clobber = gimple_build_assign (visited,
+					       build_clobber
+					       (TREE_TYPE (visited)));
+	gimple_seq_add_stmt (&ckseq, clobber);
+
 	/* If we have multiple exit edges, insert (copies of)
 	   ckseq in all of them.  */
-	for (int i = EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (cfun)->preds);
-	     i--; )
+	for (int i = chk_edges.length (); i--; )
 	  {
 	    gimple_seq seq = ckseq;
 	    /* Copy the sequence, unless we're dealing with the
 	       last edge (we're counting down to zero).  */
-	    if (i)
+	    if (i || count_chkcall)
+	      seq = gimple_seq_copy (seq);
+
+	    edge e = chk_edges[i];
+
+	    if (dump_file)
+	      {
+		if (e->dest == EXIT_BLOCK_PTR_FOR_FN (cfun))
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to exit.\n",
+			   e->src->index);
+		else
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to postcheck block %i.\n",
+			   e->src->index, e->dest->index);
+	      }
+
+	    insert_exit_check_on_edge (seq, e);
+
+	    gcc_checking_assert (!bitmap_bit_p (chkcall_blocks, e->src->index));
+	  }
+
+	sbitmap_iterator it;
+	unsigned i;
+	EXECUTE_IF_SET_IN_BITMAP (chkcall_blocks, 0, i, it)
+	  {
+	    basic_block bb = BASIC_BLOCK_FOR_FN (cfun, i);
+
+	    gimple_seq seq = ckseq;
+	    gcc_checking_assert (count_chkcall > 0);
+	    if (--count_chkcall)
 	      seq = gimple_seq_copy (seq);
 
-	    insert_exit_check (seq,
-			       EDGE_PRED (EXIT_BLOCK_PTR_FOR_FN (cfun), i));
+	    if (dump_file)
+	      fprintf (dump_file,
+		       "Inserting out-of-line check before stmt in block %i.\n",
+		       bb->index);
+
+	    insert_exit_check_in_block (seq, bb);
 	  }
+
+	gcc_checking_assert (count_chkcall == 0);
       }
     else
       {
 	/* Inline checking requires a single exit edge.  */
-	gimple *last = gsi_stmt (gsi_last (ckseq));
+	gimple *last = gimple_build_assign (visited,
+					       build_clobber
+					       (TREE_TYPE (visited)));
+	gimple_seq_add_stmt (&ckseq, last);
 
-	insert_exit_check (ckseq,
-			   single_pred_edge (EXIT_BLOCK_PTR_FOR_FN (cfun)));
+	if (!count_chkcall)
+	  {
+	    edge e = single_pred_edge (EXIT_BLOCK_PTR_FOR_FN (cfun));
+
+	    if (dump_file)
+	      {
+		if (e->dest == EXIT_BLOCK_PTR_FOR_FN (cfun))
+		  fprintf (dump_file,
+			   "Inserting out-of-line check in"
+			   " block %i's edge to postcheck block %i.\n",
+			   e->src->index, e->dest->index);
+		else
+		  fprintf (dump_file,
+			   "Inserting inline check in"
+			   " block %i's edge to exit.\n",
+			   e->src->index);
+	      }
+
+	    insert_exit_check_on_edge (ckseq, e);
+	  }
+	else
+	  {
+	    gcc_checking_assert (count_chkcall == 1);
+
+	    sbitmap_iterator it;
+	    unsigned i;
+	    EXECUTE_IF_SET_IN_BITMAP (chkcall_blocks, 0, i, it)
+	      {
+		basic_block bb = BASIC_BLOCK_FOR_FN (cfun, i);
+
+		gimple_seq seq = ckseq;
+		gcc_checking_assert (count_chkcall > 0);
+		if (--count_chkcall)
+		  seq = gimple_seq_copy (seq);
+
+		if (dump_file)
+		  fprintf (dump_file,
+			   "Inserting inline check before stmt in block %i.\n",
+			   bb->index);
+
+		insert_exit_check_in_block (seq, bb);
+	      }
+
+	    gcc_checking_assert (count_chkcall == 0);
+	  }
 
 	/* The inserted ckseq computes CKFAIL at LAST.  Now we have to
 	   conditionally trap on it.  */
@@ -540,8 +876,7 @@ public:
 	  add_bb_to_loop (trp, current_loops->tree_root);
 
 	/* Insert a conditional branch to the trap block.  If the
-	   conditional wouldn't be the last statement, split the
-	   block.  */
+	   conditional wouldn't be the last stmt, split the block.  */
 	gimple_stmt_iterator gsi = gsi_for_stmt (last);
 	if (!gsi_one_before_end_p (gsi))
 	  split_block (gsi_bb (gsi), gsi_stmt (gsi));
@@ -564,6 +899,24 @@ public:
 	if (dom_info_available_p (CDI_DOMINATORS))
 	  set_immediate_dominator (CDI_DOMINATORS, trp, gimple_bb (last));
       }
+
+    /* Insert initializers for visited at the entry.  Do this after
+       other insertions, to avoid messing with block numbers.  */
+    gimple_seq iseq = NULL;
+
+    gcall *vinit = gimple_build_call (builtin_decl_explicit
+				      (BUILT_IN_MEMSET), 3,
+				      build1 (ADDR_EXPR,
+					      build_pointer_type
+					      (TREE_TYPE (visited)),
+					      visited),
+				      integer_zero_node,
+				      TYPE_SIZE_UNIT (TREE_TYPE (visited)));
+    gimple_seq_add_stmt (&iseq, vinit);
+
+    gsi_insert_seq_on_edge_immediate (single_succ_edge
+				      (ENTRY_BLOCK_PTR_FOR_FN (cfun)),
+				      iseq);
   }
 
   /* Push onto RTCFG a (mask, index) pair to test for IBB when BB is
@@ -607,7 +960,7 @@ public:
     return false;
   }
 
-  /* Add to CKSEQ statements to clear CKPART if OBB is visited.  */
+  /* Add to CKSEQ stmts to clear CKPART if OBB is visited.  */
   void
   build_block_check (basic_block obb)
   {
@@ -632,34 +985,48 @@ public:
 
   /* Add to BB code to set its bit in VISITED, and add to RTCFG or
      CKSEQ the data or code needed to check BB's predecessors and
-     successors.  Do NOT change the CFG.  */
-  void visit (basic_block bb)
+     successors.  If CHECKPOINT, assume the block is a checkpoint,
+     whether or not it has an edge to EXIT.  If POSTCHECK, assume the
+     block post-dominates checkpoints and therefore no bitmap setting
+     or checks are to be performed in or for it.  Do NOT change the
+     CFG.  */
+  void visit (basic_block bb, bool checkpoint, bool postcheck)
   {
     /* Set the bit in VISITED when entering the block.  */
     gimple_stmt_iterator gsi = gsi_after_labels (bb);
-    gsi_insert_seq_before (&gsi, vset (bb), GSI_SAME_STMT);
+    if (!postcheck)
+      gsi_insert_seq_before (&gsi, vset (bb), GSI_SAME_STMT);
 
     if (rtcfg)
       {
-	/* Build a list of (index, mask) terminated by (NULL, 0).
-	   Consolidate masks with the same index when they're
-	   adjacent.  First, predecessors.  Count backwards, because
-	   we're going to reverse the list.  The order shouldn't
-	   matter, but let's not make it surprising.  */
-	for (int i = EDGE_COUNT (bb->preds); i--; )
-	  if (push_rtcfg_pair (EDGE_PRED (bb, i)->src, bb,
-			       ENTRY_BLOCK_PTR_FOR_FN (cfun)))
-	    break;
+	if (!postcheck)
+	  {
+	    /* Build a list of (index, mask) terminated by (NULL, 0).
+	       Consolidate masks with the same index when they're
+	       adjacent.  First, predecessors.  Count backwards, because
+	       we're going to reverse the list.  The order shouldn't
+	       matter, but let's not make it surprising.  */
+	    for (int i = EDGE_COUNT (bb->preds); i--; )
+	      if (push_rtcfg_pair (EDGE_PRED (bb, i)->src, bb,
+				   ENTRY_BLOCK_PTR_FOR_FN (cfun)))
+		break;
+	  }
 	rtcfg = tree_cons (NULL_TREE, build_int_cst (vword_type, 0), rtcfg);
 
-	/* Then, successors.  */
-	for (int i = EDGE_COUNT (bb->succs); i--; )
-	  if (push_rtcfg_pair (EDGE_SUCC (bb, i)->dest, bb,
-			       EXIT_BLOCK_PTR_FOR_FN (cfun)))
-	    break;
+	if (!postcheck)
+	  {
+	    /* Then, successors.  */
+	    if (!checkpoint
+		|| !push_rtcfg_pair (EXIT_BLOCK_PTR_FOR_FN (cfun),
+				     bb, EXIT_BLOCK_PTR_FOR_FN (cfun)))
+	      for (int i = EDGE_COUNT (bb->succs); i--; )
+		if (push_rtcfg_pair (EDGE_SUCC (bb, i)->dest, bb,
+				     EXIT_BLOCK_PTR_FOR_FN (cfun)))
+		  break;
+	  }
 	rtcfg = tree_cons (NULL_TREE, build_int_cst (vword_type, 0), rtcfg);
       }
-    else
+    else if (!postcheck)
       {
 	/* Schedule test to fail if the block was reached but somehow none
 	   of its predecessors were.  */
@@ -677,6 +1044,8 @@ public:
 	gassign *blkruns = gimple_build_assign (ckpart, unshare_expr (bit));
 	gimple_seq_add_stmt (&ckseq, blkruns);
 
+	if (checkpoint)
+	  build_block_check (EXIT_BLOCK_PTR_FOR_FN (cfun));
 	for (int i = 0, e = EDGE_COUNT (bb->succs); i < e; i++)
 	  build_block_check (EDGE_SUCC (bb, i)->dest);
 
@@ -687,21 +1056,350 @@ public:
   }
 };
 
+/* It might be useful to avoid checking before noreturn calls that are
+   known to always finish by throwing an exception, rather than by
+   ending the program or looping forever.  Such functions would have
+   to be annotated somehow, with an attribute or flag.
+   Exception-raising functions, such as C++'s __cxa_throw,
+   __cxa_rethrow, and Ada's */
+static bool
+always_throwing_noreturn_call_p (gimple *)
+{
+  return false;
+}
+
 /* Control flow redundancy hardening: record the execution path, and
    verify at exit that an expect path was taken.  */
 
 unsigned int
-pass_harden_control_flow_redundancy::execute (function *)
+pass_harden_control_flow_redundancy::execute (function *fun)
 {
-  rt_bb_visited vstd;
-
+  bool const check_at_escaping_exceptions
+    = (flag_exceptions
+       && flag_harden_control_flow_redundancy_check_exceptions);
+  bool const check_before_noreturn_calls
+    = flag_harden_control_flow_redundancy_check_noreturn > HCFRNR_NEVER;
+  bool const check_before_nothrow_noreturn_calls
+    = (check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn >= HCFRNR_NOTHROW);
+  bool const check_before_throwing_noreturn_calls
+    = (flag_exceptions
+       && check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn > HCFRNR_NOTHROW);
+  bool const check_before_always_throwing_noreturn_calls
+    = (flag_exceptions
+       && check_before_noreturn_calls
+       && flag_harden_control_flow_redundancy_check_noreturn >= HCFRNR_ALWAYS);
   basic_block bb;
-  FOR_EACH_BB_FN (bb, cfun)
-    vstd.visit (bb);
+  basic_block bb_eh_cleanup = NULL;
+
+  if (check_at_escaping_exceptions)
+    {
+      int lp_eh_cleanup = -1;
+
+      /* Record the preexisting blocks, to avoid visiting newly-created
+	 blocks.  */
+      auto_sbitmap to_visit (last_basic_block_for_fn (fun));
+      bitmap_clear (to_visit);
+
+      FOR_EACH_BB_FN (bb, fun)
+	bitmap_set_bit (to_visit, bb->index);
+
+      /* Scan the blocks for stmts with escaping exceptions, that
+	 wouldn't be denoted in the CFG, and associate them with an
+	 empty cleanup handler around the whole function.  Walk
+	 backwards, so that even when we split the block, */
+      sbitmap_iterator it;
+      unsigned i;
+      EXECUTE_IF_SET_IN_BITMAP (to_visit, 0, i, it)
+	{
+	  bb = BASIC_BLOCK_FOR_FN (fun, i);
+
+	  for (gimple_stmt_iterator gsi = gsi_last_bb (bb);
+	       !gsi_end_p (gsi); gsi_prev (&gsi))
+	    {
+	      gimple *stmt = gsi_stmt (gsi);
+	      if (!stmt_could_throw_p (fun, stmt))
+		continue;
+
+	      /* If it must not throw, or if it already has a handler,
+		 we need not worry about it.  */
+	      if (lookup_stmt_eh_lp (stmt) != 0)
+		continue;
+
+	      /* Don't split blocks at, nor add EH edges to, tail
+		 calls, we will add verification before the call
+		 anyway.  */
+	      if (is_a <gcall *> (stmt)
+		  && (gimple_call_must_tail_p (as_a <gcall *> (stmt))
+		      || gimple_call_tail_p (as_a <gcall *> (stmt))
+		      || returning_call_p (as_a <gcall *> (stmt))))
+		continue;
+
+	      if (!gsi_one_before_end_p (gsi))
+		split_block (bb, stmt);
+	      /* A resx or noreturn call needs not be associated with
+		 the cleanup handler if we're going to add checking
+		 before it.  We only test cases that didn't require
+		 block splitting because noreturn calls would always
+		 be at the end of blocks, and we test for zero
+		 successors because if there is an edge, it's not
+		 noreturn, as any EH edges would have already been
+		 caught by the lookup_stmt_eh_lp test above.  */
+	      else if (check_before_noreturn_calls
+		       && EDGE_COUNT (bb->succs) == 0
+		       && (is_a <gresx *> (stmt)
+			   ? check_before_always_throwing_noreturn_calls
+			   : (!is_a <gcall *> (stmt)
+			      || !gimple_call_noreturn_p (stmt))
+			   ? (gcc_unreachable (), false)
+			   : (!flag_exceptions
+			      || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+			   ? check_before_nothrow_noreturn_calls
+			   : always_throwing_noreturn_call_p (stmt)
+			   ? check_before_always_throwing_noreturn_calls
+			   : check_before_throwing_noreturn_calls))
+		{
+		  if (dump_file)
+		    {
+		      fprintf (dump_file,
+			       "Bypassing cleanup for noreturn stmt"
+			       " in block %i:\n",
+			       bb->index);
+		      print_gimple_stmt (dump_file, stmt, 0);
+		    }
+		  continue;
+		}
+
+	      if (!bb_eh_cleanup)
+		{
+		  bb_eh_cleanup = create_empty_bb (bb);
+		  if (dom_info_available_p (CDI_DOMINATORS))
+		    set_immediate_dominator (CDI_DOMINATORS, bb_eh_cleanup, bb);
+		  if (current_loops)
+		    add_bb_to_loop (bb_eh_cleanup, current_loops->tree_root);
+
+		  /* Make the new block an EH cleanup for the call.  */
+		  eh_region new_r = gen_eh_region_cleanup (NULL);
+		  eh_landing_pad lp = gen_eh_landing_pad (new_r);
+		  tree label = gimple_block_label (bb_eh_cleanup);
+		  lp->post_landing_pad = label;
+		  EH_LANDING_PAD_NR (label) = lp_eh_cleanup = lp->index;
+
+		  /* Just propagate the exception.
+		     We will later insert the verifier call.  */
+		  gimple_stmt_iterator ehgsi;
+		  ehgsi = gsi_after_labels (bb_eh_cleanup);
+		  gresx *resx = gimple_build_resx (new_r->index);
+		  gsi_insert_before (&ehgsi, resx, GSI_SAME_STMT);
+
+		  if (dump_file)
+		    fprintf (dump_file,
+			     "Created cleanup block %i:\n",
+			     bb_eh_cleanup->index);
+		}
+	      else if (dom_info_available_p (CDI_DOMINATORS))
+		{
+		  basic_block immdom;
+		  immdom = get_immediate_dominator (CDI_DOMINATORS,
+						    bb_eh_cleanup);
+		  if (!dominated_by_p (CDI_DOMINATORS, bb, immdom))
+		    {
+		      immdom = nearest_common_dominator (CDI_DOMINATORS,
+							 immdom, bb);
+		      set_immediate_dominator (CDI_DOMINATORS,
+					       bb_eh_cleanup, immdom);
+		    }
+		}
+
+	      if (dump_file)
+		{
+		  fprintf (dump_file,
+			   "Associated cleanup block with stmt in block %i:\n",
+			   bb->index);
+		  print_gimple_stmt (dump_file, stmt, 0);
+		}
+
+	      add_stmt_to_eh_lp (stmt, lp_eh_cleanup);
+	      /* Finally, wire the EH cleanup block into the CFG.  */
+	      make_eh_edges (stmt);
+	    }
+	}
+
+      if (bb_eh_cleanup)
+	{
+	  /* A cfg_cleanup after bb_eh_cleanup makes for a more compact
+	     rtcfg, and it avoids bb numbering differences when we split
+	     blocks because of trailing debug insns only.  */
+	  cleanup_tree_cfg ();
+	  gcc_checking_assert (EDGE_COUNT (bb_eh_cleanup->succs) == 0);
+	}
+    }
+
+  /* These record blocks with calls that are to be preceded by
+     checkpoints, such as noreturn calls (if so chosen), must-tail
+     calls, potential early-marked tail calls, and returning calls (if
+     so chosen).  */
+  int count_chkcall = 0;
+  auto_sbitmap chkcall_blocks (last_basic_block_for_fn (fun));
+  bitmap_clear (chkcall_blocks);
+
+  /* We wish to add verification at blocks without successors, such as
+     noreturn calls (raising or not) and the reraise at the cleanup
+     block, but not other reraises: they will go through the cleanup
+     block.  */
+  if (check_before_noreturn_calls)
+    FOR_EACH_BB_FN (bb, fun)
+      {
+	gimple_stmt_iterator gsi = gsi_last_bb (bb);
+	if (gsi_end_p (gsi))
+	  continue;
+	gimple *stmt = gsi_stmt (gsi);
 
-  vstd.check ();
+	if (EDGE_COUNT (bb->succs) == 0)
+	  {
+	    /* A stmt at the end of a block without any successors is
+	       either a resx or a noreturn call without a local
+	       handler.  Check that it's one of the desired
+	       checkpoints.  */
+	    if (flag_exceptions && is_a <gresx *> (stmt)
+		? (check_before_always_throwing_noreturn_calls
+		   || bb == bb_eh_cleanup)
+		: (!is_a <gcall *> (stmt)
+		   || !gimple_call_noreturn_p (stmt))
+		? (/* Catch cases in which successors would be
+		      expected.  */
+		   gcc_unreachable (), false)
+		: (!flag_exceptions
+		   || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+		? check_before_nothrow_noreturn_calls
+		: always_throwing_noreturn_call_p (stmt)
+		? check_before_always_throwing_noreturn_calls
+		: check_before_throwing_noreturn_calls)
+	      {
+		if (dump_file)
+		  {
+		    fprintf (dump_file,
+			     "Scheduling check before stmt"
+			     " in succ-less block %i:\n",
+			     bb->index);
+		    print_gimple_stmt (dump_file, stmt, 0);
+		  }
+
+		if (bitmap_set_bit (chkcall_blocks, bb->index))
+		  count_chkcall++;
+		else
+		  gcc_unreachable ();
+	      }
+	    continue;
+	  }
 
-  return 0;
+	/* If there are no exceptions, then any noreturn call must have
+	   zero successor edges.  Otherwise, check for blocks without
+	   non-EH successors, but skip those with resx stmts and edges
+	   (i.e., those other than that in bb_eh_cleanup), since those
+	   will go through bb_eh_cleanup, that will have been counted as
+	   noreturn above because it has no successors.  */
+	gcc_checking_assert (bb != bb_eh_cleanup
+			     || !check_at_escaping_exceptions);
+	if (flag_exceptions && is_a <gresx *> (stmt)
+	    ? check_before_always_throwing_noreturn_calls
+	    : (!is_a <gcall *> (stmt)
+	       || !gimple_call_noreturn_p (stmt))
+	    ? false
+	    : (!flag_exceptions
+	       || gimple_call_nothrow_p (as_a <gcall *> (stmt)))
+	    ? (/* Catch cases that should not have successors.  */
+	       gcc_unreachable (), check_before_nothrow_noreturn_calls)
+	    : always_throwing_noreturn_call_p (stmt)
+	    ? check_before_always_throwing_noreturn_calls
+	    : check_before_throwing_noreturn_calls)
+	  {
+	    gcc_checking_assert (single_succ_p (bb)
+				 && (single_succ_edge (bb)->flags & EDGE_EH));
+
+	    if (dump_file)
+	      {
+		fprintf (dump_file,
+			 "Scheduling check before stmt"
+			 " in EH-succ block %i:\n",
+			 bb->index);
+		print_gimple_stmt (dump_file, stmt, 0);
+	      }
+
+	    if (bitmap_set_bit (chkcall_blocks, bb->index))
+	      count_chkcall++;
+	    else
+	      gcc_unreachable ();
+	  }
+      }
+  else if (bb_eh_cleanup)
+    {
+      if (bitmap_set_bit (chkcall_blocks, bb_eh_cleanup->index))
+	count_chkcall++;
+      else
+	gcc_unreachable ();
+    }
+
+  gcc_checking_assert (!bb_eh_cleanup
+		       || bitmap_bit_p (chkcall_blocks, bb_eh_cleanup->index));
+
+  /* If we don't have edges to exit nor noreturn calls (including the
+     cleanup reraise), then we may skip instrumentation: that would
+     amount to a function that ends with an infinite loop.  */
+  if (!count_chkcall
+      && EDGE_COUNT (EXIT_BLOCK_PTR_FOR_FN (fun)->preds) == 0)
+    {
+      if (dump_file)
+	fprintf (dump_file,
+		 "Disabling CFR, no exit paths to check\n");
+
+      return 0;
+    }
+
+  /* Search for must-tail calls, early-marked potential tail calls,
+     and, if requested, returning calls.  As we introduce early
+     checks, */
+  int count_postchk = 0;
+  auto_sbitmap postchk_blocks (last_basic_block_for_fn (fun));
+  bitmap_clear (postchk_blocks);
+  chk_edges_t chk_edges;
+  hardcfr_sibcall_search_preds (EXIT_BLOCK_PTR_FOR_FN (fun), chk_edges,
+				count_chkcall, chkcall_blocks,
+				count_postchk, postchk_blocks,
+				NULL);
+
+  rt_bb_visited vstd (chk_edges.length () + count_chkcall);
+
+  auto_sbitmap combined_blocks (last_basic_block_for_fn (fun));
+  bitmap_copy (combined_blocks, chkcall_blocks);
+  int i;
+  edge *e;
+  FOR_EACH_VEC_ELT (chk_edges, i, e)
+    if (!bitmap_set_bit (combined_blocks, (*e)->src->index))
+      gcc_unreachable ();
+
+  /* Visit blocks in index order, because building rtcfg depends on
+     that.  Blocks must be compact, which the cleanup_cfg requirement
+     ensures.  This would also enable FOR_EACH_BB_FN to be used to
+     iterate in index order, but bb_eh_cleanup block splits and
+     insertions changes that.  */
+  gcc_checking_assert (n_basic_blocks_for_fn (fun)
+		       == last_basic_block_for_fn (fun));
+  for (int i = NUM_FIXED_BLOCKS; i < n_basic_blocks_for_fn (fun); i++)
+    {
+      bb = BASIC_BLOCK_FOR_FN (fun, i);
+      gcc_checking_assert (bb->index == i);
+      vstd.visit (bb, bitmap_bit_p (combined_blocks, i),
+		  bitmap_bit_p (postchk_blocks, i));
+    }
+
+  vstd.check (chk_edges, count_chkcall, chkcall_blocks);
+
+  return
+    TODO_update_ssa
+    | TODO_cleanup_cfg
+    | TODO_verify_il;
 }
 
 /* Instantiate a hardcfr pass.  */
diff --git a/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c b/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c
new file mode 100644
index 00000000000..a6992eb9f8e
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/harden-cfr-noret-never-O0.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -O0 -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we don't insert checking before noreturn calls.  -O0 is tested
+   separately because h is not found to be noreturn without optimization.  */
+
+#include "torture/harden-cfr-noret.c"
+
+/* No out-of-line checks.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* Only one inline check at the end of f and of h2.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c
new file mode 100644
index 00000000000..8bd2d13ac18
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-never.c
@@ -0,0 +1,18 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we don't insert checking before noreturn calls.  -O0 is tested
+   separately because h is not found to be noreturn without optimization, which
+   affects codegen for h2, so h2 is omitted here at -O0.  */
+
+#if !__OPTIMIZE__
+# define OMIT_H2
+#endif
+
+#include "harden-cfr-noret.c"
+
+
+/* No out-of-line checks.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* Only one inline check at the end of f.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c
new file mode 100644
index 00000000000..a804a6cfe59
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-noexcept.c
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fno-exceptions -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that -fno-exceptions makes for implicit nothrow in noreturn
+   handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#include "harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c
new file mode 100644
index 00000000000..f390cfdbc59
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret-nothrow.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checking before nothrow noreturn calls.  */
+
+#include "harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c
new file mode 100644
index 00000000000..fdd803109a4
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-noret.c
@@ -0,0 +1,38 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert checking before all noreturn calls.  */
+
+#ifndef ATTR_NOTHROW_OPT /* Overridden in harden-cfr-noret-noexcept.  */
+#define ATTR_NOTHROW_OPT __attribute__ ((__nothrow__))
+#endif
+
+extern void __attribute__ ((__noreturn__)) ATTR_NOTHROW_OPT g (void);
+
+void f(int i) {
+  if (i)
+    /* Out-of-line checks here...  */
+    g ();
+  /* ... and here.  */
+}
+
+void __attribute__ ((__noinline__, __noclone__))
+h(void) {
+  /* Inline check here.  */
+  g ();
+}
+
+#ifndef OMIT_H2 /* from harden-cfr-noret-never.  */
+void h2(void) {
+  /* Inline check either here, whether because of noreturn or tail call...  */
+  h ();
+  /* ... or here, if not optimizing.  */
+}
+#endif
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c
new file mode 100644
index 00000000000..6d11487bbba
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-notail.c
@@ -0,0 +1,8 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+#include "harden-cfr-tail.c"
+
+/* Inline checking after the calls, disabling tail calling.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 5 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 0 "hardcfr" } } */
diff --git a/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c b/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
index 40d76c5c163..d5467eafa9f 100644
--- a/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
+++ b/gcc/testsuite/c-c++-common/torture/harden-cfr-tail.c
@@ -1,17 +1,52 @@
 /* { dg-do compile } */
-/* { dg-options "-fharden-control-flow-redundancy -fdump-tree-hardcfr -ffat-lto-objects" } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-hardcfr-check-exceptions -fdump-tree-hardcfr -ffat-lto-objects -Wno-return-type" } */
 
-/* We'd like to check that we insert checking so as to not disrupt tail calls,
-   but unfortunately mandatory tail calls are not available in C, and optimizing
-   calls as tail calls only takes place after hardcfr.  */
+/* Check that we insert CFR checking so as to not disrupt tail calls.
+   Mandatory tail calls are not available in C, and optimizing calls as tail
+   calls only takes place after hardcfr, so we insert checking before calls
+   followed by copies and return stmts with the same return value, that might
+   (or might not) end up optimized to tail calls.  */
 
-extern int g(int i);
+extern int g (int i);
 
-int f(int i) {
+int f1(int i) {
+  /* Inline check before the returning call.  */
   return g (i);
 }
 
-/* Inline checking before the tail call.  */
-/* { dg-final { scan-tree-dump-times "__builtin_trap" 1 "hardcfr" } } */
-/* Inline checking before the tail call.  */
-/* { dg-final { scan-tree-dump-times "\\\[tail\]" 1 "hardcfr" { xfail *-*-* } } } */
+extern void g2 (int i);
+
+void f2(int i) {
+  /* Inline check before the returning call, that ignores the returned value,
+     matching the value-less return.  */
+  g2 (i);
+  return;
+}
+
+void f3(int i) {
+  /* Inline check before the returning call.  */
+  g (i);
+}
+
+void f4(int i) {
+  if (i)
+    /* Out-of-line check before the returning call.  */
+    return g2 (i);
+  /* Out-of-line check before implicit return.  */
+}
+
+int f5(int i) {
+  /* Not regarded as a returning call, returning value other than callee's
+     returned value.  */
+  g (i);
+  /* Inline check after the non-returning call.  */
+  return i;
+}
+
+/* Out-of-line checks in f4, before returning calls and before return.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking in all other functions.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 4 "hardcfr" } } */
+/* Check before tail-call in all but f5, but f4 is out-of-line.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 3 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C b/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C
new file mode 100644
index 00000000000..17ea79f7cfb
--- /dev/null
+++ b/gcc/testsuite/g++.dg/harden-cfr-throw-always-O0.C
@@ -0,0 +1,11 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects -O0" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions, and also checking before noreturn
+   calls.  h2 and h2b get an extra resx without ehcleanup.  */
+
+#include "torture/harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 16 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C b/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C
new file mode 100644
index 00000000000..f0338ccc361
--- /dev/null
+++ b/gcc/testsuite/g++.dg/harden-cfr-throw-returning-O0.C
@@ -0,0 +1,10 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -foptimize-sibling-calls -fdump-tree-hardcfr -O0" } */
+
+/* -fhardcfr-check-returning-calls gets implicitly disabled because,
+   -at O0, -foptimize-sibling-calls has no effect.  */
+
+#include "torture/harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 12 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C
new file mode 100644
index 00000000000..0d35920c7ee
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-always-no-nothrow.C
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  */
+
+#include "harden-cfr-noret-no-nothrow.C"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Bypassing cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C
new file mode 100644
index 00000000000..b7d247ff43c
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-never-no-nothrow.C
@@ -0,0 +1,18 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=never -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  Expected results for =never and =nothrow are the same,
+   since the functions are not nothrow.  */
+
+#include "harden-cfr-noret-no-nothrow.C"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Associated cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C
new file mode 100644
index 00000000000..62c58cfd406
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-noret-no-nothrow.C
@@ -0,0 +1,23 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C++ does NOT make for implicit nothrow in noreturn
+   handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#if ! __OPTIMIZE__
+void __attribute__ ((__noreturn__)) h (void);
+#endif
+
+#include "../../c-c++-common/torture/harden-cfr-noret.c"
+
+/* All 3 noreturn calls.  */
+/* { dg-final { scan-tree-dump-times "Associated cleanup" 3 "hardcfr" } } */
+/* Out-of-line checks in f.  */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check in block \[0-9]*'s edge to exit" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checks in h and h2.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C
new file mode 100644
index 00000000000..0286f6e6d3f
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-always.C
@@ -0,0 +1,20 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fhardcfr-check-noreturn-calls=always -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions, and also checking before noreturn
+   calls.  */
+
+#if ! __OPTIMIZE__
+/* Without optimization, functions with cleanups end up with an extra
+   resx that is not optimized out, so arrange to optimize them.  */
+void __attribute__ ((__optimize__ (1))) h2(void);
+void __attribute__ ((__optimize__ (1))) h2b(void);
+#endif
+
+#include "harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 14 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
+/* h, h2, h2b, and h4.  */
+/* { dg-final { scan-tree-dump-times "Bypassing" 4 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C
new file mode 100644
index 00000000000..885b0b236af
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-nocleanup.C
@@ -0,0 +1,11 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions  -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we do not insert cleanups for checking around the bodies
+   of maybe-throwing functions.  h4 doesn't get any checks, because we
+   don't have noreturn checking enabled.  */
+
+#include "harden-cfr-throw.C"
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 0 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 6 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C
new file mode 100644
index 00000000000..d150470c618
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw-returning.C
@@ -0,0 +1,31 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -foptimize-sibling-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions.  These results depend on checking before
+   returning calls, which is only enabled when sibcall optimizations
+   are enabled, so change the optimization mode to -O1 for f and f2,
+   so that -foptimize-sibling-calls can take effect and enable
+   -fhardcfr-check-returning-calls, so that we get the same results.
+   There is a separate test for -O0.  */
+
+#if ! __OPTIMIZE__
+void __attribute__ ((__optimize__ (1))) f(int i);
+void __attribute__ ((__optimize__ (1))) f2(int i);
+void __attribute__ ((__optimize__ (1))) h3(void);
+#endif
+
+#include "harden-cfr-throw.C"
+
+/* f gets out-of-line checks before the unwrapped tail call and in the
+   else edge.  */
+/* f2 gets out-of-line checks before both unwrapped tail calls.  */
+/* h gets out-of-line checks before the implicit return and in the
+   cleanup block.  */
+/* h2 and h2b get out-of-line checks before the cleanup returning
+   call, and in the cleanup block.  */
+/* h3 gets an inline check before the __cxa_end_catch returning call.  */
+/* h4 gets an inline check in the cleanup block.  */
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 10 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C b/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C
new file mode 100644
index 00000000000..992fbdad381
--- /dev/null
+++ b/gcc/testsuite/g++.dg/torture/harden-cfr-throw.C
@@ -0,0 +1,65 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-returning-calls -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that we insert cleanups for checking around the bodies of
+   maybe-throwing functions.  */
+
+extern void g (void);
+extern void g2 (void);
+
+void f(int i) {
+  if (i)
+    g ();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void f2(int i) {
+  if (i)
+    g ();
+  else
+    g2 ();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h(void) {
+  try {
+    g ();
+  } catch (...) {
+    throw;
+  }
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+struct needs_cleanup {
+  ~needs_cleanup();
+};
+
+void h2(void) {
+  needs_cleanup y; /* No check in the cleanup handler.  */
+  g();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+extern void __attribute__ ((__nothrow__)) another_cleanup (void*);
+
+void h2b(void) {
+  int x __attribute__ ((cleanup (another_cleanup)));
+  g();
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h3(void) {
+  try {
+    throw 1;
+  } catch (...) {
+  }
+  /* Out-of-line checks here, and in the implicit handler.  */
+}
+
+void h4(void) {
+  throw 1;
+  /* Inline check in the cleanup around the __cxa_throw noreturn call.  */
+}
+
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 12 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "builtin_trap" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c b/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c
new file mode 100644
index 00000000000..8e4ee1fab08
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/torture/harden-cfr-noret-no-nothrow.c
@@ -0,0 +1,15 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-noreturn-calls=nothrow -fdump-tree-hardcfr -ffat-lto-objects" } */
+
+/* Check that C makes for implicit nothrow in noreturn handling.  */
+
+#define ATTR_NOTHROW_OPT
+
+#include "../../c-c++-common/torture/harden-cfr-noret.c"
+
+/* One out-of-line check before the noreturn call in f, and another at the end
+   of f.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* One inline check in h, before the noreturn call, and another in h2, before
+   or after the call, depending on noreturn detection.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
diff --git a/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c b/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c
new file mode 100644
index 00000000000..634d98f1ffc
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/torture/harden-cfr-tail-ub.c
@@ -0,0 +1,40 @@
+/* { dg-do compile } */
+/* { dg-options "-fharden-control-flow-redundancy -fhardcfr-check-returning-calls -fno-hardcfr-check-exceptions -fdump-tree-hardcfr -ffat-lto-objects -Wno-return-type" } */
+
+/* In C only, check some additional cases (comparing with
+   c-c++-common/torture/harden-cfr-tail.c) of falling off the end of non-void
+   function.  C++ would issue an unreachable call in these cases.  */
+
+extern int g (int i);
+
+int f1(int i) {
+  /* Inline check before the returning call, that doesn't return anything.  */
+  g (i);
+  /* Implicit return without value, despite the return type; this combination
+     enables tail-calling of g, and is recognized as a returning call.  */
+}
+
+extern void g2 (int i);
+
+int f2(int i) {
+  /* Inline check before the returning call, that disregards its return
+     value.  */
+  g2 (i);
+  /* Implicit return without value, despite the return type; this combination
+     enables tail-calling of g2, and is recognized as a returning call.  */
+}
+
+int f3(int i) {
+  if (i)
+    /* Out-of-line check before the returning call.  */
+    return g (i);
+  /* Out-of-line check before implicit return.  */
+}
+
+/* Out-of-line checks in f3, before returning calls and before return.  */
+/* { dg-final { scan-tree-dump-times "hardcfr_check" 2 "hardcfr" } } */
+/* Inline checking in all other functions.  */
+/* { dg-final { scan-tree-dump-times "__builtin_trap" 2 "hardcfr" } } */
+/* Check before tail-call in all functions, but f3 is out-of-line.  */
+/* { dg-final { scan-tree-dump-times "Inserting inline check before stmt" 2 "hardcfr" } } */
+/* { dg-final { scan-tree-dump-times "Inserting out-of-line check before stmt" 1 "hardcfr" } } */
diff --git a/gcc/testsuite/gnat.dg/hardcfr.adb b/gcc/testsuite/gnat.dg/hardcfr.adb
index b578a913d75..abe1605c029 100644
--- a/gcc/testsuite/gnat.dg/hardcfr.adb
+++ b/gcc/testsuite/gnat.dg/hardcfr.adb
@@ -1,5 +1,5 @@
 --  { dg-do run }
---  { dg-options "-fharden-control-flow-redundancy -fdump-tree-hardcfr --param=hardcfr-max-blocks=22 --param=hardcfr-max-inline-blocks=12 -O0" }
+--  { dg-options "-fharden-control-flow-redundancy -fno-hardcfr-check-exceptions -fdump-tree-hardcfr --param=hardcfr-max-blocks=22 --param=hardcfr-max-inline-blocks=12 -O0" }
 
 procedure HardCFR is
    function F (I, J : Integer) return Integer is
diff --git a/libgcc/hardcfr.c b/libgcc/hardcfr.c
index 8ef29428111..55f6b995f2f 100644
--- a/libgcc/hardcfr.c
+++ b/libgcc/hardcfr.c
@@ -60,14 +60,25 @@ extern void __hardcfr_check (size_t blocks,
 			     vword const *visited,
 			     vword const *cfg);
 
+/* Compute the MASK for the bit representing BLOCK in WORDIDX's vword in a
+   visited blocks bit array.  */
+static inline void
+block2mask (size_t const block, vword *const mask, size_t *const wordidx)
+{
+  size_t wbits = __CHAR_BIT__ * sizeof (vword);
+  *wordidx = block / wbits;
+  *mask = (vword)1 << (block % wbits);
+}
 
 /* Check whether the bit corresponding to BLOCK is set in VISITED.  */
 static inline bool
 visited_p (size_t const block, vword const *const visited)
 {
-  size_t wbits = __CHAR_BIT__ * sizeof (vword);
-  vword w = visited[block / wbits];
-  return (w & ((vword)1 << (block % wbits))) != 0;
+  vword mask;
+  size_t wordidx;
+  block2mask (block, &mask, &wordidx);
+  vword w = visited[wordidx];
+  return (w & mask) != 0;
 }
 
 /* Read and consume a mask from **CFG_IT.  (Consume meaning advancing the
@@ -118,19 +129,19 @@ consume_seq (vword const **const cfg_it)
    we reach the terminator without finding any.  Consume the entire sequence
    otherwise, so that *CFG_IT points just past the terminator, which may be the
    beginning of the next sequence.  */
-static inline void
+static inline bool
 check_seq (vword const *const visited, vword const **const cfg_it)
 {
   vword mask;
   size_t wordidx;
 
   /* If the block was visited, check that at least one of the
-     preds was also visited.  */
+     preds/succs was also visited.  */
   do
     /* If we get to the end of the sequence without finding any
        match, something is amiss.  */
     if (!next_pair (cfg_it, &mask, &wordidx))
-      __builtin_trap ();
+      return false;
   /* Keep searching until we find a match, at which point the
      condition is satisfied.  */
   while (!test_mask (visited, mask, wordidx));
@@ -139,6 +150,94 @@ check_seq (vword const *const visited, vword const **const cfg_it)
      skipped the block, so as to position the iterator at the beginning of the
      next .  */
   consume_seq (cfg_it);
+
+  return true;
+}
+
+/* Print out the CFG with BLOCKS blocks, presumed to be associated with CALLER.
+   This is expected to be optimized out entirely, unless the verbose part of
+   __hardcfr_check_fail is enabled.  */
+static inline void
+__hardcfr_debug_cfg (size_t const blocks,
+		     void const *const caller,
+		     vword const *const cfg)
+{
+  __builtin_printf ("CFG at %p, for %p", cfg, caller);
+  vword const *cfg_it = cfg;
+  for (size_t i = 0; i < blocks; i++)
+    {
+      vword mask; size_t wordidx;
+      block2mask (i, &mask, &wordidx);
+      __builtin_printf ("\nblock %lu (%lu/0x%lx)\npreds: ",
+			(unsigned long)i,
+			(unsigned long)wordidx, (unsigned long)mask);
+      while (next_pair (&cfg_it, &mask, &wordidx))
+	__builtin_printf (" (%lu/0x%lx)",
+			  (unsigned long)wordidx, (unsigned long)mask);
+      __builtin_printf ("\nsuccs: ");
+      while (next_pair (&cfg_it, &mask, &wordidx))
+	__builtin_printf (" (%lu/0x%lx)",
+			  (unsigned long)wordidx, (unsigned long)mask);
+    }
+  __builtin_printf ("\n");
+}
+
+#ifndef ATTRIBUTE_UNUSED
+# define ATTRIBUTE_UNUSED __attribute__ ((__unused__))
+#endif
+
+/* This is called when an out-of-line hardcfr check fails.  All the arguments
+   are ignored, and it just traps, unless HARDCFR_VERBOSE_FAIL is enabled.  IF
+   it is, it prints the PART of the CFG, expected to have BLOCKS blocks, that
+   failed at CALLER's BLOCK, and the VISITED bitmap.  When the verbose mode is
+   enabled, it also forces __hardcfr_debug_cfg (above) to be compiled into an
+   out-of-line function, that could be called from a debugger.
+   */
+static inline void
+__hardcfr_check_fail (size_t const blocks ATTRIBUTE_UNUSED,
+		      vword const *const visited,
+		      vword const *const cfg ATTRIBUTE_UNUSED,
+		      size_t const block ATTRIBUTE_UNUSED,
+		      int const part ATTRIBUTE_UNUSED,
+		      void const *const caller ATTRIBUTE_UNUSED)
+{
+#if HARDCFR_VERBOSE_FAIL
+  static const char *parts[] = { "preds", "succs" };
+
+  vword mask; size_t wordidx;
+  block2mask (block, &mask, &wordidx);
+  __builtin_printf ("hardcfr fail at %p block %lu (%lu/0x%lx), expected %s:",
+		    caller, (unsigned long)block,
+		    (unsigned long)wordidx, (unsigned long)mask,
+		    parts[part]);
+
+  /* Skip data for previous blocks.  */
+  vword const *cfg_it = cfg;
+  for (size_t i = block; i--; )
+    {
+      consume_seq (&cfg_it);
+      consume_seq (&cfg_it);
+    }
+  for (size_t i = part; i--; )
+    consume_seq (&cfg_it);
+
+  while (next_pair (&cfg_it, &mask, &wordidx))
+    __builtin_printf (" (%lu/0x%lx)",
+		      (unsigned long)wordidx, (unsigned long)mask);
+
+  __builtin_printf ("\nvisited:");
+  block2mask (blocks, &mask, &wordidx);
+  for (size_t i = 0; i <= wordidx; i++)
+    __builtin_printf (" (%lu/0x%lx)",
+		      (unsigned long)i, (unsigned long)visited[i]);
+  __builtin_printf ("\n");
+
+  /* Reference __hardcfr_debug_cfg so that it's output out-of-line, so that it
+     can be called from a debugger.  */
+  if (!caller || caller == __hardcfr_debug_cfg)
+    return;
+#endif
+  __builtin_trap ();
 }
 
 /* Check that, for each of the BLOCKS basic blocks, if its bit is set in
@@ -168,9 +267,13 @@ __hardcfr_check (size_t const blocks,
       else
 	{
 	  /* Check predecessors.  */
-	  check_seq (visited, &cfg_it);
+	  if (!check_seq (visited, &cfg_it))
+	    __hardcfr_check_fail (blocks, visited, cfg, i, 0,
+				  __builtin_return_address (0));
 	  /* Check successors.  */
-	  check_seq (visited, &cfg_it);
+	  if (!check_seq (visited, &cfg_it))
+	    __hardcfr_check_fail (blocks, visited, cfg, i, 1,
+				  __builtin_return_address (0));
 	}
     }
 }

^ permalink raw reply	[flat|nested] 13+ messages in thread

end of thread, other threads:[~2023-06-09  6:16 UTC | newest]

Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-09-07 23:58 [gcc(refs/users/aoliva/heads/testme)] hardcfr: add optional checkpoints Alexandre Oliva
  -- strict thread matches above, loose matches on Subject: below --
2023-06-09  6:16 Alexandre Oliva
2023-06-08 10:58 Alexandre Oliva
2023-06-08 10:42 Alexandre Oliva
2023-06-08  9:16 Alexandre Oliva
2023-06-08  4:47 Alexandre Oliva
2022-10-25  2:51 Alexandre Oliva
2022-10-20  4:09 Alexandre Oliva
2022-10-06 10:31 Alexandre Oliva
2022-10-01  4:50 Alexandre Oliva
2022-09-09  7:08 Alexandre Oliva
2022-09-07 23:51 Alexandre Oliva
2022-09-06  1:28 Alexandre Oliva

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).