[PATCH] Move Asan instrumentation to sanopt pass

[PATCH] Move Asan instrumentation to sanopt pass

[Yury Gribov]

[-- Attachment #1: Type: text/plain, Size: 1447 bytes --]

Hi all,

Attached patch delays generation of Asan memory checking code
until sanopt pass. This is a first step towards global static analysis
of Asan instrumentation which would allow to
* remove redundant instrumentations
* aggregate adjacent Asan checks
* move invariant checks from loops

The patch also changes the logic behind 
asan-instrumentation-with-call-threshold
parameter to more closely match LLVM.

The patch splits build_check_stmt routine to two parts. The first one
(called from asan0/asan passes) inserts calls to internal functions
ASAN_LOAD and ASAN_STORE. The second expands those to inline checks
(in asan_expand_check_ifn).

Here are some obvious disadvantages:
* passing additional info via hidden parameter of
ASAN_{LOAD,STORE} is ugly but I'm not sure how to do this better
* delayed expansion runs after all optimization passes
so inlined Asan checks will not get a chance to be
CSE-ed, etc.; this may probably be solved by moving sanopt earlier
in the pipeline. BTW I haven't experienced notable slowdowns in my 
experiments.
* passing program pointers to ASAN_{LOAD,STORE} may damage alias analysis
because all pointers will now escape; I probably could
provide fnspec with (EAF_DIRECT | EAF_NOCLOBBER | EAF_NOESCAPE) or
even EAF_UNUSED for these functions but this does not seem
to be supported in current middle-end.

The patch was bootstrapped, regtested and asan-bootstrapped on x64.

Is this ok for trunk?

-Yury

[-- Attachment #2: asan_ifn-1.diff --]
[-- Type: text/x-diff, Size: 41364 bytes --]

commit ec53fb00ab4a762c3c4cefa886f6cd9ee549de8d
Author: Yury Gribov <y.gribov@samsung.com>
Date:   Thu Jul 17 09:45:26 2014 +0400

    Move inlining of Asan memory checks to sanopt pass.
    Change asan-instrumentation-with-call-threshold to more closely match LLVM.
    
    gcc/
    
    2014-07-17  Yury Gribov  <y.gribov@samsung.com>
    
    	* asan.c (asan_check_flags): New enum.
    	(build_check_stmt_with_calls): Removed function.
    	(build_check_stmt): Split inlining logic to
    	asan_expand_check_ifn.
    	(instrument_derefs): Rename parameter.
    	(instrument_mem_region_access): Rename parameter.
    	(instrument_strlen_call): Likewise.
    	(asan_expand_check_ifn): New function.
    	(asan_instrument): Remove old code.
    	(pass_sanopt::execute): Change handling of
    	asan-instrumentation-with-call-threshold.
    	* doc/invoke.texi (asan-instrumentation-with-call-threshold):
    	Update description.
    	* gimple_iterator.h (gsi_start_bb): Fix uninitialized
    	warnings.
    	* internal-fn.c (expand_ASAN_LOAD): New function.
    	(expand_ASAN_STORE): Likewise.
    	* internal-fn.def (ASAN_LOAD): New internal function.
    	(ASAN_STORE): Likewise.
    	* params.def (PARAM_ASAN_INSTRUMENTATION_WITH_CALL_THRESHOLD):
    	Update description.
    	(PARAM_ASAN_USE_AFTER_RETURN): Likewise.
    
    gcc/testsuite/
    
    2014-07-17  Yury Gribov  <y.gribov@samsung.com>
    
    	* c-c++-common/asan/inc.c: Update test.
    	* c-c++-common/asan/instrument-with-calls-2.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-1.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-2.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-3.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-4.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-5.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-6.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-7.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-8.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-9.c: Likewise.

diff --git a/gcc/asan.c b/gcc/asan.c
index 0d78634..7fe079d 100644
--- a/gcc/asan.c
+++ b/gcc/asan.c
@@ -243,18 +243,15 @@ static GTY(()) tree shadow_ptr_types[2];
 /* Decl for __asan_option_detect_stack_use_after_return.  */
 static GTY(()) tree asan_detect_stack_use_after_return;
 
-/* Number of instrumentations in current function so far.  */
-
-static int asan_num_accesses;
-
-/* Check whether we should replace inline instrumentation with calls.  */
-
-static inline bool
-use_calls_p ()
+/* Various flags for Asan builtins.  */
+enum asan_check_flags
 {
-  return ASAN_INSTRUMENTATION_WITH_CALL_THRESHOLD < INT_MAX
-    && asan_num_accesses >= ASAN_INSTRUMENTATION_WITH_CALL_THRESHOLD;
-}
+  ASAN_CHECK_NON_ZERO_LEN = 1 << 0,
+  ASAN_CHECK_SCALAR_ACCESS = 1 << 1,
+  ASAN_CHECK_START_INSTRUMENTED = 1 << 2,
+  ASAN_CHECK_END_INSTRUMENTED = 1 << 3,
+  ASAN_CHECK_LAST
+};
 
 /* Hashtable support for memory references used by gimple
    statements.  */
@@ -1553,55 +1550,6 @@ maybe_create_ssa_name (location_t loc, tree base, gimple_stmt_iterator *iter,
   return gimple_assign_lhs (g);
 }
 
-/* Instrument the memory access instruction using callbacks.
-   Parameters are similar to BUILD_CHECK_STMT.  */
-
-static void
-build_check_stmt_with_calls (location_t loc, tree base, tree len,
-			     HOST_WIDE_INT size_in_bytes, gimple_stmt_iterator *iter,
-			     bool before_p, bool is_store, bool is_scalar_access)
-{
-  gimple_stmt_iterator gsi = *iter;
-  tree base_ssa = maybe_create_ssa_name (loc, base, &gsi, before_p);
-
-  gimple g
-    = gimple_build_assign_with_ops (NOP_EXPR,
-				    make_ssa_name (pointer_sized_int_node, NULL),
-				    base_ssa, NULL_TREE);
-  gimple_set_location (g, loc);
-  if (before_p)
-    gsi_insert_before (&gsi, g, GSI_NEW_STMT);
-  else
-    gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-  tree base_addr = gimple_assign_lhs (g);
-
-  int nargs;
-  tree fun
-    = check_func (is_store, is_scalar_access ? size_in_bytes : -1, &nargs);
-  if (nargs == 1)
-    g = gimple_build_call (fun, 1, base_addr);
-  else
-    {
-      gcc_assert (nargs == 2);
-      g = gimple_build_assign_with_ops (NOP_EXPR,
-					make_ssa_name (pointer_sized_int_node,
-						       NULL),
-					len, NULL_TREE);
-      gimple_set_location (g, loc);
-      gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-      tree sz_arg = gimple_assign_lhs (g);
-      g = gimple_build_call (fun, nargs, base_addr, sz_arg);
-    }
-  gimple_set_location (g, loc);
-  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-
-  if (!before_p)
-    {
-      gsi_next (&gsi);
-      *iter = gsi;
-    }
-}
-
 /* Instrument the memory access instruction BASE.  Insert new
    statements before or after ITER.
 
@@ -1622,27 +1570,30 @@ build_check_stmt_with_calls (location_t loc, tree base, tree len,
    otherwise, it points to the statement logically following it.  */
 
 static void
-build_check_stmt (location_t location, tree base, tree len,
+build_check_stmt (location_t loc, tree base, tree len,
 		  HOST_WIDE_INT size_in_bytes, gimple_stmt_iterator *iter,
-		  bool non_zero_len_p, bool before_p, bool is_store,
+		  bool is_non_zero_len, bool before_p, bool is_store,
 		  bool is_scalar_access, unsigned int align = 0,
 		  bool start_instrumented = false,
 		  bool end_instrumented = false)
 {
   gimple_stmt_iterator gsi = *iter;
   gimple g;
-  tree uintptr_type
-    = build_nonstandard_integer_type (TYPE_PRECISION (TREE_TYPE (base)), 1);
 
-  gcc_assert (!(size_in_bytes > 0 && !non_zero_len_p));
+  gcc_assert (!(size_in_bytes > 0 && !is_non_zero_len));
 
   if (start_instrumented && end_instrumented)
     {
       if (!before_p)
-        gsi_next (iter);
+	gsi_next (iter);
       return;
     }
 
+  gsi = *iter;
+
+  base = unshare_expr (base);
+  base = maybe_create_ssa_name (loc, base, &gsi, before_p);
+
   if (len)
     len = unshare_expr (len);
   else
@@ -1654,9 +1605,8 @@ build_check_stmt (location_t location, tree base, tree len,
   if (size_in_bytes > 1)
     {
       if ((size_in_bytes & (size_in_bytes - 1)) != 0
-	  || !is_scalar_access
 	  || size_in_bytes > 16)
-	size_in_bytes = -1;
+	is_scalar_access = false;
       else if (align && align < size_in_bytes * BITS_PER_UNIT)
 	{
 	  /* On non-strict alignment targets, if
@@ -1667,189 +1617,33 @@ build_check_stmt (location_t location, tree base, tree len,
 	  if (size_in_bytes != 16
 	      || STRICT_ALIGNMENT
 	      || align < 8 * BITS_PER_UNIT)
-	    size_in_bytes = -1;
-	}
-    }
-
-  HOST_WIDE_INT real_size_in_bytes = size_in_bytes == -1 ? 1 : size_in_bytes;
-
-  tree shadow_ptr_type = shadow_ptr_types[real_size_in_bytes == 16 ? 1 : 0];
-  tree shadow_type = TREE_TYPE (shadow_ptr_type);
-
-  base = unshare_expr (base);
-
-  if (use_calls_p ())
-    {
-      gsi = *iter;
-      build_check_stmt_with_calls (location, base, len, size_in_bytes, iter,
-				   before_p, is_store, is_scalar_access);
-      return;
-    }
-
-  ++asan_num_accesses;
-
-  if (!non_zero_len_p)
-    {
-      gcc_assert (before_p);
-
-      /* So, the length of the memory area to asan-protect is
-	 non-constant.  Let's guard the generated instrumentation code
-	 like:
-
-	 if (len != 0)
-	   {
-	     //asan instrumentation code goes here.
-	   }
-	 // falltrough instructions, starting with *ITER.  */
-
-      g = gimple_build_cond (NE_EXPR,
-			     len,
-			     build_int_cst (TREE_TYPE (len), 0),
-			     NULL_TREE, NULL_TREE);
-      gimple_set_location (g, location);
-
-      basic_block then_bb, fallthrough_bb;
-      insert_if_then_before_iter (g, iter, /*then_more_likely_p=*/true,
-				  &then_bb, &fallthrough_bb);
-      /* Note that fallthrough_bb starts with the statement that was
-	 pointed to by ITER.  */
-
-      /* The 'then block' of the 'if (len != 0) condition is where
-	 we'll generate the asan instrumentation code now.  */
-      gsi = gsi_last_bb (then_bb);
-      build_check_stmt (location, base, len, size_in_bytes, &gsi,
-			/*non_zero_len_p*/true, /*before_p*/true, is_store,
-			is_scalar_access, align,
-			start_instrumented, end_instrumented);
-      return;
-    }
-
-  /* Get an iterator on the point where we can add the condition
-     statement for the instrumentation.  */
-  basic_block then_bb, else_bb;
-  gsi = create_cond_insert_point (&gsi, before_p,
-				  /*then_more_likely_p=*/false,
-				  /*create_then_fallthru_edge=*/false,
-				  &then_bb,
-				  &else_bb);
-
-  tree base_ssa = maybe_create_ssa_name (location, base, &gsi,
-					 /*before_p*/false);
-
-  g = gimple_build_assign_with_ops (NOP_EXPR,
-				    make_ssa_name (uintptr_type, NULL),
-				    base_ssa, NULL_TREE);
-  gimple_set_location (g, location);
-  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-  tree base_addr = gimple_assign_lhs (g);
-
-  tree t = NULL_TREE;
-  if (real_size_in_bytes >= 8)
-    {
-      tree shadow = build_shadow_mem_access (&gsi, location, base_addr,
-					     shadow_ptr_type);
-      t = shadow;
-    }
-  else
-    {
-      /* Slow path for 1, 2 and 4 byte accesses.  */
-
-      if (!start_instrumented)
-	{
-	  /* Test (shadow != 0)
-		  & ((base_addr & 7) + (real_size_in_bytes - 1)) >= shadow).  */
-	  tree shadow = build_shadow_mem_access (&gsi, location, base_addr,
-						 shadow_ptr_type);
-	  gimple shadow_test = build_assign (NE_EXPR, shadow, 0);
-	  gimple_seq seq = NULL;
-	  gimple_seq_add_stmt (&seq, shadow_test);
-	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR, base_addr, 7));
-	  gimple_seq_add_stmt (&seq, build_type_cast (shadow_type,
-						      gimple_seq_last (seq)));
-	  if (real_size_in_bytes > 1)
-	    gimple_seq_add_stmt (&seq,
-				 build_assign (PLUS_EXPR, gimple_seq_last (seq),
-					       real_size_in_bytes - 1));
-	  gimple_seq_add_stmt (&seq, build_assign (GE_EXPR,
-						   gimple_seq_last (seq),
-						   shadow));
-	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR, shadow_test,
-						   gimple_seq_last (seq)));
-	  t = gimple_assign_lhs (gimple_seq_last (seq));
-	  gimple_seq_set_location (seq, location);
-	  gsi_insert_seq_after (&gsi, seq, GSI_CONTINUE_LINKING);
-	}
-
-      /* For non-constant, misaligned or otherwise weird access sizes,
-	 check first and last byte.  */
-      if (size_in_bytes == -1 && !end_instrumented)
-	{
-	  g = gimple_build_assign_with_ops (MINUS_EXPR,
-					    make_ssa_name (uintptr_type, NULL),
-					    len,
-					    build_int_cst (uintptr_type, 1));
-	  gimple_set_location (g, location);
-	  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-	  tree last = gimple_assign_lhs (g);
-	  g = gimple_build_assign_with_ops (PLUS_EXPR,
-					    make_ssa_name (uintptr_type, NULL),
-					    base_addr,
-					    last);
-	  gimple_set_location (g, location);
-	  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-	  tree base_end_addr = gimple_assign_lhs (g);
-
-	  tree shadow = build_shadow_mem_access (&gsi, location, base_end_addr,
-						 shadow_ptr_type);
-	  gimple shadow_test = build_assign (NE_EXPR, shadow, 0);
-	  gimple_seq seq = NULL;
-	  gimple_seq_add_stmt (&seq, shadow_test);
-	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR,
-						   base_end_addr, 7));
-	  gimple_seq_add_stmt (&seq, build_type_cast (shadow_type,
-						      gimple_seq_last (seq)));
-	  gimple_seq_add_stmt (&seq, build_assign (GE_EXPR,
-						   gimple_seq_last (seq),
-						   shadow));
-	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR, shadow_test,
-						   gimple_seq_last (seq)));
-	  if (!start_instrumented)
-	    gimple_seq_add_stmt (&seq, build_assign (BIT_IOR_EXPR, t,
-						     gimple_seq_last (seq)));
-	  t = gimple_assign_lhs (gimple_seq_last (seq));
-	  gimple_seq_set_location (seq, location);
-	  gsi_insert_seq_after (&gsi, seq, GSI_CONTINUE_LINKING);
+	    is_scalar_access = false;
 	}
     }
 
-  g = gimple_build_cond (NE_EXPR, t, build_int_cst (TREE_TYPE (t), 0),
-			 NULL_TREE, NULL_TREE);
-  gimple_set_location (g, location);
-  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-
-  /* Generate call to the run-time library (e.g. __asan_report_load8).  */
-  gsi = gsi_start_bb (then_bb);
-  int nargs;
-  tree fun = report_error_func (is_store, is_scalar_access ? size_in_bytes : -1,
-				&nargs);
-  if (nargs == 1)
-      g = gimple_build_call (fun, 1, base_addr);
+  HOST_WIDE_INT flags = 0;
+  if (is_non_zero_len)
+    flags |= ASAN_CHECK_NON_ZERO_LEN;
+  if (is_scalar_access)
+    flags |= ASAN_CHECK_SCALAR_ACCESS;
+  if (start_instrumented)
+    flags |= ASAN_CHECK_START_INSTRUMENTED;
+  if (end_instrumented)
+    flags |= ASAN_CHECK_END_INSTRUMENTED;
+
+  g = gimple_build_call_internal (is_store ? IFN_ASAN_STORE : IFN_ASAN_LOAD,
+				  3,
+				  build_int_cst (integer_type_node, flags),
+				  base, len);
+  gimple_set_location (g, loc);
+  if (before_p)
+    gsi_insert_before (&gsi, g, GSI_SAME_STMT);
   else
     {
-      gcc_assert (nargs == 2);
-      g = gimple_build_assign_with_ops (NOP_EXPR,
-					make_ssa_name (pointer_sized_int_node,
-						       NULL),
-					len, NULL_TREE);
-      gimple_set_location (g, location);
       gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-      tree sz_arg = gimple_assign_lhs (g);
-      g = gimple_build_call (fun, nargs, base_addr, sz_arg);
+      gsi_next (&gsi);
+      *iter = gsi;
     }
-  gimple_set_location (g, location);
-  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-
-  *iter = gsi_start_bb (else_bb);
 }
 
 /* If T represents a memory access, add instrumentation code before ITER.
@@ -1942,7 +1736,7 @@ instrument_derefs (gimple_stmt_iterator *iter, tree t,
     {
       unsigned int align = get_object_alignment (t);
       build_check_stmt (location, base, NULL_TREE, size_in_bytes, iter,
-			/*non_zero_len_p*/size_in_bytes > 0, /*before_p=*/true,
+			/*is_non_zero_len*/size_in_bytes > 0, /*before_p=*/true,
 			is_store, /*is_scalar_access*/true, align);
       update_mem_ref_hash_table (base, size_in_bytes);
       update_mem_ref_hash_table (t, size_in_bytes);
@@ -1980,7 +1774,7 @@ instrument_mem_region_access (tree base, tree len,
   HOST_WIDE_INT size_in_bytes = tree_fits_shwi_p (len) ? tree_to_shwi (len) : -1;
 
   build_check_stmt (location, base, len, size_in_bytes, iter,
-		    /*non_zero_len_p*/size_in_bytes > 0, /*before_p*/true,
+		    /*is_non_zero_len*/size_in_bytes > 0, /*before_p*/true,
 		    is_store, /*is_scalar_access*/false, /*align*/0,
 		    start_instrumented, end_instrumented);
 
@@ -2028,6 +1822,7 @@ instrument_strlen_call (gimple_stmt_iterator *iter)
   tree str_arg = gimple_call_arg (call, 0);
   bool start_instrumented = has_mem_ref_been_instrumented (str_arg, 1);
 
+  // TODO: remove this?
   tree cptr_type = build_pointer_type (char_type_node);
   gimple str_arg_ssa =
     gimple_build_assign_with_ops (NOP_EXPR,
@@ -2037,7 +1832,7 @@ instrument_strlen_call (gimple_stmt_iterator *iter)
   gsi_insert_before (iter, str_arg_ssa, GSI_SAME_STMT);
 
   build_check_stmt (loc, gimple_assign_lhs (str_arg_ssa), NULL_TREE, 1, iter,
-		    /*non_zero_len_p*/true, /*before_p=*/true,
+		    /*is_non_zero_len*/true, /*before_p=*/true,
 		    /*is_store=*/false, /*is_scalar_access*/true, /*align*/0,
 		    start_instrumented, start_instrumented);
 
@@ -2050,7 +1845,7 @@ instrument_strlen_call (gimple_stmt_iterator *iter)
   gsi_insert_after (iter, g, GSI_NEW_STMT);
 
   build_check_stmt (loc, gimple_assign_lhs (g), NULL_TREE, 1, iter,
-		    /*non_zero_len_p*/true, /*before_p=*/false,
+		    /*is_non_zero_len*/true, /*before_p=*/false,
 		    /*is_store=*/false, /*is_scalar_access*/true, /*align*/0);
 
   return true;
@@ -2617,6 +2412,214 @@ asan_finish_file (void)
   flag_sanitize |= SANITIZE_ADDRESS;
 }
 
+/* Expand the ASAN_{LOAD,STORE} builtins.  */
+
+static void
+asan_expand_check_ifn (gimple_stmt_iterator *iter, bool is_store,
+		       bool use_calls)
+{
+  gimple g = gsi_stmt (*iter);
+  location_t loc = gimple_location (g);
+
+  HOST_WIDE_INT flags = tree_to_shwi (gimple_call_arg (g, 0));
+  gcc_assert (flags < ASAN_CHECK_LAST);
+  bool is_scalar_access = (flags & ASAN_CHECK_SCALAR_ACCESS) != 0;
+
+  tree base = gimple_call_arg (g, 1);
+  tree len = gimple_call_arg (g, 2);
+
+  HOST_WIDE_INT size_in_bytes
+    = is_scalar_access && tree_fits_shwi_p (len) ? tree_to_shwi (len) : -1;
+
+  if (use_calls)
+    {
+      /* Instrument using callbacks.  */
+      gimple g
+        = gimple_build_assign_with_ops (NOP_EXPR,
+				        make_ssa_name (pointer_sized_int_node,
+                                        NULL),
+				        base, NULL_TREE);
+      gimple_set_location (g, loc);
+      gsi_insert_before (iter, g, GSI_SAME_STMT);
+      tree base_addr = gimple_assign_lhs (g);
+
+      int nargs;
+      tree fun = check_func (is_store, size_in_bytes, &nargs);
+      if (nargs == 1)
+        g = gimple_build_call (fun, 1, base_addr);
+      else
+        {
+          gcc_assert (nargs == 2);
+          g = gimple_build_assign_with_ops (NOP_EXPR,
+					    make_ssa_name (pointer_sized_int_node,
+						           NULL),
+					    len, NULL_TREE);
+          gimple_set_location (g, loc);
+          gsi_insert_before (iter, g, GSI_SAME_STMT);
+          tree sz_arg = gimple_assign_lhs (g);
+          g = gimple_build_call (fun, nargs, base_addr, sz_arg);
+        }
+      gimple_set_location (g, loc);
+      gsi_replace (iter, g, false);
+      return;
+    }
+
+  bool is_non_zero_len = (flags & ASAN_CHECK_NON_ZERO_LEN) != 0;
+  bool start_instrumented = (flags & ASAN_CHECK_START_INSTRUMENTED) != 0;
+  bool end_instrumented = (flags & ASAN_CHECK_END_INSTRUMENTED) != 0;
+
+  HOST_WIDE_INT real_size_in_bytes = size_in_bytes == -1 ? 1 : size_in_bytes;
+
+  tree uintptr_type
+    = build_nonstandard_integer_type (TYPE_PRECISION (TREE_TYPE (base)), 1);
+
+  tree shadow_ptr_type = shadow_ptr_types[real_size_in_bytes == 16 ? 1 : 0];
+  tree shadow_type = TREE_TYPE (shadow_ptr_type);
+
+  gimple_stmt_iterator gsi = *iter;
+
+  if (!is_non_zero_len)
+    {
+      /* So, the length of the memory area to asan-protect is
+	 non-constant.  Let's guard the generated instrumentation code
+	 like:
+
+	 if (len != 0)
+	   {
+	     //asan instrumentation code goes here.
+	   }
+	 // falltrough instructions, starting with *ITER.  */
+
+      g = gimple_build_cond (NE_EXPR,
+			    len,
+			    build_int_cst (TREE_TYPE (len), 0),
+			    NULL_TREE, NULL_TREE);
+      gimple_set_location (g, loc);
+
+      basic_block then_bb, fallthrough_bb;
+      insert_if_then_before_iter (g, iter, /*then_more_likely_p=*/true,
+				 &then_bb, &fallthrough_bb);
+      /* Note that fallthrough_bb starts with the statement that was
+	pointed to by ITER.  */
+
+      /* The 'then block' of the 'if (len != 0) condition is where
+	we'll generate the asan instrumentation code now.  */
+      gsi = gsi_last_bb (then_bb);
+    }
+
+  /* Get an iterator on the point where we can add the condition
+     statement for the instrumentation.  */
+  basic_block then_bb, else_bb;
+  gsi = create_cond_insert_point (&gsi, /*before_p*/false,
+				  /*then_more_likely_p=*/false,
+				  /*create_then_fallthru_edge=*/false,
+				  &then_bb,
+				  &else_bb);
+
+  g = gimple_build_assign_with_ops (NOP_EXPR,
+				    make_ssa_name (pointer_sized_int_node,
+						   NULL),
+				    base, NULL_TREE);
+  gimple_set_location (g, loc);
+  gsi_insert_before (&gsi, g, GSI_NEW_STMT);
+  tree base_addr = gimple_assign_lhs (g);
+
+  tree t = NULL_TREE;
+  if (real_size_in_bytes >= 8)
+    {
+      tree shadow = build_shadow_mem_access (&gsi, loc, base_addr,
+					     shadow_ptr_type);
+      t = shadow;
+    }
+  else
+    {
+      /* Slow path for 1, 2 and 4 byte accesses.  */
+
+      if (!start_instrumented)
+	{
+	  /* Test (shadow != 0)
+	     & ((base_addr & 7) + (real_size_in_bytes - 1)) >= shadow).  */
+	  tree shadow = build_shadow_mem_access (&gsi, loc, base_addr,
+						 shadow_ptr_type);
+	  gimple shadow_test = build_assign (NE_EXPR, shadow, 0);
+	  gimple_seq seq = NULL;
+	  gimple_seq_add_stmt (&seq, shadow_test);
+	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR, base_addr, 7));
+	  gimple_seq_add_stmt (&seq, build_type_cast (shadow_type,
+						      gimple_seq_last (seq)));
+	  if (real_size_in_bytes > 1)
+	    gimple_seq_add_stmt (&seq,
+				 build_assign (PLUS_EXPR, gimple_seq_last (seq),
+					       real_size_in_bytes - 1));
+	  gimple_seq_add_stmt (&seq, build_assign (GE_EXPR,
+						   gimple_seq_last (seq),
+						   shadow));
+	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR, shadow_test,
+						   gimple_seq_last (seq)));
+	  t = gimple_assign_lhs (gimple_seq_last (seq));
+	  gimple_seq_set_location (seq, loc);
+	  gsi_insert_seq_after (&gsi, seq, GSI_CONTINUE_LINKING);
+	}
+
+      /* For non-constant, misaligned or otherwise weird access sizes,
+	 check first and last byte.  */
+      if (size_in_bytes == -1 && !end_instrumented)
+	{
+	  g = gimple_build_assign_with_ops (MINUS_EXPR,
+					    make_ssa_name (uintptr_type, NULL),
+					    len,
+					    build_int_cst (uintptr_type, 1));
+	  gimple_set_location (g, loc);
+	  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
+	  tree last = gimple_assign_lhs (g);
+	  g = gimple_build_assign_with_ops (PLUS_EXPR,
+					    make_ssa_name (uintptr_type, NULL),
+					    base_addr,
+					    last);
+	  gimple_set_location (g, loc);
+	  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
+	  tree base_end_addr = gimple_assign_lhs (g);
+
+	  tree shadow = build_shadow_mem_access (&gsi, loc, base_end_addr,
+						 shadow_ptr_type);
+	  gimple shadow_test = build_assign (NE_EXPR, shadow, 0);
+	  gimple_seq seq = NULL;
+	  gimple_seq_add_stmt (&seq, shadow_test);
+	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR,
+						   base_end_addr, 7));
+	  gimple_seq_add_stmt (&seq, build_type_cast (shadow_type,
+						      gimple_seq_last (seq)));
+	  gimple_seq_add_stmt (&seq, build_assign (GE_EXPR,
+						   gimple_seq_last (seq),
+						   shadow));
+	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR, shadow_test,
+						   gimple_seq_last (seq)));
+	  if (!start_instrumented)
+	    gimple_seq_add_stmt (&seq, build_assign (BIT_IOR_EXPR, t,
+						     gimple_seq_last (seq)));
+	  t = gimple_assign_lhs (gimple_seq_last (seq));
+	  gimple_seq_set_location (seq, loc);
+	  gsi_insert_seq_after (&gsi, seq, GSI_CONTINUE_LINKING);
+	}
+    }
+
+  g = gimple_build_cond (NE_EXPR, t, build_int_cst (TREE_TYPE (t), 0),
+			 NULL_TREE, NULL_TREE);
+  gimple_set_location (g, loc);
+  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
+
+  /* Generate call to the run-time library (e.g. __asan_report_load8).  */
+  gsi = gsi_start_bb (then_bb);
+  int nargs;
+  tree fun = report_error_func (is_store, size_in_bytes, &nargs);
+  g = gimple_build_call (fun, nargs, base_addr, len);
+  gimple_set_location (g, loc);
+  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
+
+  gsi_remove (iter, true);
+  *iter = gsi_start_bb (else_bb);
+}
+
 /* Instrument the current function.  */
 
 static unsigned int
@@ -2624,7 +2627,6 @@ asan_instrument (void)
 {
   if (shadow_ptr_types[0] == NULL_TREE)
     asan_init_shadow_ptr_types ();
-  asan_num_accesses = 0;
   transform_statements ();
   return 0;
 }
@@ -2744,10 +2746,35 @@ unsigned int
 pass_sanopt::execute (function *fun)
 {
   basic_block bb;
+  gimple_stmt_iterator gsi;
 
+  int asan_num_accesses = 0;
+  FOR_EACH_BB_FN (bb, fun)
+    for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi))
+      {
+ 	gimple stmt = gsi_stmt (gsi);
+	if (is_gimple_call (stmt) && gimple_call_internal_p (stmt))
+	  {
+	    enum internal_fn ifn = gimple_call_internal_fn (stmt);
+	    switch (ifn)
+	      {
+	      case IFN_ASAN_LOAD:
+	      case IFN_ASAN_STORE:
+		{
+		  ++asan_num_accesses;
+		  break;
+		}
+	      default:
+		break;
+	      }
+	    }
+        }
+
+  bool use_calls = ASAN_INSTRUMENTATION_WITH_CALL_THRESHOLD < INT_MAX
+    && asan_num_accesses >= ASAN_INSTRUMENTATION_WITH_CALL_THRESHOLD;
+    
   FOR_EACH_BB_FN (bb, fun)
     {
-      gimple_stmt_iterator gsi;
       for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi))
 	{
 	  gimple stmt = gsi_stmt (gsi);
@@ -2756,17 +2783,27 @@ pass_sanopt::execute (function *fun)
 	    continue;
 
 	  if (gimple_call_internal_p (stmt))
-	    switch (gimple_call_internal_fn (stmt))
-	      {
-	      case IFN_UBSAN_NULL:
-		ubsan_expand_null_ifn (gsi);
-		break;
-	      case IFN_UBSAN_BOUNDS:
-		ubsan_expand_bounds_ifn (&gsi);
-		break;
-	      default:
-		break;
-	      }
+	    {
+	      enum internal_fn ifn = gimple_call_internal_fn (stmt);
+	      switch (ifn)
+		{
+		case IFN_UBSAN_NULL:
+		  ubsan_expand_null_ifn (gsi);
+		  break;
+		case IFN_UBSAN_BOUNDS:
+		  ubsan_expand_bounds_ifn (&gsi);
+		  break;
+		case IFN_ASAN_LOAD:
+		case IFN_ASAN_STORE:
+		  {
+		    bool is_store = ifn == IFN_ASAN_STORE;
+		    asan_expand_check_ifn (&gsi, is_store, use_calls);
+		    break;
+		  }
+		default:
+		  break;
+		}
+	    }
 
 	  if (dump_file && (dump_flags & TDF_DETAILS))
 	    {
@@ -2775,7 +2812,7 @@ pass_sanopt::execute (function *fun)
 	      fprintf (dump_file, "\n");
 	    }
 
-	  /* ubsan_expand_bounds_ifn might move us to the end of the BB.  */
+	  /* Expansions might move us to the end of the BB.  */
 	  if (gsi_end_p (gsi))
 	    break;
 	}
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index b5e8d98..8a755df 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -10312,9 +10312,9 @@ To disable use-after-return detection use
 @option{--param asan-use-after-return=0}.
 
 @item asan-instrumentation-with-call-threshold
-Once number of memory accesses  in function becomes greater
-or equal than this number, use callbacks instead of
-generating inline code.  E.g. to disable inline code use
+If number of memory accesses in function being instrumented
+is greater or equal to this number, use callbacks instead of inline checks.
+E.g. to disable inline code use
 @option{--param asan-instrumentation-with-call-threshold=0}.
 
 @end table
diff --git a/gcc/gimple-iterator.h b/gcc/gimple-iterator.h
index 909d58b..0761bba 100644
--- a/gcc/gimple-iterator.h
+++ b/gcc/gimple-iterator.h
@@ -116,6 +116,7 @@ gsi_start_bb (basic_block bb)
   gimple_seq *seq;
 
   seq = bb_seq_addr (bb);
+  gcc_assert (seq);
   i.ptr = gimple_seq_first (*seq);
   i.seq = seq;
   i.bb = bb;
diff --git a/gcc/internal-fn.c b/gcc/internal-fn.c
index 78f59d6..f7935b2 100644
--- a/gcc/internal-fn.c
+++ b/gcc/internal-fn.c
@@ -167,6 +167,20 @@ expand_UBSAN_BOUNDS (gimple stmt ATTRIBUTE_UNUSED)
   gcc_unreachable ();
 }
 
+/* These should get expanded in the sanopt pass.  */
+
+static void
+expand_ASAN_LOAD (gimple stmt ATTRIBUTE_UNUSED)
+{
+  gcc_unreachable ();
+}
+
+static void
+expand_ASAN_STORE (gimple stmt ATTRIBUTE_UNUSED)
+{
+  gcc_unreachable ();
+}
+
 /* Add sub/add overflow checking to the statement STMT.
    CODE says whether the operation is +, or -.  */
 
diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def
index f0766bc..cfa7450 100644
--- a/gcc/internal-fn.def
+++ b/gcc/internal-fn.def
@@ -54,3 +54,5 @@ DEF_INTERNAL_FN (UBSAN_CHECK_SUB, ECF_CONST | ECF_LEAF | ECF_NOTHROW)
 DEF_INTERNAL_FN (UBSAN_CHECK_MUL, ECF_CONST | ECF_LEAF | ECF_NOTHROW)
 DEF_INTERNAL_FN (ABNORMAL_DISPATCHER, ECF_NORETURN)
 DEF_INTERNAL_FN (BUILTIN_EXPECT, ECF_CONST | ECF_LEAF | ECF_NOTHROW)
+DEF_INTERNAL_FN (ASAN_LOAD, ECF_TM_PURE | ECF_LEAF | ECF_NOTHROW)
+DEF_INTERNAL_FN (ASAN_STORE, ECF_TM_PURE | ECF_LEAF | ECF_NOTHROW)
diff --git a/gcc/params.def b/gcc/params.def
index aa1e88d..f1c44e0 100644
--- a/gcc/params.def
+++ b/gcc/params.def
@@ -1072,14 +1072,14 @@ DEFPARAM (PARAM_ASAN_MEMINTRIN,
 
 DEFPARAM (PARAM_ASAN_USE_AFTER_RETURN,
          "asan-use-after-return",
-         "Enable asan builtin functions protection",
+         "Enable asan detection of use-after-return bugs",
          1, 0, 1)
 
 DEFPARAM (PARAM_ASAN_INSTRUMENTATION_WITH_CALL_THRESHOLD,
          "asan-instrumentation-with-call-threshold",
-         "Use callbacks instead of inline code once number of accesses "
-         " in function becomes greater or equal than this threshold",
-         10000, 0, INT_MAX)
+         "Use callbacks instead of inline code if number of accesses "
+         "in function becomes greater or equal to this number",
+         7000, 0, INT_MAX)
 
 DEFPARAM (PARAM_UNINIT_CONTROL_DEP_ATTEMPTS,
 	  "uninit-control-dep-attempts",
diff --git a/gcc/testsuite/c-c++-common/asan/inc.c b/gcc/testsuite/c-c++-common/asan/inc.c
index b9c6734..865d50a 100644
--- a/gcc/testsuite/c-c++-common/asan/inc.c
+++ b/gcc/testsuite/c-c++-common/asan/inc.c
@@ -16,6 +16,6 @@ main ()
   return 0;
 }
 
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report" 1 "asan0" } }  */
-/* { dg-final { scan-tree-dump "__builtin___asan_report_load4" "asan0" } }  */
+/* { dg-final { scan-tree-dump-times "ASAN_" 1 "asan0" } }  */
+/* { dg-final { scan-tree-dump "ASAN_LOAD \\(.*, 4\\);" "asan0" } }  */
 /* { dg-final { cleanup-tree-dump "asan0" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/instrument-with-calls-2.c b/gcc/testsuite/c-c++-common/asan/instrument-with-calls-2.c
index 570f796..04fdad0 100644
--- a/gcc/testsuite/c-c++-common/asan/instrument-with-calls-2.c
+++ b/gcc/testsuite/c-c++-common/asan/instrument-with-calls-2.c
@@ -9,8 +9,8 @@ void f(int *a, int *b) {
   x = *b;
 }
 
-/* { dg-final { scan-assembler-not "__asan_store4" } } */
-/* { dg-final { scan-assembler "__asan_report_store4" } } */
+/* { dg-final { scan-assembler "__asan_store4" } } */
+/* { dg-final { scan-assembler-not "__asan_report_store4" } } */
 /* { dg-final { scan-assembler "__asan_load4" } } */
 /* { dg-final { scan-assembler-not "__asan_report_load4" } } */
 /* { dg-final { cleanup-saved-temps } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-1.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-1.c
index c7c594e..028f8d7 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-1.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-1.c
@@ -2,7 +2,7 @@
    location in the same basic block, the second reference should not
    be instrumented by the Address Sanitizer.  */
 
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -62,7 +62,7 @@ main ()
   return test0 () && test1 (0);
 }
 
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store1" 3 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 2 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load" 1 "asan0" }  } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store1" 3 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 2 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load" 1 "sanopt" }  } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-2.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-2.c
index 143312f..a58411c 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-2.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-2.c
@@ -3,7 +3,7 @@
    be instrumented by the Address Sanitizer.  But in case of access to
    overlapping regions we must be precise.  */
 
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -20,7 +20,7 @@ main ()
   __builtin_memset (tab, 1, 3);
 }
 
-/* { dg-final { scan-tree-dump-times "& 7" 3 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 2 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report" 2 "asan0" }  } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "& 7" 3 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 2 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report" 2 "sanopt" }  } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-3.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-3.c
index 420a263..5193ae0 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-3.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-3.c
@@ -1,4 +1,4 @@
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -12,7 +12,7 @@ foo (__INT32_TYPE__ *p)
   return ret; 
 }
 
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report" 2 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store" 1 "asan0" } } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report" 2 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store" 1 "sanopt" } } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-4.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-4.c
index da91cd5..c3632aa 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-4.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-4.c
@@ -1,4 +1,4 @@
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -10,8 +10,8 @@ foo  (int *a, char *b, char *c)
   /* For a total of 5 checks.  */
 }
 
-/* { dg-final { scan-tree-dump-times "& 7" 5 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 1 "asan0" } } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "& 7" 5 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 1 "sanopt" } } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-5.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-5.c
index 134be66..077ea34 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-5.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-5.c
@@ -1,4 +1,4 @@
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -10,9 +10,9 @@ foo  (int *a, char *b, char *c)
   /* For a total of 5 checks.  */
 }
 
-/* { dg-final { scan-tree-dump-times "& 7" 5 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load4" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 1 "asan0" } } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "& 7" 5 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load4" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 1 "sanopt" } } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-6.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-6.c
index 55c8ee3..6d87104 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-6.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-6.c
@@ -1,4 +1,4 @@
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -12,9 +12,9 @@ foo  (int *a, char *b, char *c)
   /* For a total of 8 checks.  */
 }
 
-/* { dg-final { scan-tree-dump-times "& 7" 8 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load4" 2 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 2 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 2 "asan0" } } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "& 7" 8 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load4" 2 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 2 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 2 "sanopt" } } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-7.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-7.c
index a04956d..5baa10d 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-7.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-7.c
@@ -1,4 +1,4 @@
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -21,7 +21,7 @@ foo  (int *a, char *b, char *c)
   return d;
 }
 
-/* { dg-final { scan-tree-dump-times "& 7" 6 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 4 "asan0" } } */
-/* { dg-final { scan-tree-dump-not "__builtin___asan_report_store" "asan0" } } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "& 7" 6 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 4 "sanopt" } } */
+/* { dg-final { scan-tree-dump-not "__builtin___asan_report_store" "sanopt" } } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-8.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-8.c
index 308a133..2a4c081 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-8.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-8.c
@@ -1,4 +1,4 @@
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -12,9 +12,9 @@ foo  (int *a, char *b, char *c)
   /* For a total of 5 checks.  */
 }
 
-/* { dg-final { scan-tree-dump-times "& 7" 5 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load4" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 1 "asan0" } } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "& 7" 5 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load4" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 1 "sanopt" } } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-9.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-9.c
index c6575ad..9449de5 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-9.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-9.c
@@ -1,4 +1,4 @@
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -9,5 +9,5 @@ f (char *a)
   return  __builtin_strlen (a);
 }
 
-/* { dg-final { scan-tree-dump-times "__asan_report_load1" 1 "asan0" } } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "__asan_report_load1" 1 "sanopt" } } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */

Re: [PATCH] Move Asan instrumentation to sanopt pass

[Jakub Jelinek]

On Fri, Jul 18, 2014 at 05:36:30PM +0400, Yury Gribov wrote:
> The patch was bootstrapped, regtested and asan-bootstrapped on x64.

Thanks for working on this.

For formatting, can you please just replace 8 spaces with tabs
in the ^+ lines in the patch?
sed '/^+/s/        /\t/g' or so.
Can you avoid using // comments in code that uses /* */ comments?

If all the ifns have a bitmask argument, one question is if we really
want ASAN_LOAD vs. ASAN_STORE, instead of a single ASAN_CHECK that
would actually have ASAN_CHECK_IS_STORE as one of the flags.

>  pass_sanopt::execute (function *fun)
>  {
>    basic_block bb;
> +  gimple_stmt_iterator gsi;
>  
> +  int asan_num_accesses = 0;

IMHO you should guard this with if (flag_sanitize & SANITIZE_ADDRESS),
to avoid the cost for -fsanitize=undefined, thread etc.

> +  FOR_EACH_BB_FN (bb, fun)
> +    for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi))
> +      {
> + 	gimple stmt = gsi_stmt (gsi);
> +	if (is_gimple_call (stmt) && gimple_call_internal_p (stmt))
> +	  {
> +	    enum internal_fn ifn = gimple_call_internal_fn (stmt);
> +	    switch (ifn)
> +	      {
> +	      case IFN_ASAN_LOAD:
> +	      case IFN_ASAN_STORE:
> +		{
> +		  ++asan_num_accesses;
> +		  break;
> +		}
> +	      default:
> +		break;
> +	      }
> +	    }
> +        }
> +
> +  bool use_calls = ASAN_INSTRUMENTATION_WITH_CALL_THRESHOLD < INT_MAX
> +    && asan_num_accesses >= ASAN_INSTRUMENTATION_WITH_CALL_THRESHOLD;
> +    

> --- a/gcc/gimple-iterator.h
> +++ b/gcc/gimple-iterator.h
> @@ -116,6 +116,7 @@ gsi_start_bb (basic_block bb)
>    gimple_seq *seq;
>  
>    seq = bb_seq_addr (bb);
> +  gcc_assert (seq);
>    i.ptr = gimple_seq_first (*seq);
>    i.seq = seq;
>    i.bb = bb;

Uh.  Can you please explain this?  That sounds weird.

	Jakub

Re: [PATCH] Move Asan instrumentation to sanopt pass

[Yury Gribov]

[-- Attachment #1: Type: text/plain, Size: 178 bytes --]

 > Attached patch delays generation of Asan memory checking
 > code until sanopt pass.

Here is an updated patch based on Jakub's review. Bootstrapped and 
regtested on x64.

-Y

[-- Attachment #2: asan_ifn-2.diff --]
[-- Type: text/x-diff, Size: 41830 bytes --]

commit 7c371a6f462e166d3f2ad89afbe2e61b1a0b799b
Author: Yury Gribov <y.gribov@samsung.com>
Date:   Thu Jul 17 09:45:26 2014 +0400

    Move inlining of Asan memory checks to sanopt pass.
    Change asan-instrumentation-with-call-threshold to more closely match LLVM.
    
    gcc/
    
    2014-07-21  Yury Gribov  <y.gribov@samsung.com>
    
    	* asan.c (asan_check_flags): New enum.
    	(build_check_stmt_with_calls): Removed function.
    	(build_check_stmt): Split inlining logic to
    	asan_expand_check_ifn.
    	(instrument_derefs): Rename parameter.
    	(instrument_mem_region_access): Rename parameter.
    	(instrument_strlen_call): Likewise.
    	(asan_expand_check_ifn): New function.
    	(asan_instrument): Remove old code.
    	(pass_sanopt::execute): Change handling of
    	asan-instrumentation-with-call-threshold.
    	(asan_clear_shadow): Fix formatting.
    	(asan_function_start): Likewise.
    	(asan_emit_stack_protection): Likewise.
    	* doc/invoke.texi (asan-instrumentation-with-call-threshold):
    	Update description.
    	* internal-fn.c (expand_ASAN_CHECK): New function.
    	* internal-fn.def (ASAN_CHECK): New internal function.
    	* params.def (PARAM_ASAN_INSTRUMENTATION_WITH_CALL_THRESHOLD):
    	Update description.
    	(PARAM_ASAN_USE_AFTER_RETURN): Likewise.
    
    gcc/testsuite/
    
    2014-07-21  Yury Gribov  <y.gribov@samsung.com>
    
    	* c-c++-common/asan/inc.c: Update test.
    	* c-c++-common/asan/instrument-with-calls-2.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-1.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-2.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-3.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-4.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-5.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-6.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-7.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-8.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-9.c: Likewise.

diff --git a/gcc/asan.c b/gcc/asan.c
index 0d78634..57d2ade 100644
--- a/gcc/asan.c
+++ b/gcc/asan.c
@@ -243,18 +243,16 @@ static GTY(()) tree shadow_ptr_types[2];
 /* Decl for __asan_option_detect_stack_use_after_return.  */
 static GTY(()) tree asan_detect_stack_use_after_return;
 
-/* Number of instrumentations in current function so far.  */
-
-static int asan_num_accesses;
-
-/* Check whether we should replace inline instrumentation with calls.  */
-
-static inline bool
-use_calls_p ()
+/* Various flags for Asan builtins.  */
+enum asan_check_flags
 {
-  return ASAN_INSTRUMENTATION_WITH_CALL_THRESHOLD < INT_MAX
-    && asan_num_accesses >= ASAN_INSTRUMENTATION_WITH_CALL_THRESHOLD;
-}
+  ASAN_CHECK_STORE = 1 << 0,
+  ASAN_CHECK_SCALAR_ACCESS = 1 << 1,
+  ASAN_CHECK_NON_ZERO_LEN = 1 << 2,
+  ASAN_CHECK_START_INSTRUMENTED = 1 << 3,
+  ASAN_CHECK_END_INSTRUMENTED = 1 << 4,
+  ASAN_CHECK_LAST
+};
 
 /* Hashtable support for memory references used by gimple
    statements.  */
@@ -943,7 +941,7 @@ asan_clear_shadow (rtx shadow_mem, HOST_WIDE_INT len)
 
   emit_move_insn (shadow_mem, const0_rtx);
   tmp = expand_simple_binop (Pmode, PLUS, addr, gen_int_mode (4, Pmode), addr,
-                             true, OPTAB_LIB_WIDEN);
+			     true, OPTAB_LIB_WIDEN);
   if (tmp != addr)
     emit_move_insn (addr, tmp);
   emit_cmp_and_jump_insns (addr, end, LT, NULL_RTX, Pmode, true, top_label);
@@ -958,7 +956,7 @@ asan_function_start (void)
   section *fnsec = function_section (current_function_decl);
   switch_to_section (fnsec);
   ASM_OUTPUT_DEBUG_LABEL (asm_out_file, "LASANPC",
-                         current_function_funcdef_no);
+			 current_function_funcdef_no);
 }
 
 /* Insert code to protect stack vars.  The prologue sequence should be emitted
@@ -1023,7 +1021,7 @@ asan_emit_stack_protection (rtx base, rtx pbase, unsigned int alignb,
     {
       use_after_return_class = floor_log2 (asan_frame_size - 1) - 5;
       /* __asan_stack_malloc_N guarantees alignment
-         N < 6 ? (64 << N) : 4096 bytes.  */
+	 N < 6 ? (64 << N) : 4096 bytes.  */
       if (alignb > (use_after_return_class < 6
 		    ? (64U << use_after_return_class) : 4096U))
 	use_after_return_class = -1;
@@ -1096,7 +1094,7 @@ asan_emit_stack_protection (rtx base, rtx pbase, unsigned int alignb,
   ASM_GENERATE_INTERNAL_LABEL (buf, "LASANPC", current_function_funcdef_no);
   id = get_identifier (buf);
   decl = build_decl (DECL_SOURCE_LOCATION (current_function_decl),
-                    VAR_DECL, id, char_type_node);
+		    VAR_DECL, id, char_type_node);
   SET_DECL_ASSEMBLER_NAME (decl, id);
   TREE_ADDRESSABLE (decl) = 1;
   TREE_READONLY (decl) = 1;
@@ -1553,55 +1551,6 @@ maybe_create_ssa_name (location_t loc, tree base, gimple_stmt_iterator *iter,
   return gimple_assign_lhs (g);
 }
 
-/* Instrument the memory access instruction using callbacks.
-   Parameters are similar to BUILD_CHECK_STMT.  */
-
-static void
-build_check_stmt_with_calls (location_t loc, tree base, tree len,
-			     HOST_WIDE_INT size_in_bytes, gimple_stmt_iterator *iter,
-			     bool before_p, bool is_store, bool is_scalar_access)
-{
-  gimple_stmt_iterator gsi = *iter;
-  tree base_ssa = maybe_create_ssa_name (loc, base, &gsi, before_p);
-
-  gimple g
-    = gimple_build_assign_with_ops (NOP_EXPR,
-				    make_ssa_name (pointer_sized_int_node, NULL),
-				    base_ssa, NULL_TREE);
-  gimple_set_location (g, loc);
-  if (before_p)
-    gsi_insert_before (&gsi, g, GSI_NEW_STMT);
-  else
-    gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-  tree base_addr = gimple_assign_lhs (g);
-
-  int nargs;
-  tree fun
-    = check_func (is_store, is_scalar_access ? size_in_bytes : -1, &nargs);
-  if (nargs == 1)
-    g = gimple_build_call (fun, 1, base_addr);
-  else
-    {
-      gcc_assert (nargs == 2);
-      g = gimple_build_assign_with_ops (NOP_EXPR,
-					make_ssa_name (pointer_sized_int_node,
-						       NULL),
-					len, NULL_TREE);
-      gimple_set_location (g, loc);
-      gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-      tree sz_arg = gimple_assign_lhs (g);
-      g = gimple_build_call (fun, nargs, base_addr, sz_arg);
-    }
-  gimple_set_location (g, loc);
-  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-
-  if (!before_p)
-    {
-      gsi_next (&gsi);
-      *iter = gsi;
-    }
-}
-
 /* Instrument the memory access instruction BASE.  Insert new
    statements before or after ITER.
 
@@ -1622,27 +1571,30 @@ build_check_stmt_with_calls (location_t loc, tree base, tree len,
    otherwise, it points to the statement logically following it.  */
 
 static void
-build_check_stmt (location_t location, tree base, tree len,
+build_check_stmt (location_t loc, tree base, tree len,
 		  HOST_WIDE_INT size_in_bytes, gimple_stmt_iterator *iter,
-		  bool non_zero_len_p, bool before_p, bool is_store,
+		  bool is_non_zero_len, bool before_p, bool is_store,
 		  bool is_scalar_access, unsigned int align = 0,
 		  bool start_instrumented = false,
 		  bool end_instrumented = false)
 {
   gimple_stmt_iterator gsi = *iter;
   gimple g;
-  tree uintptr_type
-    = build_nonstandard_integer_type (TYPE_PRECISION (TREE_TYPE (base)), 1);
 
-  gcc_assert (!(size_in_bytes > 0 && !non_zero_len_p));
+  gcc_assert (!(size_in_bytes > 0 && !is_non_zero_len));
 
   if (start_instrumented && end_instrumented)
     {
       if (!before_p)
-        gsi_next (iter);
+	gsi_next (iter);
       return;
     }
 
+  gsi = *iter;
+
+  base = unshare_expr (base);
+  base = maybe_create_ssa_name (loc, base, &gsi, before_p);
+
   if (len)
     len = unshare_expr (len);
   else
@@ -1654,9 +1606,8 @@ build_check_stmt (location_t location, tree base, tree len,
   if (size_in_bytes > 1)
     {
       if ((size_in_bytes & (size_in_bytes - 1)) != 0
-	  || !is_scalar_access
 	  || size_in_bytes > 16)
-	size_in_bytes = -1;
+	is_scalar_access = false;
       else if (align && align < size_in_bytes * BITS_PER_UNIT)
 	{
 	  /* On non-strict alignment targets, if
@@ -1667,189 +1618,34 @@ build_check_stmt (location_t location, tree base, tree len,
 	  if (size_in_bytes != 16
 	      || STRICT_ALIGNMENT
 	      || align < 8 * BITS_PER_UNIT)
-	    size_in_bytes = -1;
-	}
-    }
-
-  HOST_WIDE_INT real_size_in_bytes = size_in_bytes == -1 ? 1 : size_in_bytes;
-
-  tree shadow_ptr_type = shadow_ptr_types[real_size_in_bytes == 16 ? 1 : 0];
-  tree shadow_type = TREE_TYPE (shadow_ptr_type);
-
-  base = unshare_expr (base);
-
-  if (use_calls_p ())
-    {
-      gsi = *iter;
-      build_check_stmt_with_calls (location, base, len, size_in_bytes, iter,
-				   before_p, is_store, is_scalar_access);
-      return;
-    }
-
-  ++asan_num_accesses;
-
-  if (!non_zero_len_p)
-    {
-      gcc_assert (before_p);
-
-      /* So, the length of the memory area to asan-protect is
-	 non-constant.  Let's guard the generated instrumentation code
-	 like:
-
-	 if (len != 0)
-	   {
-	     //asan instrumentation code goes here.
-	   }
-	 // falltrough instructions, starting with *ITER.  */
-
-      g = gimple_build_cond (NE_EXPR,
-			     len,
-			     build_int_cst (TREE_TYPE (len), 0),
-			     NULL_TREE, NULL_TREE);
-      gimple_set_location (g, location);
-
-      basic_block then_bb, fallthrough_bb;
-      insert_if_then_before_iter (g, iter, /*then_more_likely_p=*/true,
-				  &then_bb, &fallthrough_bb);
-      /* Note that fallthrough_bb starts with the statement that was
-	 pointed to by ITER.  */
-
-      /* The 'then block' of the 'if (len != 0) condition is where
-	 we'll generate the asan instrumentation code now.  */
-      gsi = gsi_last_bb (then_bb);
-      build_check_stmt (location, base, len, size_in_bytes, &gsi,
-			/*non_zero_len_p*/true, /*before_p*/true, is_store,
-			is_scalar_access, align,
-			start_instrumented, end_instrumented);
-      return;
-    }
-
-  /* Get an iterator on the point where we can add the condition
-     statement for the instrumentation.  */
-  basic_block then_bb, else_bb;
-  gsi = create_cond_insert_point (&gsi, before_p,
-				  /*then_more_likely_p=*/false,
-				  /*create_then_fallthru_edge=*/false,
-				  &then_bb,
-				  &else_bb);
-
-  tree base_ssa = maybe_create_ssa_name (location, base, &gsi,
-					 /*before_p*/false);
-
-  g = gimple_build_assign_with_ops (NOP_EXPR,
-				    make_ssa_name (uintptr_type, NULL),
-				    base_ssa, NULL_TREE);
-  gimple_set_location (g, location);
-  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-  tree base_addr = gimple_assign_lhs (g);
-
-  tree t = NULL_TREE;
-  if (real_size_in_bytes >= 8)
-    {
-      tree shadow = build_shadow_mem_access (&gsi, location, base_addr,
-					     shadow_ptr_type);
-      t = shadow;
-    }
-  else
-    {
-      /* Slow path for 1, 2 and 4 byte accesses.  */
-
-      if (!start_instrumented)
-	{
-	  /* Test (shadow != 0)
-		  & ((base_addr & 7) + (real_size_in_bytes - 1)) >= shadow).  */
-	  tree shadow = build_shadow_mem_access (&gsi, location, base_addr,
-						 shadow_ptr_type);
-	  gimple shadow_test = build_assign (NE_EXPR, shadow, 0);
-	  gimple_seq seq = NULL;
-	  gimple_seq_add_stmt (&seq, shadow_test);
-	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR, base_addr, 7));
-	  gimple_seq_add_stmt (&seq, build_type_cast (shadow_type,
-						      gimple_seq_last (seq)));
-	  if (real_size_in_bytes > 1)
-	    gimple_seq_add_stmt (&seq,
-				 build_assign (PLUS_EXPR, gimple_seq_last (seq),
-					       real_size_in_bytes - 1));
-	  gimple_seq_add_stmt (&seq, build_assign (GE_EXPR,
-						   gimple_seq_last (seq),
-						   shadow));
-	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR, shadow_test,
-						   gimple_seq_last (seq)));
-	  t = gimple_assign_lhs (gimple_seq_last (seq));
-	  gimple_seq_set_location (seq, location);
-	  gsi_insert_seq_after (&gsi, seq, GSI_CONTINUE_LINKING);
-	}
-
-      /* For non-constant, misaligned or otherwise weird access sizes,
-	 check first and last byte.  */
-      if (size_in_bytes == -1 && !end_instrumented)
-	{
-	  g = gimple_build_assign_with_ops (MINUS_EXPR,
-					    make_ssa_name (uintptr_type, NULL),
-					    len,
-					    build_int_cst (uintptr_type, 1));
-	  gimple_set_location (g, location);
-	  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-	  tree last = gimple_assign_lhs (g);
-	  g = gimple_build_assign_with_ops (PLUS_EXPR,
-					    make_ssa_name (uintptr_type, NULL),
-					    base_addr,
-					    last);
-	  gimple_set_location (g, location);
-	  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-	  tree base_end_addr = gimple_assign_lhs (g);
-
-	  tree shadow = build_shadow_mem_access (&gsi, location, base_end_addr,
-						 shadow_ptr_type);
-	  gimple shadow_test = build_assign (NE_EXPR, shadow, 0);
-	  gimple_seq seq = NULL;
-	  gimple_seq_add_stmt (&seq, shadow_test);
-	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR,
-						   base_end_addr, 7));
-	  gimple_seq_add_stmt (&seq, build_type_cast (shadow_type,
-						      gimple_seq_last (seq)));
-	  gimple_seq_add_stmt (&seq, build_assign (GE_EXPR,
-						   gimple_seq_last (seq),
-						   shadow));
-	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR, shadow_test,
-						   gimple_seq_last (seq)));
-	  if (!start_instrumented)
-	    gimple_seq_add_stmt (&seq, build_assign (BIT_IOR_EXPR, t,
-						     gimple_seq_last (seq)));
-	  t = gimple_assign_lhs (gimple_seq_last (seq));
-	  gimple_seq_set_location (seq, location);
-	  gsi_insert_seq_after (&gsi, seq, GSI_CONTINUE_LINKING);
+	    is_scalar_access = false;
 	}
     }
 
-  g = gimple_build_cond (NE_EXPR, t, build_int_cst (TREE_TYPE (t), 0),
-			 NULL_TREE, NULL_TREE);
-  gimple_set_location (g, location);
-  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-
-  /* Generate call to the run-time library (e.g. __asan_report_load8).  */
-  gsi = gsi_start_bb (then_bb);
-  int nargs;
-  tree fun = report_error_func (is_store, is_scalar_access ? size_in_bytes : -1,
-				&nargs);
-  if (nargs == 1)
-      g = gimple_build_call (fun, 1, base_addr);
+  HOST_WIDE_INT flags = 0;
+  if (is_store)
+    flags |= ASAN_CHECK_STORE;
+  if (is_non_zero_len)
+    flags |= ASAN_CHECK_NON_ZERO_LEN;
+  if (is_scalar_access)
+    flags |= ASAN_CHECK_SCALAR_ACCESS;
+  if (start_instrumented)
+    flags |= ASAN_CHECK_START_INSTRUMENTED;
+  if (end_instrumented)
+    flags |= ASAN_CHECK_END_INSTRUMENTED;
+
+  g = gimple_build_call_internal (IFN_ASAN_CHECK, 3,
+				  build_int_cst (integer_type_node, flags),
+				  base, len);
+  gimple_set_location (g, loc);
+  if (before_p)
+    gsi_insert_before (&gsi, g, GSI_SAME_STMT);
   else
     {
-      gcc_assert (nargs == 2);
-      g = gimple_build_assign_with_ops (NOP_EXPR,
-					make_ssa_name (pointer_sized_int_node,
-						       NULL),
-					len, NULL_TREE);
-      gimple_set_location (g, location);
       gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-      tree sz_arg = gimple_assign_lhs (g);
-      g = gimple_build_call (fun, nargs, base_addr, sz_arg);
+      gsi_next (&gsi);
+      *iter = gsi;
     }
-  gimple_set_location (g, location);
-  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-
-  *iter = gsi_start_bb (else_bb);
 }
 
 /* If T represents a memory access, add instrumentation code before ITER.
@@ -1942,7 +1738,7 @@ instrument_derefs (gimple_stmt_iterator *iter, tree t,
     {
       unsigned int align = get_object_alignment (t);
       build_check_stmt (location, base, NULL_TREE, size_in_bytes, iter,
-			/*non_zero_len_p*/size_in_bytes > 0, /*before_p=*/true,
+			/*is_non_zero_len*/size_in_bytes > 0, /*before_p=*/true,
 			is_store, /*is_scalar_access*/true, align);
       update_mem_ref_hash_table (base, size_in_bytes);
       update_mem_ref_hash_table (t, size_in_bytes);
@@ -1980,7 +1776,7 @@ instrument_mem_region_access (tree base, tree len,
   HOST_WIDE_INT size_in_bytes = tree_fits_shwi_p (len) ? tree_to_shwi (len) : -1;
 
   build_check_stmt (location, base, len, size_in_bytes, iter,
-		    /*non_zero_len_p*/size_in_bytes > 0, /*before_p*/true,
+		    /*is_non_zero_len*/size_in_bytes > 0, /*before_p*/true,
 		    is_store, /*is_scalar_access*/false, /*align*/0,
 		    start_instrumented, end_instrumented);
 
@@ -2037,7 +1833,7 @@ instrument_strlen_call (gimple_stmt_iterator *iter)
   gsi_insert_before (iter, str_arg_ssa, GSI_SAME_STMT);
 
   build_check_stmt (loc, gimple_assign_lhs (str_arg_ssa), NULL_TREE, 1, iter,
-		    /*non_zero_len_p*/true, /*before_p=*/true,
+		    /*is_non_zero_len*/true, /*before_p=*/true,
 		    /*is_store=*/false, /*is_scalar_access*/true, /*align*/0,
 		    start_instrumented, start_instrumented);
 
@@ -2050,7 +1846,7 @@ instrument_strlen_call (gimple_stmt_iterator *iter)
   gsi_insert_after (iter, g, GSI_NEW_STMT);
 
   build_check_stmt (loc, gimple_assign_lhs (g), NULL_TREE, 1, iter,
-		    /*non_zero_len_p*/true, /*before_p=*/false,
+		    /*is_non_zero_len*/true, /*before_p=*/false,
 		    /*is_store=*/false, /*is_scalar_access*/true, /*align*/0);
 
   return true;
@@ -2617,6 +2413,213 @@ asan_finish_file (void)
   flag_sanitize |= SANITIZE_ADDRESS;
 }
 
+/* Expand the ASAN_{LOAD,STORE} builtins.  */
+
+static void
+asan_expand_check_ifn (gimple_stmt_iterator *iter, bool use_calls)
+{
+  gimple g = gsi_stmt (*iter);
+  location_t loc = gimple_location (g);
+
+  HOST_WIDE_INT flags = tree_to_shwi (gimple_call_arg (g, 0));
+  gcc_assert (flags < ASAN_CHECK_LAST);
+  bool is_scalar_access = (flags & ASAN_CHECK_SCALAR_ACCESS) != 0;
+  bool is_store = (flags & ASAN_CHECK_STORE) != 0;
+  bool is_non_zero_len = (flags & ASAN_CHECK_NON_ZERO_LEN) != 0;
+  bool start_instrumented = (flags & ASAN_CHECK_START_INSTRUMENTED) != 0;
+  bool end_instrumented = (flags & ASAN_CHECK_END_INSTRUMENTED) != 0;
+
+  tree base = gimple_call_arg (g, 1);
+  tree len = gimple_call_arg (g, 2);
+
+  HOST_WIDE_INT size_in_bytes
+    = is_scalar_access && tree_fits_shwi_p (len) ? tree_to_shwi (len) : -1;
+
+  if (use_calls)
+    {
+      /* Instrument using callbacks.  */
+      gimple g
+	= gimple_build_assign_with_ops (NOP_EXPR,
+					make_ssa_name (pointer_sized_int_node,
+					NULL),
+					base, NULL_TREE);
+      gimple_set_location (g, loc);
+      gsi_insert_before (iter, g, GSI_SAME_STMT);
+      tree base_addr = gimple_assign_lhs (g);
+
+      int nargs;
+      tree fun = check_func (is_store, size_in_bytes, &nargs);
+      if (nargs == 1)
+	g = gimple_build_call (fun, 1, base_addr);
+      else
+	{
+	  gcc_assert (nargs == 2);
+	  g = gimple_build_assign_with_ops (NOP_EXPR,
+					    make_ssa_name (pointer_sized_int_node,
+							   NULL),
+					    len, NULL_TREE);
+	  gimple_set_location (g, loc);
+	  gsi_insert_before (iter, g, GSI_SAME_STMT);
+	  tree sz_arg = gimple_assign_lhs (g);
+	  g = gimple_build_call (fun, nargs, base_addr, sz_arg);
+	}
+      gimple_set_location (g, loc);
+      gsi_replace (iter, g, false);
+      return;
+    }
+
+  HOST_WIDE_INT real_size_in_bytes = size_in_bytes == -1 ? 1 : size_in_bytes;
+
+  tree uintptr_type
+    = build_nonstandard_integer_type (TYPE_PRECISION (TREE_TYPE (base)), 1);
+
+  tree shadow_ptr_type = shadow_ptr_types[real_size_in_bytes == 16 ? 1 : 0];
+  tree shadow_type = TREE_TYPE (shadow_ptr_type);
+
+  gimple_stmt_iterator gsi = *iter;
+
+  if (!is_non_zero_len)
+    {
+      /* So, the length of the memory area to asan-protect is
+	 non-constant.  Let's guard the generated instrumentation code
+	 like:
+
+	 if (len != 0)
+	   {
+	     //asan instrumentation code goes here.
+	   }
+	 // falltrough instructions, starting with *ITER.  */
+
+      g = gimple_build_cond (NE_EXPR,
+			    len,
+			    build_int_cst (TREE_TYPE (len), 0),
+			    NULL_TREE, NULL_TREE);
+      gimple_set_location (g, loc);
+
+      basic_block then_bb, fallthrough_bb;
+      insert_if_then_before_iter (g, iter, /*then_more_likely_p=*/true,
+				 &then_bb, &fallthrough_bb);
+      /* Note that fallthrough_bb starts with the statement that was
+	pointed to by ITER.  */
+
+      /* The 'then block' of the 'if (len != 0) condition is where
+	we'll generate the asan instrumentation code now.  */
+      gsi = gsi_last_bb (then_bb);
+    }
+
+  /* Get an iterator on the point where we can add the condition
+     statement for the instrumentation.  */
+  basic_block then_bb, else_bb;
+  gsi = create_cond_insert_point (&gsi, /*before_p*/false,
+				  /*then_more_likely_p=*/false,
+				  /*create_then_fallthru_edge=*/false,
+				  &then_bb,
+				  &else_bb);
+
+  g = gimple_build_assign_with_ops (NOP_EXPR,
+				    make_ssa_name (pointer_sized_int_node,
+						   NULL),
+				    base, NULL_TREE);
+  gimple_set_location (g, loc);
+  gsi_insert_before (&gsi, g, GSI_NEW_STMT);
+  tree base_addr = gimple_assign_lhs (g);
+
+  tree t = NULL_TREE;
+  if (real_size_in_bytes >= 8)
+    {
+      tree shadow = build_shadow_mem_access (&gsi, loc, base_addr,
+					     shadow_ptr_type);
+      t = shadow;
+    }
+  else
+    {
+      /* Slow path for 1, 2 and 4 byte accesses.  */
+
+      if (!start_instrumented)
+	{
+	  /* Test (shadow != 0)
+	     & ((base_addr & 7) + (real_size_in_bytes - 1)) >= shadow).  */
+	  tree shadow = build_shadow_mem_access (&gsi, loc, base_addr,
+						 shadow_ptr_type);
+	  gimple shadow_test = build_assign (NE_EXPR, shadow, 0);
+	  gimple_seq seq = NULL;
+	  gimple_seq_add_stmt (&seq, shadow_test);
+	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR, base_addr, 7));
+	  gimple_seq_add_stmt (&seq, build_type_cast (shadow_type,
+						      gimple_seq_last (seq)));
+	  if (real_size_in_bytes > 1)
+	    gimple_seq_add_stmt (&seq,
+				 build_assign (PLUS_EXPR, gimple_seq_last (seq),
+					       real_size_in_bytes - 1));
+	  gimple_seq_add_stmt (&seq, build_assign (GE_EXPR,
+						   gimple_seq_last (seq),
+						   shadow));
+	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR, shadow_test,
+						   gimple_seq_last (seq)));
+	  t = gimple_assign_lhs (gimple_seq_last (seq));
+	  gimple_seq_set_location (seq, loc);
+	  gsi_insert_seq_after (&gsi, seq, GSI_CONTINUE_LINKING);
+	}
+
+      /* For non-constant, misaligned or otherwise weird access sizes,
+	 check first and last byte.  */
+      if (size_in_bytes == -1 && !end_instrumented)
+	{
+	  g = gimple_build_assign_with_ops (MINUS_EXPR,
+					    make_ssa_name (uintptr_type, NULL),
+					    len,
+					    build_int_cst (uintptr_type, 1));
+	  gimple_set_location (g, loc);
+	  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
+	  tree last = gimple_assign_lhs (g);
+	  g = gimple_build_assign_with_ops (PLUS_EXPR,
+					    make_ssa_name (uintptr_type, NULL),
+					    base_addr,
+					    last);
+	  gimple_set_location (g, loc);
+	  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
+	  tree base_end_addr = gimple_assign_lhs (g);
+
+	  tree shadow = build_shadow_mem_access (&gsi, loc, base_end_addr,
+						 shadow_ptr_type);
+	  gimple shadow_test = build_assign (NE_EXPR, shadow, 0);
+	  gimple_seq seq = NULL;
+	  gimple_seq_add_stmt (&seq, shadow_test);
+	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR,
+						   base_end_addr, 7));
+	  gimple_seq_add_stmt (&seq, build_type_cast (shadow_type,
+						      gimple_seq_last (seq)));
+	  gimple_seq_add_stmt (&seq, build_assign (GE_EXPR,
+						   gimple_seq_last (seq),
+						   shadow));
+	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR, shadow_test,
+						   gimple_seq_last (seq)));
+	  if (!start_instrumented)
+	    gimple_seq_add_stmt (&seq, build_assign (BIT_IOR_EXPR, t,
+						     gimple_seq_last (seq)));
+	  t = gimple_assign_lhs (gimple_seq_last (seq));
+	  gimple_seq_set_location (seq, loc);
+	  gsi_insert_seq_after (&gsi, seq, GSI_CONTINUE_LINKING);
+	}
+    }
+
+  g = gimple_build_cond (NE_EXPR, t, build_int_cst (TREE_TYPE (t), 0),
+			 NULL_TREE, NULL_TREE);
+  gimple_set_location (g, loc);
+  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
+
+  /* Generate call to the run-time library (e.g. __asan_report_load8).  */
+  gsi = gsi_start_bb (then_bb);
+  int nargs;
+  tree fun = report_error_func (is_store, size_in_bytes, &nargs);
+  g = gimple_build_call (fun, nargs, base_addr, len);
+  gimple_set_location (g, loc);
+  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
+
+  gsi_remove (iter, true);
+  *iter = gsi_start_bb (else_bb);
+}
+
 /* Instrument the current function.  */
 
 static unsigned int
@@ -2624,7 +2627,6 @@ asan_instrument (void)
 {
   if (shadow_ptr_types[0] == NULL_TREE)
     asan_init_shadow_ptr_types ();
-  asan_num_accesses = 0;
   transform_statements ();
   return 0;
 }
@@ -2745,6 +2747,23 @@ pass_sanopt::execute (function *fun)
 {
   basic_block bb;
 
+  int asan_num_accesses = 0;
+  if (flag_sanitize & SANITIZE_ADDRESS)
+    {
+      gimple_stmt_iterator gsi;
+      FOR_EACH_BB_FN (bb, fun)
+	for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi))
+	  {
+ 	    gimple stmt = gsi_stmt (gsi);
+	    if (is_gimple_call (stmt) && gimple_call_internal_p (stmt)
+		&& gimple_call_internal_fn (stmt) == IFN_ASAN_CHECK)
+	      ++asan_num_accesses;
+	  }
+    }
+
+  bool use_calls = ASAN_INSTRUMENTATION_WITH_CALL_THRESHOLD < INT_MAX
+    && asan_num_accesses >= ASAN_INSTRUMENTATION_WITH_CALL_THRESHOLD;
+
   FOR_EACH_BB_FN (bb, fun)
     {
       gimple_stmt_iterator gsi;
@@ -2756,17 +2775,25 @@ pass_sanopt::execute (function *fun)
 	    continue;
 
 	  if (gimple_call_internal_p (stmt))
-	    switch (gimple_call_internal_fn (stmt))
-	      {
-	      case IFN_UBSAN_NULL:
-		ubsan_expand_null_ifn (gsi);
-		break;
-	      case IFN_UBSAN_BOUNDS:
-		ubsan_expand_bounds_ifn (&gsi);
-		break;
-	      default:
-		break;
-	      }
+	    {
+	      enum internal_fn ifn = gimple_call_internal_fn (stmt);
+	      switch (ifn)
+		{
+		case IFN_UBSAN_NULL:
+		  ubsan_expand_null_ifn (gsi);
+		  break;
+		case IFN_UBSAN_BOUNDS:
+		  ubsan_expand_bounds_ifn (&gsi);
+		  break;
+		case IFN_ASAN_CHECK:
+		  {
+		    asan_expand_check_ifn (&gsi, use_calls);
+		    break;
+		  }
+		default:
+		  break;
+		}
+	    }
 
 	  if (dump_file && (dump_flags & TDF_DETAILS))
 	    {
@@ -2775,7 +2802,7 @@ pass_sanopt::execute (function *fun)
 	      fprintf (dump_file, "\n");
 	    }
 
-	  /* ubsan_expand_bounds_ifn might move us to the end of the BB.  */
+	  /* Expansions might move us to the end of the BB.  */
 	  if (gsi_end_p (gsi))
 	    break;
 	}
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index b5e8d98..8a755df 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -10312,9 +10312,9 @@ To disable use-after-return detection use
 @option{--param asan-use-after-return=0}.
 
 @item asan-instrumentation-with-call-threshold
-Once number of memory accesses  in function becomes greater
-or equal than this number, use callbacks instead of
-generating inline code.  E.g. to disable inline code use
+If number of memory accesses in function being instrumented
+is greater or equal to this number, use callbacks instead of inline checks.
+E.g. to disable inline code use
 @option{--param asan-instrumentation-with-call-threshold=0}.
 
 @end table
diff --git a/gcc/internal-fn.c b/gcc/internal-fn.c
index 78f59d6..7fb54ea 100644
--- a/gcc/internal-fn.c
+++ b/gcc/internal-fn.c
@@ -167,6 +167,14 @@ expand_UBSAN_BOUNDS (gimple stmt ATTRIBUTE_UNUSED)
   gcc_unreachable ();
 }
 
+/* This should get expanded in the sanopt pass.  */
+
+static void
+expand_ASAN_CHECK (gimple stmt ATTRIBUTE_UNUSED)
+{
+  gcc_unreachable ();
+}
+
 /* Add sub/add overflow checking to the statement STMT.
    CODE says whether the operation is +, or -.  */
 
diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def
index f0766bc..f0aa1b4 100644
--- a/gcc/internal-fn.def
+++ b/gcc/internal-fn.def
@@ -54,3 +54,4 @@ DEF_INTERNAL_FN (UBSAN_CHECK_SUB, ECF_CONST | ECF_LEAF | ECF_NOTHROW)
 DEF_INTERNAL_FN (UBSAN_CHECK_MUL, ECF_CONST | ECF_LEAF | ECF_NOTHROW)
 DEF_INTERNAL_FN (ABNORMAL_DISPATCHER, ECF_NORETURN)
 DEF_INTERNAL_FN (BUILTIN_EXPECT, ECF_CONST | ECF_LEAF | ECF_NOTHROW)
+DEF_INTERNAL_FN (ASAN_CHECK, ECF_TM_PURE | ECF_LEAF | ECF_NOTHROW)
diff --git a/gcc/params.def b/gcc/params.def
index aa1e88d..f1c44e0 100644
--- a/gcc/params.def
+++ b/gcc/params.def
@@ -1072,14 +1072,14 @@ DEFPARAM (PARAM_ASAN_MEMINTRIN,
 
 DEFPARAM (PARAM_ASAN_USE_AFTER_RETURN,
          "asan-use-after-return",
-         "Enable asan builtin functions protection",
+         "Enable asan detection of use-after-return bugs",
          1, 0, 1)
 
 DEFPARAM (PARAM_ASAN_INSTRUMENTATION_WITH_CALL_THRESHOLD,
          "asan-instrumentation-with-call-threshold",
-         "Use callbacks instead of inline code once number of accesses "
-         " in function becomes greater or equal than this threshold",
-         10000, 0, INT_MAX)
+         "Use callbacks instead of inline code if number of accesses "
+         "in function becomes greater or equal to this number",
+         7000, 0, INT_MAX)
 
 DEFPARAM (PARAM_UNINIT_CONTROL_DEP_ATTEMPTS,
 	  "uninit-control-dep-attempts",
diff --git a/gcc/testsuite/c-c++-common/asan/inc.c b/gcc/testsuite/c-c++-common/asan/inc.c
index b9c6734..36cc3d8 100644
--- a/gcc/testsuite/c-c++-common/asan/inc.c
+++ b/gcc/testsuite/c-c++-common/asan/inc.c
@@ -16,6 +16,6 @@ main ()
   return 0;
 }
 
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report" 1 "asan0" } }  */
-/* { dg-final { scan-tree-dump "__builtin___asan_report_load4" "asan0" } }  */
+/* { dg-final { scan-tree-dump-times "ASAN_" 1 "asan0" } }  */
+/* { dg-final { scan-tree-dump "ASAN_CHECK \\(.*, 4\\);" "asan0" } }  */
 /* { dg-final { cleanup-tree-dump "asan0" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/instrument-with-calls-2.c b/gcc/testsuite/c-c++-common/asan/instrument-with-calls-2.c
index 570f796..04fdad0 100644
--- a/gcc/testsuite/c-c++-common/asan/instrument-with-calls-2.c
+++ b/gcc/testsuite/c-c++-common/asan/instrument-with-calls-2.c
@@ -9,8 +9,8 @@ void f(int *a, int *b) {
   x = *b;
 }
 
-/* { dg-final { scan-assembler-not "__asan_store4" } } */
-/* { dg-final { scan-assembler "__asan_report_store4" } } */
+/* { dg-final { scan-assembler "__asan_store4" } } */
+/* { dg-final { scan-assembler-not "__asan_report_store4" } } */
 /* { dg-final { scan-assembler "__asan_load4" } } */
 /* { dg-final { scan-assembler-not "__asan_report_load4" } } */
 /* { dg-final { cleanup-saved-temps } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-1.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-1.c
index c7c594e..028f8d7 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-1.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-1.c
@@ -2,7 +2,7 @@
    location in the same basic block, the second reference should not
    be instrumented by the Address Sanitizer.  */
 
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -62,7 +62,7 @@ main ()
   return test0 () && test1 (0);
 }
 
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store1" 3 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 2 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load" 1 "asan0" }  } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store1" 3 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 2 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load" 1 "sanopt" }  } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-2.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-2.c
index 143312f..a58411c 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-2.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-2.c
@@ -3,7 +3,7 @@
    be instrumented by the Address Sanitizer.  But in case of access to
    overlapping regions we must be precise.  */
 
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -20,7 +20,7 @@ main ()
   __builtin_memset (tab, 1, 3);
 }
 
-/* { dg-final { scan-tree-dump-times "& 7" 3 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 2 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report" 2 "asan0" }  } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "& 7" 3 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 2 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report" 2 "sanopt" }  } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-3.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-3.c
index 420a263..5193ae0 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-3.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-3.c
@@ -1,4 +1,4 @@
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -12,7 +12,7 @@ foo (__INT32_TYPE__ *p)
   return ret; 
 }
 
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report" 2 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store" 1 "asan0" } } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report" 2 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store" 1 "sanopt" } } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-4.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-4.c
index da91cd5..c3632aa 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-4.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-4.c
@@ -1,4 +1,4 @@
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -10,8 +10,8 @@ foo  (int *a, char *b, char *c)
   /* For a total of 5 checks.  */
 }
 
-/* { dg-final { scan-tree-dump-times "& 7" 5 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 1 "asan0" } } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "& 7" 5 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 1 "sanopt" } } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-5.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-5.c
index 134be66..077ea34 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-5.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-5.c
@@ -1,4 +1,4 @@
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -10,9 +10,9 @@ foo  (int *a, char *b, char *c)
   /* For a total of 5 checks.  */
 }
 
-/* { dg-final { scan-tree-dump-times "& 7" 5 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load4" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 1 "asan0" } } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "& 7" 5 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load4" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 1 "sanopt" } } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-6.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-6.c
index 55c8ee3..6d87104 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-6.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-6.c
@@ -1,4 +1,4 @@
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -12,9 +12,9 @@ foo  (int *a, char *b, char *c)
   /* For a total of 8 checks.  */
 }
 
-/* { dg-final { scan-tree-dump-times "& 7" 8 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load4" 2 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 2 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 2 "asan0" } } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "& 7" 8 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load4" 2 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 2 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 2 "sanopt" } } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-7.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-7.c
index a04956d..5baa10d 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-7.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-7.c
@@ -1,4 +1,4 @@
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -21,7 +21,7 @@ foo  (int *a, char *b, char *c)
   return d;
 }
 
-/* { dg-final { scan-tree-dump-times "& 7" 6 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 4 "asan0" } } */
-/* { dg-final { scan-tree-dump-not "__builtin___asan_report_store" "asan0" } } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "& 7" 6 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 4 "sanopt" } } */
+/* { dg-final { scan-tree-dump-not "__builtin___asan_report_store" "sanopt" } } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-8.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-8.c
index 308a133..2a4c081 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-8.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-8.c
@@ -1,4 +1,4 @@
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -12,9 +12,9 @@ foo  (int *a, char *b, char *c)
   /* For a total of 5 checks.  */
 }
 
-/* { dg-final { scan-tree-dump-times "& 7" 5 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load4" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 1 "asan0" } } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "& 7" 5 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load4" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 1 "sanopt" } } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-9.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-9.c
index c6575ad..9449de5 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-9.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-9.c
@@ -1,4 +1,4 @@
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -9,5 +9,5 @@ f (char *a)
   return  __builtin_strlen (a);
 }
 
-/* { dg-final { scan-tree-dump-times "__asan_report_load1" 1 "asan0" } } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "__asan_report_load1" 1 "sanopt" } } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */

Re: [PATCH] Move Asan instrumentation to sanopt pass

[Richard Biener]

On Fri, Jul 18, 2014 at 3:36 PM, Yury Gribov <y.gribov@samsung.com> wrote:
> Hi all,
>
> Attached patch delays generation of Asan memory checking code
> until sanopt pass. This is a first step towards global static analysis
> of Asan instrumentation which would allow to
> * remove redundant instrumentations
> * aggregate adjacent Asan checks
> * move invariant checks from loops
>
> The patch also changes the logic behind
> asan-instrumentation-with-call-threshold
> parameter to more closely match LLVM.
>
> The patch splits build_check_stmt routine to two parts. The first one
> (called from asan0/asan passes) inserts calls to internal functions
> ASAN_LOAD and ASAN_STORE. The second expands those to inline checks
> (in asan_expand_check_ifn).
>
> Here are some obvious disadvantages:
> * passing additional info via hidden parameter of
> ASAN_{LOAD,STORE} is ugly but I'm not sure how to do this better
> * delayed expansion runs after all optimization passes
> so inlined Asan checks will not get a chance to be
> CSE-ed, etc.; this may probably be solved by moving sanopt earlier
> in the pipeline. BTW I haven't experienced notable slowdowns in my
> experiments.
> * passing program pointers to ASAN_{LOAD,STORE} may damage alias analysis
> because all pointers will now escape;

Ugh.

> I probably could
> provide fnspec with (EAF_DIRECT | EAF_NOCLOBBER | EAF_NOESCAPE) or
> even EAF_UNUSED for these functions but this does not seem
> to be supported in current middle-end.

Simply add the "fn spec" attribute to the functions?

Richard.

> The patch was bootstrapped, regtested and asan-bootstrapped on x64.
>
> Is this ok for trunk?
>
> -Yury

Re: [PATCH] Move Asan instrumentation to sanopt pass

[Yury Gribov]

On 07/22/2014 05:57 PM, Richard Biener wrote:
>> I probably could
>> provide fnspec with (EAF_DIRECT | EAF_NOCLOBBER | EAF_NOESCAPE) or
>> even EAF_UNUSED for these functions but this does not seem
>> to be supported in current middle-end.
>
> Simply add the "fn spec" attribute to the functions?

Problem is that internal functions don't seem to support this
(gimple_statement_call::fntype and gimple_statement_call::internal_fn
occupy the same field). I wasn't sure about the reasons for this
so decided change them in original patch. Do you think we should we allow
fntype for internal functions?

-Y

Re: [PATCH] Move Asan instrumentation to sanopt pass

[Richard Biener]

On Tue, Jul 22, 2014 at 4:15 PM, Yury Gribov <y.gribov@samsung.com> wrote:
> On 07/22/2014 05:57 PM, Richard Biener wrote:
>>>
>>> I probably could
>>> provide fnspec with (EAF_DIRECT | EAF_NOCLOBBER | EAF_NOESCAPE) or
>>> even EAF_UNUSED for these functions but this does not seem
>>> to be supported in current middle-end.
>>
>>
>> Simply add the "fn spec" attribute to the functions?
>
>
> Problem is that internal functions don't seem to support this
> (gimple_statement_call::fntype and gimple_statement_call::internal_fn
> occupy the same field). I wasn't sure about the reasons for this
> so decided change them in original patch. Do you think we should we allow
> fntype for internal functions?

Ah.... internal fns.  Those cannot have attributes indeed (technical
limitation).
Martin was working on putting those flags elsewhere (cgraph, though internal
functions don't have cgraph nodes either ...).  Maybe it was a bad idea to use
internal functions for ASAN.

Richard.

> -Y

Re: [PATCH] Move Asan instrumentation to sanopt pass

[Yury Gribov]

[-- Attachment #1: Type: text/plain, Size: 340 bytes --]

On 07/22/2014 12:27 PM, Yury Gribov wrote:
>> Attached patch delays generation of Asan memory checking
>> code until sanopt pass.
>
> Here is an updated patch based on Jakub's review.
> Bootstrapped and regtested on x64.

Yet another version with ASAN_CHECK changed to builtin function
(instead of internal-fn) per Richard's advice.

-Y




[-- Attachment #2: asan_ifn-3.diff --]
[-- Type: text/x-diff, Size: 43772 bytes --]

commit 18aaa0e370a09fc3a226e0d79503fc58bd770703
Author: Yury Gribov <y.gribov@samsung.com>
Date:   Thu Jul 17 09:45:26 2014 +0400

    Move inlining of Asan memory checks to sanopt pass.
    Change asan-instrumentation-with-call-threshold to more closely match LLVM.
    
    gcc/
    
    2014-07-23  Yury Gribov  <y.gribov@samsung.com>
    
    	* asan.c (asan_check_flags): New enum.
    	(asan_clear_shadow): Fix formatting.
    	(asan_function_start): Likewise.
    	(asan_emit_stack_protection): Likewise.
    	(build_check_stmt_with_calls): Removed function.
    	(build_check_stmt): Split inlining logic to
    	asan_expand_check_ifn.
    	(instrument_derefs): Rename parameter.
    	(instrument_mem_region_access): Rename parameter.
    	(instrument_strlen_call): Likewise.
    	(asan_expand_check): New function.
    	(asan_instrument): Remove old code.
    	(pass_sanopt::execute): Change handling of
    	asan-instrumentation-with-call-threshold.
    	* builtin-types.def (BT_FN_VOID_PTR_PTRMODE_INT): New type.
    	* builtins.def: Fix comment.
    	* doc/invoke.texi (asan-instrumentation-with-call-threshold):
    	Update description.
    	* params.def (PARAM_ASAN_INSTRUMENTATION_WITH_CALL_THRESHOLD):
    	Update description.
    	(PARAM_ASAN_USE_AFTER_RETURN): Likewise.
    	* sanitizer.def (BUILT_IN_ASAN_CHECK): New builtin.
    
    gcc/testsuite/
    
    2014-07-23  Yury Gribov  <y.gribov@samsung.com>
    
    	* c-c++-common/asan/inc.c: Update test.
    	* c-c++-common/asan/instrument-with-calls-2.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-1.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-2.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-3.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-4.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-5.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-6.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-7.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-8.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-9.c: Likewise.

diff --git a/gcc/asan.c b/gcc/asan.c
index 0d78634..6f0aa2d 100644
--- a/gcc/asan.c
+++ b/gcc/asan.c
@@ -243,18 +243,16 @@ static GTY(()) tree shadow_ptr_types[2];
 /* Decl for __asan_option_detect_stack_use_after_return.  */
 static GTY(()) tree asan_detect_stack_use_after_return;
 
-/* Number of instrumentations in current function so far.  */
-
-static int asan_num_accesses;
-
-/* Check whether we should replace inline instrumentation with calls.  */
-
-static inline bool
-use_calls_p ()
+/* Various flags for Asan builtins.  */
+enum asan_check_flags
 {
-  return ASAN_INSTRUMENTATION_WITH_CALL_THRESHOLD < INT_MAX
-    && asan_num_accesses >= ASAN_INSTRUMENTATION_WITH_CALL_THRESHOLD;
-}
+  ASAN_CHECK_STORE = 1 << 0,
+  ASAN_CHECK_SCALAR_ACCESS = 1 << 1,
+  ASAN_CHECK_NON_ZERO_LEN = 1 << 2,
+  ASAN_CHECK_START_INSTRUMENTED = 1 << 3,
+  ASAN_CHECK_END_INSTRUMENTED = 1 << 4,
+  ASAN_CHECK_LAST
+};
 
 /* Hashtable support for memory references used by gimple
    statements.  */
@@ -943,7 +941,7 @@ asan_clear_shadow (rtx shadow_mem, HOST_WIDE_INT len)
 
   emit_move_insn (shadow_mem, const0_rtx);
   tmp = expand_simple_binop (Pmode, PLUS, addr, gen_int_mode (4, Pmode), addr,
-                             true, OPTAB_LIB_WIDEN);
+			     true, OPTAB_LIB_WIDEN);
   if (tmp != addr)
     emit_move_insn (addr, tmp);
   emit_cmp_and_jump_insns (addr, end, LT, NULL_RTX, Pmode, true, top_label);
@@ -958,7 +956,7 @@ asan_function_start (void)
   section *fnsec = function_section (current_function_decl);
   switch_to_section (fnsec);
   ASM_OUTPUT_DEBUG_LABEL (asm_out_file, "LASANPC",
-                         current_function_funcdef_no);
+			 current_function_funcdef_no);
 }
 
 /* Insert code to protect stack vars.  The prologue sequence should be emitted
@@ -1023,7 +1021,7 @@ asan_emit_stack_protection (rtx base, rtx pbase, unsigned int alignb,
     {
       use_after_return_class = floor_log2 (asan_frame_size - 1) - 5;
       /* __asan_stack_malloc_N guarantees alignment
-         N < 6 ? (64 << N) : 4096 bytes.  */
+	 N < 6 ? (64 << N) : 4096 bytes.  */
       if (alignb > (use_after_return_class < 6
 		    ? (64U << use_after_return_class) : 4096U))
 	use_after_return_class = -1;
@@ -1096,7 +1094,7 @@ asan_emit_stack_protection (rtx base, rtx pbase, unsigned int alignb,
   ASM_GENERATE_INTERNAL_LABEL (buf, "LASANPC", current_function_funcdef_no);
   id = get_identifier (buf);
   decl = build_decl (DECL_SOURCE_LOCATION (current_function_decl),
-                    VAR_DECL, id, char_type_node);
+		    VAR_DECL, id, char_type_node);
   SET_DECL_ASSEMBLER_NAME (decl, id);
   TREE_ADDRESSABLE (decl) = 1;
   TREE_READONLY (decl) = 1;
@@ -1553,55 +1551,6 @@ maybe_create_ssa_name (location_t loc, tree base, gimple_stmt_iterator *iter,
   return gimple_assign_lhs (g);
 }
 
-/* Instrument the memory access instruction using callbacks.
-   Parameters are similar to BUILD_CHECK_STMT.  */
-
-static void
-build_check_stmt_with_calls (location_t loc, tree base, tree len,
-			     HOST_WIDE_INT size_in_bytes, gimple_stmt_iterator *iter,
-			     bool before_p, bool is_store, bool is_scalar_access)
-{
-  gimple_stmt_iterator gsi = *iter;
-  tree base_ssa = maybe_create_ssa_name (loc, base, &gsi, before_p);
-
-  gimple g
-    = gimple_build_assign_with_ops (NOP_EXPR,
-				    make_ssa_name (pointer_sized_int_node, NULL),
-				    base_ssa, NULL_TREE);
-  gimple_set_location (g, loc);
-  if (before_p)
-    gsi_insert_before (&gsi, g, GSI_NEW_STMT);
-  else
-    gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-  tree base_addr = gimple_assign_lhs (g);
-
-  int nargs;
-  tree fun
-    = check_func (is_store, is_scalar_access ? size_in_bytes : -1, &nargs);
-  if (nargs == 1)
-    g = gimple_build_call (fun, 1, base_addr);
-  else
-    {
-      gcc_assert (nargs == 2);
-      g = gimple_build_assign_with_ops (NOP_EXPR,
-					make_ssa_name (pointer_sized_int_node,
-						       NULL),
-					len, NULL_TREE);
-      gimple_set_location (g, loc);
-      gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-      tree sz_arg = gimple_assign_lhs (g);
-      g = gimple_build_call (fun, nargs, base_addr, sz_arg);
-    }
-  gimple_set_location (g, loc);
-  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-
-  if (!before_p)
-    {
-      gsi_next (&gsi);
-      *iter = gsi;
-    }
-}
-
 /* Instrument the memory access instruction BASE.  Insert new
    statements before or after ITER.
 
@@ -1622,27 +1571,30 @@ build_check_stmt_with_calls (location_t loc, tree base, tree len,
    otherwise, it points to the statement logically following it.  */
 
 static void
-build_check_stmt (location_t location, tree base, tree len,
+build_check_stmt (location_t loc, tree base, tree len,
 		  HOST_WIDE_INT size_in_bytes, gimple_stmt_iterator *iter,
-		  bool non_zero_len_p, bool before_p, bool is_store,
+		  bool is_non_zero_len, bool before_p, bool is_store,
 		  bool is_scalar_access, unsigned int align = 0,
 		  bool start_instrumented = false,
 		  bool end_instrumented = false)
 {
   gimple_stmt_iterator gsi = *iter;
   gimple g;
-  tree uintptr_type
-    = build_nonstandard_integer_type (TYPE_PRECISION (TREE_TYPE (base)), 1);
 
-  gcc_assert (!(size_in_bytes > 0 && !non_zero_len_p));
+  gcc_assert (!(size_in_bytes > 0 && !is_non_zero_len));
 
   if (start_instrumented && end_instrumented)
     {
       if (!before_p)
-        gsi_next (iter);
+	gsi_next (iter);
       return;
     }
 
+  gsi = *iter;
+
+  base = unshare_expr (base);
+  base = maybe_create_ssa_name (loc, base, &gsi, before_p);
+
   if (len)
     len = unshare_expr (len);
   else
@@ -1654,9 +1606,8 @@ build_check_stmt (location_t location, tree base, tree len,
   if (size_in_bytes > 1)
     {
       if ((size_in_bytes & (size_in_bytes - 1)) != 0
-	  || !is_scalar_access
 	  || size_in_bytes > 16)
-	size_in_bytes = -1;
+	is_scalar_access = false;
       else if (align && align < size_in_bytes * BITS_PER_UNIT)
 	{
 	  /* On non-strict alignment targets, if
@@ -1667,189 +1618,34 @@ build_check_stmt (location_t location, tree base, tree len,
 	  if (size_in_bytes != 16
 	      || STRICT_ALIGNMENT
 	      || align < 8 * BITS_PER_UNIT)
-	    size_in_bytes = -1;
+	    is_scalar_access = false;
 	}
     }
 
-  HOST_WIDE_INT real_size_in_bytes = size_in_bytes == -1 ? 1 : size_in_bytes;
-
-  tree shadow_ptr_type = shadow_ptr_types[real_size_in_bytes == 16 ? 1 : 0];
-  tree shadow_type = TREE_TYPE (shadow_ptr_type);
-
-  base = unshare_expr (base);
-
-  if (use_calls_p ())
-    {
-      gsi = *iter;
-      build_check_stmt_with_calls (location, base, len, size_in_bytes, iter,
-				   before_p, is_store, is_scalar_access);
-      return;
-    }
-
-  ++asan_num_accesses;
-
-  if (!non_zero_len_p)
-    {
-      gcc_assert (before_p);
-
-      /* So, the length of the memory area to asan-protect is
-	 non-constant.  Let's guard the generated instrumentation code
-	 like:
-
-	 if (len != 0)
-	   {
-	     //asan instrumentation code goes here.
-	   }
-	 // falltrough instructions, starting with *ITER.  */
-
-      g = gimple_build_cond (NE_EXPR,
-			     len,
-			     build_int_cst (TREE_TYPE (len), 0),
-			     NULL_TREE, NULL_TREE);
-      gimple_set_location (g, location);
-
-      basic_block then_bb, fallthrough_bb;
-      insert_if_then_before_iter (g, iter, /*then_more_likely_p=*/true,
-				  &then_bb, &fallthrough_bb);
-      /* Note that fallthrough_bb starts with the statement that was
-	 pointed to by ITER.  */
-
-      /* The 'then block' of the 'if (len != 0) condition is where
-	 we'll generate the asan instrumentation code now.  */
-      gsi = gsi_last_bb (then_bb);
-      build_check_stmt (location, base, len, size_in_bytes, &gsi,
-			/*non_zero_len_p*/true, /*before_p*/true, is_store,
-			is_scalar_access, align,
-			start_instrumented, end_instrumented);
-      return;
-    }
-
-  /* Get an iterator on the point where we can add the condition
-     statement for the instrumentation.  */
-  basic_block then_bb, else_bb;
-  gsi = create_cond_insert_point (&gsi, before_p,
-				  /*then_more_likely_p=*/false,
-				  /*create_then_fallthru_edge=*/false,
-				  &then_bb,
-				  &else_bb);
-
-  tree base_ssa = maybe_create_ssa_name (location, base, &gsi,
-					 /*before_p*/false);
-
-  g = gimple_build_assign_with_ops (NOP_EXPR,
-				    make_ssa_name (uintptr_type, NULL),
-				    base_ssa, NULL_TREE);
-  gimple_set_location (g, location);
-  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-  tree base_addr = gimple_assign_lhs (g);
-
-  tree t = NULL_TREE;
-  if (real_size_in_bytes >= 8)
-    {
-      tree shadow = build_shadow_mem_access (&gsi, location, base_addr,
-					     shadow_ptr_type);
-      t = shadow;
-    }
-  else
-    {
-      /* Slow path for 1, 2 and 4 byte accesses.  */
-
-      if (!start_instrumented)
-	{
-	  /* Test (shadow != 0)
-		  & ((base_addr & 7) + (real_size_in_bytes - 1)) >= shadow).  */
-	  tree shadow = build_shadow_mem_access (&gsi, location, base_addr,
-						 shadow_ptr_type);
-	  gimple shadow_test = build_assign (NE_EXPR, shadow, 0);
-	  gimple_seq seq = NULL;
-	  gimple_seq_add_stmt (&seq, shadow_test);
-	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR, base_addr, 7));
-	  gimple_seq_add_stmt (&seq, build_type_cast (shadow_type,
-						      gimple_seq_last (seq)));
-	  if (real_size_in_bytes > 1)
-	    gimple_seq_add_stmt (&seq,
-				 build_assign (PLUS_EXPR, gimple_seq_last (seq),
-					       real_size_in_bytes - 1));
-	  gimple_seq_add_stmt (&seq, build_assign (GE_EXPR,
-						   gimple_seq_last (seq),
-						   shadow));
-	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR, shadow_test,
-						   gimple_seq_last (seq)));
-	  t = gimple_assign_lhs (gimple_seq_last (seq));
-	  gimple_seq_set_location (seq, location);
-	  gsi_insert_seq_after (&gsi, seq, GSI_CONTINUE_LINKING);
-	}
-
-      /* For non-constant, misaligned or otherwise weird access sizes,
-	 check first and last byte.  */
-      if (size_in_bytes == -1 && !end_instrumented)
-	{
-	  g = gimple_build_assign_with_ops (MINUS_EXPR,
-					    make_ssa_name (uintptr_type, NULL),
-					    len,
-					    build_int_cst (uintptr_type, 1));
-	  gimple_set_location (g, location);
-	  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-	  tree last = gimple_assign_lhs (g);
-	  g = gimple_build_assign_with_ops (PLUS_EXPR,
-					    make_ssa_name (uintptr_type, NULL),
-					    base_addr,
-					    last);
-	  gimple_set_location (g, location);
-	  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-	  tree base_end_addr = gimple_assign_lhs (g);
-
-	  tree shadow = build_shadow_mem_access (&gsi, location, base_end_addr,
-						 shadow_ptr_type);
-	  gimple shadow_test = build_assign (NE_EXPR, shadow, 0);
-	  gimple_seq seq = NULL;
-	  gimple_seq_add_stmt (&seq, shadow_test);
-	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR,
-						   base_end_addr, 7));
-	  gimple_seq_add_stmt (&seq, build_type_cast (shadow_type,
-						      gimple_seq_last (seq)));
-	  gimple_seq_add_stmt (&seq, build_assign (GE_EXPR,
-						   gimple_seq_last (seq),
-						   shadow));
-	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR, shadow_test,
-						   gimple_seq_last (seq)));
-	  if (!start_instrumented)
-	    gimple_seq_add_stmt (&seq, build_assign (BIT_IOR_EXPR, t,
-						     gimple_seq_last (seq)));
-	  t = gimple_assign_lhs (gimple_seq_last (seq));
-	  gimple_seq_set_location (seq, location);
-	  gsi_insert_seq_after (&gsi, seq, GSI_CONTINUE_LINKING);
-	}
-    }
-
-  g = gimple_build_cond (NE_EXPR, t, build_int_cst (TREE_TYPE (t), 0),
-			 NULL_TREE, NULL_TREE);
-  gimple_set_location (g, location);
-  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-
-  /* Generate call to the run-time library (e.g. __asan_report_load8).  */
-  gsi = gsi_start_bb (then_bb);
-  int nargs;
-  tree fun = report_error_func (is_store, is_scalar_access ? size_in_bytes : -1,
-				&nargs);
-  if (nargs == 1)
-      g = gimple_build_call (fun, 1, base_addr);
+  HOST_WIDE_INT flags = 0;
+  if (is_store)
+    flags |= ASAN_CHECK_STORE;
+  if (is_non_zero_len)
+    flags |= ASAN_CHECK_NON_ZERO_LEN;
+  if (is_scalar_access)
+    flags |= ASAN_CHECK_SCALAR_ACCESS;
+  if (start_instrumented)
+    flags |= ASAN_CHECK_START_INSTRUMENTED;
+  if (end_instrumented)
+    flags |= ASAN_CHECK_END_INSTRUMENTED;
+
+  tree fun = builtin_decl_implicit (BUILT_IN_ASAN_CHECK);
+  g = gimple_build_call (fun, 3, base, len,
+			 build_int_cst (integer_type_node, flags));
+  gimple_set_location (g, loc);
+  if (before_p)
+    gsi_insert_before (&gsi, g, GSI_SAME_STMT);
   else
     {
-      gcc_assert (nargs == 2);
-      g = gimple_build_assign_with_ops (NOP_EXPR,
-					make_ssa_name (pointer_sized_int_node,
-						       NULL),
-					len, NULL_TREE);
-      gimple_set_location (g, location);
       gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-      tree sz_arg = gimple_assign_lhs (g);
-      g = gimple_build_call (fun, nargs, base_addr, sz_arg);
+      gsi_next (&gsi);
+      *iter = gsi;
     }
-  gimple_set_location (g, location);
-  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-
-  *iter = gsi_start_bb (else_bb);
 }
 
 /* If T represents a memory access, add instrumentation code before ITER.
@@ -1942,7 +1738,7 @@ instrument_derefs (gimple_stmt_iterator *iter, tree t,
     {
       unsigned int align = get_object_alignment (t);
       build_check_stmt (location, base, NULL_TREE, size_in_bytes, iter,
-			/*non_zero_len_p*/size_in_bytes > 0, /*before_p=*/true,
+			/*is_non_zero_len*/size_in_bytes > 0, /*before_p=*/true,
 			is_store, /*is_scalar_access*/true, align);
       update_mem_ref_hash_table (base, size_in_bytes);
       update_mem_ref_hash_table (t, size_in_bytes);
@@ -1980,7 +1776,7 @@ instrument_mem_region_access (tree base, tree len,
   HOST_WIDE_INT size_in_bytes = tree_fits_shwi_p (len) ? tree_to_shwi (len) : -1;
 
   build_check_stmt (location, base, len, size_in_bytes, iter,
-		    /*non_zero_len_p*/size_in_bytes > 0, /*before_p*/true,
+		    /*is_non_zero_len*/size_in_bytes > 0, /*before_p*/true,
 		    is_store, /*is_scalar_access*/false, /*align*/0,
 		    start_instrumented, end_instrumented);
 
@@ -2037,7 +1833,7 @@ instrument_strlen_call (gimple_stmt_iterator *iter)
   gsi_insert_before (iter, str_arg_ssa, GSI_SAME_STMT);
 
   build_check_stmt (loc, gimple_assign_lhs (str_arg_ssa), NULL_TREE, 1, iter,
-		    /*non_zero_len_p*/true, /*before_p=*/true,
+		    /*is_non_zero_len*/true, /*before_p=*/true,
 		    /*is_store=*/false, /*is_scalar_access*/true, /*align*/0,
 		    start_instrumented, start_instrumented);
 
@@ -2050,7 +1846,7 @@ instrument_strlen_call (gimple_stmt_iterator *iter)
   gsi_insert_after (iter, g, GSI_NEW_STMT);
 
   build_check_stmt (loc, gimple_assign_lhs (g), NULL_TREE, 1, iter,
-		    /*non_zero_len_p*/true, /*before_p=*/false,
+		    /*is_non_zero_len*/true, /*before_p=*/false,
 		    /*is_store=*/false, /*is_scalar_access*/true, /*align*/0);
 
   return true;
@@ -2414,6 +2210,10 @@ initialize_sanitizer_builtins (void)
   tree BT_FN_VOID_PTR_PTRMODE
     = build_function_type_list (void_type_node, ptr_type_node,
 				pointer_sized_int_node, NULL_TREE);
+  tree BT_FN_VOID_PTR_PTRMODE_INT
+    = build_function_type_list (void_type_node, ptr_type_node,
+				pointer_sized_int_node, integer_type_node,
+				NULL_TREE);
   tree BT_FN_VOID_INT
     = build_function_type_list (void_type_node, integer_type_node, NULL_TREE);
   tree BT_FN_BOOL_VPTR_PTR_IX_INT_INT[5];
@@ -2491,6 +2291,14 @@ initialize_sanitizer_builtins (void)
 #include "sanitizer.def"
 
 #undef DEF_SANITIZER_BUILTIN
+
+  /* Add fn spec to __asan_check.  */
+
+  decl = builtin_decl_implicit (BUILT_IN_ASAN_CHECK);
+  tree attrs = tree_cons (get_identifier ("fn spec"),
+			  build_tree_list (NULL_TREE, build_string (3, "R..")),
+			  DECL_ATTRIBUTES (decl));
+  DECL_ATTRIBUTES (decl) = attrs;
 }
 
 /* Called via htab_traverse.  Count number of emitted
@@ -2617,6 +2425,213 @@ asan_finish_file (void)
   flag_sanitize |= SANITIZE_ADDRESS;
 }
 
+/* Expand ASAN_CHECK builtin.  */
+
+static void
+asan_expand_check (gimple_stmt_iterator *iter, bool use_calls)
+{
+  gimple g = gsi_stmt (*iter);
+  location_t loc = gimple_location (g);
+
+  tree base = gimple_call_arg (g, 0);
+  tree len = gimple_call_arg (g, 1);
+
+  HOST_WIDE_INT flags = tree_to_shwi (gimple_call_arg (g, 2));
+  gcc_assert (flags < ASAN_CHECK_LAST);
+  bool is_scalar_access = (flags & ASAN_CHECK_SCALAR_ACCESS) != 0;
+  bool is_store = (flags & ASAN_CHECK_STORE) != 0;
+  bool is_non_zero_len = (flags & ASAN_CHECK_NON_ZERO_LEN) != 0;
+  bool start_instrumented = (flags & ASAN_CHECK_START_INSTRUMENTED) != 0;
+  bool end_instrumented = (flags & ASAN_CHECK_END_INSTRUMENTED) != 0;
+
+  HOST_WIDE_INT size_in_bytes
+    = is_scalar_access && tree_fits_shwi_p (len) ? tree_to_shwi (len) : -1;
+
+  if (use_calls)
+    {
+      /* Instrument using callbacks.  */
+      gimple g
+	= gimple_build_assign_with_ops (NOP_EXPR,
+					make_ssa_name (pointer_sized_int_node,
+					NULL),
+					base, NULL_TREE);
+      gimple_set_location (g, loc);
+      gsi_insert_before (iter, g, GSI_SAME_STMT);
+      tree base_addr = gimple_assign_lhs (g);
+
+      int nargs;
+      tree fun = check_func (is_store, size_in_bytes, &nargs);
+      if (nargs == 1)
+	g = gimple_build_call (fun, 1, base_addr);
+      else
+	{
+	  gcc_assert (nargs == 2);
+	  g = gimple_build_assign_with_ops (NOP_EXPR,
+					    make_ssa_name (pointer_sized_int_node,
+							   NULL),
+					    len, NULL_TREE);
+	  gimple_set_location (g, loc);
+	  gsi_insert_before (iter, g, GSI_SAME_STMT);
+	  tree sz_arg = gimple_assign_lhs (g);
+	  g = gimple_build_call (fun, nargs, base_addr, sz_arg);
+	}
+      gimple_set_location (g, loc);
+      gsi_replace (iter, g, false);
+      return;
+    }
+
+  HOST_WIDE_INT real_size_in_bytes = size_in_bytes == -1 ? 1 : size_in_bytes;
+
+  tree uintptr_type
+    = build_nonstandard_integer_type (TYPE_PRECISION (TREE_TYPE (base)), 1);
+
+  tree shadow_ptr_type = shadow_ptr_types[real_size_in_bytes == 16 ? 1 : 0];
+  tree shadow_type = TREE_TYPE (shadow_ptr_type);
+
+  gimple_stmt_iterator gsi = *iter;
+
+  if (!is_non_zero_len)
+    {
+      /* So, the length of the memory area to asan-protect is
+	 non-constant.  Let's guard the generated instrumentation code
+	 like:
+
+	 if (len != 0)
+	   {
+	     //asan instrumentation code goes here.
+	   }
+	 // falltrough instructions, starting with *ITER.  */
+
+      g = gimple_build_cond (NE_EXPR,
+			    len,
+			    build_int_cst (TREE_TYPE (len), 0),
+			    NULL_TREE, NULL_TREE);
+      gimple_set_location (g, loc);
+
+      basic_block then_bb, fallthrough_bb;
+      insert_if_then_before_iter (g, iter, /*then_more_likely_p=*/true,
+				 &then_bb, &fallthrough_bb);
+      /* Note that fallthrough_bb starts with the statement that was
+	pointed to by ITER.  */
+
+      /* The 'then block' of the 'if (len != 0) condition is where
+	we'll generate the asan instrumentation code now.  */
+      gsi = gsi_last_bb (then_bb);
+    }
+
+  /* Get an iterator on the point where we can add the condition
+     statement for the instrumentation.  */
+  basic_block then_bb, else_bb;
+  gsi = create_cond_insert_point (&gsi, /*before_p*/false,
+				  /*then_more_likely_p=*/false,
+				  /*create_then_fallthru_edge=*/false,
+				  &then_bb,
+				  &else_bb);
+
+  g = gimple_build_assign_with_ops (NOP_EXPR,
+				    make_ssa_name (pointer_sized_int_node,
+						   NULL),
+				    base, NULL_TREE);
+  gimple_set_location (g, loc);
+  gsi_insert_before (&gsi, g, GSI_NEW_STMT);
+  tree base_addr = gimple_assign_lhs (g);
+
+  tree t = NULL_TREE;
+  if (real_size_in_bytes >= 8)
+    {
+      tree shadow = build_shadow_mem_access (&gsi, loc, base_addr,
+					     shadow_ptr_type);
+      t = shadow;
+    }
+  else
+    {
+      /* Slow path for 1, 2 and 4 byte accesses.  */
+
+      if (!start_instrumented)
+	{
+	  /* Test (shadow != 0)
+	     & ((base_addr & 7) + (real_size_in_bytes - 1)) >= shadow).  */
+	  tree shadow = build_shadow_mem_access (&gsi, loc, base_addr,
+						 shadow_ptr_type);
+	  gimple shadow_test = build_assign (NE_EXPR, shadow, 0);
+	  gimple_seq seq = NULL;
+	  gimple_seq_add_stmt (&seq, shadow_test);
+	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR, base_addr, 7));
+	  gimple_seq_add_stmt (&seq, build_type_cast (shadow_type,
+						      gimple_seq_last (seq)));
+	  if (real_size_in_bytes > 1)
+	    gimple_seq_add_stmt (&seq,
+				 build_assign (PLUS_EXPR, gimple_seq_last (seq),
+					       real_size_in_bytes - 1));
+	  gimple_seq_add_stmt (&seq, build_assign (GE_EXPR,
+						   gimple_seq_last (seq),
+						   shadow));
+	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR, shadow_test,
+						   gimple_seq_last (seq)));
+	  t = gimple_assign_lhs (gimple_seq_last (seq));
+	  gimple_seq_set_location (seq, loc);
+	  gsi_insert_seq_after (&gsi, seq, GSI_CONTINUE_LINKING);
+	}
+
+      /* For non-constant, misaligned or otherwise weird access sizes,
+	 check first and last byte.  */
+      if (size_in_bytes == -1 && !end_instrumented)
+	{
+	  g = gimple_build_assign_with_ops (MINUS_EXPR,
+					    make_ssa_name (uintptr_type, NULL),
+					    len,
+					    build_int_cst (uintptr_type, 1));
+	  gimple_set_location (g, loc);
+	  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
+	  tree last = gimple_assign_lhs (g);
+	  g = gimple_build_assign_with_ops (PLUS_EXPR,
+					    make_ssa_name (uintptr_type, NULL),
+					    base_addr,
+					    last);
+	  gimple_set_location (g, loc);
+	  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
+	  tree base_end_addr = gimple_assign_lhs (g);
+
+	  tree shadow = build_shadow_mem_access (&gsi, loc, base_end_addr,
+						 shadow_ptr_type);
+	  gimple shadow_test = build_assign (NE_EXPR, shadow, 0);
+	  gimple_seq seq = NULL;
+	  gimple_seq_add_stmt (&seq, shadow_test);
+	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR,
+						   base_end_addr, 7));
+	  gimple_seq_add_stmt (&seq, build_type_cast (shadow_type,
+						      gimple_seq_last (seq)));
+	  gimple_seq_add_stmt (&seq, build_assign (GE_EXPR,
+						   gimple_seq_last (seq),
+						   shadow));
+	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR, shadow_test,
+						   gimple_seq_last (seq)));
+	  if (!start_instrumented)
+	    gimple_seq_add_stmt (&seq, build_assign (BIT_IOR_EXPR, t,
+						     gimple_seq_last (seq)));
+	  t = gimple_assign_lhs (gimple_seq_last (seq));
+	  gimple_seq_set_location (seq, loc);
+	  gsi_insert_seq_after (&gsi, seq, GSI_CONTINUE_LINKING);
+	}
+    }
+
+  g = gimple_build_cond (NE_EXPR, t, build_int_cst (TREE_TYPE (t), 0),
+			 NULL_TREE, NULL_TREE);
+  gimple_set_location (g, loc);
+  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
+
+  /* Generate call to the run-time library (e.g. __asan_report_load8).  */
+  gsi = gsi_start_bb (then_bb);
+  int nargs;
+  tree fun = report_error_func (is_store, size_in_bytes, &nargs);
+  g = gimple_build_call (fun, nargs, base_addr, len);
+  gimple_set_location (g, loc);
+  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
+
+  gsi_remove (iter, true);
+  *iter = gsi_start_bb (else_bb);
+}
+
 /* Instrument the current function.  */
 
 static unsigned int
@@ -2624,7 +2639,6 @@ asan_instrument (void)
 {
   if (shadow_ptr_types[0] == NULL_TREE)
     asan_init_shadow_ptr_types ();
-  asan_num_accesses = 0;
   transform_statements ();
   return 0;
 }
@@ -2745,6 +2759,22 @@ pass_sanopt::execute (function *fun)
 {
   basic_block bb;
 
+  int asan_num_accesses = 0;
+  if (flag_sanitize & SANITIZE_ADDRESS)
+    {
+      gimple_stmt_iterator gsi;
+      FOR_EACH_BB_FN (bb, fun)
+	for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi))
+	  {
+ 	    gimple stmt = gsi_stmt (gsi);
+	    if (gimple_call_builtin_p (stmt, BUILT_IN_ASAN_CHECK))
+	      ++asan_num_accesses;
+	  }
+    }
+
+  bool use_calls = ASAN_INSTRUMENTATION_WITH_CALL_THRESHOLD < INT_MAX
+    && asan_num_accesses >= ASAN_INSTRUMENTATION_WITH_CALL_THRESHOLD;
+
   FOR_EACH_BB_FN (bb, fun)
     {
       gimple_stmt_iterator gsi;
@@ -2756,17 +2786,22 @@ pass_sanopt::execute (function *fun)
 	    continue;
 
 	  if (gimple_call_internal_p (stmt))
-	    switch (gimple_call_internal_fn (stmt))
-	      {
-	      case IFN_UBSAN_NULL:
-		ubsan_expand_null_ifn (gsi);
-		break;
-	      case IFN_UBSAN_BOUNDS:
-		ubsan_expand_bounds_ifn (&gsi);
-		break;
-	      default:
-		break;
-	      }
+	    {
+	      enum internal_fn ifn = gimple_call_internal_fn (stmt);
+	      switch (ifn)
+		{
+		case IFN_UBSAN_NULL:
+		  ubsan_expand_null_ifn (gsi);
+		  break;
+		case IFN_UBSAN_BOUNDS:
+		  ubsan_expand_bounds_ifn (&gsi);
+		  break;
+		default:
+		  break;
+		}
+	    }
+	  else if (gimple_call_builtin_p (stmt, BUILT_IN_ASAN_CHECK))
+	    asan_expand_check (&gsi, use_calls);
 
 	  if (dump_file && (dump_flags & TDF_DETAILS))
 	    {
@@ -2775,7 +2810,7 @@ pass_sanopt::execute (function *fun)
 	      fprintf (dump_file, "\n");
 	    }
 
-	  /* ubsan_expand_bounds_ifn might move us to the end of the BB.  */
+	  /* Expansions might move us to the end of the BB.  */
 	  if (gsi_end_p (gsi))
 	    break;
 	}
diff --git a/gcc/builtin-types.def b/gcc/builtin-types.def
index f09c335..1690130 100644
--- a/gcc/builtin-types.def
+++ b/gcc/builtin-types.def
@@ -422,6 +422,7 @@ DEF_FUNCTION_TYPE_3 (BT_FN_VOID_VPTR_I4_INT, BT_VOID, BT_VOLATILE_PTR, BT_I4, BT
 DEF_FUNCTION_TYPE_3 (BT_FN_VOID_VPTR_I8_INT, BT_VOID, BT_VOLATILE_PTR, BT_I8, BT_INT)
 DEF_FUNCTION_TYPE_3 (BT_FN_VOID_VPTR_I16_INT, BT_VOID, BT_VOLATILE_PTR, BT_I16, BT_INT)
 DEF_FUNCTION_TYPE_3 (BT_FN_INT_PTRPTR_SIZE_SIZE, BT_INT, BT_PTR_PTR, BT_SIZE, BT_SIZE)
+DEF_FUNCTION_TYPE_3 (BT_FN_VOID_PTR_PTRMODE_INT, BT_VOID, BT_PTR, BT_PTRMODE, BT_INT)
 
 DEF_FUNCTION_TYPE_4 (BT_FN_SIZE_CONST_PTR_SIZE_SIZE_FILEPTR,
 		     BT_SIZE, BT_CONST_PTR, BT_SIZE, BT_SIZE, BT_FILEPTR)
diff --git a/gcc/builtins.def b/gcc/builtins.def
index cd823a3..fbfc3c0 100644
--- a/gcc/builtins.def
+++ b/gcc/builtins.def
@@ -169,8 +169,8 @@ along with GCC; see the file COPYING3.  If not see
 	       true, true, true, ATTRS, false, flag_tm)
 
 /* Builtin used by the implementation of libsanitizer. These
-   functions are mapped to the actual implementation of the 
-   libtsan library. */
+   functions are mapped to the actual implementation in
+   lib*san libraries. */
 #undef DEF_SANITIZER_BUILTIN
 #define DEF_SANITIZER_BUILTIN(ENUM, NAME, TYPE, ATTRS) \
   DEF_BUILTIN (ENUM, "__builtin_" NAME, BUILT_IN_NORMAL, TYPE, TYPE,    \
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index b5e8d98..8a755df 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -10312,9 +10312,9 @@ To disable use-after-return detection use
 @option{--param asan-use-after-return=0}.
 
 @item asan-instrumentation-with-call-threshold
-Once number of memory accesses  in function becomes greater
-or equal than this number, use callbacks instead of
-generating inline code.  E.g. to disable inline code use
+If number of memory accesses in function being instrumented
+is greater or equal to this number, use callbacks instead of inline checks.
+E.g. to disable inline code use
 @option{--param asan-instrumentation-with-call-threshold=0}.
 
 @end table
diff --git a/gcc/params.def b/gcc/params.def
index aa1e88d..f1c44e0 100644
--- a/gcc/params.def
+++ b/gcc/params.def
@@ -1072,14 +1072,14 @@ DEFPARAM (PARAM_ASAN_MEMINTRIN,
 
 DEFPARAM (PARAM_ASAN_USE_AFTER_RETURN,
          "asan-use-after-return",
-         "Enable asan builtin functions protection",
+         "Enable asan detection of use-after-return bugs",
          1, 0, 1)
 
 DEFPARAM (PARAM_ASAN_INSTRUMENTATION_WITH_CALL_THRESHOLD,
          "asan-instrumentation-with-call-threshold",
-         "Use callbacks instead of inline code once number of accesses "
-         " in function becomes greater or equal than this threshold",
-         10000, 0, INT_MAX)
+         "Use callbacks instead of inline code if number of accesses "
+         "in function becomes greater or equal to this number",
+         7000, 0, INT_MAX)
 
 DEFPARAM (PARAM_UNINIT_CONTROL_DEP_ATTEMPTS,
 	  "uninit-control-dep-attempts",
diff --git a/gcc/sanitizer.def b/gcc/sanitizer.def
index 1f5ef21..5dfc369 100644
--- a/gcc/sanitizer.def
+++ b/gcc/sanitizer.def
@@ -27,6 +27,8 @@ along with GCC; see the file COPYING3.  If not see
    for other FEs by asan.c.  */
 
 /* Address Sanitizer */
+DEF_SANITIZER_BUILTIN (BUILT_IN_ASAN_CHECK, "__asan_check",
+		       BT_FN_VOID_PTR_PTRMODE_INT, ATTR_NOTHROW_LEAF_LIST)
 DEF_SANITIZER_BUILTIN(BUILT_IN_ASAN_INIT, "__asan_init_v3",
 		      BT_FN_VOID, ATTR_NOTHROW_LEAF_LIST)
 /* Do not reorder the BUILT_IN_ASAN_{REPORT,CHECK}* builtins, e.g. cfgcleanup.c
diff --git a/gcc/testsuite/c-c++-common/asan/inc.c b/gcc/testsuite/c-c++-common/asan/inc.c
index b9c6734..147709e 100644
--- a/gcc/testsuite/c-c++-common/asan/inc.c
+++ b/gcc/testsuite/c-c++-common/asan/inc.c
@@ -16,6 +16,6 @@ main ()
   return 0;
 }
 
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report" 1 "asan0" } }  */
-/* { dg-final { scan-tree-dump "__builtin___asan_report_load4" "asan0" } }  */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_check" 1 "asan0" } }  */
+/* { dg-final { scan-tree-dump "__builtin___asan_check \\(.*, 4," "asan0" } }  */
 /* { dg-final { cleanup-tree-dump "asan0" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/instrument-with-calls-2.c b/gcc/testsuite/c-c++-common/asan/instrument-with-calls-2.c
index 570f796..04fdad0 100644
--- a/gcc/testsuite/c-c++-common/asan/instrument-with-calls-2.c
+++ b/gcc/testsuite/c-c++-common/asan/instrument-with-calls-2.c
@@ -9,8 +9,8 @@ void f(int *a, int *b) {
   x = *b;
 }
 
-/* { dg-final { scan-assembler-not "__asan_store4" } } */
-/* { dg-final { scan-assembler "__asan_report_store4" } } */
+/* { dg-final { scan-assembler "__asan_store4" } } */
+/* { dg-final { scan-assembler-not "__asan_report_store4" } } */
 /* { dg-final { scan-assembler "__asan_load4" } } */
 /* { dg-final { scan-assembler-not "__asan_report_load4" } } */
 /* { dg-final { cleanup-saved-temps } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-1.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-1.c
index c7c594e..028f8d7 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-1.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-1.c
@@ -2,7 +2,7 @@
    location in the same basic block, the second reference should not
    be instrumented by the Address Sanitizer.  */
 
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -62,7 +62,7 @@ main ()
   return test0 () && test1 (0);
 }
 
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store1" 3 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 2 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load" 1 "asan0" }  } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store1" 3 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 2 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load" 1 "sanopt" }  } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-2.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-2.c
index 143312f..a58411c 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-2.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-2.c
@@ -3,7 +3,7 @@
    be instrumented by the Address Sanitizer.  But in case of access to
    overlapping regions we must be precise.  */
 
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -20,7 +20,7 @@ main ()
   __builtin_memset (tab, 1, 3);
 }
 
-/* { dg-final { scan-tree-dump-times "& 7" 3 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 2 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report" 2 "asan0" }  } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "& 7" 3 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 2 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report" 2 "sanopt" }  } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-3.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-3.c
index 420a263..5193ae0 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-3.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-3.c
@@ -1,4 +1,4 @@
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -12,7 +12,7 @@ foo (__INT32_TYPE__ *p)
   return ret; 
 }
 
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report" 2 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store" 1 "asan0" } } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report" 2 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store" 1 "sanopt" } } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-4.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-4.c
index da91cd5..c3632aa 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-4.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-4.c
@@ -1,4 +1,4 @@
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -10,8 +10,8 @@ foo  (int *a, char *b, char *c)
   /* For a total of 5 checks.  */
 }
 
-/* { dg-final { scan-tree-dump-times "& 7" 5 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 1 "asan0" } } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "& 7" 5 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 1 "sanopt" } } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-5.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-5.c
index 134be66..077ea34 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-5.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-5.c
@@ -1,4 +1,4 @@
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -10,9 +10,9 @@ foo  (int *a, char *b, char *c)
   /* For a total of 5 checks.  */
 }
 
-/* { dg-final { scan-tree-dump-times "& 7" 5 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load4" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 1 "asan0" } } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "& 7" 5 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load4" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 1 "sanopt" } } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-6.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-6.c
index 55c8ee3..6d87104 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-6.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-6.c
@@ -1,4 +1,4 @@
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -12,9 +12,9 @@ foo  (int *a, char *b, char *c)
   /* For a total of 8 checks.  */
 }
 
-/* { dg-final { scan-tree-dump-times "& 7" 8 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load4" 2 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 2 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 2 "asan0" } } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "& 7" 8 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load4" 2 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 2 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 2 "sanopt" } } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-7.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-7.c
index a04956d..5baa10d 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-7.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-7.c
@@ -1,4 +1,4 @@
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -21,7 +21,7 @@ foo  (int *a, char *b, char *c)
   return d;
 }
 
-/* { dg-final { scan-tree-dump-times "& 7" 6 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 4 "asan0" } } */
-/* { dg-final { scan-tree-dump-not "__builtin___asan_report_store" "asan0" } } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "& 7" 6 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 4 "sanopt" } } */
+/* { dg-final { scan-tree-dump-not "__builtin___asan_report_store" "sanopt" } } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-8.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-8.c
index 308a133..2a4c081 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-8.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-8.c
@@ -1,4 +1,4 @@
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -12,9 +12,9 @@ foo  (int *a, char *b, char *c)
   /* For a total of 5 checks.  */
 }
 
-/* { dg-final { scan-tree-dump-times "& 7" 5 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load4" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 1 "asan0" } } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "& 7" 5 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load4" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 1 "sanopt" } } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-9.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-9.c
index c6575ad..9449de5 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-9.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-9.c
@@ -1,4 +1,4 @@
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -9,5 +9,5 @@ f (char *a)
   return  __builtin_strlen (a);
 }
 
-/* { dg-final { scan-tree-dump-times "__asan_report_load1" 1 "asan0" } } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "__asan_report_load1" 1 "sanopt" } } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */

Re: [PATCH] Move Asan instrumentation to sanopt pass

[Jakub Jelinek]

On Tue, Jul 22, 2014 at 04:20:04PM +0200, Richard Biener wrote:
> On Tue, Jul 22, 2014 at 4:15 PM, Yury Gribov <y.gribov@samsung.com> wrote:
> > On 07/22/2014 05:57 PM, Richard Biener wrote:
> >>>
> >>> I probably could
> >>> provide fnspec with (EAF_DIRECT | EAF_NOCLOBBER | EAF_NOESCAPE) or
> >>> even EAF_UNUSED for these functions but this does not seem
> >>> to be supported in current middle-end.
> >>
> >>
> >> Simply add the "fn spec" attribute to the functions?
> >
> >
> > Problem is that internal functions don't seem to support this
> > (gimple_statement_call::fntype and gimple_statement_call::internal_fn
> > occupy the same field). I wasn't sure about the reasons for this
> > so decided change them in original patch. Do you think we should we allow
> > fntype for internal functions?
> 
> Ah.... internal fns.  Those cannot have attributes indeed (technical
> limitation).
> Martin was working on putting those flags elsewhere (cgraph, though internal
> functions don't have cgraph nodes either ...).  Maybe it was a bad idea to use
> internal functions for ASAN.

But if it is a builtin function, then users can use it directly, which is
not really desirable.
For internal-fn, we already support ECF* constants, guess either we could
add support for EAF* too, through internal-fn.def, or we need support for
normal builtins that are inaccessible to users (but using . or space
in names is too ugly IMHO for that).

	Jakub

Re: [PATCH] Move Asan instrumentation to sanopt pass

[Yury Gribov]

On 07/24/2014 12:09 AM, Jakub Jelinek wrote:
>> Ah.... internal fns.  Those cannot have attributes indeed (technical
>> limitation).
>> Martin was working on putting those flags elsewhere (cgraph, though internal
>> functions don't have cgraph nodes either ...).  Maybe it was a bad idea to use
>> internal functions for ASAN.
>
> For internal-fn, we already support ECF* constants, guess either we could
> add support for EAF* too, through internal-fn.def,

Just hack in EAFs or support full-featured declarations in internals?
The latter looks more appropriate but would increase size of
internal function calls by one word (namely 
gimple_statement_call::internal_fn).

> or we need support for
> normal builtins that are inaccessible to users (but using . or space
> in names is too ugly IMHO for that).

We could be just another flag in DEF_BUILTIN.
Although my understanding was that we already have
user-inaccessible builtins and these are internal functions.

-Y

Re: [PATCH] Move Asan instrumentation to sanopt pass

[Jakub Jelinek]

On Thu, Jul 24, 2014 at 10:21:14AM +0400, Yury Gribov wrote:
> On 07/24/2014 12:09 AM, Jakub Jelinek wrote:
> >>Ah.... internal fns.  Those cannot have attributes indeed (technical
> >>limitation).
> >>Martin was working on putting those flags elsewhere (cgraph, though internal
> >>functions don't have cgraph nodes either ...).  Maybe it was a bad idea to use
> >>internal functions for ASAN.
> >
> >For internal-fn, we already support ECF* constants, guess either we could
> >add support for EAF* too, through internal-fn.def,
> 
> Just hack in EAFs or support full-featured declarations in internals?
> The latter looks more appropriate but would increase size of
> internal function calls by one word (namely
> gimple_statement_call::internal_fn).

Internal functions have internal-fn-id instead of a declaration, the whole
point of them is that they don't have function types etc. and it solely
about the types used for arguments and return value in the IL.
So, either support for just EAF*, or perhaps support for DECL_ATTRIBUTES
for internal-fns, say by having some tree array where you'd store what you
stick into DECL_ATTRIBUTES normally.

	Jakub

Re: [PATCH] Move Asan instrumentation to sanopt pass

[Yury Gribov]

[-- Attachment #1: Type: text/plain, Size: 364 bytes --]

On 07/24/2014 11:48 AM, Jakub Jelinek wrote:
 > So, either support for just EAF*, or perhaps support for DECL_ATTRIBUTES
 > for internal-fns, say by having some tree array where you'd store 
what you
 > stick into DECL_ATTRIBUTES normally.

I'd prefer to avoid attributes. Would something like this be enough?
(not yet fully tested, just ran asan.exp tests).


-Y

[-- Attachment #2: ifn-arg-flags-1.diff --]
[-- Type: text/x-diff, Size: 8721 bytes --]

diff --git a/gcc/gimple.c b/gcc/gimple.c
index 4a9d379..aa0caab 100644
--- a/gcc/gimple.c
+++ b/gcc/gimple.c
@@ -1348,6 +1348,9 @@ gimple_call_fnspec (const_gimple stmt)
 int
 gimple_call_arg_flags (const_gimple stmt, unsigned arg)
 {
+  if (gimple_call_internal_p (stmt))
+    return internal_fn_arg_flags (gimple_call_internal_fn (stmt), arg);
+
   tree attr = gimple_call_fnspec (stmt);
 
   if (!attr || 1 + arg >= (unsigned) TREE_STRING_LENGTH (attr))
diff --git a/gcc/internal-fn.c b/gcc/internal-fn.c
index 7fb54ea..3f9b448 100644
--- a/gcc/internal-fn.c
+++ b/gcc/internal-fn.c
@@ -40,7 +40,7 @@ along with GCC; see the file COPYING3.  If not see
 
 /* The names of each internal function, indexed by function number.  */
 const char *const internal_fn_name_array[] = {
-#define DEF_INTERNAL_FN(CODE, FLAGS) #CODE,
+#define DEF_INTERNAL_FN(CODE, FLAGS, NARGS, ARG_FLAGS) #CODE,
 #include "internal-fn.def"
 #undef DEF_INTERNAL_FN
   "<invalid-fn>"
@@ -48,12 +48,63 @@ const char *const internal_fn_name_array[] = {
 
 /* The ECF_* flags of each internal function, indexed by function number.  */
 const int internal_fn_flags_array[] = {
-#define DEF_INTERNAL_FN(CODE, FLAGS) FLAGS,
+#define DEF_INTERNAL_FN(CODE, FLAGS, NARGS, ARG_FLAGS) FLAGS,
 #include "internal-fn.def"
 #undef DEF_INTERNAL_FN
   0
 };
 
+/* The EAF_* flags for arguments of internal functions,
+   indexed indirectly via INTERNAL_FN_ARG_FLAGS_ARRAY.  */
+static GTY(()) int *internal_fn_arg_flags_array;
+
+/* Maps internal function to offset in internal_fn_arg_flags_array.  */
+static size_t internal_fn_arg_flags_offsets[IFN_LAST];
+static size_t max_arg_offset;
+
+void
+init_internal_fns ()
+{
+  max_arg_offset = 0;
+#define DEF_INTERNAL_FN(CODE, FLAGS, NARGS, ARG_FLAGS) max_arg_offset += NARGS;
+#include "internal-fn.def"
+#undef DEF_INTERNAL_FN
+  internal_fn_arg_flags_array
+    = (int *) xmalloc (max_arg_offset * sizeof (internal_fn_arg_flags_array[0]));
+
+  size_t offset = 0;
+#define APPEND_ARG_FLAGS0() ;
+#define APPEND_ARG_FLAGS1(F1)                 \
+  internal_fn_arg_flags_array[offset++] = F1;
+#define APPEND_ARG_FLAGS2(F1, F2)             \
+  internal_fn_arg_flags_array[offset++] = F1; \
+  internal_fn_arg_flags_array[offset++] = F2;
+#define APPEND_ARG_FLAGS3(F1, F2, F3)         \
+  internal_fn_arg_flags_array[offset++] = F1; \
+  internal_fn_arg_flags_array[offset++] = F2; \
+  internal_fn_arg_flags_array[offset++] = F3;
+#define APPEND_ARG_FLAGS4(F1, F2, F3, F4)     \
+  internal_fn_arg_flags_array[offset++] = F1; \
+  internal_fn_arg_flags_array[offset++] = F2; \
+  internal_fn_arg_flags_array[offset++] = F3; \
+  internal_fn_arg_flags_array[offset++] = F4;
+#define DEF_INTERNAL_FN(CODE, FLAGS, NARGS, ARG_FLAGS)      \
+  internal_fn_arg_flags_offsets[(int) IFN_##CODE] = offset; \
+  APPEND_ARG_FLAGS##NARGS ARG_FLAGS
+#include "internal-fn.def"
+#undef DEF_INTERNAL_FN
+}
+
+/* Return the EAF_* flags for N-th argument of function FN.  */
+
+int
+internal_fn_arg_flags (enum internal_fn fn, int arg)
+{
+  size_t offset = internal_fn_arg_flags_offsets[(int) fn] + arg;
+  gcc_assert (offset < max_arg_offset);
+  return internal_fn_arg_flags_array[offset];
+}
+
 /* ARRAY_TYPE is an array of vector modes.  Return the associated insn
    for load-lanes-style optab OPTAB.  The insn must exist.  */
 
@@ -905,7 +956,7 @@ expand_BUILTIN_EXPECT (gimple stmt)
 
    where STMT is the statement that performs the call. */
 static void (*const internal_fn_expanders[]) (gimple) = {
-#define DEF_INTERNAL_FN(CODE, FLAGS) expand_##CODE,
+#define DEF_INTERNAL_FN(CODE, FLAGS, NARGS, ARG_FLAGS) expand_##CODE,
 #include "internal-fn.def"
 #undef DEF_INTERNAL_FN
   0
diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def
index f0aa1b4..a2861a6 100644
--- a/gcc/internal-fn.def
+++ b/gcc/internal-fn.def
@@ -28,30 +28,34 @@ along with GCC; see the file COPYING3.  If not see
 
    Each entry in this file has the form:
 
-     DEF_INTERNAL_FN (NAME, FLAGS)
+     DEF_INTERNAL_FN (NAME, FLAGS, NARGS, ARG_FLAGS)
 
-   where NAME is the name of the function and FLAGS is a set of
-   ECF_* flags.  Each entry must have a corresponding expander
-   of the form:
+   where NAME is the name of the function, FLAGS is a set of
+   ECF_* flags and ARG_FLAGS holds comma-separated list of sets
+   of EAF_* flags.
+   
+   Each entry must have a corresponding expander of the form:
 
      void expand_NAME (gimple stmt)
 
    where STMT is the statement that performs the call.  */
 
-DEF_INTERNAL_FN (LOAD_LANES, ECF_CONST | ECF_LEAF)
-DEF_INTERNAL_FN (STORE_LANES, ECF_CONST | ECF_LEAF)
-DEF_INTERNAL_FN (GOMP_SIMD_LANE, ECF_NOVOPS | ECF_LEAF | ECF_NOTHROW)
-DEF_INTERNAL_FN (GOMP_SIMD_VF, ECF_CONST | ECF_LEAF | ECF_NOTHROW)
-DEF_INTERNAL_FN (GOMP_SIMD_LAST_LANE, ECF_CONST | ECF_LEAF | ECF_NOTHROW)
-DEF_INTERNAL_FN (LOOP_VECTORIZED, ECF_NOVOPS | ECF_LEAF | ECF_NOTHROW)
-DEF_INTERNAL_FN (MASK_LOAD, ECF_PURE | ECF_LEAF)
-DEF_INTERNAL_FN (MASK_STORE, ECF_LEAF)
-DEF_INTERNAL_FN (ANNOTATE,  ECF_CONST | ECF_LEAF | ECF_NOTHROW)
-DEF_INTERNAL_FN (UBSAN_NULL, ECF_LEAF | ECF_NOTHROW)
-DEF_INTERNAL_FN (UBSAN_BOUNDS, ECF_LEAF | ECF_NOTHROW)
-DEF_INTERNAL_FN (UBSAN_CHECK_ADD, ECF_CONST | ECF_LEAF | ECF_NOTHROW)
-DEF_INTERNAL_FN (UBSAN_CHECK_SUB, ECF_CONST | ECF_LEAF | ECF_NOTHROW)
-DEF_INTERNAL_FN (UBSAN_CHECK_MUL, ECF_CONST | ECF_LEAF | ECF_NOTHROW)
-DEF_INTERNAL_FN (ABNORMAL_DISPATCHER, ECF_NORETURN)
-DEF_INTERNAL_FN (BUILTIN_EXPECT, ECF_CONST | ECF_LEAF | ECF_NOTHROW)
-DEF_INTERNAL_FN (ASAN_CHECK, ECF_TM_PURE | ECF_LEAF | ECF_NOTHROW)
+DEF_INTERNAL_FN (LOAD_LANES, ECF_CONST | ECF_LEAF, 1, (0))
+DEF_INTERNAL_FN (STORE_LANES, ECF_CONST | ECF_LEAF, 1, (0))
+DEF_INTERNAL_FN (GOMP_SIMD_LANE, ECF_NOVOPS | ECF_LEAF | ECF_NOTHROW, 1, (0))
+DEF_INTERNAL_FN (GOMP_SIMD_VF, ECF_CONST | ECF_LEAF | ECF_NOTHROW, 1, (0))
+DEF_INTERNAL_FN (GOMP_SIMD_LAST_LANE, ECF_CONST | ECF_LEAF | ECF_NOTHROW, \
+		 2, (0, 0))
+DEF_INTERNAL_FN (LOOP_VECTORIZED, ECF_NOVOPS | ECF_LEAF | ECF_NOTHROW, 2, (0, 0))
+DEF_INTERNAL_FN (MASK_LOAD, ECF_PURE | ECF_LEAF, 3, (0, 0, 0))
+DEF_INTERNAL_FN (MASK_STORE, ECF_LEAF, 4, (0, 0, 0, 0))
+DEF_INTERNAL_FN (ANNOTATE,  ECF_CONST | ECF_LEAF | ECF_NOTHROW, 2, (0, 0))
+DEF_INTERNAL_FN (UBSAN_NULL, ECF_LEAF | ECF_NOTHROW, 2, (0, 0))
+DEF_INTERNAL_FN (UBSAN_BOUNDS, ECF_LEAF | ECF_NOTHROW, 3, (0, 0, 0))
+DEF_INTERNAL_FN (UBSAN_CHECK_ADD, ECF_CONST | ECF_LEAF | ECF_NOTHROW, 2, (0, 0))
+DEF_INTERNAL_FN (UBSAN_CHECK_SUB, ECF_CONST | ECF_LEAF | ECF_NOTHROW, 2, (0, 0))
+DEF_INTERNAL_FN (UBSAN_CHECK_MUL, ECF_CONST | ECF_LEAF | ECF_NOTHROW, 2, (0, 0))
+DEF_INTERNAL_FN (ABNORMAL_DISPATCHER, ECF_NORETURN, 1, (0))
+DEF_INTERNAL_FN (BUILTIN_EXPECT, ECF_CONST | ECF_LEAF | ECF_NOTHROW, 1, (0))
+DEF_INTERNAL_FN (ASAN_CHECK, ECF_TM_PURE | ECF_LEAF | ECF_NOTHROW, \
+		 3, (EAF_DIRECT | EAF_NOCLOBBER | EAF_NOESCAPE, 0, 0))
diff --git a/gcc/internal-fn.h b/gcc/internal-fn.h
index 2dcf44e..e476aac 100644
--- a/gcc/internal-fn.h
+++ b/gcc/internal-fn.h
@@ -20,6 +20,8 @@ along with GCC; see the file COPYING3.  If not see
 #ifndef GCC_INTERNAL_FN_H
 #define GCC_INTERNAL_FN_H
 
+extern void init_internal_fns ();
+
 /* Return the name of internal function FN.  The name is only meaningful
    for dumps; it has no linkage.  */
 
@@ -41,6 +43,8 @@ internal_fn_flags (enum internal_fn fn)
   return internal_fn_flags_array[(int) fn];
 }
 
+int internal_fn_arg_flags (enum internal_fn, int);
+
 extern void expand_internal_call (gimple);
 
 #endif
diff --git a/gcc/tree-core.h b/gcc/tree-core.h
index 34c48fa..cc46b81 100644
--- a/gcc/tree-core.h
+++ b/gcc/tree-core.h
@@ -667,7 +667,7 @@ enum annot_expr_kind {
 
 /* Internal functions.  */
 enum internal_fn {
-#define DEF_INTERNAL_FN(CODE, FLAGS) IFN_##CODE,
+#define DEF_INTERNAL_FN(CODE, FLAGS, NARGS, ARG_FLAGS) IFN_##CODE,
 #include "internal-fn.def"
 #undef DEF_INTERNAL_FN
   IFN_LAST
diff --git a/gcc/tree.c b/gcc/tree.c
index 10063a4..5520b8b 100644
--- a/gcc/tree.c
+++ b/gcc/tree.c
@@ -9920,8 +9920,9 @@ local_define_builtin (const char *name, tree type, enum built_in_function code,
 }
 
 /* Call this function after instantiating all builtins that the language
-   front end cares about.  This will build the rest of the builtins that
-   are relied upon by the tree optimizers and the middle-end.  */
+   front end cares about.  This will build the rest of the builtins
+   and internal function that are relied upon by the tree optimizers and
+   the middle-end.  */
 
 void
 build_common_builtin_nodes (void)
@@ -10154,6 +10155,8 @@ build_common_builtin_nodes (void)
 			      ECF_CONST | ECF_NOTHROW | ECF_LEAF);
       }
   }
+
+  init_internal_fns ();
 }
 
 /* HACK.  GROSS.  This is absolutely disgusting.  I wish there was a

Re: [PATCH] Move Asan instrumentation to sanopt pass

[Jakub Jelinek]

On Mon, Jul 28, 2014 at 06:14:08PM +0400, Yury Gribov wrote:
> On 07/24/2014 11:48 AM, Jakub Jelinek wrote:
> > So, either support for just EAF*, or perhaps support for DECL_ATTRIBUTES
> > for internal-fns, say by having some tree array where you'd store what you
> > stick into DECL_ATTRIBUTES normally.
> 
> I'd prefer to avoid attributes. Would something like this be enough?
> (not yet fully tested, just ran asan.exp tests).

I think I'd prefer if you'd specify the fnspec as either NULL (most internal
functions) or normal string and reuse the string parsing in
gimple_call_arg_flags.
So, you'd just replace the:
  tree attr = gimple_call_fnspec (stmt);

  if (!attr || 1 + arg >= (unsigned) TREE_STRING_LENGTH (attr))
    return 0;

  switch (TREE_STRING_POINTER (attr)[1 + arg])
with:
  const char *fnspec;
  if (gimple_call_arg_flags (stmt))
    {
      fnspec = ...;
      if (!fnspec || 1 + arg >= strlen (fnspec))
	return 0;
    }
  else
    {
      tree attr = gimple_call_fnspec (stmt);

      if (!attr || 1 + arg >= (unsigned) TREE_STRING_LENGTH (attr))
	return 0;

      fnspec = TREE_STRING_POINTER (attr);
    }
  switch (fnspec[1 + arg])

	Jakub

Re: [PATCH] Move Asan instrumentation to sanopt pass

[Richard Biener]

On Fri, Aug 1, 2014 at 9:17 AM, Jakub Jelinek <jakub@redhat.com> wrote:
> On Mon, Jul 28, 2014 at 06:14:08PM +0400, Yury Gribov wrote:
>> On 07/24/2014 11:48 AM, Jakub Jelinek wrote:
>> > So, either support for just EAF*, or perhaps support for DECL_ATTRIBUTES
>> > for internal-fns, say by having some tree array where you'd store what you
>> > stick into DECL_ATTRIBUTES normally.
>>
>> I'd prefer to avoid attributes. Would something like this be enough?
>> (not yet fully tested, just ran asan.exp tests).
>
> I think I'd prefer if you'd specify the fnspec as either NULL (most internal
> functions) or normal string and reuse the string parsing in
> gimple_call_arg_flags.

Yeah, me too.

Richard.

> So, you'd just replace the:
>   tree attr = gimple_call_fnspec (stmt);
>
>   if (!attr || 1 + arg >= (unsigned) TREE_STRING_LENGTH (attr))
>     return 0;
>
>   switch (TREE_STRING_POINTER (attr)[1 + arg])
> with:
>   const char *fnspec;
>   if (gimple_call_arg_flags (stmt))
>     {
>       fnspec = ...;
>       if (!fnspec || 1 + arg >= strlen (fnspec))
>         return 0;
>     }
>   else
>     {
>       tree attr = gimple_call_fnspec (stmt);
>
>       if (!attr || 1 + arg >= (unsigned) TREE_STRING_LENGTH (attr))
>         return 0;
>
>       fnspec = TREE_STRING_POINTER (attr);
>     }
>   switch (fnspec[1 + arg])
>
>         Jakub

[PATCH 2/2] Move Asan instrumentation to sanopt pass

[Yury Gribov]

[-- Attachment #1: Type: text/plain, Size: 417 bytes --]

Hi all,

Attached patch delays generation of Asan memory checking code
until sanopt pass. This is a first step towards global static analysis
of Asan instrumentation (hope to send it for review next week).

The patch also changes the logic behind 
asan-instrumentation-with-call-threshold parameter to more closely match 
LLVM.

The patch was bootstrapped, regtested and asan-bootstrapped on x64.

Ok for trunk?

-Y


[-- Attachment #2: asan_ifn-4.diff --]
[-- Type: text/x-diff, Size: 41655 bytes --]

commit 22a79d39656d3d600e97c7ae599b86b6d25efe00
Author: Yury Gribov <y.gribov@samsung.com>
Date:   Wed Aug 6 11:06:24 2014 +0400

    Move inlining of Asan memory checks to sanopt pass.
    Change asan-instrumentation-with-call-threshold to more closely match LLVM.
    
    gcc/
    
    2014-08-08  Yury Gribov  <y.gribov@samsung.com>
    
    	* asan.c (asan_check_flags): New enum.
    	(build_check_stmt_with_calls): Removed function.
    	(build_check_stmt): Split inlining logic to
    	asan_expand_check_ifn.
    	(instrument_derefs): Rename parameter.
    	(instrument_mem_region_access): Rename parameter.
    	(instrument_strlen_call): Likewise.
    	(asan_expand_check_ifn): New function.
    	(asan_instrument): Remove old code.
    	(pass_sanopt::execute): Change handling of
    	asan-instrumentation-with-call-threshold.
    	(asan_clear_shadow): Fix formatting.
    	(asan_function_start): Likewise.
    	(asan_emit_stack_protection): Likewise.
    	* doc/invoke.texi (asan-instrumentation-with-call-threshold):
    	Update description.
    	* internal-fn.c (expand_ASAN_CHECK): New function.
    	* internal-fn.def (ASAN_CHECK): New internal function.
    	* params.def (PARAM_ASAN_INSTRUMENTATION_WITH_CALL_THRESHOLD):
    	Update description.
    	(PARAM_ASAN_USE_AFTER_RETURN): Likewise.
    
    gcc/testsuite/
    
    2014-07-21  Yury Gribov  <y.gribov@samsung.com>
    
    	* c-c++-common/asan/inc.c: Update test.
    	* c-c++-common/asan/instrument-with-calls-2.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-1.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-2.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-3.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-4.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-5.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-6.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-7.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-8.c: Likewise.
    	* c-c++-common/asan/no-redundant-instrumentation-9.c: Likewise.

diff --git a/gcc/asan.c b/gcc/asan.c
index 76f21bd..4e6f438 100644
--- a/gcc/asan.c
+++ b/gcc/asan.c
@@ -244,18 +244,16 @@ static GTY(()) tree shadow_ptr_types[2];
 /* Decl for __asan_option_detect_stack_use_after_return.  */
 static GTY(()) tree asan_detect_stack_use_after_return;
 
-/* Number of instrumentations in current function so far.  */
-
-static int asan_num_accesses;
-
-/* Check whether we should replace inline instrumentation with calls.  */
-
-static inline bool
-use_calls_p ()
+/* Various flags for Asan builtins.  */
+enum asan_check_flags
 {
-  return ASAN_INSTRUMENTATION_WITH_CALL_THRESHOLD < INT_MAX
-    && asan_num_accesses >= ASAN_INSTRUMENTATION_WITH_CALL_THRESHOLD;
-}
+  ASAN_CHECK_STORE = 1 << 0,
+  ASAN_CHECK_SCALAR_ACCESS = 1 << 1,
+  ASAN_CHECK_NON_ZERO_LEN = 1 << 2,
+  ASAN_CHECK_START_INSTRUMENTED = 1 << 3,
+  ASAN_CHECK_END_INSTRUMENTED = 1 << 4,
+  ASAN_CHECK_LAST
+};
 
 /* Hashtable support for memory references used by gimple
    statements.  */
@@ -945,7 +943,7 @@ asan_clear_shadow (rtx shadow_mem, HOST_WIDE_INT len)
 
   emit_move_insn (shadow_mem, const0_rtx);
   tmp = expand_simple_binop (Pmode, PLUS, addr, gen_int_mode (4, Pmode), addr,
-                             true, OPTAB_LIB_WIDEN);
+			     true, OPTAB_LIB_WIDEN);
   if (tmp != addr)
     emit_move_insn (addr, tmp);
   emit_cmp_and_jump_insns (addr, end, LT, NULL_RTX, Pmode, true, top_label);
@@ -960,7 +958,7 @@ asan_function_start (void)
   section *fnsec = function_section (current_function_decl);
   switch_to_section (fnsec);
   ASM_OUTPUT_DEBUG_LABEL (asm_out_file, "LASANPC",
-                         current_function_funcdef_no);
+			 current_function_funcdef_no);
 }
 
 /* Insert code to protect stack vars.  The prologue sequence should be emitted
@@ -1025,7 +1023,7 @@ asan_emit_stack_protection (rtx base, rtx pbase, unsigned int alignb,
     {
       use_after_return_class = floor_log2 (asan_frame_size - 1) - 5;
       /* __asan_stack_malloc_N guarantees alignment
-         N < 6 ? (64 << N) : 4096 bytes.  */
+	 N < 6 ? (64 << N) : 4096 bytes.  */
       if (alignb > (use_after_return_class < 6
 		    ? (64U << use_after_return_class) : 4096U))
 	use_after_return_class = -1;
@@ -1098,7 +1096,7 @@ asan_emit_stack_protection (rtx base, rtx pbase, unsigned int alignb,
   ASM_GENERATE_INTERNAL_LABEL (buf, "LASANPC", current_function_funcdef_no);
   id = get_identifier (buf);
   decl = build_decl (DECL_SOURCE_LOCATION (current_function_decl),
-                    VAR_DECL, id, char_type_node);
+		    VAR_DECL, id, char_type_node);
   SET_DECL_ASSEMBLER_NAME (decl, id);
   TREE_ADDRESSABLE (decl) = 1;
   TREE_READONLY (decl) = 1;
@@ -1555,55 +1553,6 @@ maybe_create_ssa_name (location_t loc, tree base, gimple_stmt_iterator *iter,
   return gimple_assign_lhs (g);
 }
 
-/* Instrument the memory access instruction using callbacks.
-   Parameters are similar to BUILD_CHECK_STMT.  */
-
-static void
-build_check_stmt_with_calls (location_t loc, tree base, tree len,
-			     HOST_WIDE_INT size_in_bytes, gimple_stmt_iterator *iter,
-			     bool before_p, bool is_store, bool is_scalar_access)
-{
-  gimple_stmt_iterator gsi = *iter;
-  tree base_ssa = maybe_create_ssa_name (loc, base, &gsi, before_p);
-
-  gimple g
-    = gimple_build_assign_with_ops (NOP_EXPR,
-				    make_ssa_name (pointer_sized_int_node, NULL),
-				    base_ssa, NULL_TREE);
-  gimple_set_location (g, loc);
-  if (before_p)
-    gsi_insert_before (&gsi, g, GSI_NEW_STMT);
-  else
-    gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-  tree base_addr = gimple_assign_lhs (g);
-
-  int nargs;
-  tree fun
-    = check_func (is_store, is_scalar_access ? size_in_bytes : -1, &nargs);
-  if (nargs == 1)
-    g = gimple_build_call (fun, 1, base_addr);
-  else
-    {
-      gcc_assert (nargs == 2);
-      g = gimple_build_assign_with_ops (NOP_EXPR,
-					make_ssa_name (pointer_sized_int_node,
-						       NULL),
-					len, NULL_TREE);
-      gimple_set_location (g, loc);
-      gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-      tree sz_arg = gimple_assign_lhs (g);
-      g = gimple_build_call (fun, nargs, base_addr, sz_arg);
-    }
-  gimple_set_location (g, loc);
-  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-
-  if (!before_p)
-    {
-      gsi_next (&gsi);
-      *iter = gsi;
-    }
-}
-
 /* Instrument the memory access instruction BASE.  Insert new
    statements before or after ITER.
 
@@ -1624,27 +1573,30 @@ build_check_stmt_with_calls (location_t loc, tree base, tree len,
    otherwise, it points to the statement logically following it.  */
 
 static void
-build_check_stmt (location_t location, tree base, tree len,
+build_check_stmt (location_t loc, tree base, tree len,
 		  HOST_WIDE_INT size_in_bytes, gimple_stmt_iterator *iter,
-		  bool non_zero_len_p, bool before_p, bool is_store,
+		  bool is_non_zero_len, bool before_p, bool is_store,
 		  bool is_scalar_access, unsigned int align = 0,
 		  bool start_instrumented = false,
 		  bool end_instrumented = false)
 {
   gimple_stmt_iterator gsi = *iter;
   gimple g;
-  tree uintptr_type
-    = build_nonstandard_integer_type (TYPE_PRECISION (TREE_TYPE (base)), 1);
 
-  gcc_assert (!(size_in_bytes > 0 && !non_zero_len_p));
+  gcc_assert (!(size_in_bytes > 0 && !is_non_zero_len));
 
   if (start_instrumented && end_instrumented)
     {
       if (!before_p)
-        gsi_next (iter);
+	gsi_next (iter);
       return;
     }
 
+  gsi = *iter;
+
+  base = unshare_expr (base);
+  base = maybe_create_ssa_name (loc, base, &gsi, before_p);
+
   if (len)
     len = unshare_expr (len);
   else
@@ -1656,9 +1608,8 @@ build_check_stmt (location_t location, tree base, tree len,
   if (size_in_bytes > 1)
     {
       if ((size_in_bytes & (size_in_bytes - 1)) != 0
-	  || !is_scalar_access
 	  || size_in_bytes > 16)
-	size_in_bytes = -1;
+	is_scalar_access = false;
       else if (align && align < size_in_bytes * BITS_PER_UNIT)
 	{
 	  /* On non-strict alignment targets, if
@@ -1669,189 +1620,34 @@ build_check_stmt (location_t location, tree base, tree len,
 	  if (size_in_bytes != 16
 	      || STRICT_ALIGNMENT
 	      || align < 8 * BITS_PER_UNIT)
-	    size_in_bytes = -1;
-	}
-    }
-
-  HOST_WIDE_INT real_size_in_bytes = size_in_bytes == -1 ? 1 : size_in_bytes;
-
-  tree shadow_ptr_type = shadow_ptr_types[real_size_in_bytes == 16 ? 1 : 0];
-  tree shadow_type = TREE_TYPE (shadow_ptr_type);
-
-  base = unshare_expr (base);
-
-  if (use_calls_p ())
-    {
-      gsi = *iter;
-      build_check_stmt_with_calls (location, base, len, size_in_bytes, iter,
-				   before_p, is_store, is_scalar_access);
-      return;
-    }
-
-  ++asan_num_accesses;
-
-  if (!non_zero_len_p)
-    {
-      gcc_assert (before_p);
-
-      /* So, the length of the memory area to asan-protect is
-	 non-constant.  Let's guard the generated instrumentation code
-	 like:
-
-	 if (len != 0)
-	   {
-	     //asan instrumentation code goes here.
-	   }
-	 // falltrough instructions, starting with *ITER.  */
-
-      g = gimple_build_cond (NE_EXPR,
-			     len,
-			     build_int_cst (TREE_TYPE (len), 0),
-			     NULL_TREE, NULL_TREE);
-      gimple_set_location (g, location);
-
-      basic_block then_bb, fallthrough_bb;
-      insert_if_then_before_iter (g, iter, /*then_more_likely_p=*/true,
-				  &then_bb, &fallthrough_bb);
-      /* Note that fallthrough_bb starts with the statement that was
-	 pointed to by ITER.  */
-
-      /* The 'then block' of the 'if (len != 0) condition is where
-	 we'll generate the asan instrumentation code now.  */
-      gsi = gsi_last_bb (then_bb);
-      build_check_stmt (location, base, len, size_in_bytes, &gsi,
-			/*non_zero_len_p*/true, /*before_p*/true, is_store,
-			is_scalar_access, align,
-			start_instrumented, end_instrumented);
-      return;
-    }
-
-  /* Get an iterator on the point where we can add the condition
-     statement for the instrumentation.  */
-  basic_block then_bb, else_bb;
-  gsi = create_cond_insert_point (&gsi, before_p,
-				  /*then_more_likely_p=*/false,
-				  /*create_then_fallthru_edge=*/false,
-				  &then_bb,
-				  &else_bb);
-
-  tree base_ssa = maybe_create_ssa_name (location, base, &gsi,
-					 /*before_p*/false);
-
-  g = gimple_build_assign_with_ops (NOP_EXPR,
-				    make_ssa_name (uintptr_type, NULL),
-				    base_ssa, NULL_TREE);
-  gimple_set_location (g, location);
-  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-  tree base_addr = gimple_assign_lhs (g);
-
-  tree t = NULL_TREE;
-  if (real_size_in_bytes >= 8)
-    {
-      tree shadow = build_shadow_mem_access (&gsi, location, base_addr,
-					     shadow_ptr_type);
-      t = shadow;
-    }
-  else
-    {
-      /* Slow path for 1, 2 and 4 byte accesses.  */
-
-      if (!start_instrumented)
-	{
-	  /* Test (shadow != 0)
-		  & ((base_addr & 7) + (real_size_in_bytes - 1)) >= shadow).  */
-	  tree shadow = build_shadow_mem_access (&gsi, location, base_addr,
-						 shadow_ptr_type);
-	  gimple shadow_test = build_assign (NE_EXPR, shadow, 0);
-	  gimple_seq seq = NULL;
-	  gimple_seq_add_stmt (&seq, shadow_test);
-	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR, base_addr, 7));
-	  gimple_seq_add_stmt (&seq, build_type_cast (shadow_type,
-						      gimple_seq_last (seq)));
-	  if (real_size_in_bytes > 1)
-	    gimple_seq_add_stmt (&seq,
-				 build_assign (PLUS_EXPR, gimple_seq_last (seq),
-					       real_size_in_bytes - 1));
-	  gimple_seq_add_stmt (&seq, build_assign (GE_EXPR,
-						   gimple_seq_last (seq),
-						   shadow));
-	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR, shadow_test,
-						   gimple_seq_last (seq)));
-	  t = gimple_assign_lhs (gimple_seq_last (seq));
-	  gimple_seq_set_location (seq, location);
-	  gsi_insert_seq_after (&gsi, seq, GSI_CONTINUE_LINKING);
-	}
-
-      /* For non-constant, misaligned or otherwise weird access sizes,
-	 check first and last byte.  */
-      if (size_in_bytes == -1 && !end_instrumented)
-	{
-	  g = gimple_build_assign_with_ops (MINUS_EXPR,
-					    make_ssa_name (uintptr_type, NULL),
-					    len,
-					    build_int_cst (uintptr_type, 1));
-	  gimple_set_location (g, location);
-	  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-	  tree last = gimple_assign_lhs (g);
-	  g = gimple_build_assign_with_ops (PLUS_EXPR,
-					    make_ssa_name (uintptr_type, NULL),
-					    base_addr,
-					    last);
-	  gimple_set_location (g, location);
-	  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-	  tree base_end_addr = gimple_assign_lhs (g);
-
-	  tree shadow = build_shadow_mem_access (&gsi, location, base_end_addr,
-						 shadow_ptr_type);
-	  gimple shadow_test = build_assign (NE_EXPR, shadow, 0);
-	  gimple_seq seq = NULL;
-	  gimple_seq_add_stmt (&seq, shadow_test);
-	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR,
-						   base_end_addr, 7));
-	  gimple_seq_add_stmt (&seq, build_type_cast (shadow_type,
-						      gimple_seq_last (seq)));
-	  gimple_seq_add_stmt (&seq, build_assign (GE_EXPR,
-						   gimple_seq_last (seq),
-						   shadow));
-	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR, shadow_test,
-						   gimple_seq_last (seq)));
-	  if (!start_instrumented)
-	    gimple_seq_add_stmt (&seq, build_assign (BIT_IOR_EXPR, t,
-						     gimple_seq_last (seq)));
-	  t = gimple_assign_lhs (gimple_seq_last (seq));
-	  gimple_seq_set_location (seq, location);
-	  gsi_insert_seq_after (&gsi, seq, GSI_CONTINUE_LINKING);
+	    is_scalar_access = false;
 	}
     }
 
-  g = gimple_build_cond (NE_EXPR, t, build_int_cst (TREE_TYPE (t), 0),
-			 NULL_TREE, NULL_TREE);
-  gimple_set_location (g, location);
-  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-
-  /* Generate call to the run-time library (e.g. __asan_report_load8).  */
-  gsi = gsi_start_bb (then_bb);
-  int nargs;
-  tree fun = report_error_func (is_store, is_scalar_access ? size_in_bytes : -1,
-				&nargs);
-  if (nargs == 1)
-      g = gimple_build_call (fun, 1, base_addr);
+  HOST_WIDE_INT flags = 0;
+  if (is_store)
+    flags |= ASAN_CHECK_STORE;
+  if (is_non_zero_len)
+    flags |= ASAN_CHECK_NON_ZERO_LEN;
+  if (is_scalar_access)
+    flags |= ASAN_CHECK_SCALAR_ACCESS;
+  if (start_instrumented)
+    flags |= ASAN_CHECK_START_INSTRUMENTED;
+  if (end_instrumented)
+    flags |= ASAN_CHECK_END_INSTRUMENTED;
+
+  g = gimple_build_call_internal (IFN_ASAN_CHECK, 3,
+				  build_int_cst (integer_type_node, flags),
+				  base, len);
+  gimple_set_location (g, loc);
+  if (before_p)
+    gsi_insert_before (&gsi, g, GSI_SAME_STMT);
   else
     {
-      gcc_assert (nargs == 2);
-      g = gimple_build_assign_with_ops (NOP_EXPR,
-					make_ssa_name (pointer_sized_int_node,
-						       NULL),
-					len, NULL_TREE);
-      gimple_set_location (g, location);
       gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-      tree sz_arg = gimple_assign_lhs (g);
-      g = gimple_build_call (fun, nargs, base_addr, sz_arg);
+      gsi_next (&gsi);
+      *iter = gsi;
     }
-  gimple_set_location (g, location);
-  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
-
-  *iter = gsi_start_bb (else_bb);
 }
 
 /* If T represents a memory access, add instrumentation code before ITER.
@@ -1944,7 +1740,7 @@ instrument_derefs (gimple_stmt_iterator *iter, tree t,
     {
       unsigned int align = get_object_alignment (t);
       build_check_stmt (location, base, NULL_TREE, size_in_bytes, iter,
-			/*non_zero_len_p*/size_in_bytes > 0, /*before_p=*/true,
+			/*is_non_zero_len*/size_in_bytes > 0, /*before_p=*/true,
 			is_store, /*is_scalar_access*/true, align);
       update_mem_ref_hash_table (base, size_in_bytes);
       update_mem_ref_hash_table (t, size_in_bytes);
@@ -1982,7 +1778,7 @@ instrument_mem_region_access (tree base, tree len,
   HOST_WIDE_INT size_in_bytes = tree_fits_shwi_p (len) ? tree_to_shwi (len) : -1;
 
   build_check_stmt (location, base, len, size_in_bytes, iter,
-		    /*non_zero_len_p*/size_in_bytes > 0, /*before_p*/true,
+		    /*is_non_zero_len*/size_in_bytes > 0, /*before_p*/true,
 		    is_store, /*is_scalar_access*/false, /*align*/0,
 		    start_instrumented, end_instrumented);
 
@@ -2039,7 +1835,7 @@ instrument_strlen_call (gimple_stmt_iterator *iter)
   gsi_insert_before (iter, str_arg_ssa, GSI_SAME_STMT);
 
   build_check_stmt (loc, gimple_assign_lhs (str_arg_ssa), NULL_TREE, 1, iter,
-		    /*non_zero_len_p*/true, /*before_p=*/true,
+		    /*is_non_zero_len*/true, /*before_p=*/true,
 		    /*is_store=*/false, /*is_scalar_access*/true, /*align*/0,
 		    start_instrumented, start_instrumented);
 
@@ -2052,7 +1848,7 @@ instrument_strlen_call (gimple_stmt_iterator *iter)
   gsi_insert_after (iter, g, GSI_NEW_STMT);
 
   build_check_stmt (loc, gimple_assign_lhs (g), NULL_TREE, 1, iter,
-		    /*non_zero_len_p*/true, /*before_p=*/false,
+		    /*is_non_zero_len*/true, /*before_p=*/false,
 		    /*is_store=*/false, /*is_scalar_access*/true, /*align*/0);
 
   return true;
@@ -2619,6 +2415,215 @@ asan_finish_file (void)
   flag_sanitize |= SANITIZE_ADDRESS;
 }
 
+/* Expand the ASAN_{LOAD,STORE} builtins.  */
+
+static bool
+asan_expand_check_ifn (gimple_stmt_iterator *iter, bool use_calls)
+{
+  gimple g = gsi_stmt (*iter);
+  location_t loc = gimple_location (g);
+
+  HOST_WIDE_INT flags = tree_to_shwi (gimple_call_arg (g, 0));
+  gcc_assert (flags < ASAN_CHECK_LAST);
+  bool is_scalar_access = (flags & ASAN_CHECK_SCALAR_ACCESS) != 0;
+  bool is_store = (flags & ASAN_CHECK_STORE) != 0;
+  bool is_non_zero_len = (flags & ASAN_CHECK_NON_ZERO_LEN) != 0;
+  bool start_instrumented = (flags & ASAN_CHECK_START_INSTRUMENTED) != 0;
+  bool end_instrumented = (flags & ASAN_CHECK_END_INSTRUMENTED) != 0;
+
+  tree base = gimple_call_arg (g, 1);
+  tree len = gimple_call_arg (g, 2);
+
+  HOST_WIDE_INT size_in_bytes
+    = is_scalar_access && tree_fits_shwi_p (len) ? tree_to_shwi (len) : -1;
+
+  if (use_calls)
+    {
+      /* Instrument using callbacks.  */
+      gimple g
+	= gimple_build_assign_with_ops (NOP_EXPR,
+					make_ssa_name (pointer_sized_int_node,
+					NULL),
+					base, NULL_TREE);
+      gimple_set_location (g, loc);
+      gsi_insert_before (iter, g, GSI_SAME_STMT);
+      tree base_addr = gimple_assign_lhs (g);
+
+      int nargs;
+      tree fun = check_func (is_store, size_in_bytes, &nargs);
+      if (nargs == 1)
+	g = gimple_build_call (fun, 1, base_addr);
+      else
+	{
+	  gcc_assert (nargs == 2);
+	  g = gimple_build_assign_with_ops (NOP_EXPR,
+					    make_ssa_name (pointer_sized_int_node,
+							   NULL),
+					    len, NULL_TREE);
+	  gimple_set_location (g, loc);
+	  gsi_insert_before (iter, g, GSI_SAME_STMT);
+	  tree sz_arg = gimple_assign_lhs (g);
+	  g = gimple_build_call (fun, nargs, base_addr, sz_arg);
+	}
+      gimple_set_location (g, loc);
+      gsi_replace (iter, g, false);
+      return false;
+    }
+
+  HOST_WIDE_INT real_size_in_bytes = size_in_bytes == -1 ? 1 : size_in_bytes;
+
+  tree uintptr_type
+    = build_nonstandard_integer_type (TYPE_PRECISION (TREE_TYPE (base)), 1);
+
+  tree shadow_ptr_type = shadow_ptr_types[real_size_in_bytes == 16 ? 1 : 0];
+  tree shadow_type = TREE_TYPE (shadow_ptr_type);
+
+  gimple_stmt_iterator gsi = *iter;
+
+  if (!is_non_zero_len)
+    {
+      /* So, the length of the memory area to asan-protect is
+	 non-constant.  Let's guard the generated instrumentation code
+	 like:
+
+	 if (len != 0)
+	   {
+	     //asan instrumentation code goes here.
+	   }
+	 // falltrough instructions, starting with *ITER.  */
+
+      g = gimple_build_cond (NE_EXPR,
+			    len,
+			    build_int_cst (TREE_TYPE (len), 0),
+			    NULL_TREE, NULL_TREE);
+      gimple_set_location (g, loc);
+
+      basic_block then_bb, fallthrough_bb;
+      insert_if_then_before_iter (g, iter, /*then_more_likely_p=*/true,
+				 &then_bb, &fallthrough_bb);
+      /* Note that fallthrough_bb starts with the statement that was
+	pointed to by ITER.  */
+
+      /* The 'then block' of the 'if (len != 0) condition is where
+	we'll generate the asan instrumentation code now.  */
+      gsi = gsi_last_bb (then_bb);
+    }
+
+  /* Get an iterator on the point where we can add the condition
+     statement for the instrumentation.  */
+  basic_block then_bb, else_bb;
+  gsi = create_cond_insert_point (&gsi, /*before_p*/false,
+				  /*then_more_likely_p=*/false,
+				  /*create_then_fallthru_edge=*/false,
+				  &then_bb,
+				  &else_bb);
+
+  g = gimple_build_assign_with_ops (NOP_EXPR,
+				    make_ssa_name (pointer_sized_int_node,
+						   NULL),
+				    base, NULL_TREE);
+  gimple_set_location (g, loc);
+  gsi_insert_before (&gsi, g, GSI_NEW_STMT);
+  tree base_addr = gimple_assign_lhs (g);
+
+  tree t = NULL_TREE;
+  if (real_size_in_bytes >= 8)
+    {
+      tree shadow = build_shadow_mem_access (&gsi, loc, base_addr,
+					     shadow_ptr_type);
+      t = shadow;
+    }
+  else
+    {
+      /* Slow path for 1, 2 and 4 byte accesses.  */
+
+      if (!start_instrumented)
+	{
+	  /* Test (shadow != 0)
+	     & ((base_addr & 7) + (real_size_in_bytes - 1)) >= shadow).  */
+	  tree shadow = build_shadow_mem_access (&gsi, loc, base_addr,
+						 shadow_ptr_type);
+	  gimple shadow_test = build_assign (NE_EXPR, shadow, 0);
+	  gimple_seq seq = NULL;
+	  gimple_seq_add_stmt (&seq, shadow_test);
+	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR, base_addr, 7));
+	  gimple_seq_add_stmt (&seq, build_type_cast (shadow_type,
+						      gimple_seq_last (seq)));
+	  if (real_size_in_bytes > 1)
+	    gimple_seq_add_stmt (&seq,
+				 build_assign (PLUS_EXPR, gimple_seq_last (seq),
+					       real_size_in_bytes - 1));
+	  gimple_seq_add_stmt (&seq, build_assign (GE_EXPR,
+						   gimple_seq_last (seq),
+						   shadow));
+	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR, shadow_test,
+						   gimple_seq_last (seq)));
+	  t = gimple_assign_lhs (gimple_seq_last (seq));
+	  gimple_seq_set_location (seq, loc);
+	  gsi_insert_seq_after (&gsi, seq, GSI_CONTINUE_LINKING);
+	}
+
+      /* For non-constant, misaligned or otherwise weird access sizes,
+	 check first and last byte.  */
+      if (size_in_bytes == -1 && !end_instrumented)
+	{
+	  g = gimple_build_assign_with_ops (MINUS_EXPR,
+					    make_ssa_name (uintptr_type, NULL),
+					    len,
+					    build_int_cst (uintptr_type, 1));
+	  gimple_set_location (g, loc);
+	  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
+	  tree last = gimple_assign_lhs (g);
+	  g = gimple_build_assign_with_ops (PLUS_EXPR,
+					    make_ssa_name (uintptr_type, NULL),
+					    base_addr,
+					    last);
+	  gimple_set_location (g, loc);
+	  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
+	  tree base_end_addr = gimple_assign_lhs (g);
+
+	  tree shadow = build_shadow_mem_access (&gsi, loc, base_end_addr,
+						 shadow_ptr_type);
+	  gimple shadow_test = build_assign (NE_EXPR, shadow, 0);
+	  gimple_seq seq = NULL;
+	  gimple_seq_add_stmt (&seq, shadow_test);
+	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR,
+						   base_end_addr, 7));
+	  gimple_seq_add_stmt (&seq, build_type_cast (shadow_type,
+						      gimple_seq_last (seq)));
+	  gimple_seq_add_stmt (&seq, build_assign (GE_EXPR,
+						   gimple_seq_last (seq),
+						   shadow));
+	  gimple_seq_add_stmt (&seq, build_assign (BIT_AND_EXPR, shadow_test,
+						   gimple_seq_last (seq)));
+	  if (!start_instrumented)
+	    gimple_seq_add_stmt (&seq, build_assign (BIT_IOR_EXPR, t,
+						     gimple_seq_last (seq)));
+	  t = gimple_assign_lhs (gimple_seq_last (seq));
+	  gimple_seq_set_location (seq, loc);
+	  gsi_insert_seq_after (&gsi, seq, GSI_CONTINUE_LINKING);
+	}
+    }
+
+  g = gimple_build_cond (NE_EXPR, t, build_int_cst (TREE_TYPE (t), 0),
+			 NULL_TREE, NULL_TREE);
+  gimple_set_location (g, loc);
+  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
+
+  /* Generate call to the run-time library (e.g. __asan_report_load8).  */
+  gsi = gsi_start_bb (then_bb);
+  int nargs;
+  tree fun = report_error_func (is_store, size_in_bytes, &nargs);
+  g = gimple_build_call (fun, nargs, base_addr, len);
+  gimple_set_location (g, loc);
+  gsi_insert_after (&gsi, g, GSI_NEW_STMT);
+
+  gsi_remove (iter, true);
+  *iter = gsi_start_bb (else_bb);
+
+  return true;
+}
+
 /* Instrument the current function.  */
 
 static unsigned int
@@ -2626,7 +2631,6 @@ asan_instrument (void)
 {
   if (shadow_ptr_types[0] == NULL_TREE)
     asan_init_shadow_ptr_types ();
-  asan_num_accesses = 0;
   transform_statements ();
   return 0;
 }
@@ -2747,6 +2751,23 @@ pass_sanopt::execute (function *fun)
 {
   basic_block bb;
 
+  int asan_num_accesses = 0;
+  if (flag_sanitize & SANITIZE_ADDRESS)
+    {
+      gimple_stmt_iterator gsi;
+      FOR_EACH_BB_FN (bb, fun)
+	for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi))
+	  {
+ 	    gimple stmt = gsi_stmt (gsi);
+	    if (is_gimple_call (stmt) && gimple_call_internal_p (stmt)
+		&& gimple_call_internal_fn (stmt) == IFN_ASAN_CHECK)
+	      ++asan_num_accesses;
+	  }
+    }
+
+  bool use_calls = ASAN_INSTRUMENTATION_WITH_CALL_THRESHOLD < INT_MAX
+    && asan_num_accesses >= ASAN_INSTRUMENTATION_WITH_CALL_THRESHOLD;
+
   FOR_EACH_BB_FN (bb, fun)
     {
       gimple_stmt_iterator gsi;
@@ -2762,17 +2783,25 @@ pass_sanopt::execute (function *fun)
 	    }
 
 	  if (gimple_call_internal_p (stmt))
-	    switch (gimple_call_internal_fn (stmt))
-	      {
-	      case IFN_UBSAN_NULL:
-		no_next = ubsan_expand_null_ifn (&gsi);
-		break;
-	      case IFN_UBSAN_BOUNDS:
-		no_next = ubsan_expand_bounds_ifn (&gsi);
-		break;
-	      default:
-		break;
-	      }
+	    {
+	      enum internal_fn ifn = gimple_call_internal_fn (stmt);
+	      switch (ifn)
+		{
+		case IFN_UBSAN_NULL:
+		  no_next = ubsan_expand_null_ifn (&gsi);
+		  break;
+		case IFN_UBSAN_BOUNDS:
+		  no_next = ubsan_expand_bounds_ifn (&gsi);
+		  break;
+		case IFN_ASAN_CHECK:
+		  {
+		    no_next = asan_expand_check_ifn (&gsi, use_calls);
+		    break;
+		  }
+		default:
+		  break;
+		}
+	    }
 
 	  if (dump_file && (dump_flags & TDF_DETAILS))
 	    {
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index 4f327df..e444cbc0 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -10375,9 +10375,9 @@ To disable use-after-return detection use
 @option{--param asan-use-after-return=0}.
 
 @item asan-instrumentation-with-call-threshold
-Once number of memory accesses  in function becomes greater
-or equal than this number, use callbacks instead of
-generating inline code.  E.g. to disable inline code use
+If number of memory accesses in function being instrumented
+is greater or equal to this number, use callbacks instead of inline checks.
+E.g. to disable inline code use
 @option{--param asan-instrumentation-with-call-threshold=0}.
 
 @end table
diff --git a/gcc/internal-fn.c b/gcc/internal-fn.c
index b58b868..fa6ac45 100644
--- a/gcc/internal-fn.c
+++ b/gcc/internal-fn.c
@@ -181,6 +181,14 @@ expand_UBSAN_BOUNDS (gimple stmt ATTRIBUTE_UNUSED)
   gcc_unreachable ();
 }
 
+/* This should get expanded in the sanopt pass.  */
+
+static void
+expand_ASAN_CHECK (gimple stmt ATTRIBUTE_UNUSED)
+{
+  gcc_unreachable ();
+}
+
 /* Add sub/add overflow checking to the statement STMT.
    CODE says whether the operation is +, or -.  */
 
diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def
index 9857542..7ae60f3 100644
--- a/gcc/internal-fn.def
+++ b/gcc/internal-fn.def
@@ -55,3 +55,4 @@ DEF_INTERNAL_FN (UBSAN_CHECK_SUB, ECF_CONST | ECF_LEAF | ECF_NOTHROW, NULL)
 DEF_INTERNAL_FN (UBSAN_CHECK_MUL, ECF_CONST | ECF_LEAF | ECF_NOTHROW, NULL)
 DEF_INTERNAL_FN (ABNORMAL_DISPATCHER, ECF_NORETURN, NULL)
 DEF_INTERNAL_FN (BUILTIN_EXPECT, ECF_CONST | ECF_LEAF | ECF_NOTHROW, NULL)
+DEF_INTERNAL_FN (ASAN_CHECK, ECF_TM_PURE | ECF_LEAF | ECF_NOTHROW, ".W..")
diff --git a/gcc/params.def b/gcc/params.def
index cad00e2..aefdd07 100644
--- a/gcc/params.def
+++ b/gcc/params.def
@@ -1085,14 +1085,14 @@ DEFPARAM (PARAM_ASAN_MEMINTRIN,
 
 DEFPARAM (PARAM_ASAN_USE_AFTER_RETURN,
          "asan-use-after-return",
-         "Enable asan builtin functions protection",
+         "Enable asan detection of use-after-return bugs",
          1, 0, 1)
 
 DEFPARAM (PARAM_ASAN_INSTRUMENTATION_WITH_CALL_THRESHOLD,
          "asan-instrumentation-with-call-threshold",
-         "Use callbacks instead of inline code once number of accesses "
-         " in function becomes greater or equal than this threshold",
-         10000, 0, INT_MAX)
+         "Use callbacks instead of inline code if number of accesses "
+         "in function becomes greater or equal to this number",
+         7000, 0, INT_MAX)
 
 DEFPARAM (PARAM_UNINIT_CONTROL_DEP_ATTEMPTS,
 	  "uninit-control-dep-attempts",
diff --git a/gcc/testsuite/c-c++-common/asan/inc.c b/gcc/testsuite/c-c++-common/asan/inc.c
index b9c6734..36cc3d8 100644
--- a/gcc/testsuite/c-c++-common/asan/inc.c
+++ b/gcc/testsuite/c-c++-common/asan/inc.c
@@ -16,6 +16,6 @@ main ()
   return 0;
 }
 
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report" 1 "asan0" } }  */
-/* { dg-final { scan-tree-dump "__builtin___asan_report_load4" "asan0" } }  */
+/* { dg-final { scan-tree-dump-times "ASAN_" 1 "asan0" } }  */
+/* { dg-final { scan-tree-dump "ASAN_CHECK \\(.*, 4\\);" "asan0" } }  */
 /* { dg-final { cleanup-tree-dump "asan0" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/instrument-with-calls-2.c b/gcc/testsuite/c-c++-common/asan/instrument-with-calls-2.c
index 570f796..04fdad0 100644
--- a/gcc/testsuite/c-c++-common/asan/instrument-with-calls-2.c
+++ b/gcc/testsuite/c-c++-common/asan/instrument-with-calls-2.c
@@ -9,8 +9,8 @@ void f(int *a, int *b) {
   x = *b;
 }
 
-/* { dg-final { scan-assembler-not "__asan_store4" } } */
-/* { dg-final { scan-assembler "__asan_report_store4" } } */
+/* { dg-final { scan-assembler "__asan_store4" } } */
+/* { dg-final { scan-assembler-not "__asan_report_store4" } } */
 /* { dg-final { scan-assembler "__asan_load4" } } */
 /* { dg-final { scan-assembler-not "__asan_report_load4" } } */
 /* { dg-final { cleanup-saved-temps } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-1.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-1.c
index c7c594e..028f8d7 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-1.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-1.c
@@ -2,7 +2,7 @@
    location in the same basic block, the second reference should not
    be instrumented by the Address Sanitizer.  */
 
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -62,7 +62,7 @@ main ()
   return test0 () && test1 (0);
 }
 
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store1" 3 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 2 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load" 1 "asan0" }  } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store1" 3 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 2 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load" 1 "sanopt" }  } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-2.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-2.c
index 143312f..a58411c 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-2.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-2.c
@@ -3,7 +3,7 @@
    be instrumented by the Address Sanitizer.  But in case of access to
    overlapping regions we must be precise.  */
 
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -20,7 +20,7 @@ main ()
   __builtin_memset (tab, 1, 3);
 }
 
-/* { dg-final { scan-tree-dump-times "& 7" 3 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 2 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report" 2 "asan0" }  } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "& 7" 3 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 2 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report" 2 "sanopt" }  } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-3.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-3.c
index 420a263..5193ae0 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-3.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-3.c
@@ -1,4 +1,4 @@
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -12,7 +12,7 @@ foo (__INT32_TYPE__ *p)
   return ret; 
 }
 
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report" 2 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store" 1 "asan0" } } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report" 2 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store" 1 "sanopt" } } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-4.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-4.c
index da91cd5..c3632aa 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-4.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-4.c
@@ -1,4 +1,4 @@
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -10,8 +10,8 @@ foo  (int *a, char *b, char *c)
   /* For a total of 5 checks.  */
 }
 
-/* { dg-final { scan-tree-dump-times "& 7" 5 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 1 "asan0" } } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "& 7" 5 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 1 "sanopt" } } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-5.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-5.c
index 134be66..077ea34 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-5.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-5.c
@@ -1,4 +1,4 @@
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -10,9 +10,9 @@ foo  (int *a, char *b, char *c)
   /* For a total of 5 checks.  */
 }
 
-/* { dg-final { scan-tree-dump-times "& 7" 5 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load4" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 1 "asan0" } } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "& 7" 5 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load4" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 1 "sanopt" } } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-6.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-6.c
index 55c8ee3..6d87104 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-6.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-6.c
@@ -1,4 +1,4 @@
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -12,9 +12,9 @@ foo  (int *a, char *b, char *c)
   /* For a total of 8 checks.  */
 }
 
-/* { dg-final { scan-tree-dump-times "& 7" 8 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load4" 2 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 2 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 2 "asan0" } } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "& 7" 8 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load4" 2 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 2 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 2 "sanopt" } } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-7.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-7.c
index a04956d..5baa10d 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-7.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-7.c
@@ -1,4 +1,4 @@
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -21,7 +21,7 @@ foo  (int *a, char *b, char *c)
   return d;
 }
 
-/* { dg-final { scan-tree-dump-times "& 7" 6 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 4 "asan0" } } */
-/* { dg-final { scan-tree-dump-not "__builtin___asan_report_store" "asan0" } } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "& 7" 6 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 4 "sanopt" } } */
+/* { dg-final { scan-tree-dump-not "__builtin___asan_report_store" "sanopt" } } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-8.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-8.c
index 308a133..2a4c081 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-8.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-8.c
@@ -1,4 +1,4 @@
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -12,9 +12,9 @@ foo  (int *a, char *b, char *c)
   /* For a total of 5 checks.  */
 }
 
-/* { dg-final { scan-tree-dump-times "& 7" 5 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load4" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 1 "asan0" } } */
-/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 1 "asan0" } } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "& 7" 5 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load1" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load4" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_load_n" 1 "sanopt" } } */
+/* { dg-final { scan-tree-dump-times "__builtin___asan_report_store_n" 1 "sanopt" } } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */
diff --git a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-9.c b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-9.c
index c6575ad..9449de5 100644
--- a/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-9.c
+++ b/gcc/testsuite/c-c++-common/asan/no-redundant-instrumentation-9.c
@@ -1,4 +1,4 @@
-/* { dg-options "-fdump-tree-asan0" } */
+/* { dg-options "-fdump-tree-sanopt" } */
 /* { dg-do compile } */
 /* { dg-skip-if "" { *-*-* } { "*" } { "-O0" } } */
 
@@ -9,5 +9,5 @@ f (char *a)
   return  __builtin_strlen (a);
 }
 
-/* { dg-final { scan-tree-dump-times "__asan_report_load1" 1 "asan0" } } */
-/* { dg-final { cleanup-tree-dump "asan0" } } */
+/* { dg-final { scan-tree-dump-times "__asan_report_load1" 1 "sanopt" } } */
+/* { dg-final { cleanup-tree-dump "sanopt" } } */

[PATCH 1/2] Support fnspecs for internal functions

[Yury Gribov]

[-- Attachment #1: Type: text/plain, Size: 503 bytes --]

Hi all,

Following discussion in 
https://gcc.gnu.org/ml/gcc-patches/2014-08/msg00016.html this patch adds 
support for fnspecs to internal function machinery which should 
(hopefully) allow compiler to perform better analysis and optimization 
of this construct.

For now the only internal functions using this machinery would be 
UBSAN_NULL and ASAN_CHECK (both would take EAF_DIRECT and EAF_NOESCAPE 
for their pointer argument).

The patch was bootstrapped and regtested on x64.

Ok for trunk?

-Y


[-- Attachment #2: ifn_fnspec-1.diff --]
[-- Type: text/x-diff, Size: 9046 bytes --]

commit b90c4abf0b804c60b548f94fe59118d029363140
Author: Yury Gribov <y.gribov@samsung.com>
Date:   Wed Aug 6 11:05:39 2014 +0400

    Added fnspec to internal functions.
    
    2014-08-08  Yury Gribov  <y.gribov@samsung.com>
    
    	* gimple.c (gimple_call_fnspec): Support internal functions.
    	(gimple_call_return_flags): Use const.
    	* Makefile.in: Add internal-fn.h to list of GC files.
    	* internal-fn.def: Add fnspec information.
    	* internal-fn.h (internal_fn_fnspec): New function.
    	(init_internal_fns): Declare new function.
    	* internal-fn.c (internal_fn_fnspec_array): New global variable.
    	(init_internal_fns): New function.
    	* tree-core.h: Update macro call.
    	* tree.c (build_common_builtin_nodes): Initialize internal fns.

diff --git a/gcc/Makefile.in b/gcc/Makefile.in
index a2fb5f5..67b497f 100644
--- a/gcc/Makefile.in
+++ b/gcc/Makefile.in
@@ -2297,7 +2297,9 @@ GTFILES = $(CPP_ID_DATA_H) $(srcdir)/input.h $(srcdir)/coretypes.h \
   $(srcdir)/vtable-verify.c \
   $(srcdir)/asan.c \
   $(srcdir)/ubsan.c \
-  $(srcdir)/tsan.c $(srcdir)/ipa-devirt.c \
+  $(srcdir)/tsan.c \
+  $(srcdir)/ipa-devirt.c \
+  $(srcdir)/internal-fn.h \
   @all_gtfiles@
 
 # Compute the list of GT header files from the corresponding C sources,
diff --git a/gcc/gimple.c b/gcc/gimple.c
index 4a9d379..4763a59 100644
--- a/gcc/gimple.c
+++ b/gcc/gimple.c
@@ -1327,11 +1327,14 @@ gimple_call_flags (const_gimple stmt)
 
 /* Return the "fn spec" string for call STMT.  */
 
-static tree
+static const_tree
 gimple_call_fnspec (const_gimple stmt)
 {
   tree type, attr;
 
+  if (gimple_call_internal_p (stmt))
+    return internal_fn_fnspec (gimple_call_internal_fn (stmt));
+
   type = gimple_call_fntype (stmt);
   if (!type)
     return NULL_TREE;
@@ -1348,7 +1351,7 @@ gimple_call_fnspec (const_gimple stmt)
 int
 gimple_call_arg_flags (const_gimple stmt, unsigned arg)
 {
-  tree attr = gimple_call_fnspec (stmt);
+  const_tree attr = gimple_call_fnspec (stmt);
 
   if (!attr || 1 + arg >= (unsigned) TREE_STRING_LENGTH (attr))
     return 0;
@@ -1382,7 +1385,7 @@ gimple_call_arg_flags (const_gimple stmt, unsigned arg)
 int
 gimple_call_return_flags (const_gimple stmt)
 {
-  tree attr;
+  const_tree attr;
 
   if (gimple_call_flags (stmt) & ECF_MALLOC)
     return ERF_NOALIAS;
diff --git a/gcc/internal-fn.c b/gcc/internal-fn.c
index 78f59d6..b58b868 100644
--- a/gcc/internal-fn.c
+++ b/gcc/internal-fn.c
@@ -40,7 +40,7 @@ along with GCC; see the file COPYING3.  If not see
 
 /* The names of each internal function, indexed by function number.  */
 const char *const internal_fn_name_array[] = {
-#define DEF_INTERNAL_FN(CODE, FLAGS) #CODE,
+#define DEF_INTERNAL_FN(CODE, FLAGS, FNSPEC) #CODE,
 #include "internal-fn.def"
 #undef DEF_INTERNAL_FN
   "<invalid-fn>"
@@ -48,12 +48,26 @@ const char *const internal_fn_name_array[] = {
 
 /* The ECF_* flags of each internal function, indexed by function number.  */
 const int internal_fn_flags_array[] = {
-#define DEF_INTERNAL_FN(CODE, FLAGS) FLAGS,
+#define DEF_INTERNAL_FN(CODE, FLAGS, FNSPEC) FLAGS,
 #include "internal-fn.def"
 #undef DEF_INTERNAL_FN
   0
 };
 
+/* Fnspec of each internal function, indexed by function number.  */
+const_tree internal_fn_fnspec_array[IFN_LAST + 1];
+
+void
+init_internal_fns ()
+{
+#define DEF_INTERNAL_FN(CODE, FLAGS, FNSPEC) \
+  if (FNSPEC) internal_fn_fnspec_array[IFN_##CODE] = \
+    build_string ((int) sizeof (FNSPEC) + 1, FNSPEC ? FNSPEC : "");
+#include "internal-fn.def"
+#undef DEF_INTERNAL_FN
+  internal_fn_fnspec_array[IFN_LAST] = 0;
+}
+
 /* ARRAY_TYPE is an array of vector modes.  Return the associated insn
    for load-lanes-style optab OPTAB.  The insn must exist.  */
 
@@ -897,7 +911,7 @@ expand_BUILTIN_EXPECT (gimple stmt)
 
    where STMT is the statement that performs the call. */
 static void (*const internal_fn_expanders[]) (gimple) = {
-#define DEF_INTERNAL_FN(CODE, FLAGS) expand_##CODE,
+#define DEF_INTERNAL_FN(CODE, FLAGS, FNSPEC) expand_##CODE,
 #include "internal-fn.def"
 #undef DEF_INTERNAL_FN
   0
diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def
index f0766bc..9857542 100644
--- a/gcc/internal-fn.def
+++ b/gcc/internal-fn.def
@@ -28,29 +28,30 @@ along with GCC; see the file COPYING3.  If not see
 
    Each entry in this file has the form:
 
-     DEF_INTERNAL_FN (NAME, FLAGS)
+     DEF_INTERNAL_FN (NAME, FLAGS, FNSPEC)
 
-   where NAME is the name of the function and FLAGS is a set of
-   ECF_* flags.  Each entry must have a corresponding expander
-   of the form:
+   where NAME is the name of the function, FLAGS is a set of
+   ECF_* flags and FNSPEC is a string describing functions fnspec.
+   
+   Each entry must have a corresponding expander of the form:
 
      void expand_NAME (gimple stmt)
 
    where STMT is the statement that performs the call.  */
 
-DEF_INTERNAL_FN (LOAD_LANES, ECF_CONST | ECF_LEAF)
-DEF_INTERNAL_FN (STORE_LANES, ECF_CONST | ECF_LEAF)
-DEF_INTERNAL_FN (GOMP_SIMD_LANE, ECF_NOVOPS | ECF_LEAF | ECF_NOTHROW)
-DEF_INTERNAL_FN (GOMP_SIMD_VF, ECF_CONST | ECF_LEAF | ECF_NOTHROW)
-DEF_INTERNAL_FN (GOMP_SIMD_LAST_LANE, ECF_CONST | ECF_LEAF | ECF_NOTHROW)
-DEF_INTERNAL_FN (LOOP_VECTORIZED, ECF_NOVOPS | ECF_LEAF | ECF_NOTHROW)
-DEF_INTERNAL_FN (MASK_LOAD, ECF_PURE | ECF_LEAF)
-DEF_INTERNAL_FN (MASK_STORE, ECF_LEAF)
-DEF_INTERNAL_FN (ANNOTATE,  ECF_CONST | ECF_LEAF | ECF_NOTHROW)
-DEF_INTERNAL_FN (UBSAN_NULL, ECF_LEAF | ECF_NOTHROW)
-DEF_INTERNAL_FN (UBSAN_BOUNDS, ECF_LEAF | ECF_NOTHROW)
-DEF_INTERNAL_FN (UBSAN_CHECK_ADD, ECF_CONST | ECF_LEAF | ECF_NOTHROW)
-DEF_INTERNAL_FN (UBSAN_CHECK_SUB, ECF_CONST | ECF_LEAF | ECF_NOTHROW)
-DEF_INTERNAL_FN (UBSAN_CHECK_MUL, ECF_CONST | ECF_LEAF | ECF_NOTHROW)
-DEF_INTERNAL_FN (ABNORMAL_DISPATCHER, ECF_NORETURN)
-DEF_INTERNAL_FN (BUILTIN_EXPECT, ECF_CONST | ECF_LEAF | ECF_NOTHROW)
+DEF_INTERNAL_FN (LOAD_LANES, ECF_CONST | ECF_LEAF, NULL)
+DEF_INTERNAL_FN (STORE_LANES, ECF_CONST | ECF_LEAF, NULL)
+DEF_INTERNAL_FN (GOMP_SIMD_LANE, ECF_NOVOPS | ECF_LEAF | ECF_NOTHROW, NULL)
+DEF_INTERNAL_FN (GOMP_SIMD_VF, ECF_CONST | ECF_LEAF | ECF_NOTHROW, NULL)
+DEF_INTERNAL_FN (GOMP_SIMD_LAST_LANE, ECF_CONST | ECF_LEAF | ECF_NOTHROW, NULL)
+DEF_INTERNAL_FN (LOOP_VECTORIZED, ECF_NOVOPS | ECF_LEAF | ECF_NOTHROW, NULL)
+DEF_INTERNAL_FN (MASK_LOAD, ECF_PURE | ECF_LEAF, NULL)
+DEF_INTERNAL_FN (MASK_STORE, ECF_LEAF, NULL)
+DEF_INTERNAL_FN (ANNOTATE,  ECF_CONST | ECF_LEAF | ECF_NOTHROW, NULL)
+DEF_INTERNAL_FN (UBSAN_NULL, ECF_LEAF | ECF_NOTHROW, ".W.")
+DEF_INTERNAL_FN (UBSAN_BOUNDS, ECF_LEAF | ECF_NOTHROW, NULL)
+DEF_INTERNAL_FN (UBSAN_CHECK_ADD, ECF_CONST | ECF_LEAF | ECF_NOTHROW, NULL)
+DEF_INTERNAL_FN (UBSAN_CHECK_SUB, ECF_CONST | ECF_LEAF | ECF_NOTHROW, NULL)
+DEF_INTERNAL_FN (UBSAN_CHECK_MUL, ECF_CONST | ECF_LEAF | ECF_NOTHROW, NULL)
+DEF_INTERNAL_FN (ABNORMAL_DISPATCHER, ECF_NORETURN, NULL)
+DEF_INTERNAL_FN (BUILTIN_EXPECT, ECF_CONST | ECF_LEAF | ECF_NOTHROW, NULL)
diff --git a/gcc/internal-fn.h b/gcc/internal-fn.h
index 2dcf44e..14573aa 100644
--- a/gcc/internal-fn.h
+++ b/gcc/internal-fn.h
@@ -20,6 +20,12 @@ along with GCC; see the file COPYING3.  If not see
 #ifndef GCC_INTERNAL_FN_H
 #define GCC_INTERNAL_FN_H
 
+#include "coretypes.h"
+
+/* Initialize internal function tables.  */
+
+extern void init_internal_fns ();
+
 /* Return the name of internal function FN.  The name is only meaningful
    for dumps; it has no linkage.  */
 
@@ -41,6 +47,16 @@ internal_fn_flags (enum internal_fn fn)
   return internal_fn_flags_array[(int) fn];
 }
 
+/* Return fnspec for function FN.  */
+
+extern GTY(()) const_tree internal_fn_fnspec_array[IFN_LAST + 1];
+
+static inline const_tree
+internal_fn_fnspec (enum internal_fn fn)
+{
+  return internal_fn_fnspec_array[(int) fn];
+}
+
 extern void expand_internal_call (gimple);
 
 #endif
diff --git a/gcc/tree-core.h b/gcc/tree-core.h
index 9d462d1..9e9ca8c 100644
--- a/gcc/tree-core.h
+++ b/gcc/tree-core.h
@@ -667,7 +667,7 @@ enum annot_expr_kind {
 
 /* Internal functions.  */
 enum internal_fn {
-#define DEF_INTERNAL_FN(CODE, FLAGS) IFN_##CODE,
+#define DEF_INTERNAL_FN(CODE, FLAGS, FNSPEC) IFN_##CODE,
 #include "internal-fn.def"
 #undef DEF_INTERNAL_FN
   IFN_LAST
diff --git a/gcc/tree.c b/gcc/tree.c
index d95cf78..1545a86 100644
--- a/gcc/tree.c
+++ b/gcc/tree.c
@@ -9870,8 +9870,9 @@ local_define_builtin (const char *name, tree type, enum built_in_function code,
 }
 
 /* Call this function after instantiating all builtins that the language
-   front end cares about.  This will build the rest of the builtins that
-   are relied upon by the tree optimizers and the middle-end.  */
+   front end cares about.  This will build the rest of the builtins
+   and internal function that are relied upon by the tree optimizers and
+   the middle-end.  */
 
 void
 build_common_builtin_nodes (void)
@@ -10104,6 +10105,8 @@ build_common_builtin_nodes (void)
 			      ECF_CONST | ECF_NOTHROW | ECF_LEAF);
       }
   }
+
+  init_internal_fns ();
 }
 
 /* HACK.  GROSS.  This is absolutely disgusting.  I wish there was a

Re: [PATCH 1/2] Support fnspecs for internal functions

[Jakub Jelinek]

On Fri, Aug 08, 2014 at 10:11:25AM +0400, Yury Gribov wrote:
> commit b90c4abf0b804c60b548f94fe59118d029363140
> Author: Yury Gribov <y.gribov@samsung.com>
> Date:   Wed Aug 6 11:05:39 2014 +0400
> 
>     Added fnspec to internal functions.
>     
>     2014-08-08  Yury Gribov  <y.gribov@samsung.com>
>     
>     	* gimple.c (gimple_call_fnspec): Support internal functions.
>     	(gimple_call_return_flags): Use const.
>     	* Makefile.in: Add internal-fn.h to list of GC files.
>     	* internal-fn.def: Add fnspec information.
>     	* internal-fn.h (internal_fn_fnspec): New function.
>     	(init_internal_fns): Declare new function.
>     	* internal-fn.c (internal_fn_fnspec_array): New global variable.
>     	(init_internal_fns): New function.
>     	* tree-core.h: Update macro call.
>     	* tree.c (build_common_builtin_nodes): Initialize internal fns.

Ok, thanks, with a minor nit:

> --- a/gcc/tree.c
> +++ b/gcc/tree.c
> @@ -9870,8 +9870,9 @@ local_define_builtin (const char *name, tree type, enum built_in_function code,
>  }
>  
>  /* Call this function after instantiating all builtins that the language
> -   front end cares about.  This will build the rest of the builtins that
> -   are relied upon by the tree optimizers and the middle-end.  */
> +   front end cares about.  This will build the rest of the builtins
> +   and internal function that are relied upon by the tree optimizers and

s/function/functions/

	Jakub

Re: [PATCH 2/2] Move Asan instrumentation to sanopt pass

[Jakub Jelinek]

On Fri, Aug 08, 2014 at 10:11:30AM +0400, Yury Gribov wrote:
> commit 22a79d39656d3d600e97c7ae599b86b6d25efe00
> Author: Yury Gribov <y.gribov@samsung.com>
> Date:   Wed Aug 6 11:06:24 2014 +0400
> 
>     Move inlining of Asan memory checks to sanopt pass.
>     Change asan-instrumentation-with-call-threshold to more closely match LLVM.
>     
>     gcc/
>     
>     2014-08-08  Yury Gribov  <y.gribov@samsung.com>
>     
>     	* asan.c (asan_check_flags): New enum.
>     	(build_check_stmt_with_calls): Removed function.
>     	(build_check_stmt): Split inlining logic to
>     	asan_expand_check_ifn.
>     	(instrument_derefs): Rename parameter.
>     	(instrument_mem_region_access): Rename parameter.
>     	(instrument_strlen_call): Likewise.
>     	(asan_expand_check_ifn): New function.
>     	(asan_instrument): Remove old code.
>     	(pass_sanopt::execute): Change handling of
>     	asan-instrumentation-with-call-threshold.
>     	(asan_clear_shadow): Fix formatting.
>     	(asan_function_start): Likewise.
>     	(asan_emit_stack_protection): Likewise.
>     	* doc/invoke.texi (asan-instrumentation-with-call-threshold):
>     	Update description.
>     	* internal-fn.c (expand_ASAN_CHECK): New function.
>     	* internal-fn.def (ASAN_CHECK): New internal function.
>     	* params.def (PARAM_ASAN_INSTRUMENTATION_WITH_CALL_THRESHOLD):
>     	Update description.
>     	(PARAM_ASAN_USE_AFTER_RETURN): Likewise.
>     
>     gcc/testsuite/
>     
>     2014-07-21  Yury Gribov  <y.gribov@samsung.com>
>     
>     	* c-c++-common/asan/inc.c: Update test.
>     	* c-c++-common/asan/instrument-with-calls-2.c: Likewise.
>     	* c-c++-common/asan/no-redundant-instrumentation-1.c: Likewise.
>     	* c-c++-common/asan/no-redundant-instrumentation-2.c: Likewise.
>     	* c-c++-common/asan/no-redundant-instrumentation-3.c: Likewise.
>     	* c-c++-common/asan/no-redundant-instrumentation-4.c: Likewise.
>     	* c-c++-common/asan/no-redundant-instrumentation-5.c: Likewise.
>     	* c-c++-common/asan/no-redundant-instrumentation-6.c: Likewise.
>     	* c-c++-common/asan/no-redundant-instrumentation-7.c: Likewise.
>     	* c-c++-common/asan/no-redundant-instrumentation-8.c: Likewise.
>     	* c-c++-common/asan/no-redundant-instrumentation-9.c: Likewise.

Ok, thanks.

	Jakub

Re: [PATCH 2/2] Move Asan instrumentation to sanopt pass

[Yury Gribov]

On 08/08/2014 12:09 PM, Jakub Jelinek wrote:
 > Ok, thanks.

Done in r213807.

-Y

Re: [PATCH 1/2] Support fnspecs for internal functions

[Yury Gribov]

 > On 08/08/2014 12:07 PM, Jakub Jelinek wrote:
 > Ok, thanks, with a minor nit:

Done in r213806 (the nit fixed in patch 2/2).

-Y