ubsan built-in function types

ubsan built-in function types

[Martin Sebor]

Most sanitizer built-in argument types are all of pointer types.
For example:

   BUILT_IN_UBSAN_HANDLE_SHIFT_OUT_OF_BOUNDS
as
   BT_FN_VOID_PTR_PTR_PTR

or

   BUILT_IN_UBSAN_HANDLE_VLA_BOUND_NOT_POSITIVE
as
   BT_FN_VOID_PTR_PTR.

But some calls to these functions are made with some arguments
of integer types.  For instance, the sanitized code for the shift
expression below:

   int f (int i, int j)
   {
     return i << j;
   }

is this:

   <bb 3> :
   _9 = (unsigned long) j.0_13;
   _10 = (unsigned long) i.1_15;
   # .MEM_17 = VDEF <.MEM_16(D)>
   __builtin___ubsan_handle_shift_out_of_bounds (&*.Lubsan_data0, _10, _9);

As a result, gimple_call_builtin_p() returns false for such calls
because the arguments don't match the expected types.  Assuming
the function types are that way on purpose, is it expected that
gimple_call_builtin_p() fail for these calls?  If so, what's
the recommended way to test a statement to see if it's a sanitizer
built-in?

ASAN uses gimple_call_builtin_p (stmt, BUILT_IN_NORMAL)) to see
if a statement is the result of instrumentation (in
has_stmt_been_instrumented_p) and this test fails for the same
reason.  I don't know if it matters, I was just looking for
a way to check that succeeds even for these calls.

Thanks
Martin

Re: ubsan built-in function types

[Richard Biener]

On Fri, Jul 2, 2021 at 6:33 PM Martin Sebor via Gcc <gcc@gcc.gnu.org> wrote:
>
> Most sanitizer built-in argument types are all of pointer types.
> For example:
>
>    BUILT_IN_UBSAN_HANDLE_SHIFT_OUT_OF_BOUNDS
> as
>    BT_FN_VOID_PTR_PTR_PTR
>
> or
>
>    BUILT_IN_UBSAN_HANDLE_VLA_BOUND_NOT_POSITIVE
> as
>    BT_FN_VOID_PTR_PTR.
>
> But some calls to these functions are made with some arguments
> of integer types.  For instance, the sanitized code for the shift
> expression below:
>
>    int f (int i, int j)
>    {
>      return i << j;
>    }
>
> is this:
>
>    <bb 3> :
>    _9 = (unsigned long) j.0_13;
>    _10 = (unsigned long) i.1_15;
>    # .MEM_17 = VDEF <.MEM_16(D)>
>    __builtin___ubsan_handle_shift_out_of_bounds (&*.Lubsan_data0, _10, _9);
>
> As a result, gimple_call_builtin_p() returns false for such calls
> because the arguments don't match the expected types.  Assuming
> the function types are that way on purpose, is it expected that
> gimple_call_builtin_p() fail for these calls?

This API uses gimple_builtin_call_types_compatible to guard these
kind of mismatches which makes consumers not need to do extensive
argument verification checks.  So, yes.

> If so, what's
> the recommended way to test a statement to see if it's a sanitizer
> built-in?

Use the head part of the above API:

bool
gimple_call_builtin_p (const gimple *stmt, enum built_in_function code)
{
  tree fndecl;
  if (is_gimple_call (stmt)
      && (fndecl = gimple_call_fndecl (stmt)) != NULL_TREE
      && fndecl_built_in_p (fndecl, code))

and return true instead of checking for compatible types.

> ASAN uses gimple_call_builtin_p (stmt, BUILT_IN_NORMAL)) to see
> if a statement is the result of instrumentation (in
> has_stmt_been_instrumented_p) and this test fails for the same
> reason.  I don't know if it matters, I was just looking for
> a way to check that succeeds even for these calls.
>
> Thanks
> Martin