[Bug sanitizer/109134] New: UBSan signed integer overflow check missing

[Bug sanitizer/109134] New: UBSan signed integer overflow check missing

[mpolacek at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109134

            Bug ID: 109134
           Summary: UBSan signed integer overflow check missing
           Product: gcc
           Version: 12.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: sanitizer
          Assignee: unassigned at gcc dot gnu.org
          Reporter: mpolacek at gcc dot gnu.org
                CC: dodji at gcc dot gnu.org, dvyukov at gcc dot gnu.org,
                    jakub at gcc dot gnu.org, kcc at gcc dot gnu.org, marxin at gcc dot gnu.org
  Target Milestone: ---

Related to bug 109107, but not a dup and my patch doesn't fix this one:

/* { dg-do run { target int32 } } */
/* { dg-options "-fsanitize=signed-integer-overflow" } */

#define INT_MIN (-__INT_MAX__ - 1)
const int a = INT_MIN;
const unsigned int b = 676540;
int d = 1;

__attribute__((noipa)) int
foo ()
{
  int c = b - (a - (short) d);
  return c;
}

int
main ()
{
  foo ();
  return 0;
}

With -O0 we give
runtime error: signed integer overflow: -2147483648 - 1 cannot be represented
in type 'int'
but with -O nothing.

We fold

int c = (int) ((unsigned int) b - (unsigned int) ((int) a - (int) (short int)
d));

into

int c = (int) ((unsigned int) (short int) d + 2148160188);

hiding the overflow.

[Bug sanitizer/109134] UBSan signed integer overflow check missing

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109134

--- Comment #1 from Richard Biener <rguenth at gcc dot gnu.org> ---
Having that extra TYPE_OVERFLOW_SANITIZED really isn't sustainable.  But the
main issue is that the instrumentation happens after (some) folding has taken
place.

We probably also have still code second-guessing TYPE_OVERFLOW_UNDEFINED
if !TYPE_OVERFLOW_WRAPS or the other way around - it would be best to
transition to TYPE_OVERFLOW_BEHAVIOR () yielding an enum, there can
only be one of { wraps, undefined, traps, sanitized } and transforms
relying on undefined overflow shall not trigger when it's 'sanitized'.

If it were not for the second-guessing code then adjusting
TYPE_OVERFLOW_UNDEFINED to false when sanitizing should work (but
TYPE_OVERFLOW_WRAPS should still return false as well!).  It might
also work to magically switch back to undefined after instrumentation.