[gcc r13-6475] c-family: Fix up -Wsign-compare BIT_NOT_EXPR handling [PR107465]

[gcc r13-6475] c-family: Fix up -Wsign-compare BIT_NOT_EXPR handling [PR107465]

[Jakub Jelinek]

https://gcc.gnu.org/g:daaf74a714c41c8dbaf9954bcc58462c63062b4f

commit r13-6475-gdaaf74a714c41c8dbaf9954bcc58462c63062b4f
Author: Jakub Jelinek <jakub@redhat.com>
Date:   Sat Mar 4 10:18:37 2023 +0100

    c-family: Fix up -Wsign-compare BIT_NOT_EXPR handling [PR107465]
    
    The following patch fixes multiple bugs in warn_for_sign_compare related to
    the BIT_NOT_EXPR related warnings.
    My understanding is that what those 3 warnings are meant to warn (since 1995
    apparently) is the case where we have BIT_NOT_EXPR of a zero-extended
    value, so in result_type the value is something like:
    0b11111111XXXXXXXX (e.g. ~ of a 8->16 bit zero extension)
    0b000000000000000011111111XXXXXXXX (e.g. ~ of a 8->16 bit zero extension
    then zero extended to 32 bits)
    0b111111111111111111111111XXXXXXXX (e.g. ~ of a 8->16 bit zero extension
    then sign extended to 32 bits)
    and the intention of the warning is to warn when this is compared against
    something that has some 0 bits at the place where the above has guaranteed
    1 bits, either ensured through comparison against constant where we know
    the bits exactly, or through zero extension from some narrower type where
    again we know at least some upper bits are zero extended.
    The bugs in the warning code are:
    1) misunderstanding of the {,c_common_}get_narrower APIs - the unsignedp
       it sets is only meaningful if the function actually returns something
       narrower (in that case it says whether the narrower value is then
       sign (0) or zero (1) extended to the originally passed value.
       Though op0 or op1 at this point might be already narrower than
       result_type, and if the function doesn't return anything narrower,
       it all depends on whether the passed in op{0,1} had TYPE_UNSIGNED
       type or not
    2) the code didn't check at all whether the BIT_NOT_EXPR operand
       was actually zero extended (i.e. that it was narrower and unsignedp
       was set to 1 for it), all it did is check that unsignedp from the
       call was 1.  But that isn't well defined thing, if the argument
       is returned as is, the function sets unsignedp to 0, but if there
       is e.g. a useless cast to the same or compatible type in between,
       it can return 1 if the cast is unsigned; now, if BIT_NOT_EXPR
       operand is not zero extended, we know nothing at all about any bits
       in the operand containing BIT_NOT_EXPR, so there is nothing to warn
       about
    3) the code was actually testing both operands after calling
       c_common_get_narrower on them and on the one with BIT_NOT_EXPR
       again for constants; I think that is just wrong in case the BIT_NOT_EXPR
       operand wouldn't be fully folded, the warning makes sense only if the
       other operand not having BIT_NOT_EXPR in it is constant
    4) as can be seen from the above bit pattern examples, the upper bits above
       (in the patch arg0) aren't always all 1s, there could be some zero extension
       above it and from it one would have 0s, so that needs to be taken into
       account for the choice which constant bits to test for being always set
       otherwise warning is emitted, or for the zero extension guaranteed zero
       bits
    5) the patch also simplifies the handling, we only do it if one but not
       both operands are BIT_NOT_EXPR after first {,c_common_}get_narrower,
       so we can just use std::swap to ensure it is the first one
    6) the code compared bits against HOST_BITS_PER_LONG, which made sense
       back in 1995 when the values were stored into long, but now that they
       are HOST_WIDE_INT should test HOST_BITS_PER_WIDE_INT (or we could rewrite
       the stuff to wide_int, not done in the patch)
    
    2023-03-04  Jakub Jelinek  <jakub@redhat.com>
    
            PR c/107465
            * c-warn.cc (warn_for_sign_compare): If c_common_get_narrower
            doesn't return a narrower result, use TYPE_UNSIGNED to set unsignedp0
            and unsignedp1.  For the one BIT_NOT_EXPR case vs. one without,
            only check for constant in the non-BIT_NOT_EXPR operand, use std::swap
            to simplify the code, only warn if BIT_NOT_EXPR operand is extended
            from narrower unsigned, fix up computation of mask for the constant
            cases and for unsigned other operand case handle differently
            BIT_NOT_EXPR result being sign vs. zero extended.
    
            * c-c++-common/Wsign-compare-2.c: New test.
            * c-c++-common/pr107465.c: New test.

Diff:
---
 gcc/c-family/c-warn.cc                       |  87 ++++++++++++++--------
 gcc/testsuite/c-c++-common/Wsign-compare-2.c | 106 +++++++++++++++++++++++++++
 gcc/testsuite/c-c++-common/pr107465.c        |  22 ++++++
 3 files changed, 184 insertions(+), 31 deletions(-)

diff --git a/gcc/c-family/c-warn.cc b/gcc/c-family/c-warn.cc
index 2941cf4c2f9..70d9bb20097 100644
--- a/gcc/c-family/c-warn.cc
+++ b/gcc/c-family/c-warn.cc
@@ -2345,42 +2345,50 @@ warn_for_sign_compare (location_t location,
      have all bits set that are set in the ~ operand when it is
      extended.  */
 
-  op0 = c_common_get_narrower (op0, &unsignedp0);
-  op1 = c_common_get_narrower (op1, &unsignedp1);
+  tree arg0 = c_common_get_narrower (op0, &unsignedp0);
+  if (TYPE_PRECISION (TREE_TYPE (arg0)) == TYPE_PRECISION (TREE_TYPE (op0)))
+    unsignedp0 = TYPE_UNSIGNED (TREE_TYPE (op0));
+  op0 = arg0;
+  tree arg1 = c_common_get_narrower (op1, &unsignedp1);
+  if (TYPE_PRECISION (TREE_TYPE (arg1)) == TYPE_PRECISION (TREE_TYPE (op1)))
+    unsignedp1 = TYPE_UNSIGNED (TREE_TYPE (op1));
+  op1 = arg1;
 
   if ((TREE_CODE (op0) == BIT_NOT_EXPR)
       ^ (TREE_CODE (op1) == BIT_NOT_EXPR))
     {
-      if (TREE_CODE (op0) == BIT_NOT_EXPR)
-	op0 = c_common_get_narrower (TREE_OPERAND (op0, 0), &unsignedp0);
       if (TREE_CODE (op1) == BIT_NOT_EXPR)
-	op1 = c_common_get_narrower (TREE_OPERAND (op1, 0), &unsignedp1);
-
-      if (tree_fits_shwi_p (op0) || tree_fits_shwi_p (op1))
 	{
-	  tree primop;
-	  HOST_WIDE_INT constant, mask;
-	  int unsignedp;
-	  unsigned int bits;
+	  std::swap (op0, op1);
+	  std::swap (unsignedp0, unsignedp1);
+	}
 
-	  if (tree_fits_shwi_p (op0))
-	    {
-	      primop = op1;
-	      unsignedp = unsignedp1;
-	      constant = tree_to_shwi (op0);
-	    }
-	  else
-	    {
-	      primop = op0;
-	      unsignedp = unsignedp0;
-	      constant = tree_to_shwi (op1);
-	    }
+      int unsignedp;
+      arg0 = c_common_get_narrower (TREE_OPERAND (op0, 0), &unsignedp);
 
-	  bits = TYPE_PRECISION (TREE_TYPE (primop));
-	  if (bits < TYPE_PRECISION (result_type)
-	      && bits < HOST_BITS_PER_LONG && unsignedp)
+      /* For these warnings, we need BIT_NOT_EXPR operand to be
+	 zero extended from narrower type to BIT_NOT_EXPR's type.
+	 In that case, all those bits above the narrower's type
+	 are after BIT_NOT_EXPR set to 1.  */
+      if (tree_fits_shwi_p (op1))
+	{
+	  HOST_WIDE_INT constant = tree_to_shwi (op1);
+	  unsigned int bits = TYPE_PRECISION (TREE_TYPE (arg0));
+	  if (unsignedp
+	      && bits < TYPE_PRECISION (TREE_TYPE (op0))
+	      && bits < HOST_BITS_PER_WIDE_INT)
 	    {
-	      mask = HOST_WIDE_INT_M1U << bits;
+	      HOST_WIDE_INT mask = HOST_WIDE_INT_M1U << bits;
+	      if (unsignedp0)
+		{
+		  bits = TYPE_PRECISION (TREE_TYPE (op0));
+		  if (bits < TYPE_PRECISION (result_type)
+		      && bits < HOST_BITS_PER_WIDE_INT)
+		    mask &= ~(HOST_WIDE_INT_M1U << bits);
+		}
+	      bits = TYPE_PRECISION (result_type);
+	      if (bits < HOST_BITS_PER_WIDE_INT)
+		mask &= ~(HOST_WIDE_INT_M1U << bits);
 	      if ((mask & constant) != mask)
 		{
 		  if (constant == 0)
@@ -2394,11 +2402,28 @@ warn_for_sign_compare (location_t location,
 		}
 	    }
 	}
-      else if (unsignedp0 && unsignedp1
-	       && (TYPE_PRECISION (TREE_TYPE (op0))
-		   < TYPE_PRECISION (result_type))
+      else if ((TYPE_PRECISION (TREE_TYPE (arg0))
+		< TYPE_PRECISION (TREE_TYPE (op0)))
+	       && unsignedp
+	       && unsignedp1
+	       /* If unsignedp0, the BIT_NOT_EXPR result is
+		  zero extended, so say if op0 is unsigned char
+		  variable, BIT_NOT_EXPR is unsigned short and
+		  result type int and op0 has value 0x55, the
+		  int value will be 0xffaa, or for op0 0xaa it
+		  will be 0xff55.  In these cases, warn if
+		  op1 is unsigned and narrower than unsigned short.
+		  While if unsignedp0 is false, the BIT_NOT_EXPR
+		  result is sign extended and because of the
+		  above TYPE_PRECISION comparison we know the
+		  MSB of BIT_NOT_EXPR is set (perhaps with some
+		  further bits below it).  The sign extension will
+		  then ensure all bits above BIT_NOT_EXPR up to
+		  result_type's precision are set.  */
 	       && (TYPE_PRECISION (TREE_TYPE (op1))
-		   < TYPE_PRECISION (result_type)))
+		   < TYPE_PRECISION (unsignedp0
+				     ? TREE_TYPE (op0)
+				     : result_type)))
 	warning_at (location, OPT_Wsign_compare,
 		    "comparison of promoted bitwise complement "
 		    "of an unsigned value with unsigned");
diff --git a/gcc/testsuite/c-c++-common/Wsign-compare-2.c b/gcc/testsuite/c-c++-common/Wsign-compare-2.c
new file mode 100644
index 00000000000..23933d09cdc
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/Wsign-compare-2.c
@@ -0,0 +1,106 @@
+/* { dg-do compile { target { ilp32 || lp64 } } } */
+/* { dg-options "-Wsign-compare" } */
+
+int
+f1 (unsigned char x)
+{
+  return (unsigned short) (~(unsigned short) x) == 0;		/* { dg-warning "promoted bitwise complement of an unsigned value is always nonzero" "" { target c } } */
+}
+
+int
+f2 (unsigned char x)
+{
+  return (unsigned short) (~(unsigned short) x) == 5;		/* { dg-warning "comparison of promoted bitwise complement of an unsigned value with constant" "" { target c }  } */
+}
+
+int
+f3 (unsigned char x)
+{
+  return (unsigned int) (~(unsigned short) x) == 0xffff0005U;	/* { dg-warning "comparison of promoted bitwise complement of an unsigned value with constant" } */
+}
+
+int
+f4 (unsigned char x)
+{
+  return (unsigned int) (~(unsigned short) x) == 0xffff0005ULL;	/* { dg-warning "comparison of promoted bitwise complement of an unsigned value with constant" } */
+}
+
+int
+f5 (unsigned char x)
+{
+  return (unsigned int) (~(unsigned short) x) == 0xffffff05U;	/* { dg-bogus "comparison of promoted bitwise complement of an unsigned value with constant" } */
+}
+
+int
+f6 (unsigned char x)
+{
+  return (unsigned int) (~(unsigned short) x) == 0xffffff05ULL;	/* { dg-bogus "comparison of promoted bitwise complement of an unsigned value with constant" } */
+}
+
+int
+f7 (unsigned char x)
+{
+  return (unsigned long long) (~(unsigned short) x) == 0xffffffffffffff05ULL;	/* { dg-bogus "comparison of promoted bitwise complement of an unsigned value with constant" } */
+}
+
+typedef unsigned short T;
+
+int
+f8 (T x)
+{
+  return (unsigned short) (~(unsigned short) x) == 0;		/* { dg-bogus "promoted bitwise complement of an unsigned value is always nonzero" } */
+}
+
+int
+f9 (T x)
+{
+  return (unsigned short) (~(unsigned short) x) == 5;		/* { dg-bogus "comparison of promoted bitwise complement of an unsigned value with constant" } */
+}
+
+int
+f10 (T x, unsigned char y)
+{
+  return (unsigned short) (~(unsigned short) x) == y;		/* { dg-bogus "comparison of promoted bitwise complement of an unsigned value with unsigned" } */
+}
+
+int
+f11 (T x, unsigned char y)
+{
+  return (unsigned short) (~(unsigned short) x) == y;		/* { dg-bogus "comparison of promoted bitwise complement of an unsigned value with unsigned" } */
+}
+
+int
+f12 (unsigned char x, unsigned char y)
+{
+  return (unsigned short) (~(unsigned short) x) == y;		/* { dg-warning "comparison of promoted bitwise complement of an unsigned value with unsigned" "" { target c } } */
+}
+
+int
+f13 (unsigned char x, unsigned char y)
+{
+  return (unsigned short) (~(unsigned short) x) == y;		/* { dg-warning "comparison of promoted bitwise complement of an unsigned value with unsigned" "" { target c } } */
+}
+
+int
+f14 (unsigned char x, unsigned int y)
+{
+  return (unsigned long long) (~x) == y;			/* { dg-warning "comparison of promoted bitwise complement of an unsigned value with unsigned" } */
+}
+
+int
+f15 (unsigned short x, unsigned int y)
+{
+  return (long long) (~x) == y;					/* { dg-warning "comparison of promoted bitwise complement of an unsigned value with unsigned" } */
+}
+
+int
+f16 (unsigned char x, unsigned short y)
+{
+  return (unsigned short) (~(unsigned short) x) == y;		/* { dg-bogus "comparison of promoted bitwise complement of an unsigned value with unsigned" } */
+}
+
+int
+f17 (unsigned char x, unsigned short y)
+{
+  return (unsigned short) (~(unsigned short) x) == y;		/* { dg-bogus "comparison of promoted bitwise complement of an unsigned value with unsigned" } */
+}
diff --git a/gcc/testsuite/c-c++-common/pr107465.c b/gcc/testsuite/c-c++-common/pr107465.c
new file mode 100644
index 00000000000..7c9ff24bc83
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/pr107465.c
@@ -0,0 +1,22 @@
+/* PR c/107465 */
+/* { dg-do compile } */
+/* { dg-options "-Wsign-compare" } */
+
+void baz (void);
+typedef unsigned short T;
+
+#if __SIZEOF_SHORT__ * __CHAR_BIT__ == 16
+void
+foo (unsigned short x)
+{
+  if (!(x ^ 0xFFFF))
+    baz ();
+}
+
+void
+bar (T x)
+{
+  if (!(x ^ 0xFFFF))	/* { dg-bogus "promoted bitwise complement of an unsigned value is always nonzero" } */
+    baz ();
+}
+#endif