[PATCH] c++: Optimize away NULLPTR_TYPE comparisons [PR101443]

[Jakub Jelinek <jakub@redhat.com>]

Hi!

Comparisons of NULLPTR_TYPE operands cause all kinds of problems in the
middle-end and in fold-const.c, various optimizations assume that if they
see e.g. a non-equality comparison with one of the operands being
INTEGER_CST and it is not INTEGRAL_TYPE_P (which has TYPE_{MIN,MAX}_VALUE),
they can build_int_cst (type, 1) to find a successor.

The following patch fixes it by making sure they don't appear in the IL,
optimize them away at cp_fold time as all can be folded.

Bootstrapped/regtested on x86_64-linux and i686-linux, ok for trunk?

Though, I've just noticed that clang++ rejects the non-equality comparisons
instead, foo () > 0 with
invalid operands to binary expression ('decltype(nullptr)' (aka 'nullptr_t') and 'int')
and foo () > nullptr with
invalid operands to binary expression ('decltype(nullptr)' (aka 'nullptr_t') and 'nullptr_t')

Shall we reject those too, in addition or instead of parts of this patch?
If so, wouldn't this patch be still useful for backports, I bet we don't
want to start reject it on the release branches when we used to accept it.

2021-07-15  Jakub Jelinek  <jakub@redhat.com>

	PR c++/101443
	* cp-gimplify.c (cp_fold): For comparisons with NULLPTR_TYPE
	operands, fold them right away to true or false.

	* g++.dg/cpp0x/nullptr46.C: New test.

--- gcc/cp/cp-gimplify.c.jj	2021-06-25 10:36:22.141020337 +0200
+++ gcc/cp/cp-gimplify.c	2021-07-14 12:04:24.221860756 +0200
@@ -2424,6 +2424,32 @@ cp_fold (tree x)
       op0 = cp_fold_maybe_rvalue (TREE_OPERAND (x, 0), rval_ops);
       op1 = cp_fold_rvalue (TREE_OPERAND (x, 1));
 
+      /* decltype(nullptr) has only one value, so optimize away all comparisons
+	 with that type right away, keeping them in the IL causes troubles for
+	 various optimizations.  */
+      if (COMPARISON_CLASS_P (org_x)
+	  && TREE_CODE (TREE_TYPE (op0)) == NULLPTR_TYPE
+	  && TREE_CODE (TREE_TYPE (op1)) == NULLPTR_TYPE)
+	{
+	  switch (code)
+	    {
+	    case EQ_EXPR:
+	    case LE_EXPR:
+	    case GE_EXPR:
+	      x = constant_boolean_node (true, TREE_TYPE (x));
+	      break;
+	    case NE_EXPR:
+	    case LT_EXPR:
+	    case GT_EXPR:
+	      x = constant_boolean_node (false, TREE_TYPE (x));
+	      break;
+	    default:
+	      gcc_unreachable ();
+	    }
+	  return omit_two_operands_loc (loc, TREE_TYPE (x), x,
+					op0, op1);
+	}
+
       if (op0 != TREE_OPERAND (x, 0) || op1 != TREE_OPERAND (x, 1))
 	{
 	  if (op0 == error_mark_node || op1 == error_mark_node)
--- gcc/testsuite/g++.dg/cpp0x/nullptr46.C.jj	2021-07-14 11:48:03.917122727 +0200
+++ gcc/testsuite/g++.dg/cpp0x/nullptr46.C	2021-07-14 11:46:52.261092097 +0200
@@ -0,0 +1,11 @@
+// PR c++/101443
+// { dg-do compile { target c++11 } }
+// { dg-options "-O2" }
+
+decltype(nullptr) foo ();
+
+bool
+bar ()
+{
+  return foo () > nullptr || foo () < nullptr;
+}

	Jakub