[gcc r9-10089] c++: Optimize away NULLPTR_TYPE comparisons [PR101443]

[Jakub Jelinek <jakub@gcc.gnu.org>]

https://gcc.gnu.org/g:c910d7919529b490098319ab84c5c0132e8809f0

commit r9-10089-gc910d7919529b490098319ab84c5c0132e8809f0
Author: Jakub Jelinek <jakub@redhat.com>
Date:   Thu Jul 15 18:53:20 2021 +0200

    c++: Optimize away NULLPTR_TYPE comparisons [PR101443]
    
    Comparisons of NULLPTR_TYPE operands cause all kinds of problems in the
    middle-end and in fold-const.c, various optimizations assume that if they
    see e.g. a non-equality comparison with one of the operands being
    INTEGER_CST and it is not INTEGRAL_TYPE_P (which has TYPE_{MIN,MAX}_VALUE),
    they can build_int_cst (type, 1) to find a successor.
    
    The following patch fixes it by making sure they don't appear in the IL,
    optimize them away at cp_fold time as all can be folded.
    
    Though, I've just noticed that clang++ rejects the non-equality comparisons
    instead, foo () > 0 with
    invalid operands to binary expression ('decltype(nullptr)' (aka 'nullptr_t') and 'int')
    and foo () > nullptr with
    invalid operands to binary expression ('decltype(nullptr)' (aka 'nullptr_t') and 'nullptr_t')
    
    Shall we reject those too, in addition or instead of parts of this patch?
    If so, wouldn't this patch be still useful for backports, I bet we don't
    want to start reject it on the release branches when we used to accept it.
    
    2021-07-15  Jakub Jelinek  <jakub@redhat.com>
    
            PR c++/101443
            * cp-gimplify.c (cp_fold): For comparisons with NULLPTR_TYPE
            operands, fold them right away to true or false.
    
            * g++.dg/cpp0x/nullptr46.C: New test.
    
    (cherry picked from commit 7094a69bd62a14dfa311eaa2fea468f221c7c9f3)

Diff:
---
 gcc/cp/cp-gimplify.c                   | 26 ++++++++++++++++++++++++++
 gcc/testsuite/g++.dg/cpp0x/nullptr46.C | 11 +++++++++++
 2 files changed, 37 insertions(+)

diff --git a/gcc/cp/cp-gimplify.c b/gcc/cp/cp-gimplify.c
index 5dde4f005bf..c8f090d0da8 100644
--- a/gcc/cp/cp-gimplify.c
+++ b/gcc/cp/cp-gimplify.c
@@ -2537,6 +2537,32 @@ cp_fold (tree x)
       op0 = cp_fold_maybe_rvalue (TREE_OPERAND (x, 0), rval_ops);
       op1 = cp_fold_rvalue (TREE_OPERAND (x, 1));
 
+      /* decltype(nullptr) has only one value, so optimize away all comparisons
+	 with that type right away, keeping them in the IL causes troubles for
+	 various optimizations.  */
+      if (COMPARISON_CLASS_P (org_x)
+	  && TREE_CODE (TREE_TYPE (op0)) == NULLPTR_TYPE
+	  && TREE_CODE (TREE_TYPE (op1)) == NULLPTR_TYPE)
+	{
+	  switch (code)
+	    {
+	    case EQ_EXPR:
+	    case LE_EXPR:
+	    case GE_EXPR:
+	      x = constant_boolean_node (true, TREE_TYPE (x));
+	      break;
+	    case NE_EXPR:
+	    case LT_EXPR:
+	    case GT_EXPR:
+	      x = constant_boolean_node (false, TREE_TYPE (x));
+	      break;
+	    default:
+	      gcc_unreachable ();
+	    }
+	  return omit_two_operands_loc (loc, TREE_TYPE (x), x,
+					op0, op1);
+	}
+
       if (op0 != TREE_OPERAND (x, 0) || op1 != TREE_OPERAND (x, 1))
 	{
 	  if (op0 == error_mark_node || op1 == error_mark_node)
diff --git a/gcc/testsuite/g++.dg/cpp0x/nullptr46.C b/gcc/testsuite/g++.dg/cpp0x/nullptr46.C
new file mode 100644
index 00000000000..1514cee3c3b
--- /dev/null
+++ b/gcc/testsuite/g++.dg/cpp0x/nullptr46.C
@@ -0,0 +1,11 @@
+// PR c++/101443
+// { dg-do compile { target c++11 } }
+// { dg-options "-O2" }
+
+decltype(nullptr) foo ();
+
+bool
+bar ()
+{
+  return foo () > nullptr || foo () < nullptr;
+}