[PATCH] powerpc: Fix up *branch_anddi3_dot for -m32 -mpowerpc64 [PR109566]

[Jakub Jelinek <jakub@redhat.com>]

Hi!

The following testcase reduced from newlib ICEs on powerpc-linux,
with -O2 -m32 -mpowerpc64 since r12-6433 PR102239 optimization was
added and on the original testcase since some ranger improvements in
GCC 13 made it no longer latent on newlib.
The problem is that the *branch_anddi3_dot define_insn_and_split
relies on the *rotldi3_mask_dot define_insn_and_split being recognized
during splitting.  The rs6000_is_valid_rotate_dot_mask function checks whether
the mask is a CONST_INT which is a valid mask, but *rotl<mode>3_mask_dot in
addition to checking that it is a valid mask also has
  (<MODE>mode == Pmode || UINTVAL (operands[3]) <= 0x7fffffff)
test in the condition.  For TARGET_64BIT that doesn't add any further
requirements, but for !TARGET_64BIT && TARGET_POWERPC64 if the AND
second operand is larger than INT_MAX it will not be recognized.

The rs6000_is_valid_rotate_dot_mask function is used solely in one spot,
condition of *branch_anddi3_dot, so the following patch adjusts it
to check for that as well.

Bootstrapped/regtested on powerpc64-linux (-m32/-m64) and powerpc64le-linux,
ok for trunk/13.1/12.3?

2023-04-24  Jakub Jelinek  <jakub@redhat.com>

	PR target/109566
	* config/rs6000/rs6000.cc (rs6000_is_valid_rotate_dot_mask): For
	!TARGET_64BIT, don't return true if UINTVAL (mask) << (63 - nb)
	is larger than signed int maximum.

	* gcc.target/powerpc/pr109566.c: New test.

--- gcc/config/rs6000/rs6000.cc.jj	2023-04-04 10:33:47.433201866 +0200
+++ gcc/config/rs6000/rs6000.cc	2023-04-24 12:31:07.237031550 +0200
@@ -11409,7 +11409,16 @@ bool
 rs6000_is_valid_rotate_dot_mask (rtx mask, machine_mode mode)
 {
   int nb, ne;
-  return rs6000_is_valid_mask (mask, &nb, &ne, mode) && nb >= ne && ne > 0;
+  if (rs6000_is_valid_mask (mask, &nb, &ne, mode) && nb >= ne && ne > 0)
+    {
+      if (TARGET_64BIT)
+	return true;
+      /* *rotldi3_mask_dot requires for -m32 -mpowerpc64 that the mask is
+	 <= 0x7ffffff.  */
+      return (UINTVAL (mask) << (63 - nb)) <= 0x7fffffff;
+    }
+  else
+    return false;
 }
 
 /* Return whether MASK (a CONST_INT) is a valid mask for any rlwinm, rldicl,
--- gcc/testsuite/gcc.target/powerpc/pr109566.c.jj	2023-04-24 12:54:48.293266468 +0200
+++ gcc/testsuite/gcc.target/powerpc/pr109566.c	2023-04-24 12:34:34.306006418 +0200
@@ -0,0 +1,15 @@
+/* PR target/109566 */
+/* { dg-do compile } */
+/* { dg-options "-O2 -mpowerpc64" } */
+
+void
+foo (double x)
+{
+  union { double d; unsigned i; } u;
+  u.d = x;
+  if (u.i & 0x7ff00000)
+    return;
+  else
+    for (;;)
+      ;
+}

	Jakub