[gcc r11-9523] tree-optimization/103489 - fix ICE when bool pattern recog fails

[Richard Biener <rguenth@gcc.gnu.org>]

https://gcc.gnu.org/g:2c7d8ca8317981c2eb21eb1e85e0f55d3f71aff1

commit r11-9523-g2c7d8ca8317981c2eb21eb1e85e0f55d3f71aff1
Author: Richard Biener <rguenther@suse.de>
Date:   Tue Nov 30 14:08:19 2021 +0100

    tree-optimization/103489 - fix ICE when bool pattern recog fails
    
    bool pattern recog currently does not handle cycles correctly
    and when it fails we can ICE later vectorizing PHIs with
    mismatched bool and non-bool vector types.  The following avoids
    blindly trusting bool pattern recog here and verifies things
    more thoroughly in vectorizable_phi.  A bool pattern recog fix
    is for GCC 13.
    
    2021-11-30  Richard Biener  <rguenther@suse.de>
    
            PR tree-optimization/103489
            * tree-vect-loop.c (vectorizable_phi): Verify argument
            vector type compatibility to mitigate bool pattern recog
            bug.
    
            * gcc.dg/torture/pr103489.c: New testcase.
    
    (cherry picked from commit 0194d92c35ca8b3aa850b805d9becb4491cf6bec)

Diff:
---
 gcc/testsuite/gcc.dg/torture/pr103489.c | 12 ++++++++++++
 gcc/tree-vect-loop.c                    | 18 ++++++++++++++++++
 2 files changed, 30 insertions(+)

diff --git a/gcc/testsuite/gcc.dg/torture/pr103489.c b/gcc/testsuite/gcc.dg/torture/pr103489.c
new file mode 100644
index 00000000000..cd62623ece2
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/torture/pr103489.c
@@ -0,0 +1,12 @@
+/* { dg-do compile } */
+/* { dg-additional-options "-ftree-vectorize" } */
+
+_Bool a[80];
+short b, f;
+void g(short h[][8][16])
+{
+  for (_Bool c = 0; c < b;)
+    for (_Bool d = 0; d < (_Bool)f; d = 1)
+      for (short e = 0; e < 16; e++)
+        a[e] = h[b][1][e];
+}
diff --git a/gcc/tree-vect-loop.c b/gcc/tree-vect-loop.c
index b77b4f7f83f..468b9aed68c 100644
--- a/gcc/tree-vect-loop.c
+++ b/gcc/tree-vect-loop.c
@@ -7792,6 +7792,24 @@ vectorizable_phi (vec_info *,
 			       "incompatible vector types for invariants\n");
 	    return false;
 	  }
+	else if (SLP_TREE_DEF_TYPE (child) == vect_internal_def
+		 && !useless_type_conversion_p (vectype,
+						SLP_TREE_VECTYPE (child)))
+	  {
+	    /* With bools we can have mask and non-mask precision vectors,
+	       while pattern recog is supposed to guarantee consistency here
+	       bugs in it can cause mismatches (PR103489 for example).
+	       Deal with them here instead of ICEing later.  */
+	    if (dump_enabled_p ())
+	      dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
+			       "incompatible vector type setup from "
+			       "bool pattern detection\n");
+	    gcc_checking_assert
+	      (VECTOR_BOOLEAN_TYPE_P (SLP_TREE_VECTYPE (child))
+	       != VECTOR_BOOLEAN_TYPE_P (vectype));
+	    return false;
+	  }
+
       /* For single-argument PHIs assume coalescing which means zero cost
 	 for the scalar and the vector PHIs.  This avoids artificially
 	 favoring the vector path (but may pessimize it in some cases).  */