[Bug tree-optimization/94727] [10 Regression] GCC produces incorrect code with -O3 since r10-5071-g02d895504cc59be0

["rsandifo at gcc dot gnu.org" <gcc-bugzilla@gcc.gnu.org>]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94727

--- Comment #5 from rsandifo at gcc dot gnu.org <rsandifo at gcc dot gnu.org> ---
Well, this is a bit of mess (surprise).  We have a "<" comparison
between two booleans that are leaves of the SLP tree, so
vectorizable_comparison falls back on:

  /* Invariant comparison.  */
  if (!vectype)
    {
      vectype = get_vectype_for_scalar_type (vinfo, TREE_TYPE (rhs1),
                                             slp_node);
      if (maybe_ne (TYPE_VECTOR_SUBPARTS (vectype), nunits))
        return false;
    }

rhs1 and rhs2 are *unsigned* boolean types, so we get back a vector
of unsigned integers.  All is well, and "<" works as expected without
the need for:

  /* Boolean values may have another representation in vectors
     and therefore we prefer bit operations over comparison for
     them (which also works for scalar masks).  We store opcodes
     to use in bitop1 and bitop2.  Statement is vectorized as
       BITOP2 (rhs1 BITOP1 rhs2) or
       rhs1 BITOP2 (BITOP1 rhs2)
     depending on bitop1 and bitop2 arity.  */
  bool swap_p = false;
  if (VECTOR_BOOLEAN_TYPE_P (vectype))
    {

However, we then defer to vect_get_slp_defs to get the actual operands.
The expected vector type is not part of this interface.  The request
goes to vect_get_constant_vectors, which does:

  if (VECT_SCALAR_BOOLEAN_TYPE_P (TREE_TYPE (op))
      && vect_mask_constant_operand_p (stmt_vinfo))
    vector_type = truth_type_for (stmt_vectype);
  else
    vector_type = get_vectype_for_scalar_type (vinfo, TREE_TYPE (op), op_node);

So the function gives back a vector of mask types, which here are
vectors of *signed* booleans.  This means that "<" gives:

  true (-1) < false (0)

and so the boolean fixup above was needed after all.

I'm going to try:

diff --git a/gcc/tree-vect-stmts.c b/gcc/tree-vect-stmts.c
index 7f3a9fb5fb3..88a1e2c51d2 100644
--- a/gcc/tree-vect-stmts.c
+++ b/gcc/tree-vect-stmts.c
@@ -10566,8 +10566,11 @@ vectorizable_comparison (stmt_vec_info stmt_info,
gimple_stmt_iterator *gsi,
   /* Invariant comparison.  */
   if (!vectype)
     {
-      vectype = get_vectype_for_scalar_type (vinfo, TREE_TYPE (rhs1),
-                                            slp_node);
+      if (VECT_SCALAR_BOOLEAN_TYPE_P (TREE_TYPE (rhs1)))
+       vectype = mask_type;
+      else
+       vectype = get_vectype_for_scalar_type (vinfo, TREE_TYPE (rhs1),
+                                              slp_node);
       if (!vectype || maybe_ne (TYPE_VECTOR_SUBPARTS (vectype), nunits))
        return false;
     }

which does at least fix the testcase.