[Bug tree-optimization/104595] unvectorized loop due to bool condition loaded from memory

["rguenth at gcc dot gnu.org" <gcc-bugzilla@gcc.gnu.org>]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104595

--- Comment #6 from Richard Biener <rguenth at gcc dot gnu.org> ---
The patch doesn't really work, it regresses some AVX512 stuff in i386.exp,
notably

FAIL: gcc.target/i386/avx512vl-vdivps-2.c (internal compiler error: in
vect_get_
vec_defs_for_operand, at tree-vect-stmts.cc:1508)
...
FAIL: gcc.target/i386/mask-pack-prefer128.c scan-tree-dump-times vect
"vectorized 1 loops" 10
FAIL: gcc.target/i386/mask-pack-prefer256.c scan-tree-dump-times vect
"vectorized 1 loops" 10
FAIL: gcc.target/i386/mask-pack.c scan-tree-dump-times vect "vectorized 1
loops" 10
FAIL: gcc.target/i386/mask-unpack.c scan-tree-dump-times vect "vectorized 1
loops" 10


There's an additional case:

#define N 256
typedef short T;
extern T a[N];
extern T b[N];
extern T c[N];
extern _Bool pb[N];

void predicate_by_bool()
{
  for (int i = 0; i < N; i++)
    c[i] = pb[i] ? a[i] : b[i];
}

where we expect vect_recog_mask_conversion_pattern to trigger here.  That
case can be fixed with

@@ -4658,9 +4660,9 @@ vect_recog_mask_conversion_pattern (vec_info *vinfo,

       if (TREE_CODE (rhs1) == SSA_NAME)
        {
-         rhs1_type = integer_type_for_mask (rhs1, vinfo);
-         if (!rhs1_type)
+         if (integer_type_for_mask (rhs1, vinfo))
            return NULL;
+         rhs1_type = TREE_TYPE (rhs1);
        }
       else if (COMPARISON_CLASS_P (rhs1))
        {

but we then run into the original issue again:

t.c:10:6: missed:   not vectorized: relevant stmt not supported: patt_28 =
(<signed-boolean:16>) _1;

The cruical difference between working and not working is the _1 != 0 ?: vs.
_1 ?:  - I think we do have a duplicate bugreport here, possibly involving
combinations of different from memory bools.

Trying to make bool pattern recog inserting the relevant compensation code
is really iffy, the mask conversion pattern confuses things here - what
we are missing seems really be transforming the leafs.

Note that we do not try to patter-recog a pattern thus we cannot at the moment
have both, vect_recog_bool_pattern and vect_recog_mask_conversion_pattern
at the same time on the ?: stmt.

Note IIRC vect_recog_bool_pattern has to come last but it's now after
vect_recog_mask_conversion_pattern.  Unfortunately swapping things does
not make vect_recog_bool_pattern recognize and fixup

  patt_24 = (<signed-boolean:16>) _1;