[Bug tree-optimization/105490] unvectorized loop due to bool condition loaded from memory and different size data

["rguenther at suse dot de" <gcc-bugzilla@gcc.gnu.org>]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105490

--- Comment #3 from rguenther at suse dot de <rguenther at suse dot de> ---
On Wed, 30 Aug 2023, pinskia at gcc dot gnu.org wrote:

> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105490
> 
> --- Comment #2 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
> Here is an even odder case:
> ```
> #define N 256
> typedef short T;
> extern T a[N];
> extern T b[N];
> extern T c[N];
> extern _Bool pb[N];
> extern _Bool pb1[N];
> extern _Bool pb2[N];
> 
> void predicate_by_booland()
> {
>   for (int i = 0; i < N; i++)
>     c[i] = ((pb1[i] != pb[i]) != pb2[i]) ? a[i] : b[i];
> }
> ```
> This vectorizes currently with `-O3` but not with `-O3 -fno-tree-vrp`.
> IR with -fno-tree-vrp:
> ```
>   _1 = pb1[i_15];
>   _2 = pb[i_15];
>   _3 = _1 != _2;
>   _4 = pb2[i_15];
>   iftmp.0_10 = a[i_15];
>   _5 = _3 != _4;
>   iftmp.0_9 = b[i_15];
> ```
> IR without (VRP turned on):
> ```
>   _1 = pb1[i_15];
>   _2 = pb[i_15];
>   _3 = _1 ^ _2;
>   _4 = pb2[i_15];
>   iftmp.0_10 = a[i_15];
>   _5 = _3 != _4;
>   iftmp.0_9 = b[i_15];
> ```
> 
> So it is even more confusing ...

This is usually due to limits/bugs in the vectorizer bool pattern
recognition code which is supposed to replace "mask" uses of
bool with x ? -1 : 0 (or make sure comparisons produce them)
and data uses with x ? 1 : 0 (or make sure "data" stmts produce them).
There are defenses in vectorizable_* to detect cases where that did
go wrong (which it does sometimes), leading to missed vectorizations
(or wrong code in the worst case).

It's a mess.