[Bug tree-optimization/107096] New: Fully masking vectorization with AVX512 ICEs gcc.dg/vect/vect-over-widen-*.c

["rguenth at gcc dot gnu.org" <gcc-bugzilla@gcc.gnu.org>]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107096

            Bug ID: 107096
           Summary: Fully masking vectorization with AVX512 ICEs
                    gcc.dg/vect/vect-over-widen-*.c
           Product: gcc
           Version: 13.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: tree-optimization
          Assignee: unassigned at gcc dot gnu.org
          Reporter: rguenth at gcc dot gnu.org
  Target Milestone: ---

With like

/home/rguenther/src/trunk/gcc/testsuite/gcc.dg/vect/vect-over-widen-2.c:11:1:
error: conversion of register to a different size in 'view_convert_expr'
VIEW_CONVERT_EXPR<vector(8) <signed-boolean:1>>(loop_mask_11);

_164 = VIEW_CONVERT_EXPR<vector(8) <signed-boolean:1>>(loop_mask_11);
/home/rguenther/src/trunk/gcc/testsuite/gcc.dg/vect/vect-over-widen-2.c:11:1:
error: conversion of register to a different size in 'view_convert_expr'
VIEW_CONVERT_EXPR<vector(8) <signed-boolean:1>>(loop_mask_8);

_167 = VIEW_CONVERT_EXPR<vector(8) <signed-boolean:1>>(loop_mask_8);
/home/rguenther/src/trunk/gcc/testsuite/gcc.dg/vect/vect-over-widen-2.c:11:1:
error: conversion of register to a different size in 'view_convert_expr'
VIEW_CONVERT_EXPR<vector(8) <signed-boolean:1>>(loop_mask_118);

_170 = VIEW_CONVERT_EXPR<vector(8) <signed-boolean:1>>(loop_mask_118);
/home/rguenther/src/trunk/gcc/testsuite/gcc.dg/vect/vect-over-widen-2.c:11:1:
error: conversion of register to a different size in 'view_convert_expr'
VIEW_CONVERT_EXPR<vector(8) <signed-boolean:1>>(loop_mask_119);

_173 = VIEW_CONVERT_EXPR<vector(8) <signed-boolean:1>>(loop_mask_119);
during GIMPLE pass: vect
dump file: ./vect-over-widen-2.c.172t.vect
/home/rguenther/src/trunk/gcc/testsuite/gcc.dg/vect/vect-over-widen-2.c:11:1:
internal compiler error: verify_gimple failed


The issue is that in vect_get_loop_mask we do

  if (maybe_ne (TYPE_VECTOR_SUBPARTS (mask_type),
                TYPE_VECTOR_SUBPARTS (vectype)))
    {
      /* A loop mask for data type X can be reused for data type Y
         if X has N times more elements than Y and if Y's elements
         are N times bigger than X's.  In this case each sequence
         of N elements in the loop mask will be all-zero or all-one.
         We can then view-convert the mask so that each sequence of
         N elements is replaced by a single element.  */
      gcc_assert (multiple_p (TYPE_VECTOR_SUBPARTS (mask_type),
                              TYPE_VECTOR_SUBPARTS (vectype)));
      gimple_seq seq = NULL;
      mask_type = truth_type_for (vectype);
      mask = gimple_build (&seq, VIEW_CONVERT_EXPR, mask_type, mask);

but that ends up VIEW_CONVERTing vector(16) <signed-boolean:1> to
vector(8) <signed-boolean:1>, in this case a HImode mask to a QImode mask.

If I understand the comment correctly then it wants to re-use a mask
for V16QI for V8HI indicating that the mask bits should be set in pairs
for the V16QImask.  But then I don't understand how VIEW_CONVERTing
should perform the desired conversion.  It might be OK if the mask is
a byte mask (like for classic AVX2) but for AVX512 it would require
unpacking even/odd bits.

The actual error is likely in the setup of the loop masks though, thinking
we could do this re-use.