[Bug tree-optimization/110461] New: [14 regression] ICE when building openh264 with new vector_type checking

[Bug tree-optimization/110461] New: [14 regression] ICE when building openh264 with new vector_type checking

[sjames at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110461

            Bug ID: 110461
           Summary: [14 regression] ICE when building openh264 with new
                    vector_type checking
           Product: gcc
           Version: 14.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: tree-optimization
          Assignee: unassigned at gcc dot gnu.org
          Reporter: sjames at gcc dot gnu.org
  Target Milestone: ---

Created attachment 55414
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=55414&action=edit
encode_mb_aux.ii

g++ -c encode_mb_aux.ii -O2 is enough to repro.

```
aarch64-unknown-linux-gnu-g++ -O3 -pipe -mcpu=native -fdiagnostics-color=always
 -DNDEBUG -DHAVE_NEON_AARCH64 -Wall -fno-strict-aliasing -fPIC -MMD -MP
-fstack-protector-all -march=armv8-a -DGENERATED_VERSION_HEADER -O3 -pipe
-mcpu=native -fdiagnostics-color=always -Wno-class-memaccess -I./codec/api/wels
-I./codec/common/inc -Icodec/common/inc  -I./codec/encoder/core/inc
-I./codec/encoder/plus/inc -I./codec/processing/interface -c -o
codec/encoder/core/src/encode_mb_aux.o codec/encoder/core/src/encode_mb_aux.cpp
during GIMPLE pass: vect
codec/encoder/core/src/encode_mb_aux.cpp: In function ‘void
WelsEnc::WelsQuantFour4x4Max_c(int16_t*, const int16_t*, const int16_t*,
int16_t*)’:
codec/encoder/core/src/encode_mb_aux.cpp:209:6: internal compiler error: tree
check: expected none of vector_type, have vector_type in gimple_simplify_144,
at gimple-match-3.cc:1027
  209 | void WelsQuantFour4x4Max_c (int16_t* pDct, const int16_t* pFF, const
int16_t* pMF, int16_t* pMax) {
      |      ^~~~~~~~~~~~~~~~~~~~~
0xaaaac959b02b tree_not_check_failed(tree_node const*, char const*, int, char
const*, ...)
       
/usr/src/debug/sys-devel/gcc-14.0.0.9999/gcc-14.0.0.9999/gcc/tree.cc:8936
0xaaaacb1c6c53 tree_not_check(tree_node*, char const*, int, char const*,
tree_code)
       
/usr/src/debug/sys-devel/gcc-14.0.0.9999/gcc-14.0.0.9999/gcc/tree.h:3581
0xaaaacb1c6c53 gimple_simplify_144(gimple_match_op*, gimple**, tree_node*
(*)(tree_node*), tree_node*, tree_node**, tree_code)
       
/usr/src/debug/sys-devel/gcc-14.0.0.9999/build/gcc/gimple-match-3.cc:1027
0xaaaacb1a21bb gimple_simplify_BIT_XOR_EXPR(gimple_match_op*, gimple**,
tree_node* (*)(tree_node*), code_helper, tree_node*, tree_node*, tree_node*)
       
/usr/src/debug/sys-devel/gcc-14.0.0.9999/build/gcc/gimple-match-2.cc:9569
0xaaaaca74debf gimple_resimplify2
       
/usr/src/debug/sys-devel/gcc-14.0.0.9999/gcc-14.0.0.9999/gcc/gimple-match-exports.cc:967
0xaaaaca74e687 gimple_simplify(gimple*, gimple_match_op*, gimple**, tree_node*
(*)(tree_node*), tree_node* (*)(tree_node*))
       
/usr/src/debug/sys-devel/gcc-14.0.0.9999/gcc-14.0.0.9999/gcc/gimple-match-exports.cc:834
0xaaaac9c8e5b7 gimple_fold_stmt_to_constant_1(gimple*, tree_node*
(*)(tree_node*), tree_node* (*)(tree_node*))
       
/usr/src/debug/sys-devel/gcc-14.0.0.9999/gcc-14.0.0.9999/gcc/gimple-fold.cc:7472
0xaaaaca2cfe97 try_to_simplify
       
/usr/src/debug/sys-devel/gcc-14.0.0.9999/gcc-14.0.0.9999/gcc/tree-ssa-sccvn.cc:6096
0xaaaaca2cfe97 visit_stmt
       
/usr/src/debug/sys-devel/gcc-14.0.0.9999/gcc-14.0.0.9999/gcc/tree-ssa-sccvn.cc:6139
0xaaaaca2d0e6f process_bb
       
/usr/src/debug/sys-devel/gcc-14.0.0.9999/gcc-14.0.0.9999/gcc/tree-ssa-sccvn.cc:7945
0xaaaaca2d2823 do_rpo_vn_1
       
/usr/src/debug/sys-devel/gcc-14.0.0.9999/gcc-14.0.0.9999/gcc/tree-ssa-sccvn.cc:8544
0xaaaaca2d448f do_rpo_vn(function*, edge_def*, bitmap_head*, bool, bool,
vn_lookup_kind)
       
/usr/src/debug/sys-devel/gcc-14.0.0.9999/gcc-14.0.0.9999/gcc/tree-ssa-sccvn.cc:8646
0xaaaaca3ba1f3 execute
       
/usr/src/debug/sys-devel/gcc-14.0.0.9999/gcc-14.0.0.9999/gcc/tree-vectorizer.cc:1385
Please submit a full bug report, with preprocessed source (by using
-freport-bug).
Please include the complete backtrace with any bug report.
See <https://bugs.gentoo.org/> for instructions.
```

```
gcc (Gentoo 14.0.0 p, commit 6cb33e2f39e289ec4f25f845d8153053147c5c49) 14.0.0
20230628 (experimental) c7e87e82435b918084f305386b12b8fbcdcf3307
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
```

[Bug tree-optimization/110461] [14 regression] ICE when building openh264 with new vector_type checking

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110461

--- Comment #1 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
I think it is:
/* Try to fold (type) X op CST -> (type) (X op ((type-x) CST))

[Bug tree-optimization/110461] [14 regression] ICE when building openh264 with new vector_type checking

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110461

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |14.0
           Keywords|                            |ice-on-valid-code

[Bug tree-optimization/110461] [14 regression] ICE when building openh264 with new vector_type checking

[sjames at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110461

--- Comment #2 from Sam James <sjames at gcc dot gnu.org> ---
Created attachment 55417
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=55417&action=edit
reduced.ii

[Bug tree-optimization/110461] [14 regression] ICE when building openh264 with new vector_type checking

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110461

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
     Ever confirmed|0                           |1
   Last reconfirmed|                            |2023-06-28

--- Comment #3 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
Slightly different reduced testcase:
```
void WelsQuantFour4x4Max_c(short *pDct, int *iSign1) {
  for (int k = 0; k < 4; k++) {
    int iSign = iSign1[k];
    for (int i = 0; i < 16; i++) {
      short t = (iSign ^ (int)pDct[i]) >> 16;
      pDct[i] = (iSign ^ (int)t);
    }
    pDct += 16;
  }
}

```

[Bug tree-optimization/110461] [14 regression] ICE when building openh264 with new vector_type checking

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110461

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |rguenth at gcc dot gnu.org

--- Comment #4 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
Yes it is that pattern, specifically :
/* Try to fold (type) X op CST -> (type) (X op ((type-x) CST))
   when profitable.
   For bitwise binary operations apply operand conversions to the
   binary operation result instead of to the operands.  This allows
   to combine successive conversions and bitwise binary operations.
   We combine the above two cases by using a conditional convert.  */
(for bitop (bit_and bit_ior bit_xor)
 (simplify
  (bitop (convert@2 @0) (convert?@3 @1))
  (if (((TREE_CODE (@1) == INTEGER_CST
         && INTEGRAL_TYPE_P (TREE_TYPE (@0))
         && (int_fits_type_p (@1, TREE_TYPE (@0))
             || tree_nop_conversion_p (TREE_TYPE (@0), type)))
        || types_match (@0, @1))
       && !POINTER_TYPE_P (TREE_TYPE (@0))
       && TREE_CODE (TREE_TYPE (@0)) != OFFSET_TYPE
       /* ???  This transform conflicts with fold-const.cc doing
          Convert (T)(x & c) into (T)x & (T)c, if c is an integer
          constants (if x has signed type, the sign bit cannot be set
          in c).  This folds extension into the BIT_AND_EXPR.
          Restrict it to GIMPLE to avoid endless recursions.  */
       && (bitop != BIT_AND_EXPR || GIMPLE)
       && (/* That's a good idea if the conversion widens the operand, thus
              after hoisting the conversion the operation will be narrower.
              It is also a good if the conversion is a nop as moves the
              conversion to one side; allowing for combining of the
conversions.  */
           TYPE_PRECISION (TREE_TYPE (@0)) < TYPE_PRECISION (type)
           /* The conversion check for being a nop can only be done at the
gimple
              level as fold_binary has some re-association code which can
conflict
              with this if there is a "constant" which is not a full
INTEGER_CST.  */
           || (GIMPLE && TYPE_PRECISION (TREE_TYPE (@0)) == TYPE_PRECISION
(type))

Those 2 above TYPE_PRECISION .

[Bug tree-optimization/110461] [14 regression] ICE when building openh264 with new vector_type checking

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110461

--- Comment #5 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
Here is a better testcase which fails even at -O1 and does not depend on the
vectorizer:
```
typedef int v4si __attribute__ ((vector_size (4*sizeof(int))));
typedef short v4hi __attribute__ ((vector_size (4*sizeof(short))));

v4hi f(v4hi a, v4hi b)
{
        auto t = __builtin_convertvector (a, v4si);
        auto t1 = __builtin_convertvector (b, v4si);
        t ^= t1;
        return __builtin_convertvector (t, v4hi);
}
```

[Bug tree-optimization/110461] [14 regression] ICE when building openh264 with new vector_type checking

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110461

--- Comment #6 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
*** Bug 110465 has been marked as a duplicate of this bug. ***

[Bug tree-optimization/110461] [14 regression] ICE when building openh264 with new vector_type checking

[dcb314 at hotmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110461

David Binderman <dcb314 at hotmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dcb314 at hotmail dot com

--- Comment #7 from David Binderman <dcb314 at hotmail dot com> ---
I get something similar with -O1 -march=znver1. 

The problem seems to have started in the last couple of days:

../results.20230626.asan.ubsan/bin/gcc
../results.20230628.asan.ubsan/bin/gcc
during GIMPLE pass: fre
buildData/keep/in.14426.c: In function ‘func_63’:
buildData/keep/in.14426.c:465:18: internal compiler error: tree check: expected
none of vector_type, have vector_type in gimple_simplify_144, at
gimple-match-3.cc:1027
0x11656f9 tree_not_check_failed(tree_node const*, char const*, int, char
const*, ...)
        ../../trunk.year/gcc/tree.cc:8936
0x1dc7c8b gimple_simplify_144(gimple_match_op*, gimple**, tree_node*
(*)(tree_node*), tree_node*, tree_node**, tree_code)
        /home/dcb38/gcc/working/gcc/gimple-match-3.cc:1027

[Bug tree-optimization/110461] [14 regression] ICE when building openh264 with new vector_type checking

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110461

--- Comment #8 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
(In reply to David Binderman from comment #7)
> I get something similar with -O1 -march=znver1. 
> 
> The problem seems to have started in the last couple of days:

Yes it is exactly the same issue. Also it started with r14-2150-gfe48f2651334bc
(which just added the extra checking to make sure TYPE_PRECISION was not being
used for vector types).

[Bug tree-optimization/110461] [14 regression] ICE when building openh264 with new vector_type checking

[seurer at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110461

seurer at gcc dot gnu.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |seurer at gcc dot gnu.org

--- Comment #9 from seurer at gcc dot gnu.org ---
I am seeing probably the same error with one gcc test case and it started with 

g:fe48f2651334bc4d96b6df6b2bb6b29fcb732a83, r14-2150-gfe48f2651334bc

as that wasn't mentioned here that I saw.

make  -k check-gcc RUNTESTFLAGS="compile.exp=gcc.c-torture/compile/pr46883.c"
FAIL: gcc.c-torture/compile/pr46883.c   -O3 -fomit-frame-pointer -funroll-loops
-fpeel-loops -ftracer -finline-functions  (internal compiler error: tree check:
expected none of vector_type, have vector_type in vect_recog_rotate_pattern, at
tree-vect-patterns.cc:3634)
FAIL: gcc.c-torture/compile/pr46883.c   -O3 -fomit-frame-pointer -funroll-loops
-fpeel-loops -ftracer -finline-functions  (test for excess errors)
FAIL: gcc.c-torture/compile/pr46883.c   -O3 -g  (internal compiler error: tree
check: expected none of vector_type, have vector_type in
vect_recog_rotate_pattern, at tree-vect-patterns.cc:3634)
FAIL: gcc.c-torture/compile/pr46883.c   -O3 -g  (test for excess errors)
# of expected passes            6
# of unexpected failures        4

spawn -ignore SIGHUP /home/seurer/gcc/git/build/gcc-test/gcc/xgcc
-B/home/seurer/gcc/git/build/gcc-test/gcc/ -fdiagnostics-plain-output -O3
-fomit-frame-pointer -funroll-loops -fpeel-loops -ftracer -finline-functions -w
-c -o pr46883.o
/home/seurer/gcc/git/gcc-test/gcc/testsuite/gcc.c-torture/compile/pr46883.c
during GIMPLE pass: slp
/home/seurer/gcc/git/gcc-test/gcc/testsuite/gcc.c-torture/compile/pr46883.c: In
function 'bar':
/home/seurer/gcc/git/gcc-test/gcc/testsuite/gcc.c-torture/compile/pr46883.c:1:6:
internal compiler error: tree check: expected none of vector_type, have
vector_type in vect_recog_rotate_pattern, at tree-vect-patterns.cc:3634
0x10291b5f tree_not_check_failed(tree_node const*, char const*, int, char
const*, ...)
        /home/seurer/gcc/git/gcc-test/gcc/tree.cc:8936
0x11b9f833 tree_not_check(tree_node*, char const*, int, char const*, tree_code)
        /home/seurer/gcc/git/gcc-test/gcc/tree.h:3581
0x11b9f833 vect_recog_rotate_pattern
        /home/seurer/gcc/git/gcc-test/gcc/tree-vect-patterns.cc:3634
0x11b9cdeb vect_pattern_recog_1
        /home/seurer/gcc/git/gcc-test/gcc/tree-vect-patterns.cc:6834
0x11ba5387 vect_pattern_recog(vec_info*)
        /home/seurer/gcc/git/gcc-test/gcc/tree-vect-patterns.cc:6991
0x110a001f vect_slp_analyze_bb_1
        /home/seurer/gcc/git/gcc-test/gcc/tree-vect-slp.cc:7353
0x110a001f vect_slp_region
        /home/seurer/gcc/git/gcc-test/gcc/tree-vect-slp.cc:7462
0x110a2733 vect_slp_bbs
        /home/seurer/gcc/git/gcc-test/gcc/tree-vect-slp.cc:7653
0x110a2b7b vect_slp_function(function*)
        /home/seurer/gcc/git/gcc-test/gcc/tree-vect-slp.cc:7754
0x110b1b73 execute
        /home/seurer/gcc/git/gcc-test/gcc/tree-vectorizer.cc:1529

[Bug tree-optimization/110461] [14 regression] ICE when building openh264 with new vector_type checking

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110461

--- Comment #10 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
I audited match.pd and only one other sticks out but I can't get get it to
happen.
The other one should have happened with:
```
typedef int v4si __attribute__ ((vector_size (4*sizeof(int))));
typedef short v4hi __attribute__ ((vector_size (4*sizeof(short))));

v4si f(v4si a, v4si b)
{
  v4hi t = __builtin_convertvector (a, v4hi);
  v4si t1 = __builtin_convertvector (t, v4si);
  return t1;
}
```

But VEC_CONVERT gets lowered to something interesting (which I will put in a
different bug report).

[Bug tree-optimization/110461] [14 regression] ICE when building openh264 with new vector_type checking

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110461

--- Comment #11 from Richard Biener <rguenth at gcc dot gnu.org> ---
(In reply to Andrew Pinski from comment #4)
> Yes it is that pattern, specifically :
> /* Try to fold (type) X op CST -> (type) (X op ((type-x) CST))
>    when profitable.
>    For bitwise binary operations apply operand conversions to the
>    binary operation result instead of to the operands.  This allows
>    to combine successive conversions and bitwise binary operations.
>    We combine the above two cases by using a conditional convert.  */
> (for bitop (bit_and bit_ior bit_xor)
>  (simplify
>   (bitop (convert@2 @0) (convert?@3 @1))
>   (if (((TREE_CODE (@1) == INTEGER_CST
>          && INTEGRAL_TYPE_P (TREE_TYPE (@0))
>          && (int_fits_type_p (@1, TREE_TYPE (@0))
>              || tree_nop_conversion_p (TREE_TYPE (@0), type)))
>         || types_match (@0, @1))
>        && !POINTER_TYPE_P (TREE_TYPE (@0))
>        && TREE_CODE (TREE_TYPE (@0)) != OFFSET_TYPE
>        /* ???  This transform conflicts with fold-const.cc doing
>           Convert (T)(x & c) into (T)x & (T)c, if c is an integer
>           constants (if x has signed type, the sign bit cannot be set
>           in c).  This folds extension into the BIT_AND_EXPR.
>           Restrict it to GIMPLE to avoid endless recursions.  */
>        && (bitop != BIT_AND_EXPR || GIMPLE)
>        && (/* That's a good idea if the conversion widens the operand, thus
>               after hoisting the conversion the operation will be narrower.
>               It is also a good if the conversion is a nop as moves the
>               conversion to one side; allowing for combining of the
> conversions.  */
>            TYPE_PRECISION (TREE_TYPE (@0)) < TYPE_PRECISION (type)
>            /* The conversion check for being a nop can only be done at the
> gimple
>               level as fold_binary has some re-association code which can
> conflict
>               with this if there is a "constant" which is not a full
> INTEGER_CST.  */
>            || (GIMPLE && TYPE_PRECISION (TREE_TYPE (@0)) == TYPE_PRECISION
> (type))
> 
> Those 2 above TYPE_PRECISION .

For vectors this also lacks a check the resulting bit operation is supported.
I'm testing a patch.

[Bug tree-optimization/110461] [14 regression] ICE when building openh264 with new vector_type checking

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110461

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED
           Assignee|unassigned at gcc dot gnu.org      |rguenth at gcc dot gnu.org

[Bug tree-optimization/110461] [14 regression] ICE when building openh264 with new vector_type checking

[dcb314 at hotmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110461

--- Comment #12 from David Binderman <dcb314 at hotmail dot com> ---
Here is a test case which seems to demonstrate the problem:

$ /home/dcb38/gcc/results/bin/gcc -c -w -O1 -march=znver1 bug935.c
during GIMPLE pass: fre
bug935.c: In function ‘func_63’:
bug935.c:5:6: internal compiler error: tree check: expected none of
vector_type,
 have vector_type in gimple_simplify_144, at gimple-match-3.cc:1027
    5 | void func_63() {
      |      ^~~~~~~
0x11656f9 tree_not_check_failed(tree_node const*, char const*, int, char
const*,
 ...)
        ../../trunk.year/gcc/tree.cc:8936
0x1dc7c8b gimple_simplify_144(gimple_match_op*, gimple**, tree_node*
(*)(tree_no
de*), tree_node*, tree_node**, tree_code)
        /home/dcb38/gcc/working/gcc/gimple-match-3.cc:1027
0x1db5e1a gimple_simplify_BIT_AND_EXPR(gimple_match_op*, gimple**, tree_node*
(*
)(tree_node*), code_helper, tree_node*, tree_node*, tree_node*)
        /home/dcb38/gcc/working/gcc/gimple-match-2.cc:16999

[Bug tree-optimization/110461] [14 regression] ICE when building openh264 with new vector_type checking

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110461

--- Comment #13 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Richard Biener <rguenth@gcc.gnu.org>:

https://gcc.gnu.org/g:1e6f1659bd7337e91a88086f8092ada01e80ac94

commit r14-2182-g1e6f1659bd7337e91a88086f8092ada01e80ac94
Author: Richard Biener <rguenther@suse.de>
Date:   Thu Jun 29 09:15:27 2023 +0200

    middle-end/110461 - pattern applying wrongly to vectors

    The following guards a match.pd pattern that wasn't supposed to
    apply to vectors and thus runs into TYPE_PRECISION checking.  For
    vector support the constant case is lacking and the pattern would
    have missing optab support checking for the result operation.

            PR middle-end/110461
            * match.pd (bitop (convert@2 @0) (convert?@3 @1)): Disable
            for VECTOR_TYPE_P.

            * gcc.dg/pr110461.c: New testcase.

[Bug tree-optimization/110461] [14 regression] ICE when building openh264 with new vector_type checking

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110461

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|---                         |FIXED

--- Comment #14 from Richard Biener <rguenth at gcc dot gnu.org> ---
Fixed.  Please report persisting unrelated issues separately.