[Bug tree-optimization/114231] New: [14 regression] ICE when building libjxl

[Bug tree-optimization/114231] New: [14 regression] ICE when building libjxl

[sjames at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114231

            Bug ID: 114231
           Summary: [14 regression] ICE when building libjxl
           Product: gcc
           Version: 14.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: tree-optimization
          Assignee: unassigned at gcc dot gnu.org
          Reporter: sjames at gcc dot gnu.org
  Target Milestone: ---

Created attachment 57606
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=57606&action=edit
enc_modular.ii.xz

Originally reported downstream in Gentoo by tdr.

```
$ g++ -c enc_modular.ii -mrtm -mshstk -march=sapphirerapids -O3
-fno-vect-cost-model
during GIMPLE pass: slp
/var/tmp/portage/media-libs/libjxl-0.9.1-r1/work/libjxl-0.9.1/lib/jxl/enc_modular.cc:
In member function ‘jxl::Status
jxl::ModularFrameEncoder::PrepareStreamParams(const jxl::Rect&, const
jxl::CompressParams&, int, int, const jxl::ModularStreamId&, bool)’:
/var/tmp/portage/media-libs/libjxl-0.9.1-r1/work/libjxl-0.9.1/lib/jxl/enc_modular.cc:1294:8:
internal compiler error: in vect_transform_stmt, at tree-vect-stmts.cc:13361
 1294 | Status ModularFrameEncoder::PrepareStreamParams(const Rect& rect,
      |        ^~~~~~~~~~~~~~~~~~~
0x55bdf7edf2b0 vect_transform_stmt(vec_info*, _stmt_vec_info*,
gimple_stmt_iterator*, _slp_tree*, _slp_instance*)
       
/usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/tree-vect-stmts.cc:13361
0x55bdf99cd64d vect_schedule_slp_node
       
/usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/tree-vect-slp.cc:9410
0x55bdf99cd64d vect_schedule_slp_node
       
/usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/tree-vect-slp.cc:9203
0x55bdf99ccaa2 vect_schedule_scc
       
/usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/tree-vect-slp.cc:9645
0x55bdf96e6382 vect_schedule_slp(vec_info*, vec<_slp_instance*, va_heap,
vl_ptr> const&)
       
/usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/tree-vect-slp.cc:9790
0x55bdf9468b27 vect_slp_region
       
/usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/tree-vect-slp.cc:7911
0x55bdf94648c3 vect_slp_bbs
       
/usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/tree-vect-slp.cc:8011
0x55bdf94626be vect_slp_function(function*)
       
/usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/tree-vect-slp.cc:8127
0x55bdf9461e1c execute
       
/usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/tree-vectorizer.cc:1533
Please submit a full bug report, with preprocessed source (by using
-freport-bug).
Please include the complete backtrace with any bug report.
See <https://bugs.gentoo.org/> for instructions.
```

```
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-pc-linux-gnu/14/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with:
/var/tmp/portage/sys-devel/gcc-14.0.9999/work/gcc-14.0.9999/configure
--host=x86_64-pc-linux-gnu --build=x86_64-pc-linux-gnu --prefix=/usr
--bindir=/usr/x86_64-pc-linux-gnu/gcc-bin/14
--includedir=/usr/lib/gcc/x86_64-pc-linux-gnu/14/include
--datadir=/usr/share/gcc-data/x86_64-pc-linux-gnu/14
--mandir=/usr/share/gcc-data/x86_64-pc-linux-gnu/14/man
--infodir=/usr/share/gcc-data/x86_64-pc-linux-gnu/14/info
--with-gxx-include-dir=/usr/lib/gcc/x86_64-pc-linux-gnu/14/include/g++-v14
--disable-silent-rules --disable-dependency-tracking
--with-python-dir=/share/gcc-data/x86_64-pc-linux-gnu/14/python
--enable-languages=c,c++,fortran,rust --enable-obsolete --enable-secureplt
--disable-werror --with-system-zlib --enable-nls --without-included-gettext
--disable-libunwind-exceptions --enable-checking=yes,extra,rtl
--with-bugurl=https://bugs.gentoo.org/ --with-pkgversion='Gentoo Hardened
14.0.9999 p, commit c8305c9bdf09abe3e2f89783fe62f2e4049468fa'
--with-gcc-major-version-only --enable-libstdcxx-time --enable-lto
--disable-libstdcxx-pch --enable-shared --enable-threads=posix
--enable-__cxa_atexit --enable-clocale=gnu --enable-multilib
--with-multilib-list=m32,m64 --disable-fixed-point --enable-targets=all
--enable-libgomp --disable-libssp --disable-libada --disable-cet
--disable-systemtap --enable-valgrind-annotations --disable-vtable-verify
--disable-libvtv --with-zstd --with-isl --disable-isl-version-check
--enable-default-pie --enable-host-pie --enable-host-bind-now
--enable-default-ssp --disable-fixincludes --with-build-config='bootstrap-O3
bootstrap-lto'
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 14.0.1 20240304 (experimental)
eae6b63b5b5426f943f58b5ae0bf0a6068ca8ad6 (Gentoo Hardened 14.0.9999 p, commit
c8305c9bdf09abe3e2f89783fe62f2e4049468fa)
```

[Bug tree-optimization/114231] [14 regression] ICE when building libjxl

[sjames at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114231

--- Comment #1 from Sam James <sjames at gcc dot gnu.org> ---
Created attachment 57607
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=57607&action=edit
reduced.ii

`g++ -c reduced.ii -march=sapphirerapids -O2 -fno-vect-cost-model` is enough
for the reduced version.

[Bug tree-optimization/114231] [14 regression] ICE when building libjxl

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114231

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |ice-on-valid-code
   Target Milestone|---                         |14.0
                 CC|                            |pinskia at gcc dot gnu.org

[Bug tree-optimization/114231] [14 regression] ICE when building libjxl

[sjames at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114231

--- Comment #2 from Sam James <sjames at gcc dot gnu.org> ---
(In reply to Sam James from comment #1)
> Created attachment 57607 [details]
> reduced.ii
> 
> `g++ -c reduced.ii -march=sapphirerapids -O2 -fno-vect-cost-model` is enough
> for the reduced version.

in fact, g++ -c reduced.ii -O2 -fno-vect-cost-model is enough

[Bug tree-optimization/114231] [14 regression] ICE when building libjxl

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114231

--- Comment #3 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
vectorizable_shift

[Bug tree-optimization/114231] [14 regression] ICE when building libjxl

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114231

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
     Ever confirmed|0                           |1
             Status|UNCONFIRMED                 |NEW
   Last reconfirmed|                            |2024-03-05

--- Comment #4 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
Further reduced:
```
static inline
long ClampedSize(long begin, unsigned long size_max) {
  return begin + size_max ? size_max : 0;
}

void f(long*);
int ff[2];
void k(unsigned long x, unsigned long y) {
  long t = x >> ff[0];
  long t1 = ff[1];
  long t2 = y >> ff[0];
  long t3 = ClampedSize(t, t2);
  long t4[2];
  t4[0] = t1;
  t4[1] = t3;
  f(t4);
}
```

[Bug tree-optimization/114231] [14 regression] ICE when building libjxl

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114231

--- Comment #5 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
Little more reduced:
```

void f(long*);
int ff[2];
void f1(long, long);
void k(unsigned long x, unsigned long y) {
  long t = x >> ff[0];
  long t1 = ff[1];
  unsigned long t2 = y >> ff[0];
  long t3 = t+t2 ? t2 : 0;
  f1(t1, t3);
}
```

/app/example.cpp:9:14: missed: unusable type for last operand in vector/vector
shift/rotate.

Note if you change the type of ff to long, this works.

[Bug tree-optimization/114231] [14 regression] ICE when building libjxl

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114231

--- Comment #6 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
A little better reduced (this time only 1 BB even):
```

void f(long*);
int ff[2];
void f2(long, long, unsigned long);
void k(unsigned long x, unsigned long y) {
  long t = x >> ff[0];
  long t1 = ff[1];
  unsigned long t2 = y >> ff[0];
  f2(t1, t+t2, t2);
}
```

[Bug tree-optimization/114231] [12/13/14 regression] ICE when building libjxl

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114231

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|14.0                        |12.4
           Keywords|                            |needs-bisection
      Known to work|                            |11.1.0
            Summary|[14 regression] ICE when    |[12/13/14 regression] ICE
                   |building libjxl             |when building libjxl
             Target|                            |aarch64 x86_64

--- Comment #7 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
So my reduced testcase fails for aarch64 since GCC 12 but for x86_64 only on
the trunk. I suspect the commit that it will bisect to on x86_64 is just
enabling the pattern for x86_64.

So if anyone does a bisect, please try on aarch64.

[Bug tree-optimization/114231] [12/13/14 regression] ICE when building libjxl

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114231

--- Comment #8 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
(In reply to Andrew Pinski from comment #7)
> So my reduced testcase fails for aarch64 since GCC 12 but for x86_64 only on
> the trunk. I suspect the commit that it will bisect to on x86_64 is just
> enabling the pattern for x86_64.
> 
> So if anyone does a bisect, please try on aarch64.

Note also use `-O3 -fno-vect-cost-model` for the options since -O2 might catch
when the vectorizer is turned on for -O2.

[Bug tree-optimization/114231] [12/13/14 regression] ICE when building libjxl

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114231

--- Comment #9 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
The testcases are kinda of flaky because the following fails only on the trunk
for aarch64:
```

void f(long*);
int ff[2];
long tt[4];
unsigned long ttt;
void k(long x, long y) {
  long t = x >> ff[0];
  long t1 = ff[1];
  long t2 = y >> ff[0];
  tt[0] = t1;
  tt[1] = t+t2;
  tt[2] = t2;
}
```

[Bug tree-optimization/114231] [12/13/14 regression] ICE when building libjxl

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114231

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jakub at gcc dot gnu.org,
                   |                            |rguenth at gcc dot gnu.org

--- Comment #10 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
On x86_64 the #c6 testcase with -O3 -fno-vect-cost-model started to ICE with
r14-5603-g2b59e2b4dff42118fe3a505f07b9a6aa4cf53bdf
For aarch64 same testcase, my bet is
r12-1551-g3dfa4fe9f1a089b2b3906c83e22a1b39c49d937c
though I've only verified r12-1529 works and r12-1573 ICEs and there are no IL
differences before slp2 which newly ICEs.

[Bug tree-optimization/114231] [12/13/14 regression] ICE when building libjxl

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114231

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Assignee|unassigned at gcc dot gnu.org      |rguenth at gcc dot gnu.org
             Status|NEW                         |ASSIGNED
           Priority|P3                          |P2

--- Comment #11 from Richard Biener <rguenth at gcc dot gnu.org> ---
I will have a look.

[Bug tree-optimization/114231] [12/13/14 regression] ICE when building libjxl

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114231

--- Comment #12 from Richard Biener <rguenth at gcc dot gnu.org> ---
So the immediate reason is that between analysis and transform whether we
consider the shift vectorizable changes.  That's because we code generated
a live lane which ended up changing operands in stmts we will vectorize
(there's some odd broken handling there I think).  That's because we
match up the scalar from the SLP node with the scalar in the scalar GIMPLE
IL (which changed) here:

  stmt_vec_info op1_def_stmt_info;
  slp_tree slp_op1;
  if (!vect_is_simple_use (vinfo, stmt_info, slp_node, 1, &op1, &slp_op1,
                           &dt[1], &op1_vectype, &op1_def_stmt_info))
...
          FOR_EACH_VEC_ELT (stmts, k, slpstmt_info)
            {
              gassign *slpstmt = as_a <gassign *> (slpstmt_info->stmt);
              if (!operand_equal_p (gimple_assign_rhs2 (slpstmt), op1, 0))
                scalar_shift_arg = false;
            } 

which is a bit fragile.

But the underlying issue seems to be the live lane stuff.

Ah, and that's because we do the reduction discovery on the original
scalar stmt while live lane extraction honors patterns when checking
whether the stmt is vectorized ...

I have a patch, not sure how big the fallout might be though.

[Bug tree-optimization/114231] [12/13/14 regression] ICE when building libjxl

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114231

--- Comment #13 from GCC Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Richard Biener <rguenth@gcc.gnu.org>:

https://gcc.gnu.org/g:7890836de20912bd92afaf5abbeaf9d8c5b86542

commit r14-9316-g7890836de20912bd92afaf5abbeaf9d8c5b86542
Author: Richard Biener <rguenther@suse.de>
Date:   Tue Mar 5 10:55:56 2024 +0100

    tree-optimization/114231 - use patterns for BB SLP discovery root stmts

    The following makes sure to use recognized patterns when vectorizing
    roots during BB SLP discovery.  We need to apply those late since
    during root discovery we've not yet done pattern recognition.
    All parts of the vectorizer assume patterns get used, for the testcase
    we mix this up when doing live lane computation.

            PR tree-optimization/114231
            * tree-vect-slp.cc (vect_analyze_slp): Lookup patterns when
            processing a BB SLP root.

            * gcc.dg/vect/pr114231.c: New testcase.

[Bug tree-optimization/114231] [12/13 Regression] ICE when building libjxl

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114231

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|[12/13/14 regression] ICE   |[12/13 Regression] ICE when
                   |when building libjxl        |building libjxl
      Known to work|                            |14.0

--- Comment #14 from Richard Biener <rguenth at gcc dot gnu.org> ---
Fixed on trunk sofar.

[Bug tree-optimization/114231] [12/13 Regression] ICE when building libjxl

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114231

--- Comment #15 from GCC Commits <cvs-commit at gcc dot gnu.org> ---
The releases/gcc-13 branch has been updated by Richard Biener
<rguenth@gcc.gnu.org>:

https://gcc.gnu.org/g:04fffbaa87997ac893a9aa68b674c938ba3ecddb

commit r13-8486-g04fffbaa87997ac893a9aa68b674c938ba3ecddb
Author: Richard Biener <rguenther@suse.de>
Date:   Tue Mar 5 10:55:56 2024 +0100

    tree-optimization/114231 - use patterns for BB SLP discovery root stmts

    The following makes sure to use recognized patterns when vectorizing
    roots during BB SLP discovery.  We need to apply those late since
    during root discovery we've not yet done pattern recognition.
    All parts of the vectorizer assume patterns get used, for the testcase
    we mix this up when doing live lane computation.

            PR tree-optimization/114231
            * tree-vect-slp.cc (vect_analyze_slp): Lookup patterns when
            processing a BB SLP root.

            * gcc.dg/vect/pr114231.c: New testcase.

[Bug tree-optimization/114231] [12 Regression] ICE when building libjxl

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114231

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
      Known to fail|                            |13.2.0
      Known to work|                            |13.2.1
            Summary|[12/13 Regression] ICE when |[12 Regression] ICE when
                   |building libjxl             |building libjxl

--- Comment #16 from Richard Biener <rguenth at gcc dot gnu.org> ---
The simpler patch should be enough for the simpler code in GCC 13 and before.