[Bug tree-optimization/99873] New: [11 Regression] GCC no longer makes as much use of ST3

[Bug tree-optimization/99873] New: [11 Regression] GCC no longer makes as much use of ST3

[rsandifo at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99873

            Bug ID: 99873
           Summary: [11 Regression] GCC no longer makes as much use of ST3
           Product: gcc
           Version: 11.0
            Status: UNCONFIRMED
          Keywords: missed-optimization
          Severity: normal
          Priority: P3
         Component: tree-optimization
          Assignee: unassigned at gcc dot gnu.org
          Reporter: rsandifo at gcc dot gnu.org
                CC: rguenth at gcc dot gnu.org
  Target Milestone: ---
            Target: aarch64*-*-*

For:

void
f (int *restrict x, int *restrict y, int *restrict z, int n)
{
  for (int i = 0; i < n; i += 3)
    {
      x[i] = y[i] + z[i];
      x[i + 1] = y[i + 1] - z[i + 1];
      x[i + 2] = y[i + 2] | z[i + 2];
    }
}

GCC 10 produced a nice loop using LD3 and ST3:

.L4:
        ld3     {v4.4s - v6.4s}, [x4], 48
        ld3     {v16.4s - v18.4s}, [x6], 48
        add     v1.4s, v16.4s, v4.4s
        sub     v2.4s, v5.4s, v17.4s
        orr     v3.16b, v18.16b, v6.16b
        st3     {v1.4s - v3.4s}, [x5], 48
        cmp     x8, x4
        bne     .L4

But GCC 11 instead uses lane stores:

.L4:
        ld3     {v4.4s - v6.4s}, [x9], 48
        mov     x8, x4
        ld3     {v16.4s - v18.4s}, [x11], 48
        add     x16, x4, 24
        add     x15, x4, 36
        add     x14, x4, 16
        add     x13, x4, 28
        add     x12, x4, 40
        add     v2.4s, v16.4s, v4.4s
        add     x7, x4, 20
        sub     v1.4s, v5.4s, v17.4s
        add     x6, x4, 32
        orr     v0.16b, v18.16b, v6.16b
        add     x5, x4, 44
        add     x4, x4, 48
        str     s2, [x8], 12
        st1     {v2.s}[1], [x8]
        st1     {v2.s}[2], [x16]
        st1     {v2.s}[3], [x15]
        str     s1, [x4, -44]
        st1     {v1.s}[1], [x14]
        st1     {v1.s}[2], [x13]
        st1     {v1.s}[3], [x12]
        str     s0, [x4, -40]
        st1     {v0.s}[1], [x7]
        st1     {v0.s}[2], [x6]
        st1     {v0.s}[3], [x5]
        cmp     x10, x9
        bne     .L4

I think this is due to r11-3966 optimistically splitting store groups
in a way that we can't recover from if SLP subsequently fails.

Maybe the easiest thing for GCC 11 would be to block the split if the
target supports IFN_STORE_LANES for the group size and element type.
That restores the above case.  Of the two tests affected by r11-3966,
vect-complex-5.c seems better with LD2 & ST4, while the motivating
case (pr97428.c) still uses SLP as intended.

[Bug tree-optimization/99873] [11 Regression] GCC no longer makes as much use of ST3

[rsandifo at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99873

rsandifo at gcc dot gnu.org <rsandifo at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |ASSIGNED
           Assignee|unassigned at gcc dot gnu.org      |rsandifo at gcc dot gnu.org
   Last reconfirmed|                            |2021-04-01
     Ever confirmed|0                           |1

--- Comment #1 from rsandifo at gcc dot gnu.org <rsandifo at gcc dot gnu.org> ---
Testing a patch.

[Bug tree-optimization/99873] [11 Regression] GCC no longer makes as much use of ST3

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99873

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |11.0

--- Comment #2 from Richard Biener <rguenth at gcc dot gnu.org> ---
We can also undo the splitting if SLP doesn't work out (keep the original
DR analysis chaining somewhere).

[Bug tree-optimization/99873] [11 Regression] GCC no longer makes as much use of ST3

[rsandifo at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99873

--- Comment #3 from rsandifo at gcc dot gnu.org <rsandifo at gcc dot gnu.org> ---
(In reply to Richard Biener from comment #2)
> We can also undo the splitting if SLP doesn't work out (keep the original
> DR analysis chaining somewhere).
Yeah, that sounds like something we should do for the cases that
can't use store-lanes.  So far though, I've not seen any cases that
are better with the split group than with the store-lanes version,
so I think we want the skip even if SLP would succeed.

[Bug tree-optimization/99873] [11 Regression] GCC no longer makes as much use of ST3

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99873

--- Comment #4 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Richard Sandiford <rsandifo@gcc.gnu.org>:

https://gcc.gnu.org/g:5c5b31975e62b4c52d76dc5efd9dc717a361c710

commit r11-8029-g5c5b31975e62b4c52d76dc5efd9dc717a361c710
Author: Richard Sandiford <richard.sandiford@arm.com>
Date:   Wed Apr 7 15:21:55 2021 +0100

    vect: Don't split store groups if we have IFN_STORE_LANES [PR99873]

    As noted in the PR, we were no longer using ST3 for the testcase and
    instead stored each lane individually.  This is because we'd split
    the store group during SLP and couldn't recover when SLP failed.

    However, we can also get better code with ST3 and ST4 even if SLP would
    have succeeded, such as for vect-complex-5.c.  I'm not sure exactly
    where the cut-off point is, but it seems reasonable to allow the split
    if either of the new groups would operate on full vectors *within*
    rather than across scalar loop iterations.

    E.g. on a Cortex-A57, pr99873_3.c performs better using ST4 while
    pr99873_2.c performs better with SLP.

    Another factor is that SLP can handle smaller iteration counts than
    IFN_STORE_LANES can, but we don't have the infrastructure to choose
    reliably based on that.

    gcc/
            PR tree-optimization/99873
            * tree-vect-slp.c (vect_slp_prefer_store_lanes_p): New function.
            (vect_build_slp_instance): Don't split store groups that could
            use IFN_STORE_LANES.

    gcc/testsuite/
            * gcc.dg/vect/slp-21.c: Only expect 2 of the loops to use SLP
            if IFN_STORE_LANES is available.
            * gcc.dg/vect/vect-complex-5.c: Expect no loops to use SLP if
            IFN_STORE_LANES is available.
            * gcc.target/aarch64/pr99873_1.c: New test.
            * gcc.target/aarch64/pr99873_2.c: Likewise.
            * gcc.target/aarch64/pr99873_3.c: Likewise.
            * gcc.target/aarch64/sve/pr99873_1.c: Likewise.
            * gcc.target/aarch64/sve/pr99873_2.c: Likewise.
            * gcc.target/aarch64/sve/pr99873_3.c: Likewise.

[Bug tree-optimization/99873] [11 Regression] GCC no longer makes as much use of ST3

[rsandifo at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99873

rsandifo at gcc dot gnu.org <rsandifo at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|ASSIGNED                    |RESOLVED

--- Comment #5 from rsandifo at gcc dot gnu.org <rsandifo at gcc dot gnu.org> ---
Fixed on trunk.