From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: by sourceware.org (Postfix, from userid 1816) id 3CE1B3858D35; Wed, 21 Jun 2023 12:46:22 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 3CE1B3858D35 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1687351582; bh=L4xEWydQO7BB/hV5yFmhZni2Oz9iZWDu7GA15gG/yNQ=; h=From:To:Subject:Date:From; b=luXF5whRHaIE3q5bpOLz+cn+M86s8jK0KcSSWfUyhSCCTz8HsZ5elw+Qiw1qF60ws ElTYN4Y4+quRR60teudQnhzIkoBdMsAHBg9kHO9fboDgut8XD/QGuiQCAv3Jv6CfDT 5+Wcaind7VrDorPglniU3xDGmWaK1VfO06f4Pj6s= MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="utf-8" From: Kyrylo Tkachov To: gcc-cvs@gcc.gnu.org Subject: [gcc r14-2019] aarch64: Avoid same input and output Z register for gather loads X-Act-Checkin: gcc X-Git-Author: Kyrylo Tkachov X-Git-Refname: refs/heads/master X-Git-Oldrev: 4d9d207c668e757adefcfe5368d86bcc008fa4db X-Git-Newrev: b375c5340b55d6e7ee703717831dd27528962570 Message-Id: <20230621124622.3CE1B3858D35@sourceware.org> Date: Wed, 21 Jun 2023 12:46:22 +0000 (GMT) List-Id: https://gcc.gnu.org/g:b375c5340b55d6e7ee703717831dd27528962570 commit r14-2019-gb375c5340b55d6e7ee703717831dd27528962570 Author: Kyrylo Tkachov Date: Wed Jun 21 13:43:26 2023 +0100 aarch64: Avoid same input and output Z register for gather loads The architecture recommends that load-gather instructions avoid using the same Z register for the load address and the destination, and the Software Optimization Guides for Arm cores recommend that as well. This means that for code like: svuint64_t food (svbool_t p, uint64_t *in, svint64_t offsets, svuint64_t a) { return svadd_u64_x (p, a, svld1_gather_offset(p, in, offsets)); } we'll want to avoid generating the current: food: ld1d z0.d, p0/z, [x0, z0.d] // Z0 reused as input and output. add z0.d, z1.d, z0.d ret However, we still want to avoid generating extra moves where there were none before, so the tight aarch64-sve-acle.exp tests for load gathers should still pass as they are. This patch implements that recommendation for the load gather patterns by: * duplicating the alternatives * marking the output operand as early clobber * Tying the input Z register operand in the original alternatives to 0 * Penalising the original alternatives with '?' This results in a large-ish patch in terms of diff lines but the new compact syntax (thanks Tamar) makes it quite a readable an regular change. The benchmark numbers on a Neoverse V1 on fprate look okay: diff 503.bwaves_r 0.00% 507.cactuBSSN_r 0.00% 508.namd_r 0.00% 510.parest_r 0.55% 511.povray_r 0.22% 519.lbm_r 0.00% 521.wrf_r 0.00% 526.blender_r 0.00% 527.cam4_r 0.56% 538.imagick_r 0.00% 544.nab_r 0.00% 549.fotonik3d_r 0.00% 554.roms_r 0.00% fprate 0.10% Bootstrapped and tested on aarch64-none-linux-gnu. gcc/ChangeLog: * config/aarch64/aarch64-sve.md (mask_gather_load): Add alternatives to prefer to avoid same input and output Z register. (mask_gather_load): Likewise. (*mask_gather_load_xtw_unpacked): Likewise. (*mask_gather_load_sxtw): Likewise. (*mask_gather_load_uxtw): Likewise. (@aarch64_gather_load_): Likewise. (@aarch64_gather_load_): Likewise. (*aarch64_gather_load_ _xtw_unpacked): Likewise. (*aarch64_gather_load_ _sxtw): Likewise. (*aarch64_gather_load_ _uxtw): Likewise. (@aarch64_ldff1_gather): Likewise. (@aarch64_ldff1_gather): Likewise. (*aarch64_ldff1_gather_sxtw): Likewise. (*aarch64_ldff1_gather_uxtw): Likewise. (@aarch64_ldff1_gather_ ): Likewise. (@aarch64_ldff1_gather_ ): Likewise. (*aarch64_ldff1_gather_ _sxtw): Likewise. (*aarch64_ldff1_gather_ _uxtw): Likewise. * config/aarch64/aarch64-sve2.md (@aarch64_gather_ldnt): Likewise. (@aarch64_gather_ldnt_ ): Likewise. gcc/testsuite/ChangeLog: * gcc.target/aarch64/sve/gather_earlyclobber.c: New test. * gcc.target/aarch64/sve2/gather_earlyclobber.c: New test. Diff: --- gcc/config/aarch64/aarch64-sve.md | 194 ++++++++++++++------- gcc/config/aarch64/aarch64-sve2.md | 12 +- .../gcc.target/aarch64/sve/gather_earlyclobber.c | 96 ++++++++++ .../gcc.target/aarch64/sve2/gather_earlyclobber.c | 32 ++++ 4 files changed, 263 insertions(+), 71 deletions(-) diff --git a/gcc/config/aarch64/aarch64-sve.md b/gcc/config/aarch64/aarch64-sve.md index 955bbdfd129..da5534c3e32 100644 --- a/gcc/config/aarch64/aarch64-sve.md +++ b/gcc/config/aarch64/aarch64-sve.md @@ -1429,12 +1429,18 @@ UNSPEC_LD1_GATHER))] "TARGET_SVE" {@ [cons: =0, 1, 2, 3, 4, 5 ] - [w, Z, w, Ui1, Ui1, Upl ] ld1\t%0.s, %5/z, [%2.s] - [w, vgw, w, Ui1, Ui1, Upl ] ld1\t%0.s, %5/z, [%2.s, #%1] - [w, rk, w, Z, Ui1, Upl ] ld1\t%0.s, %5/z, [%1, %2.s, sxtw] - [w, rk, w, Ui1, Ui1, Upl ] ld1\t%0.s, %5/z, [%1, %2.s, uxtw] - [w, rk, w, Z, i, Upl ] ld1\t%0.s, %5/z, [%1, %2.s, sxtw %p4] - [w, rk, w, Ui1, i, Upl ] ld1\t%0.s, %5/z, [%1, %2.s, uxtw %p4] + [&w, Z, w, Ui1, Ui1, Upl] ld1\t%0.s, %5/z, [%2.s] + [?w, Z, 0, Ui1, Ui1, Upl] ^ + [&w, vgw, w, Ui1, Ui1, Upl] ld1\t%0.s, %5/z, [%2.s, #%1] + [?w, vgw, 0, Ui1, Ui1, Upl] ^ + [&w, rk, w, Z, Ui1, Upl] ld1\t%0.s, %5/z, [%1, %2.s, sxtw] + [?w, rk, 0, Z, Ui1, Upl] ^ + [&w, rk, w, Ui1, Ui1, Upl] ld1\t%0.s, %5/z, [%1, %2.s, uxtw] + [?w, rk, 0, Ui1, Ui1, Upl] ^ + [&w, rk, w, Z, i, Upl] ld1\t%0.s, %5/z, [%1, %2.s, sxtw %p4] + [?w, rk, 0, Z, i, Upl] ^ + [&w, rk, w, Ui1, i, Upl] ld1\t%0.s, %5/z, [%1, %2.s, uxtw %p4] + [?w, rk, 0, Ui1, i, Upl] ^ } ) @@ -1451,11 +1457,15 @@ (mem:BLK (scratch))] UNSPEC_LD1_GATHER))] "TARGET_SVE" - {@ [cons: =0, 1, 2, 3, 4, 5 ] - [w, Z, w, i, Ui1, Upl ] ld1\t%0.d, %5/z, [%2.d] - [w, vgd, w, i, Ui1, Upl ] ld1\t%0.d, %5/z, [%2.d, #%1] - [w, rk, w, i, Ui1, Upl ] ld1\t%0.d, %5/z, [%1, %2.d] - [w, rk, w, i, i, Upl ] ld1\t%0.d, %5/z, [%1, %2.d, lsl %p4] + {@ [cons: =0, 1, 2, 3, 4, 5] + [&w, Z, w, i, Ui1, Upl] ld1\t%0.d, %5/z, [%2.d] + [?w, Z, 0, i, Ui1, Upl] ^ + [&w, vgd, w, i, Ui1, Upl] ld1\t%0.d, %5/z, [%2.d, #%1] + [?w, vgd, 0, i, Ui1, Upl] ^ + [&w, rk, w, i, Ui1, Upl] ld1\t%0.d, %5/z, [%1, %2.d] + [?w, rk, 0, i, Ui1, Upl] ^ + [&w, rk, w, i, i, Upl] ld1\t%0.d, %5/z, [%1, %2.d, lsl %p4] + [?w, rk, 0, i, i, Upl] ^ } ) @@ -1475,9 +1485,11 @@ (mem:BLK (scratch))] UNSPEC_LD1_GATHER))] "TARGET_SVE" - {@ [cons: =0, 1, 2, 3, 4, 5 ] - [w, rk, w, i, Ui1, Upl ] ld1\t%0.d, %5/z, [%1, %2.d, xtw] - [w, rk, w, i, i, Upl ] ld1\t%0.d, %5/z, [%1, %2.d, xtw %p4] + {@ [cons: =0, 1, 2, 3, 4, 5] + [&w, rk, w, i, Ui1, Upl ] ld1\t%0.d, %5/z, [%1, %2.d, xtw] + [?w, rk, 0, i, Ui1, Upl ] ^ + [&w, rk, w, i, i, Upl ] ld1\t%0.d, %5/z, [%1, %2.d, xtw %p4] + [?w, rk, 0, i, i, Upl ] ^ } "&& !CONSTANT_P (operands[6])" { @@ -1503,9 +1515,11 @@ (mem:BLK (scratch))] UNSPEC_LD1_GATHER))] "TARGET_SVE" - {@ [cons: =0, 1, 2, 3, 4, 5 ] - [w, rk, w, i, Ui1, Upl ] ld1\t%0.d, %5/z, [%1, %2.d, sxtw] - [w, rk, w, i, i, Upl ] ld1\t%0.d, %5/z, [%1, %2.d, sxtw %p4] + {@ [cons: =0, 1, 2, 3, 4, 5] + [&w, rk, w, i, Ui1, Upl ] ld1\t%0.d, %5/z, [%1, %2.d, sxtw] + [?w, rk, 0, i, Ui1, Upl ] ^ + [&w, rk, w, i, i, Upl ] ld1\t%0.d, %5/z, [%1, %2.d, sxtw %p4] + [?w, rk, 0, i, i, Upl ] ^ } "&& !CONSTANT_P (operands[6])" { @@ -1528,9 +1542,11 @@ (mem:BLK (scratch))] UNSPEC_LD1_GATHER))] "TARGET_SVE" - {@ [cons: =0, 1, 2, 3, 4, 5 ] - [w, rk, w, i, Ui1, Upl ] ld1\t%0.d, %5/z, [%1, %2.d, uxtw] - [w, rk, w, i, i, Upl ] ld1\t%0.d, %5/z, [%1, %2.d, uxtw %p4] + {@ [cons: =0, 1, 2, 3, 4, 5] + [&w, rk, w, i, Ui1, Upl ] ld1\t%0.d, %5/z, [%1, %2.d, uxtw] + [?w, rk, 0, i, Ui1, Upl ] ^ + [&w, rk, w, i, i, Upl ] ld1\t%0.d, %5/z, [%1, %2.d, uxtw %p4] + [?w, rk, 0, i, i, Upl ] ^ } ) @@ -1563,13 +1579,19 @@ UNSPEC_LD1_GATHER))] UNSPEC_PRED_X))] "TARGET_SVE && (~ & ) == 0" - {@ [cons: =0, 1, 2, 3, 4, 5, 6 ] - [w, Z, w, Ui1, Ui1, Upl, UplDnm] ld1\t%0.s, %5/z, [%2.s] - [w, vg, w, Ui1, Ui1, Upl, UplDnm] ld1\t%0.s, %5/z, [%2.s, #%1] - [w, rk, w, Z, Ui1, Upl, UplDnm] ld1\t%0.s, %5/z, [%1, %2.s, sxtw] - [w, rk, w, Ui1, Ui1, Upl, UplDnm] ld1\t%0.s, %5/z, [%1, %2.s, uxtw] - [w, rk, w, Z, i, Upl, UplDnm] ld1\t%0.s, %5/z, [%1, %2.s, sxtw %p4] - [w, rk, w, Ui1, i, Upl, UplDnm] ld1\t%0.s, %5/z, [%1, %2.s, uxtw %p4] + {@ [cons: =0, 1, 2, 3, 4, 5, 6] + [&w, Z, w, Ui1, Ui1, Upl, UplDnm] ld1\t%0.s, %5/z, [%2.s] + [?w, Z, 0, Ui1, Ui1, Upl, UplDnm] ^ + [&w, vg, w, Ui1, Ui1, Upl, UplDnm] ld1\t%0.s, %5/z, [%2.s, #%1] + [?w, vg, 0, Ui1, Ui1, Upl, UplDnm] ^ + [&w, rk, w, Z, Ui1, Upl, UplDnm] ld1\t%0.s, %5/z, [%1, %2.s, sxtw] + [?w, rk, 0, Z, Ui1, Upl, UplDnm] ^ + [&w, rk, w, Ui1, Ui1, Upl, UplDnm] ld1\t%0.s, %5/z, [%1, %2.s, uxtw] + [?w, rk, 0, Ui1, Ui1, Upl, UplDnm] ^ + [&w, rk, w, Z, i, Upl, UplDnm] ld1\t%0.s, %5/z, [%1, %2.s, sxtw %p4] + [?w, rk, 0, Z, i, Upl, UplDnm] ^ + [&w, rk, w, Ui1, i, Upl, UplDnm] ld1\t%0.s, %5/z, [%1, %2.s, uxtw %p4] + [?w, rk, 0, Ui1, i, Upl, UplDnm] ^ } "&& !CONSTANT_P (operands[6])" { @@ -1595,10 +1617,14 @@ UNSPEC_PRED_X))] "TARGET_SVE && (~ & ) == 0" {@ [cons: =0, 1, 2, 3, 4, 5, 6] - [w, Z, w, i, Ui1, Upl, UplDnm] ld1\t%0.d, %5/z, [%2.d] - [w, vg, w, i, Ui1, Upl, UplDnm] ld1\t%0.d, %5/z, [%2.d, #%1] - [w, rk, w, i, Ui1, Upl, UplDnm] ld1\t%0.d, %5/z, [%1, %2.d] - [w, rk, w, i, i, Upl, UplDnm] ld1\t%0.d, %5/z, [%1, %2.d, lsl %p4] + [&w, Z, w, i, Ui1, Upl, UplDnm] ld1\t%0.d, %5/z, [%2.d] + [?w, Z, 0, i, Ui1, Upl, UplDnm] ^ + [&w, vg, w, i, Ui1, Upl, UplDnm] ld1\t%0.d, %5/z, [%2.d, #%1] + [?w, vg, 0, i, Ui1, Upl, UplDnm] ^ + [&w, rk, w, i, Ui1, Upl, UplDnm] ld1\t%0.d, %5/z, [%1, %2.d] + [?w, rk, 0, i, Ui1, Upl, UplDnm] ^ + [&w, rk, w, i, i, Upl, UplDnm] ld1\t%0.d, %5/z, [%1, %2.d, lsl %p4] + [?w, rk, 0, i, i, Upl, UplDnm] ^ } "&& !CONSTANT_P (operands[6])" { @@ -1627,8 +1653,10 @@ UNSPEC_PRED_X))] "TARGET_SVE && (~ & ) == 0" {@ [cons: =0, 1, 2, 3, 4, 5] - [w, rk, w, i, Ui1, Upl ] ld1\t%0.d, %5/z, [%1, %2.d, xtw] - [w, rk, w, i, i, Upl ] ld1\t%0.d, %5/z, [%1, %2.d, xtw %p4] + [&w, rk, w, i, Ui1, Upl ] ld1\t%0.d, %5/z, [%1, %2.d, xtw] + [?w, rk, 0, i, Ui1, Upl ] ^ + [&w, rk, w, i, i, Upl ] ld1\t%0.d, %5/z, [%1, %2.d, xtw %p4] + [?w, rk, 0, i, i, Upl ] ^ } "&& (!CONSTANT_P (operands[6]) || !CONSTANT_P (operands[7]))" { @@ -1660,8 +1688,10 @@ UNSPEC_PRED_X))] "TARGET_SVE && (~ & ) == 0" {@ [cons: =0, 1, 2, 3, 4, 5] - [w, rk, w, i, Ui1, Upl ] ld1\t%0.d, %5/z, [%1, %2.d, sxtw] - [w, rk, w, i, i, Upl ] ld1\t%0.d, %5/z, [%1, %2.d, sxtw %p4] + [&w, rk, w, i, Ui1, Upl ] ld1\t%0.d, %5/z, [%1, %2.d, sxtw] + [?w, rk, 0, i, Ui1, Upl ] ^ + [&w, rk, w, i, i, Upl ] ld1\t%0.d, %5/z, [%1, %2.d, sxtw %p4] + [?w, rk, 0, i, i, Upl ] ^ } "&& (!CONSTANT_P (operands[6]) || !CONSTANT_P (operands[7]))" { @@ -1690,8 +1720,10 @@ UNSPEC_PRED_X))] "TARGET_SVE && (~ & ) == 0" {@ [cons: =0, 1, 2, 3, 4, 5] - [w, rk, w, i, Ui1, Upl ] ld1\t%0.d, %5/z, [%1, %2.d, uxtw] - [w, rk, w, i, i, Upl ] ld1\t%0.d, %5/z, [%1, %2.d, uxtw %p4] + [&w, rk, w, i, Ui1, Upl ] ld1\t%0.d, %5/z, [%1, %2.d, uxtw] + [?w, rk, 0, i, Ui1, Upl ] ^ + [&w, rk, w, i, i, Upl ] ld1\t%0.d, %5/z, [%1, %2.d, uxtw %p4] + [?w, rk, 0, i, i, Upl ] ^ } "&& !CONSTANT_P (operands[7])" { @@ -1721,13 +1753,19 @@ (reg:VNx16BI FFRT_REGNUM)] UNSPEC_LDFF1_GATHER))] "TARGET_SVE" - {@ [cons: =0, 1, 2, 3, 4, 5 ] - [w, Z, w, i, Ui1, Upl] ldff1w\t%0.s, %5/z, [%2.s] - [w, vgw, w, i, Ui1, Upl] ldff1w\t%0.s, %5/z, [%2.s, #%1] - [w, rk, w, Z, Ui1, Upl] ldff1w\t%0.s, %5/z, [%1, %2.s, sxtw] - [w, rk, w, Ui1, Ui1, Upl] ldff1w\t%0.s, %5/z, [%1, %2.s, uxtw] - [w, rk, w, Z, i, Upl] ldff1w\t%0.s, %5/z, [%1, %2.s, sxtw %p4] - [w, rk, w, Ui1, i, Upl] ldff1w\t%0.s, %5/z, [%1, %2.s, uxtw %p4] + {@ [cons: =0, 1, 2, 3, 4, 5 ] + [&w, Z, w, i, Ui1, Upl] ldff1w\t%0.s, %5/z, [%2.s] + [?w, Z, 0, i, Ui1, Upl] ^ + [&w, vgw, w, i, Ui1, Upl] ldff1w\t%0.s, %5/z, [%2.s, #%1] + [?w, vgw, 0, i, Ui1, Upl] ^ + [&w, rk, w, Z, Ui1, Upl] ldff1w\t%0.s, %5/z, [%1, %2.s, sxtw] + [?w, rk, 0, Z, Ui1, Upl] ^ + [&w, rk, w, Ui1, Ui1, Upl] ldff1w\t%0.s, %5/z, [%1, %2.s, uxtw] + [?w, rk, 0, Ui1, Ui1, Upl] ^ + [&w, rk, w, Z, i, Upl] ldff1w\t%0.s, %5/z, [%1, %2.s, sxtw %p4] + [?w, rk, 0, Z, i, Upl] ^ + [&w, rk, w, Ui1, i, Upl] ldff1w\t%0.s, %5/z, [%1, %2.s, uxtw %p4] + [?w, rk, 0, Ui1, i, Upl] ^ } ) @@ -1745,11 +1783,15 @@ (reg:VNx16BI FFRT_REGNUM)] UNSPEC_LDFF1_GATHER))] "TARGET_SVE" - {@ [cons: =0, 1, 2, 3, 4, 5] - [w, Z, w, i, Ui1, Upl ] ldff1d\t%0.d, %5/z, [%2.d] - [w, vgd, w, i, Ui1, Upl ] ldff1d\t%0.d, %5/z, [%2.d, #%1] - [w, rk, w, i, Ui1, Upl ] ldff1d\t%0.d, %5/z, [%1, %2.d] - [w, rk, w, i, i, Upl ] ldff1d\t%0.d, %5/z, [%1, %2.d, lsl %p4] + {@ [cons: =0, 1, 2, 3, 4, 5 ] + [&w, Z, w, i, Ui1, Upl ] ldff1d\t%0.d, %5/z, [%2.d] + [?w, Z, 0, i, Ui1, Upl ] ^ + [&w, vgd, w, i, Ui1, Upl ] ldff1d\t%0.d, %5/z, [%2.d, #%1] + [?w, vgd, 0, i, Ui1, Upl ] ^ + [&w, rk, w, i, Ui1, Upl ] ldff1d\t%0.d, %5/z, [%1, %2.d] + [?w, rk, 0, i, Ui1, Upl ] ^ + [&w, rk, w, i, i, Upl ] ldff1d\t%0.d, %5/z, [%1, %2.d, lsl %p4] + [?w, rk, 0, i, i, Upl ] ^ } ) @@ -1772,8 +1814,10 @@ UNSPEC_LDFF1_GATHER))] "TARGET_SVE" {@ [cons: =0, 1, 2, 3, 4, 5] - [w, rk, w, i, Ui1, Upl ] ldff1d\t%0.d, %5/z, [%1, %2.d, sxtw] - [w, rk, w, i, i, Upl ] ldff1d\t%0.d, %5/z, [%1, %2.d, sxtw %p4] + [&w, rk, w, i, Ui1, Upl ] ldff1d\t%0.d, %5/z, [%1, %2.d, sxtw] + [?w, rk, 0, i, Ui1, Upl ] ^ + [&w, rk, w, i, i, Upl ] ldff1d\t%0.d, %5/z, [%1, %2.d, sxtw %p4] + [?w, rk, 0, i, i, Upl ] ^ } "&& !CONSTANT_P (operands[6])" { @@ -1797,8 +1841,10 @@ UNSPEC_LDFF1_GATHER))] "TARGET_SVE" {@ [cons: =0, 1, 2, 3, 4, 5] - [w, rk, w, i, Ui1, Upl ] ldff1d\t%0.d, %5/z, [%1, %2.d, uxtw] - [w, rk, w, i, i, Upl ] ldff1d\t%0.d, %5/z, [%1, %2.d, uxtw %p4] + [&w, rk, w, i, Ui1, Upl ] ldff1d\t%0.d, %5/z, [%1, %2.d, uxtw] + [?w, rk, 0, i, Ui1, Upl ] ^ + [&w, rk, w, i, i, Upl ] ldff1d\t%0.d, %5/z, [%1, %2.d, uxtw %p4] + [?w, rk, 0, i, i, Upl ] ^ } ) @@ -1833,12 +1879,18 @@ UNSPEC_PRED_X))] "TARGET_SVE" {@ [cons: =0, 1, 2, 3, 4, 5, 6] - [w, Z, w, i, Ui1, Upl, UplDnm] ldff1\t%0.s, %5/z, [%2.s] - [w, vg, w, i, Ui1, Upl, UplDnm] ldff1\t%0.s, %5/z, [%2.s, #%1] - [w, rk, w, Z, Ui1, Upl, UplDnm] ldff1\t%0.s, %5/z, [%1, %2.s, sxtw] - [w, rk, w, Ui1, Ui1, Upl, UplDnm] ldff1\t%0.s, %5/z, [%1, %2.s, uxtw] - [w, rk, w, Z, i, Upl, UplDnm] ldff1\t%0.s, %5/z, [%1, %2.s, sxtw %p4] - [w, rk, w, Ui1, i, Upl, UplDnm] ldff1\t%0.s, %5/z, [%1, %2.s, uxtw %p4] + [&w, Z, w, i, Ui1, Upl, UplDnm] ldff1\t%0.s, %5/z, [%2.s] + [?w, Z, 0, i, Ui1, Upl, UplDnm] ^ + [&w, vg, w, i, Ui1, Upl, UplDnm] ldff1\t%0.s, %5/z, [%2.s, #%1] + [?w, vg, 0, i, Ui1, Upl, UplDnm] ^ + [&w, rk, w, Z, Ui1, Upl, UplDnm] ldff1\t%0.s, %5/z, [%1, %2.s, sxtw] + [?w, rk, 0, Z, Ui1, Upl, UplDnm] ^ + [&w, rk, w, Ui1, Ui1, Upl, UplDnm] ldff1\t%0.s, %5/z, [%1, %2.s, uxtw] + [?w, rk, 0, Ui1, Ui1, Upl, UplDnm] ^ + [&w, rk, w, Z, i, Upl, UplDnm] ldff1\t%0.s, %5/z, [%1, %2.s, sxtw %p4] + [?w, rk, 0, Z, i, Upl, UplDnm] ^ + [&w, rk, w, Ui1, i, Upl, UplDnm] ldff1\t%0.s, %5/z, [%1, %2.s, uxtw %p4] + [?w, rk, 0, Ui1, i, Upl, UplDnm] ^ } "&& !CONSTANT_P (operands[6])" { @@ -1865,10 +1917,14 @@ UNSPEC_PRED_X))] "TARGET_SVE" {@ [cons: =0, 1, 2, 3, 4, 5, 6] - [w, Z, w, i, Ui1, Upl, UplDnm] ldff1\t%0.d, %5/z, [%2.d] - [w, vg, w, i, Ui1, Upl, UplDnm] ldff1\t%0.d, %5/z, [%2.d, #%1] - [w, rk, w, i, Ui1, Upl, UplDnm] ldff1\t%0.d, %5/z, [%1, %2.d] - [w, rk, w, i, i, Upl, UplDnm] ldff1\t%0.d, %5/z, [%1, %2.d, lsl %p4] + [&w, Z, w, i, Ui1, Upl, UplDnm] ldff1\t%0.d, %5/z, [%2.d] + [?w, Z, 0, i, Ui1, Upl, UplDnm] ^ + [&w, vg, w, i, Ui1, Upl, UplDnm] ldff1\t%0.d, %5/z, [%2.d, #%1] + [?w, vg, 0, i, Ui1, Upl, UplDnm] ^ + [&w, rk, w, i, Ui1, Upl, UplDnm] ldff1\t%0.d, %5/z, [%1, %2.d] + [?w, rk, 0, i, Ui1, Upl, UplDnm] ^ + [&w, rk, w, i, i, Upl, UplDnm] ldff1\t%0.d, %5/z, [%1, %2.d, lsl %p4] + [?w, rk, w, i, i, Upl, UplDnm] ^ } "&& !CONSTANT_P (operands[6])" { @@ -1899,8 +1955,10 @@ UNSPEC_PRED_X))] "TARGET_SVE" {@ [cons: =0, 1, 2, 3, 4, 5] - [w, rk, w, i, Ui1, Upl ] ldff1\t%0.d, %5/z, [%1, %2.d, sxtw] - [w, rk, w, i, i, Upl ] ldff1\t%0.d, %5/z, [%1, %2.d, sxtw %p4] + [&w, rk, w, i, Ui1, Upl ] ldff1\t%0.d, %5/z, [%1, %2.d, sxtw] + [?w, rk, 0, i, Ui1, Upl ] ^ + [&w, rk, w, i, i, Upl ] ldff1\t%0.d, %5/z, [%1, %2.d, sxtw %p4] + [?w, rk, 0, i, i, Upl ] ^ } "&& (!CONSTANT_P (operands[6]) || !CONSTANT_P (operands[7]))" { @@ -1929,8 +1987,10 @@ UNSPEC_PRED_X))] "TARGET_SVE" {@ [cons: =0, 1, 2, 3, 4, 5] - [w, rk, w, i, Ui1, Upl ] ldff1\t%0.d, %5/z, [%1, %2.d, uxtw] - [w, rk, w, i, i, Upl ] ldff1\t%0.d, %5/z, [%1, %2.d, uxtw %p4] + [&w, rk, w, i, Ui1, Upl ] ldff1\t%0.d, %5/z, [%1, %2.d, uxtw] + [?w, rk, 0, i, Ui1, Upl ] ^ + [&w, rk, w, i, i, Upl ] ldff1\t%0.d, %5/z, [%1, %2.d, uxtw %p4] + [?w, rk, 0, i, i, Upl ] ^ } "&& !CONSTANT_P (operands[7])" { diff --git a/gcc/config/aarch64/aarch64-sve2.md b/gcc/config/aarch64/aarch64-sve2.md index f84ff743fc7..7a77e9b7502 100644 --- a/gcc/config/aarch64/aarch64-sve2.md +++ b/gcc/config/aarch64/aarch64-sve2.md @@ -111,8 +111,10 @@ UNSPEC_LDNT1_GATHER))] "TARGET_SVE2" {@ [cons: =0, 1, 2, 3] - [w, Upl, Z, w ] ldnt1\t%0., %1/z, [%3.] - [w, Upl, r, w ] ldnt1\t%0., %1/z, [%3., %2] + [&w, Upl, Z, w ] ldnt1\t%0., %1/z, [%3.] + [?w, Upl, Z, 0 ] ^ + [&w, Upl, r, w ] ldnt1\t%0., %1/z, [%3., %2] + [?w, Upl, r, 0 ] ^ } ) @@ -132,8 +134,10 @@ "TARGET_SVE2 && (~ & ) == 0" {@ [cons: =0, 1, 2, 3, 4] - [w, Upl, Z, w, UplDnm] ldnt1\t%0., %1/z, [%3.] - [w, Upl, r, w, UplDnm] ldnt1\t%0., %1/z, [%3., %2] + [&w, Upl, Z, w, UplDnm] ldnt1\t%0., %1/z, [%3.] + [?w, Upl, Z, 0, UplDnm] ^ + [&w, Upl, r, w, UplDnm] ldnt1\t%0., %1/z, [%3., %2] + [?w, Upl, r, 0, UplDnm] ^ } "&& !CONSTANT_P (operands[4])" { diff --git a/gcc/testsuite/gcc.target/aarch64/sve/gather_earlyclobber.c b/gcc/testsuite/gcc.target/aarch64/sve/gather_earlyclobber.c new file mode 100644 index 00000000000..db4a1db7f81 --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sve/gather_earlyclobber.c @@ -0,0 +1,96 @@ +/* { dg-do compile } */ +/* { dg-additional-options "-O2" } */ +/* { dg-final { check-function-bodies "**" "" } } */ + +#include + +/* +** food: +** ld1d (?:z[1-9][0-9]*)\.d, p0/z, \[x0, z0\.d\] +** ... +** ret +*/ + +svuint64_t +food (svbool_t p, uint64_t *in, svint64_t offsets, svuint64_t a) +{ + return svadd_u64_x (p, a, svld1_gather_offset(p, in, offsets)); +} + +/* +** foodb: +** ld1d (?:z[1-9][0-9]*)\.d, p0/z, \[z0\.d\] +** ... +** ret +*/ + +svuint64_t +foodb (svbool_t p, svuint64_t bases, svuint64_t a) +{ + return svadd_u64_x (p, a, svld1_gather_u64 (p, bases)); +} + +/* +** foodff: +** ldff1d (?:z[1-9][0-9]*)\.d, p0/z, \[z0\.d\] +** ... +** ret +*/ + +svuint64_t +foodff (svbool_t p, svuint64_t bases, svuint64_t a) +{ + return svadd_u64_x (p, a, svldff1_gather_u64 (p, bases)); +} + +/* +** foow: +** ld1w (?:z[1-9][0-9]*)\.s, p0/z, \[z0\.s\] +** ... +** ret +*/ + +svuint32_t +foow (svbool_t p, svuint32_t bases, svuint32_t a) +{ + return svadd_u32_x (p, a, svld1_gather_u32 (p, bases)); +} + +/* +** foowff: +** ldff1w (?:z[1-9][0-9]*)\.s, p0/z, \[z0\.s\] +** ... +** ret +*/ + +svuint32_t +foowff (svbool_t p, svuint32_t bases, svuint32_t a) +{ + return svadd_u32_x (p, a, svldff1_gather_u32 (p, bases)); +} + +/* +** fooubd: +** ld1b (?:z[1-9][0-9]*)\.d, p0/z, \[x0, z0\.d\] +** ... +** ret +*/ + +svuint64_t +fooubd (svbool_t p, uint8_t *base, svuint64_t offsets, svuint64_t a) +{ + return svadd_u64_x (p, a, svld1ub_gather_offset_u64 (p, base, offsets)); +} + +/* +** foosbd: +** ld1sb (?:z[1-9][0-9]*)\.d, p0/z, \[x0, z0\.d\] +** ... +** ret +*/ +svint64_t +foosbd (svbool_t p, int8_t *base, svint64_t offsets, svint64_t a) +{ + return svadd_s64_x (p, a, svld1sb_gather_offset_s64 (p, base, offsets)); +} + diff --git a/gcc/testsuite/gcc.target/aarch64/sve2/gather_earlyclobber.c b/gcc/testsuite/gcc.target/aarch64/sve2/gather_earlyclobber.c new file mode 100644 index 00000000000..a6003ccb896 --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sve2/gather_earlyclobber.c @@ -0,0 +1,32 @@ +/* { dg-do compile } */ +/* { dg-additional-options "-O2" } */ +/* { dg-final { check-function-bodies "**" "" } } */ + +#include + +/* +** foownt: +** ldnt1w (?:z[1-9][0-9]*)\.s, p0/z, \[z0\.s\] +** ... +** ret +*/ + +svuint32_t +foownt (svbool_t p, svuint32_t bases, svuint32_t a) +{ + return svadd_u32_x (p, a, svldnt1_gather_u32 (p, bases)); +} + +/* +** foodbnt: +** ldnt1d (?:z[1-9][0-9]*)\.d, p0/z, \[z0\.d\] +** ... +** ret +*/ + +svuint64_t +foodbnt (svbool_t p, svuint64_t bases, svuint64_t a) +{ + return svadd_u64_x (p, a, svldnt1_gather_u64 (p, bases)); +} +