From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by sourceware.org (Postfix) with ESMTP id AC4243858D33 for ; Tue, 21 Nov 2023 10:39:19 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org AC4243858D33 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=arm.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org AC4243858D33 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1700563162; cv=none; b=GSD5MPaHVzSWPzx1idcUapqU5MbZ/qUR0LzGGimusoVorEBe9fNA5He6tOH4RikAM9CzzbUnRn24OnGvpnraJq9W7qHMHm2uRhuCCSc2W1ijNay2V6EbStuOK/vuMTALjX5lJYYxNC6d7WTK69rwYRE+3ohXdO1czdJx8J82+UQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1700563162; c=relaxed/simple; bh=I8MemJC1HLiQZ6gEnevsxzR2Q6VnoJw6XsMVXcAiZTg=; h=From:To:Subject:Date:Message-ID:MIME-Version; b=oZoOJYmfyqhwAQVW5cofK+tjxrqeI4+QNhjg/s3rJx4Uhg2A7kPv2SkVgpReJyfV6ViA7+azRns49g+wBXBi5Uf/VxANZHfo1BypVk5+bdxJQerukFAdGjrlP1D2OpyLjn8R5E/tI1L9ypVCfqgaCauDrq14Qfm9hqxwZlEjx5Q= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id AA1C8FEC; Tue, 21 Nov 2023 02:40:05 -0800 (PST) Received: from localhost (e121540-lin.manchester.arm.com [10.32.110.72]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 02E173F6C4; Tue, 21 Nov 2023 02:39:17 -0800 (PST) From: Richard Sandiford To: Tamar Christina Mail-Followup-To: Tamar Christina ,gcc-patches@gcc.gnu.org, nd@arm.com, Richard.Earnshaw@arm.com, Marcus.Shawcroft@arm.com, Kyrylo.Tkachov@arm.com, richard.sandiford@arm.com Cc: gcc-patches@gcc.gnu.org, nd@arm.com, Richard.Earnshaw@arm.com, Marcus.Shawcroft@arm.com, Kyrylo.Tkachov@arm.com Subject: Re: [PATCH]AArch64 Add pattern for unsigned widenings (uxtl) to zip{1,2} References: Date: Tue, 21 Nov 2023 10:39:16 +0000 In-Reply-To: (Tamar Christina's message of "Wed, 15 Nov 2023 14:41:42 +0000") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Status: No, score=-22.7 required=5.0 tests=BAYES_00,GIT_PATCH_0,KAM_DMARC_NONE,KAM_DMARC_STATUS,KAM_LAZY_DOMAIN_SECURITY,KAM_LOTSOFHASH,KAM_SHORT,SPF_HELO_NONE,SPF_NONE,TXREP,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: Tamar Christina writes: > Hi All, > > This changes unpack instructions to use zip{1,2} when doing a zero-extending > widening operation. Permutes generally have a higher throughput than the > widening operations. Zeros are shuffled into the top half of the registers. > > The testcase > > void d2 (unsigned * restrict a, unsigned short *b, int n) > { > for (int i = 0; i < (n & -8); i++) > a[i] = b[i]; > } > > now generates: > > movi v1.4s, 0 > .L3: > ldr q0, [x1], 16 > zip1 v2.8h, v0.8h, v1.8h > zip2 v0.8h, v0.8h, v1.8h > stp q2, q0, [x0] > add x0, x0, 32 > cmp x1, x2 > bne .L3 > > > instead of: > > .L3: > ldr q0, [x1], 16 > uxtl v1.4s, v0.4h > uxtl2 v0.4s, v0.8h > stp q1, q0, [x0] > add x0, x0, 32 > cmp x1, x2 > bne .L3 > > Since we need the extra 0 register we do this only for the vectorizer's lo/hi > pairs when we know the 0 will be floated outside of the loop. The patterns are used by BB SLP as well, so we don't know for certain that there's a containing loop. We could provide patterns that match zips with zero, to allow the zero to be combined back into the pair if the zero doesn't get hoisted. The zip-with-zero patterns would need to have a higher cost than plain zips, so that 2 zips + movi has the same cost as 2 zips with zero. I guess that means that the zips with zero should have a cost of 1.5 insns. The late combine pass would then be able to get rid of the zero if it is stuck in the same basic block (or a block with the same execution frequency), but would keep a hoisted zero. But if we do one thing all the time or another thing all the time, I agree it's better to do the zips all the time. If this turns out to be bad for non-Neoverse cores, we can add some sort of tuning flag. But I agree that we should use zips unconditionally until we know of a specific core that doesn't want it. > This gives an 8% speed-up in Imagick in SPECCPU 2017 on Neoverse V2. > > Bootstrapped Regtested on aarch64-none-linux-gnu and no issues. > > Ok for master? > > Thanks, > Tamar > > gcc/ChangeLog: > > * config/aarch64/aarch64-simd.md (vec_unpack_lo_ vec_unpack_lo_ (vec_unpacku_lo_ vec_unpacku_lo_ (aarch64_usubw__zip): New. > (aarch64_uaddw__zip): New. > * config/aarch64/iterators.md (PERM_EXTEND, perm_index): New. > (perm_hilo): Add UNSPEC_ZIP1, UNSPEC_ZIP2. > > gcc/testsuite/ChangeLog: > > * gcc.target/aarch64/simd/vmovl_high_1.c: Update codegen. > * gcc.target/aarch64/uxtl-combine-1.c: New test. > * gcc.target/aarch64/uxtl-combine-2.c: New test. > * gcc.target/aarch64/uxtl-combine-3.c: New test. > * gcc.target/aarch64/uxtl-combine-4.c: New test. > * gcc.target/aarch64/uxtl-combine-5.c: New test. > * gcc.target/aarch64/uxtl-combine-6.c: New test. > > --- inline copy of patch -- > diff --git a/gcc/config/aarch64/aarch64-simd.md b/gcc/config/aarch64/aarch64-simd.md > index 81ff5bad03d598fa0d48df93d172a28bc0d1d92e..3d811007dd94dcd9176d6021a41a196c12fe9c3f 100644 > --- a/gcc/config/aarch64/aarch64-simd.md > +++ b/gcc/config/aarch64/aarch64-simd.md > @@ -1988,26 +1988,60 @@ (define_insn "aarch64_simd_vec_unpack_hi_" > [(set_attr "type" "neon_shift_imm_long")] > ) > > -(define_expand "vec_unpack_hi_" > +(define_expand "vec_unpacku_hi_" > [(match_operand: 0 "register_operand") > - (ANY_EXTEND: (match_operand:VQW 1 "register_operand"))] > + (match_operand:VQW 1 "register_operand")] > + "TARGET_SIMD" > + { > + rtx res = gen_reg_rtx (mode); > + rtx tmp = aarch64_gen_shareable_zero (mode); > + if (BYTES_BIG_ENDIAN) > + emit_insn (gen_aarch64_zip2 (res, tmp, operands[1])); > + else > + emit_insn (gen_aarch64_zip2 (res, operands[1], tmp)); > + emit_move_insn (operands[0], > + simplify_gen_subreg (mode, res, mode, 0)); > + DONE; > + } > +) > + > +(define_expand "vec_unpacks_hi_" > + [(match_operand: 0 "register_operand") > + (match_operand:VQW 1 "register_operand")] > "TARGET_SIMD" > { > rtx p = aarch64_simd_vect_par_cnst_half (mode, , true); > - emit_insn (gen_aarch64_simd_vec_unpack_hi_ (operands[0], > - operands[1], p)); > + emit_insn (gen_aarch64_simd_vec_unpacks_hi_ (operands[0], > + operands[1], p)); > + DONE; > + } > +) > + > +(define_expand "vec_unpacku_lo_" > + [(match_operand: 0 "register_operand") > + (match_operand:VQW 1 "register_operand")] > + "TARGET_SIMD" > + { > + rtx res = gen_reg_rtx (mode); > + rtx tmp = aarch64_gen_shareable_zero (mode); > + if (BYTES_BIG_ENDIAN) > + emit_insn (gen_aarch64_zip1 (res, tmp, operands[1])); > + else > + emit_insn (gen_aarch64_zip1 (res, operands[1], tmp)); > + emit_move_insn (operands[0], > + simplify_gen_subreg (mode, res, mode, 0)); > DONE; > } > ) > > -(define_expand "vec_unpack_lo_" > +(define_expand "vec_unpacks_lo_" > [(match_operand: 0 "register_operand") > - (ANY_EXTEND: (match_operand:VQW 1 "register_operand"))] > + (match_operand:VQW 1 "register_operand")] > "TARGET_SIMD" > { > rtx p = aarch64_simd_vect_par_cnst_half (mode, , false); > - emit_insn (gen_aarch64_simd_vec_unpack_lo_ (operands[0], > - operands[1], p)); > + emit_insn (gen_aarch64_simd_vec_unpacks_lo_ (operands[0], > + operands[1], p)); > DONE; > } > ) > @@ -4735,6 +4769,34 @@ (define_insn "aarch64_subw2_internal" > [(set_attr "type" "neon_sub_widen")] > ) > > +(define_insn "aarch64_usubw__zip" > + [(set (match_operand: 0 "register_operand" "=w") > + (minus: > + (match_operand: 1 "register_operand" "w") > + (subreg: > + (unspec: [ > + (match_operand:VQW 2 "register_operand" "w") > + (match_operand:VQW 3 "aarch64_simd_imm_zero" "Dz") IMO it's better to leave the Dz constraint out. The predicate does all the work here. OK with that change, thanks. Richard > + ] PERM_EXTEND) 0)))] > + "TARGET_SIMD" > + "usubw\\t%0., %1., %2." > + [(set_attr "type" "neon_sub_widen")] > +) > + > +(define_insn "aarch64_uaddw__zip" > + [(set (match_operand: 0 "register_operand" "=w") > + (plus: > + (subreg: > + (unspec: [ > + (match_operand:VQW 2 "register_operand" "w") > + (match_operand:VQW 3 "aarch64_simd_imm_zero" "Dz") > + ] PERM_EXTEND) 0) > + (match_operand: 1 "register_operand" "w")))] > + "TARGET_SIMD" > + "uaddw\\t%0., %1., %2." > + [(set_attr "type" "neon_add_widen")] > +) > + > (define_insn "aarch64_addw" > [(set (match_operand: 0 "register_operand" "=w") > (plus: > diff --git a/gcc/config/aarch64/iterators.md b/gcc/config/aarch64/iterators.md > index f9e2210095ea9d6d9c96971222a7757a2f418c2d..de281671aedc5141c69063f14cf0fbec5adecb04 100644 > --- a/gcc/config/aarch64/iterators.md > +++ b/gcc/config/aarch64/iterators.md > @@ -2674,6 +2674,9 @@ (define_int_iterator PERMUTEQ [UNSPEC_ZIP1Q UNSPEC_ZIP2Q > (define_int_iterator OPTAB_PERMUTE [UNSPEC_ZIP1 UNSPEC_ZIP2 > UNSPEC_UZP1 UNSPEC_UZP2]) > > +;; Permutes for zero extends > +(define_int_iterator PERM_EXTEND [UNSPEC_ZIP1 UNSPEC_ZIP2]) > + > (define_int_iterator REVERSE [UNSPEC_REV64 UNSPEC_REV32 UNSPEC_REV16]) > > (define_int_iterator FRINT [UNSPEC_FRINTZ UNSPEC_FRINTP UNSPEC_FRINTM > @@ -3496,7 +3499,10 @@ (define_int_attr rev_op [(UNSPEC_REV64 "64") (UNSPEC_REV32 "32") > (UNSPEC_REV16 "16")]) > > (define_int_attr perm_hilo [(UNSPEC_UNPACKSHI "hi") (UNSPEC_UNPACKUHI "hi") > - (UNSPEC_UNPACKSLO "lo") (UNSPEC_UNPACKULO "lo")]) > + (UNSPEC_UNPACKSLO "lo") (UNSPEC_UNPACKULO "lo") > + (UNSPEC_ZIP2 "hi") (UNSPEC_ZIP1 "lo")]) > + > +(define_int_attr perm_index [(UNSPEC_ZIP2 "2") (UNSPEC_ZIP1 "")]) > > ;; Return true if the associated optab refers to the high-numbered lanes, > ;; false if it refers to the low-numbered lanes. The convention is for > diff --git a/gcc/testsuite/gcc.target/aarch64/simd/vmovl_high_1.c b/gcc/testsuite/gcc.target/aarch64/simd/vmovl_high_1.c > index d45bb83e3503d512c443f37a446d30d188719a96..a2d09eaee0de5a3d3409330c5c26a3b5315e84eb 100644 > --- a/gcc/testsuite/gcc.target/aarch64/simd/vmovl_high_1.c > +++ b/gcc/testsuite/gcc.target/aarch64/simd/vmovl_high_1.c > @@ -22,11 +22,11 @@ FUNC (int32x4_t, int64x2_t, s32) > /* { dg-final { scan-assembler-times {sxtl2\tv0\.2d, v0\.4s} 1} } */ > > FUNC (uint8x16_t, uint16x8_t, u8) > -/* { dg-final { scan-assembler-times {uxtl2\tv0\.8h, v0\.16b} 1} } */ > +/* { dg-final { scan-assembler-times {zip2\tv0\.16b, v0\.16b} 1} } */ > > FUNC (uint16x8_t, uint32x4_t, u16) > -/* { dg-final { scan-assembler-times {uxtl2\tv0\.4s, v0\.8h} 1} } */ > +/* { dg-final { scan-assembler-times {zip2\tv0\.8h, v0\.8h} 1} } */ > > FUNC (uint32x4_t, uint64x2_t, u32) > -/* { dg-final { scan-assembler-times {uxtl2\tv0\.2d, v0\.4s} 1} } */ > +/* { dg-final { scan-assembler-times {zip2\tv0\.4s, v0\.4s} 1} } */ > > diff --git a/gcc/testsuite/gcc.target/aarch64/uxtl-combine-1.c b/gcc/testsuite/gcc.target/aarch64/uxtl-combine-1.c > new file mode 100755 > index 0000000000000000000000000000000000000000..68fa9a09fe55f5a72355e23c90e781a898c5975e > --- /dev/null > +++ b/gcc/testsuite/gcc.target/aarch64/uxtl-combine-1.c > @@ -0,0 +1,20 @@ > +/* { dg-do assemble } */ > +/* { dg-options "-O3 --save-temps --param=vect-epilogues-nomask=0" } */ > + > +#pragma GCC target "+nosve" > + > +#define SIGN unsigned > +#define TYPE1 char > +#define TYPE2 short > + > +void d2 (SIGN TYPE2 * restrict a, SIGN TYPE1 *b, int n) > +{ > + for (int i = 0; i < (n & -8); i++) > + a[i] = b[i]; > +} > + > +/* { dg-final { scan-assembler-times {\tzip1\t} 1 } } */ > +/* { dg-final { scan-assembler-times {\tzip2\t} 1 } } */ > +/* { dg-final { scan-assembler-not {\tuxtl\t} } } */ > +/* { dg-final { scan-assembler-not {\tuxtl2\t} } } */ > + > diff --git a/gcc/testsuite/gcc.target/aarch64/uxtl-combine-2.c b/gcc/testsuite/gcc.target/aarch64/uxtl-combine-2.c > new file mode 100755 > index 0000000000000000000000000000000000000000..af8a89085cfca800b41970a8410bc91b84a31d07 > --- /dev/null > +++ b/gcc/testsuite/gcc.target/aarch64/uxtl-combine-2.c > @@ -0,0 +1,20 @@ > +/* { dg-do assemble } */ > +/* { dg-options "-O3 --save-temps --param=vect-epilogues-nomask=0" } */ > + > +#pragma GCC target "+nosve" > + > +#define SIGN unsigned > +#define TYPE1 short > +#define TYPE2 int > + > +void d2 (SIGN TYPE2 * restrict a, SIGN TYPE1 *b, int n) > +{ > + for (int i = 0; i < (n & -8); i++) > + a[i] = b[i]; > +} > + > +/* { dg-final { scan-assembler-times {\tzip1\t} 1 } } */ > +/* { dg-final { scan-assembler-times {\tzip2\t} 1 } } */ > +/* { dg-final { scan-assembler-not {\tuxtl\t} } } */ > +/* { dg-final { scan-assembler-not {\tuxtl2\t} } } */ > + > diff --git a/gcc/testsuite/gcc.target/aarch64/uxtl-combine-3.c b/gcc/testsuite/gcc.target/aarch64/uxtl-combine-3.c > new file mode 100755 > index 0000000000000000000000000000000000000000..cdae6d09529b743857a092f53a07111df64775d7 > --- /dev/null > +++ b/gcc/testsuite/gcc.target/aarch64/uxtl-combine-3.c > @@ -0,0 +1,20 @@ > +/* { dg-do assemble } */ > +/* { dg-options "-O3 --save-temps --param=vect-epilogues-nomask=0" } */ > + > +#pragma GCC target "+nosve" > + > +#define SIGN unsigned > +#define TYPE1 int > +#define TYPE2 long long > + > +void d2 (SIGN TYPE2 * restrict a, SIGN TYPE1 *b, int n) > +{ > + for (int i = 0; i < (n & -8); i++) > + a[i] = b[i]; > +} > + > +/* { dg-final { scan-assembler-times {\tzip1\t} 1 } } */ > +/* { dg-final { scan-assembler-times {\tzip2\t} 1 } } */ > +/* { dg-final { scan-assembler-not {\tuxtl\t} } } */ > +/* { dg-final { scan-assembler-not {\tuxtl2\t} } } */ > + > diff --git a/gcc/testsuite/gcc.target/aarch64/uxtl-combine-4.c b/gcc/testsuite/gcc.target/aarch64/uxtl-combine-4.c > new file mode 100755 > index 0000000000000000000000000000000000000000..e1a9c4f5661a36ec7b2c5dc6f0fd85c42fcaac39 > --- /dev/null > +++ b/gcc/testsuite/gcc.target/aarch64/uxtl-combine-4.c > @@ -0,0 +1,20 @@ > +/* { dg-do assemble } */ > +/* { dg-options "-O3 --save-temps --param=vect-epilogues-nomask=0" } */ > + > +#pragma GCC target "+nosve" > + > +#define SIGN signed > +#define TYPE1 char > +#define TYPE2 short > + > +void d2 (SIGN TYPE2 * restrict a, SIGN TYPE1 *b, int n) > +{ > + for (int i = 0; i < (n & -8); i++) > + a[i] = b[i]; > +} > + > +/* { dg-final { scan-assembler-not {\tzip1\t} } } */ > +/* { dg-final { scan-assembler-not {\tzip2\t} } } */ > +/* { dg-final { scan-assembler-times {\tsxtl\t} 1 } } */ > +/* { dg-final { scan-assembler-time {\tsxtl2\t} 1 } } */ > + > diff --git a/gcc/testsuite/gcc.target/aarch64/uxtl-combine-5.c b/gcc/testsuite/gcc.target/aarch64/uxtl-combine-5.c > new file mode 100755 > index 0000000000000000000000000000000000000000..92b09ba4abba80f240ac175be2ef880968534975 > --- /dev/null > +++ b/gcc/testsuite/gcc.target/aarch64/uxtl-combine-5.c > @@ -0,0 +1,20 @@ > +/* { dg-do assemble } */ > +/* { dg-options "-O3 --save-temps --param=vect-epilogues-nomask=0" } */ > + > +#pragma GCC target "+nosve" > + > +#define SIGN signed > +#define TYPE1 short > +#define TYPE2 int > + > +void d2 (SIGN TYPE2 * restrict a, SIGN TYPE1 *b, int n) > +{ > + for (int i = 0; i < (n & -8); i++) > + a[i] = b[i]; > +} > + > +/* { dg-final { scan-assembler-not {\tzip1\t} } } */ > +/* { dg-final { scan-assembler-not {\tzip2\t} } } */ > +/* { dg-final { scan-assembler-times {\tsxtl\t} 1 } } */ > +/* { dg-final { scan-assembler-time {\tsxtl2\t} 1 } } */ > + > diff --git a/gcc/testsuite/gcc.target/aarch64/uxtl-combine-6.c b/gcc/testsuite/gcc.target/aarch64/uxtl-combine-6.c > new file mode 100755 > index 0000000000000000000000000000000000000000..5c6e635f29d1e52f51f5b75a477f7d8744f32ca3 > --- /dev/null > +++ b/gcc/testsuite/gcc.target/aarch64/uxtl-combine-6.c > @@ -0,0 +1,20 @@ > +/* { dg-do assemble } */ > +/* { dg-options "-O3 --save-temps --param=vect-epilogues-nomask=0" } */ > + > +#pragma GCC target "+nosve" > + > +#define SIGN signed > +#define TYPE1 int > +#define TYPE2 long long > + > +void d2 (SIGN TYPE2 * restrict a, SIGN TYPE1 *b, int n) > +{ > + for (int i = 0; i < (n & -8); i++) > + a[i] = b[i]; > +} > + > +/* { dg-final { scan-assembler-not {\tzip1\t} } } */ > +/* { dg-final { scan-assembler-not {\tzip2\t} } } */ > +/* { dg-final { scan-assembler-times {\tsxtl\t} 1 } } */ > +/* { dg-final { scan-assembler-time {\tsxtl2\t} 1 } } */ > +