On Sat, Jun 8, 2024 at 3:41 PM Richard Sandiford <richard.sandiford@arm.com>
wrote:

> Mariam Arutunian <mariamarutunian@gmail.com> writes:
> > This patch introduces two new expanders for the aarch64 backend,
> > dedicated to generate optimized code for CRC computations.
> > The new expanders are designed to leverage specific hardware capabilities
> > to achieve faster CRC calculations,
> > particularly using the pmul or crc32 instructions when supported by the
> > target architecture.
>
> Thanks for porting this to aarch64!
>
> > Expander 1: Bit-Forward CRC (crc<ALLI:mode><ALLX:mode>4)
> > For targets that support pmul instruction (TARGET_AES),
> > the expander will generate code that uses the pmul (crypto_pmulldi)
> > instruction for CRC computation.
> >
> > Expander 2: Bit-Reversed CRC (crc_rev<ALLI:mode><ALLX:mode>4)
> > The expander first checks if the target supports the CRC32 instruction
> set
> > (TARGET_CRC32)
> > and the polynomial in use is 0x1EDC6F41 (iSCSI). If the conditions are
> met,
> > it emits calls to the corresponding crc32 instruction (crc32b, crc32h,
> > crc32w, or crc32x depending on the data size).
> > If the target does not support crc32 but supports pmul, it then uses the
> > pmul (crypto_pmulldi) instruction for bit-reversed CRC computation.
> >
> > Otherwise table-based CRC is generated.
> >
> >   gcc/config/aarch64/
> >
> >     * aarch64-protos.h (aarch64_expand_crc_using_clmul): New extern
> > function declaration.
> >     (aarch64_expand_reversed_crc_using_clmul):  Likewise.
> >     * aarch64.cc (aarch64_expand_crc_using_clmul): New function.
> >     (aarch64_expand_reversed_crc_using_clmul):  Likewise.
> >     * aarch64.md (UNSPEC_CRC, UNSPEC_CRC_REV):  New unspecs.
> >     (crc_rev<ALLI:mode><ALLX:mode>4): New expander for reversed CRC.
> >     (crc<ALLI:mode><ALLX:mode>4): New expander for reversed CRC.
> >     * iterators.md (crc_data_type): New mode attribute.
> >
> >   gcc/testsuite/gcc.target/aarch64/
> >
> >     * crc-1-pmul.c: Likewise.
> >     * crc-10-pmul.c: Likewise.
> >     * crc-12-pmul.c: Likewise.
> >     * crc-13-pmul.c: Likewise.
> >     * crc-14-pmul.c: Likewise.
> >     * crc-17-pmul.c: Likewise.
> >     * crc-18-pmul.c: Likewise.
> >     * crc-21-pmul.c: Likewise.
> >     * crc-22-pmul.c: Likewise.
> >     * crc-23-pmul.c: Likewise.
> >     * crc-4-pmul.c: Likewise.
> >     * crc-5-pmul.c: Likewise.
> >     * crc-6-pmul.c: Likewise.
> >     * crc-7-pmul.c: Likewise.
> >     * crc-8-pmul.c: Likewise.
> >     * crc-9-pmul.c: Likewise.
> >     * crc-CCIT-data16-pmul.c: Likewise.
> >     * crc-CCIT-data8-pmul.c: Likewise.
> >     * crc-coremark-16bitdata-pmul.c: Likewise.
> >     * crc-crc32-data16.c: New test.
> >     * crc-crc32-data32.c: Likewise.
> >     * crc-crc32-data8.c: Likewise.
> >
> > Signed-off-by: Mariam Arutunian <mariamarutunian@gmail.com
> > diff --git a/gcc/config/aarch64/aarch64-protos.h
> b/gcc/config/aarch64/aarch64-protos.h
> > index 1d3f94c813e..167e1140f0d 100644
> > --- a/gcc/config/aarch64/aarch64-protos.h
> > +++ b/gcc/config/aarch64/aarch64-protos.h
> > @@ -1117,5 +1117,8 @@ extern void mingw_pe_encode_section_info (tree,
> rtx, int);
> >
> >  bool aarch64_optimize_mode_switching (aarch64_mode_entity);
> >  void aarch64_restore_za (rtx);
> > +void aarch64_expand_crc_using_clmul (rtx *);
> > +void aarch64_expand_reversed_crc_using_clmul (rtx *);
> > +
> >
> >  #endif /* GCC_AARCH64_PROTOS_H */
> > diff --git a/gcc/config/aarch64/aarch64.cc
> b/gcc/config/aarch64/aarch64.cc
> > index ee12d8897a8..05cd0296d38 100644
> > --- a/gcc/config/aarch64/aarch64.cc
> > +++ b/gcc/config/aarch64/aarch64.cc
> > @@ -30265,6 +30265,135 @@ aarch64_retrieve_sysreg (const char *regname,
> bool write_p, bool is128op)
> >    return sysreg->encoding;
> >  }
> >
> > +/* Generate assembly to calculate CRC
> > +   using carry-less multiplication instruction.
> > +   OPERANDS[1] is input CRC,
> > +   OPERANDS[2] is data (message),
> > +   OPERANDS[3] is the polynomial without the leading 1.  */
> > +
> > +void
> > +aarch64_expand_crc_using_clmul (rtx *operands)
>
> This should probably be pmul rather than clmul.
>
> > +{
> > +  /* Check and keep arguments.  */
> > +  gcc_assert (!CONST_INT_P (operands[0]));
> > +  gcc_assert (CONST_INT_P (operands[3]));
> > +  rtx crc = operands[1];
> > +  rtx data = operands[2];
> > +  rtx polynomial = operands[3];
> > +
> > +  unsigned HOST_WIDE_INT
> > +      crc_size = GET_MODE_BITSIZE (GET_MODE (operands[0])).to_constant
> ();
> > +  gcc_assert (crc_size <= 32);
> > +  unsigned HOST_WIDE_INT
> > +      data_size = GET_MODE_BITSIZE (GET_MODE (data)).to_constant ();
>
> We could instead make the interface:
>
> void
> aarch64_expand_crc_using_pmul (scalar_mode crc_mode, scalar_mode data_mode,
>                                rtx *operands)
>
> so that the lines above don't need the to_constant.  This should "just
> work" on the .md file side, since the modes being passed are naturally
> scalar_mode.
>
> I think it'd be worth asserting also that data_size <= crc_size.
> (Although we could handle any MAX (data_size, crc_size) <= 32
> with some adjustment.)
>
> > +
> > +  /* Calculate the quotient.  */
> > +  unsigned HOST_WIDE_INT
> > +      q = gf2n_poly_long_div_quotient (UINTVAL (polynomial), crc_size +
> 1);
> > +
> > +  /* CRC calculation's main part.  */
> > +  if (crc_size > data_size)
> > +    crc = expand_shift (RSHIFT_EXPR, DImode, crc, crc_size - data_size,
> > +                     NULL_RTX, 1);
> > +
> > +  rtx t0 = gen_reg_rtx (DImode);
> > +  aarch64_emit_move (t0, gen_int_mode (q, DImode));
>
> It's only a minor simplification, but this could instead be:
>
>   rtx t0 = force_reg (DImode, gen_int_mode (q, DImode));
>
> > +  rtx t1 = gen_reg_rtx (DImode);
> > +  aarch64_emit_move (t1, polynomial);
>
> If polynomial is a constant operand of mode crc_mode, GCC's standard
> CONST_INT representation is to sign-extend the constant to 64 bits.
> E.g. a QImode value of 0b1000_0000 would be represented as -128.
>
> I think here we want the zero-extended form, so it might be safer to do:
>
>   polynomial = simplify_gen_unary (ZERO_EXTEND, DImode, polynomial,
> crc_mode);
>   rtx t1 = force_reg (DImode, polynomial);
>
> > +
> > +  rtx a0 = expand_binop (DImode, xor_optab, crc, data, NULL_RTX, 1,
> > +                      OPTAB_WIDEN);
> > +
> > +  rtx clmul_res = gen_reg_rtx (TImode);
> > +  emit_insn (gen_aarch64_crypto_pmulldi (clmul_res, a0, t0));
> > +  a0 = gen_lowpart (DImode, clmul_res);
> > +
> > +  a0 = expand_shift (RSHIFT_EXPR, DImode, a0, crc_size, NULL_RTX, 1);
> > +
> > +  emit_insn (gen_aarch64_crypto_pmulldi (clmul_res, a0, t1));
> > +  a0 = gen_lowpart (DImode, clmul_res);
> > +
> > +  if (crc_size > data_size)
> > +    {
> > +      rtx crc_part = expand_shift (LSHIFT_EXPR, DImode, operands[1],
> data_size,
> > +                                NULL_RTX, 0);
> > +      a0 =  expand_binop (DImode, xor_optab, a0, crc_part, NULL_RTX, 1,
> > +                       OPTAB_DIRECT);
>
> Formatting nit: extra space after "a0 = "
>
> > +    }
> > +  /* Zero upper bits beyond crc_size.  */
> > +  rtx num_shift = gen_int_mode (64 - crc_size, DImode);
> > +  a0 = expand_shift (LSHIFT_EXPR, DImode, a0, 64 - crc_size,  NULL_RTX,
> 0);
> > +  a0 = expand_shift (RSHIFT_EXPR, DImode, a0, 64 - crc_size,  NULL_RTX,
> 1);
>
> Rather than shift left and then right, I think we should just AND:
>
>   rtx mask = gen_int_mode (GET_MODE_MASK (crc_mode), DImode);
>   a0 = expand_binop (DImode, and_optab, a0, mask, NULL_RTX, 1,
> OPTAB_DIRECT);
>
> That said, it looks like operands[0] has crc_mode.  The register bits
> above crc_size therefore shouldn't matter, since they're undefined on read.
> E.g. even though (reg:SI R) is stored in an X register, only the low 32
> bits are defined; the upper 32 bits can be any value.
>
> So I'd expect we could replace this and...
>
> > +
> > +  rtx tgt = simplify_gen_subreg (DImode, operands[0],
> > +                              GET_MODE (operands[0]), 0);
> > +  aarch64_emit_move (tgt, a0);
>
> ...this with just:
>
>   aarch64_emitmove (operands[0], gen_lowpart (crc_mode, a0));
>
> Perhaps that would break down if operands[0] is a subreg with
> SUBREG_PROMOTED_VAR_P set, but I think it's up to target-independent
> code to handle that case.
>
> > @@ -4543,6 +4545,63 @@
> >    [(set_attr "type" "crc")]
> >  )
> >
> > +;; Reversed CRC
> > +(define_expand "crc_rev<ALLI:mode><ALLX:mode>4"
> > +      ;; return value (calculated CRC)
> > +  [(set (match_operand:ALLX 0 "register_operand" "=r")
> > +                   ;; initial CRC
> > +     (unspec:ALLX [(match_operand:ALLX 1 "register_operand" "r")
> > +                   ;; data
> > +                   (match_operand:ALLI 2 "register_operand" "r")
> > +                   ;; polynomial without leading 1
> > +                   (match_operand:ALLX 3)]
> > +     UNSPEC_CRC_REV))]
>
> Since we (rightly) never generate the RTL above, I think this can just be:
>
> (define_expand "crc_rev<ALLI:mode><ALLX:mode>4"
>   [;; return value (calculated CRC)
>    (match_operand:ALLX 0 "register_operand")
>    ;; initial CRC
>    (match_operand:ALLX 1 "register_operand")
>    ;; data
>    (match_operand:ALLI 2 "register_operand")
>    ;; polynomial without leading 1
>    (match_operand:ALLX 3)]
>
> without the unspec and constraints.
>
> > +  ""
> > +  {
> > +    /* If the polynomial is the same as the polynomial of crc32
> instruction,
> > +       put that instruction.  crc32 uses iSCSI polynomial
> (0x1EDC6F41).  */
> > +    if (TARGET_CRC32 && INTVAL (operands[3]) == 517762881)
>
> The hex constant feels a little easier to read.  I think it'd also
> be worth checking <ALLX:MODE>mode == SImode, even though it's currently
> redundant (given that no other choice would allow that polynomial).
>
> > +      {
> > +     rtx crc_result = gen_reg_rtx (SImode);
> > +     rtx crc = operands[1];
> > +     rtx data = operands[2];
> > +     emit_insn (gen_aarch64_crc32c<ALLI:crc_data_type> (crc_result, crc,
> > +                                                        data));
> > +     emit_move_insn (operands[0],
> > +                     gen_lowpart (GET_MODE (operands[0]), crc_result));
>
> If operands[0] has ALLX mode (== SImode), it looks like we should be
> able to use operands[0] directly as the result of the CRC32C.
>
> FWIW, there's also CRC32 for the HDLC etc. polynomial 0x04C11DB7.
>
> > +      }
> > +    else if (TARGET_AES)
>
> I think we also need to check <ALLI:sizen> <= <ALLX:sizen> for this.
> Similarly for the unreversed CRC pattern.
>
> Thanks again for doing this.  I realise RISC-V is the lead target for
> this work, so you've gone above and beyond by doing a full AArch64
> port too.  It'd be perfectly valid to ask Arm developers to deal
> with the comments above, so please let me know if you'd prefer that.
> The patch looks close to ready to me though.
>

Thanks for your suggestions and explanations, and thank you for recognizing
my work. I'll resolve all the comments.


Best regards,
Mariam


> Richard
>
> > +      aarch64_expand_reversed_crc_using_clmul (operands);
> > +    else
> > +      {
> > +     /* Otherwise, generate table-based CRC.  */
> > +     expand_reversed_crc_table_based (operands[0], operands[1],
> operands[2],
> > +                                      operands[3], GET_MODE
> (operands[2]),
> > +
> generate_reflecting_code_standard);
> > +      }
> > +    DONE;
> > +  }
> > +)
> > +
> > +;; Bit-forward CRC
> > +(define_expand "crc<ALLI:mode><ALLX:mode>4"
> > +      ;; return value (calculated CRC)
> > +  [(set (match_operand:ALLX 0 "register_operand" "=r")
> > +                   ;; initial CRC
> > +     (unspec:ALLX [(match_operand:ALLX 1 "register_operand" "r")
> > +                   ; data
> > +                   (match_operand:ALLI 2 "register_operand" "r")
> > +                   ;; polynomial without leading 1
> > +                   (match_operand:ALLX 3)]
> > +     UNSPEC_CRC))]
> > +  "TARGET_AES"
> > +  {
> > +    aarch64_expand_crc_using_clmul (operands);
> > +    DONE;
> > +  }
> > +)
> > +
> > +
> >  (define_insn "*csinc2<mode>_insn"
> >    [(set (match_operand:GPI 0 "register_operand" "=r")
> >          (plus:GPI (match_operand 2 "aarch64_comparison_operation" "")
> > diff --git a/gcc/config/aarch64/iterators.md
> b/gcc/config/aarch64/iterators.md
> > index 99cde46f1ba..86e4863d684 100644
> > --- a/gcc/config/aarch64/iterators.md
> > +++ b/gcc/config/aarch64/iterators.md
> > @@ -1276,6 +1276,10 @@
> >  ;; Map a mode to a specific constraint character.
> >  (define_mode_attr cmode [(QI "q") (HI "h") (SI "s") (DI "d")])
> >
> > +;; Map a mode to a specific constraint character for calling
> > +;; appropriate version of crc.
> > +(define_mode_attr crc_data_type [(QI "b") (HI "h") (SI "w") (DI "x")])
> > +
> >  ;; Map modes to Usg and Usj constraints for SISD right shifts
> >  (define_mode_attr cmode_simd [(SI "g") (DI "j")])
> >
> > diff --git a/gcc/testsuite/gcc.target/aarch64/crc-1-pmul.c
> b/gcc/testsuite/gcc.target/aarch64/crc-1-pmul.c
> > new file mode 100644
> > index 00000000000..2bea6280762
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/aarch64/crc-1-pmul.c
> > @@ -0,0 +1,8 @@
> > +/* { dg-do run } */
> > +/* { dg-options "-march=armv8-a+crypto -O2 -fdump-rtl-dfinish
> -fdump-tree-crc-details -fdisable-tree-phiopt2 -fdisable-tree-phiopt3" } */
> > +
> > +#include "../../gcc.c-torture/execute/crc-1.c"
> > +
> > +/* { dg-final { scan-tree-dump "calculates CRC!" "crc"} } */
> > +/* { dg-final { scan-tree-dump-times "Couldn't generate faster CRC
> code." 0 "crc"} } */
> > +/* { dg-final { scan-rtl-dump "pmull" "dfinish"} } */
> > \ No newline at end of file
> > diff --git a/gcc/testsuite/gcc.target/aarch64/crc-10-pmul.c
> b/gcc/testsuite/gcc.target/aarch64/crc-10-pmul.c
> > new file mode 100644
> > index 00000000000..846eecbaa85
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/aarch64/crc-10-pmul.c
> > @@ -0,0 +1,9 @@
> > +/* { dg-do run } */
> > +/* { dg-options "-march=armv8-a+crypto -O2 -fdump-rtl-dfinish
> -fdump-tree-crc-details" } */
> > +/* { dg-skip-if "" { *-*-* } { "-flto"} } */
> > +
> > +#include "../../gcc.c-torture/execute/crc-10.c"
> > +
> > +/* { dg-final { scan-tree-dump "calculates CRC!" "crc"} } */
> > +/* { dg-final { scan-tree-dump-times "Couldn't generate faster CRC
> code." 0 "crc"} } */
> > +/* { dg-final { scan-rtl-dump "pmull" "dfinish"} } */
> > diff --git a/gcc/testsuite/gcc.target/aarch64/crc-12-pmul.c
> b/gcc/testsuite/gcc.target/aarch64/crc-12-pmul.c
> > new file mode 100644
> > index 00000000000..0eea6aa6741
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/aarch64/crc-12-pmul.c
> > @@ -0,0 +1,9 @@
> > +/* { dg-do run } */
> > +/* { dg-options "-march=armv8-a+crypto -O2 -fdump-rtl-dfinish
> -fdump-tree-crc-details -fdisable-tree-phiopt2 -fdisable-tree-phiopt3" } */
> > +/* { dg-skip-if "" { *-*-* } { "-flto"} } */
> > +
> > +#include "../../gcc.c-torture/execute/crc-12.c"
> > +
> > +/* { dg-final { scan-tree-dump "calculates CRC!" "crc"} } */
> > +/* { dg-final { scan-tree-dump-times "Couldn't generate faster CRC
> code." 0 "crc"} } */
> > +/* { dg-final { scan-rtl-dump "pmull" "dfinish"} } */
> > diff --git a/gcc/testsuite/gcc.target/aarch64/crc-13-pmul.c
> b/gcc/testsuite/gcc.target/aarch64/crc-13-pmul.c
> > new file mode 100644
> > index 00000000000..7ff8fbcb665
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/aarch64/crc-13-pmul.c
> > @@ -0,0 +1,9 @@
> > +/* { dg-do run } */
> > +/* { dg-options "-march=armv8-a+crypto -O2 -fdump-rtl-dfinish
> -fdump-tree-crc-details" } */
> > +/* { dg-skip-if "" { *-*-* } { "-flto"} } */
> > +
> > +#include "../../gcc.c-torture/execute/crc-13.c"
> > +
> > +/* { dg-final { scan-tree-dump "calculates CRC!" "crc"} } */
> > +/* { dg-final { scan-tree-dump-times "Couldn't generate faster CRC
> code." 0 "crc"} } */
> > +/* { dg-final { scan-rtl-dump "pmull" "dfinish"} } */
> > diff --git a/gcc/testsuite/gcc.target/aarch64/crc-14-pmul.c
> b/gcc/testsuite/gcc.target/aarch64/crc-14-pmul.c
> > new file mode 100644
> > index 00000000000..80766daf487
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/aarch64/crc-14-pmul.c
> > @@ -0,0 +1,9 @@
> > +/* { dg-do run } */
> > +/* { dg-options "-march=armv8-a+crypto -O2 -fdump-rtl-dfinish
> -fdump-tree-crc-details" } */
> > +/* { dg-skip-if "" { *-*-* } { "-flto"} } */
> > +
> > +#include "../../gcc.c-torture/execute/crc-14.c"
> > +
> > +/* { dg-final { scan-tree-dump "calculates CRC!" "crc"} } */
> > +/* { dg-final { scan-tree-dump-times "Couldn't generate faster CRC
> code." 0 "crc"} } */
> > +/* { dg-final { scan-rtl-dump "pmull" "dfinish"} } */
> > diff --git a/gcc/testsuite/gcc.target/aarch64/crc-17-pmul.c
> b/gcc/testsuite/gcc.target/aarch64/crc-17-pmul.c
> > new file mode 100644
> > index 00000000000..0e32fffa0b6
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/aarch64/crc-17-pmul.c
> > @@ -0,0 +1,9 @@
> > +/* { dg-do run } */
> > +/* { dg-options "-march=armv8-a+crypto -O2 -fdump-rtl-dfinish
> -fdump-tree-crc-details" } */
> > +/* { dg-skip-if "" { *-*-* } { "-flto"} } */
> > +
> > +#include "../../gcc.c-torture/execute/crc-17.c"
> > +
> > +/* { dg-final { scan-tree-dump "calculates CRC!" "crc"} } */
> > +/* { dg-final { scan-tree-dump-times "Couldn't generate faster CRC
> code." 0 "crc"} } */
> > +/* { dg-final { scan-rtl-dump "pmull" "dfinish"} } */
> > diff --git a/gcc/testsuite/gcc.target/aarch64/crc-18-pmul.c
> b/gcc/testsuite/gcc.target/aarch64/crc-18-pmul.c
> > new file mode 100644
> > index 00000000000..87f4c63b5ea
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/aarch64/crc-18-pmul.c
> > @@ -0,0 +1,9 @@
> > +/* { dg-do run } */
> > +/* { dg-options "-march=armv8-a+crypto -O2 -fdump-rtl-dfinish
> -fdump-tree-crc-details" } */
> > +/* { dg-skip-if "" { *-*-* } { "-flto"} } */
> > +
> > +#include "../../gcc.c-torture/execute/crc-18.c"
> > +
> > +/* { dg-final { scan-tree-dump "calculates CRC!" "crc"} } */
> > +/* { dg-final { scan-tree-dump-times "Couldn't generate faster CRC
> code." 0 "crc"} } */
> > +/* { dg-final { scan-rtl-dump "pmull" "dfinish"} } */
> > diff --git a/gcc/testsuite/gcc.target/aarch64/crc-21-pmul.c
> b/gcc/testsuite/gcc.target/aarch64/crc-21-pmul.c
> > new file mode 100644
> > index 00000000000..6eeac8cf97f
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/aarch64/crc-21-pmul.c
> > @@ -0,0 +1,9 @@
> > +/* { dg-do run } */
> > +/* { dg-options "-march=armv8-a+crypto -O2 -fdump-rtl-dfinish
> -fdump-tree-crc-details" } */
> > +/* { dg-skip-if "" { *-*-* } { "-flto"} } */
> > +
> > +#include "../../gcc.c-torture/execute/crc-21.c"
> > +
> > +/* { dg-final { scan-tree-dump "calculates CRC!" "crc"} } */
> > +/* { dg-final { scan-tree-dump-times "Couldn't generate faster CRC
> code." 0 "crc"} } */
> > +/* { dg-final { scan-rtl-dump "pmull" "dfinish"} } */
> > diff --git a/gcc/testsuite/gcc.target/aarch64/crc-22-pmul.c
> b/gcc/testsuite/gcc.target/aarch64/crc-22-pmul.c
> > new file mode 100644
> > index 00000000000..76e3c00ce9f
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/aarch64/crc-22-pmul.c
> > @@ -0,0 +1,9 @@
> > +/* { dg-do run } */
> > +/* { dg-options "-march=armv8-a+crypto -O2 -fdump-rtl-dfinish
> -fdump-tree-crc-details" } */
> > +/* { dg-skip-if "" { *-*-* } { "-flto"} } */
> > +
> > +#include "../../gcc.c-torture/execute/crc-22.c"
> > +
> > +/* { dg-final { scan-tree-dump "calculates CRC!" "crc"} } */
> > +/* { dg-final { scan-tree-dump-times "Couldn't generate faster CRC
> code." 0 "crc"} } */
> > +/* { dg-final { scan-rtl-dump "pmull" "dfinish"} } */
> > diff --git a/gcc/testsuite/gcc.target/aarch64/crc-23-pmul.c
> b/gcc/testsuite/gcc.target/aarch64/crc-23-pmul.c
> > new file mode 100644
> > index 00000000000..e3a5e99ffba
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/aarch64/crc-23-pmul.c
> > @@ -0,0 +1,9 @@
> > +/* { dg-do run } */
> > +/* { dg-options "-march=armv8-a+crypto -O2 -fdump-rtl-dfinish
> -fdump-tree-crc-details" } */
> > +/* { dg-skip-if "" { *-*-* } { "-flto"} } */
> > +
> > +#include "../../gcc.c-torture/execute/crc-23.c"
> > +
> > +/* { dg-final { scan-tree-dump "calculates CRC!" "crc"} } */
> > +/* { dg-final { scan-tree-dump-times "Couldn't generate faster CRC
> code." 0 "crc"} } */
> > +/* { dg-final { scan-rtl-dump "pmull" "dfinish"} } */
> > diff --git a/gcc/testsuite/gcc.target/aarch64/crc-4-pmul.c
> b/gcc/testsuite/gcc.target/aarch64/crc-4-pmul.c
> > new file mode 100644
> > index 00000000000..528006c0099
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/aarch64/crc-4-pmul.c
> > @@ -0,0 +1,9 @@
> > +/* { dg-do run } */
> > +/* { dg-options "-march=armv8-a+crypto -O2 -fdump-rtl-dfinish
> -fdump-tree-crc-details" } */
> > +/* { dg-skip-if "" { *-*-* } { "-flto"} } */
> > +
> > +#include "../../gcc.c-torture/execute/crc-4.c"
> > +
> > +/* { dg-final { scan-tree-dump "calculates CRC!" "crc"} } */
> > +/* { dg-final { scan-tree-dump-times "Couldn't generate faster CRC
> code." 0 "crc"} } */
> > +/* { dg-final { scan-rtl-dump "pmull" "dfinish"} } */
> > diff --git a/gcc/testsuite/gcc.target/aarch64/crc-5-pmul.c
> b/gcc/testsuite/gcc.target/aarch64/crc-5-pmul.c
> > new file mode 100644
> > index 00000000000..41e1f8202bc
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/aarch64/crc-5-pmul.c
> > @@ -0,0 +1,9 @@
> > +/* { dg-do run } */
> > +/* { dg-options "-march=armv8-a+crypto -O2 -w -fdump-rtl-dfinish
> -fdump-tree-crc-details" } */
> > +/* { dg-skip-if "" { *-*-* } { "-flto"} } */
> > +
> > +#include "../../gcc.c-torture/execute/crc-5.c"
> > +
> > +/* { dg-final { scan-tree-dump "calculates CRC!" "crc"} } */
> > +/* { dg-final { scan-tree-dump-times "Couldn't generate faster CRC
> code." 0 "crc"} } */
> > +/* { dg-final { scan-rtl-dump "pmull" "dfinish"} } */
> > \ No newline at end of file
> > diff --git a/gcc/testsuite/gcc.target/aarch64/crc-6-pmul.c
> b/gcc/testsuite/gcc.target/aarch64/crc-6-pmul.c
> > new file mode 100644
> > index 00000000000..83db99ccb8b
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/aarch64/crc-6-pmul.c
> > @@ -0,0 +1,9 @@
> > +/* { dg-do run } */
> > +/* { dg-options "-march=armv8-a+crypto -O2 -fdump-rtl-dfinish
> -fdump-tree-crc-details" } */
> > +/* { dg-skip-if "" { *-*-* } { "-flto"} } */
> > +
> > +#include "../../gcc.c-torture/execute/crc-6.c"
> > +
> > +/* { dg-final { scan-tree-dump "calculates CRC!" "crc"} } */
> > +/* { dg-final { scan-tree-dump-times "Couldn't generate faster CRC
> code." 0 "crc"} } */
> > +/* { dg-final { scan-rtl-dump "pmull" "dfinish"} } */
> > \ No newline at end of file
> > diff --git a/gcc/testsuite/gcc.target/aarch64/crc-7-pmul.c
> b/gcc/testsuite/gcc.target/aarch64/crc-7-pmul.c
> > new file mode 100644
> > index 00000000000..7ad777aac8c
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/aarch64/crc-7-pmul.c
> > @@ -0,0 +1,9 @@
> > +/* { dg-do run } */
> > +/* { dg-options "-march=armv8-a+crypto -O2 -fdump-rtl-dfinish
> -fdump-tree-crc-details" } */
> > +/* { dg-skip-if "" { *-*-* } { "-flto"} } */
> > +
> > +#include "../../gcc.c-torture/execute/crc-7.c"
> > +
> > +/* { dg-final { scan-tree-dump "calculates CRC!" "crc"} } */
> > +/* { dg-final { scan-tree-dump-times "Couldn't generate faster CRC
> code." 0 "crc"} } */
> > +/* { dg-final { scan-rtl-dump "pmull" "dfinish"} } */
> > diff --git a/gcc/testsuite/gcc.target/aarch64/crc-8-pmul.c
> b/gcc/testsuite/gcc.target/aarch64/crc-8-pmul.c
> > new file mode 100644
> > index 00000000000..da1b619c418
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/aarch64/crc-8-pmul.c
> > @@ -0,0 +1,9 @@
> > +/* { dg-do run } */
> > +/* { dg-options "-march=armv8-a+crypto -O2 -fdump-rtl-dfinish
> -fdump-tree-crc-details" } */
> > +/* { dg-skip-if "" { *-*-* } { "-flto"} } */
> > +
> > +#include "../../gcc.c-torture/execute/crc-8.c"
> > +
> > +/* { dg-final { scan-tree-dump "calculates CRC!" "crc"} } */
> > +/* { dg-final { scan-tree-dump-times "Couldn't generate faster CRC
> code." 0 "crc"} } */
> > +/* { dg-final { scan-rtl-dump "pmull" "dfinish"} } */
> > diff --git a/gcc/testsuite/gcc.target/aarch64/crc-9-pmul.c
> b/gcc/testsuite/gcc.target/aarch64/crc-9-pmul.c
> > new file mode 100644
> > index 00000000000..33bbe0bfb26
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/aarch64/crc-9-pmul.c
> > @@ -0,0 +1,9 @@
> > +/* { dg-do run } */
> > +/* { dg-options "-march=armv8-a+crypto -O2 -fdump-rtl-dfinish
> -fdump-tree-crc-details" } */
> > +/* { dg-skip-if "" { *-*-* } { "-flto"} } */
> > +
> > +#include "../../gcc.c-torture/execute/crc-9.c"
> > +
> > +/* { dg-final { scan-tree-dump "calculates CRC!" "crc"} } */
> > +/* { dg-final { scan-tree-dump-times "Couldn't generate faster CRC
> code." 0 "crc"} } */
> > +/* { dg-final { scan-rtl-dump "pmull" "dfinish"} } */
> > diff --git a/gcc/testsuite/gcc.target/aarch64/crc-CCIT-data16-pmul.c
> b/gcc/testsuite/gcc.target/aarch64/crc-CCIT-data16-pmul.c
> > new file mode 100644
> > index 00000000000..0c452c1c0f4
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/aarch64/crc-CCIT-data16-pmul.c
> > @@ -0,0 +1,9 @@
> > +/* { dg-do run } */
> > +/* { dg-options "-w -march=armv8-a+crypto -O2 -fdump-rtl-dfinish
> -fdump-tree-crc-details" } */
> > +/* { dg-skip-if "" { *-*-* } { "-flto"} } */
> > +
> > +#include "../../gcc.c-torture/execute/crc-CCIT-data16.c"
> > +
> > +/* { dg-final { scan-tree-dump "calculates CRC!" "crc"} } */
> > +/* { dg-final { scan-tree-dump-times "Couldn't generate faster CRC
> code." 0 "crc"} } */
> > +/* { dg-final { scan-rtl-dump "pmull" "dfinish"} } */
> > \ No newline at end of file
> > diff --git a/gcc/testsuite/gcc.target/aarch64/crc-CCIT-data8-pmul.c
> b/gcc/testsuite/gcc.target/aarch64/crc-CCIT-data8-pmul.c
> > new file mode 100644
> > index 00000000000..87a0b4489a2
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/aarch64/crc-CCIT-data8-pmul.c
> > @@ -0,0 +1,9 @@
> > +/* { dg-do run } */
> > +/* { dg-options "-w -march=armv8-a+crypto -O2 -fdump-rtl-dfinish
> -fdump-tree-crc-details" } */
> > +/* { dg-skip-if "" { *-*-* } { "-flto" } } */
> > +
> > +#include "../../gcc.c-torture/execute/crc-CCIT-data8.c"
> > +
> > +/* { dg-final { scan-tree-dump "calculates CRC!" "crc"} } */
> > +/* { dg-final { scan-tree-dump-times "Couldn't generate faster CRC
> code." 0 "crc"} } */
> > +/* { dg-final { scan-rtl-dump "pmull" "dfinish"} } */
> > \ No newline at end of file
> > diff --git
> a/gcc/testsuite/gcc.target/aarch64/crc-coremark-16bitdata-pmul.c
> b/gcc/testsuite/gcc.target/aarch64/crc-coremark-16bitdata-pmul.c
> > new file mode 100644
> > index 00000000000..75ed5aff80b
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/aarch64/crc-coremark-16bitdata-pmul.c
> > @@ -0,0 +1,9 @@
> > +/* { dg-do run } */
> > +/* { dg-options "-w -march=armv8-a+crypto -O2 -fdump-rtl-dfinish
> -fdump-tree-crc-details" } */
> > +/* { dg-skip-if "" { *-*-* } { "-flto"} } */
> > +
> > +#include "../../gcc.c-torture/execute/crc-coremark16-data16.c"
> > +
> > +/* { dg-final { scan-tree-dump "calculates CRC!" "crc"} } */
> > +/* { dg-final { scan-tree-dump-times "Couldn't generate faster CRC
> code." 0 "crc"} } */
> > +/* { dg-final { scan-rtl-dump "pmull" "dfinish"} } */
> > \ No newline at end of file
> > diff --git a/gcc/testsuite/gcc.target/aarch64/crc-crc32-data16.c
> b/gcc/testsuite/gcc.target/aarch64/crc-crc32-data16.c
> > new file mode 100644
> > index 00000000000..d5aeee7c0c4
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/aarch64/crc-crc32-data16.c
> > @@ -0,0 +1,53 @@
> > +/* { dg-do run } */
> > +/* { dg-options "-march=armv8-a+crc -O2 -fdump-rtl-dfinish
> -fdump-tree-crc-details" } */
> > +/* { dg-skip-if "" { *-*-* } { "-flto"} } */
> > +
> > +#include <stdint.h>
> > +#include <stdlib.h>
> > +
> > +__attribute__ ((noinline,optimize(0)))
> > +uint32_t _crc32_O0 (uint32_t crc, uint16_t data) {
> > +  int i;
> > +  crc = crc ^ data;
> > +
> > +  for (i = 0; i < 8; i++) {
> > +      if (crc & 1)
> > +     crc = (crc >> 1) ^ 0x82F63B78;
> > +      else
> > +     crc = (crc >> 1);
> > +    }
> > +
> > +  return crc;
> > +}
> > +
> > +uint32_t _crc32 (uint32_t crc, uint16_t data) {
> > +  int i;
> > +  crc = crc ^ data;
> > +
> > +  for (i = 0; i < 8; i++) {
> > +      if (crc & 1)
> > +     crc = (crc >> 1) ^ 0x82F63B78;
> > +      else
> > +     crc = (crc >> 1);
> > +    }
> > +
> > +  return crc;
> > +}
> > +
> > +int main ()
> > +{
> > +  uint32_t crc = 0x0D800D80;
> > +  for (uint16_t i = 0; i < 0xffff; i++)
> > +    {
> > +      uint32_t res1 = _crc32_O0 (crc, i);
> > +      uint32_t res2 = _crc32 (crc, i);
> > +      if (res1 != res2)
> > +     abort ();
> > +      crc = res1;
> > +    }
> > +}
> > +
> > +/* { dg-final { scan-tree-dump "calculates CRC!" "crc"} } */
> > +/* { dg-final { scan-tree-dump-times "Couldn't generate faster CRC
> code." 0 "crc"} } */
> > +/* { dg-final { scan-rtl-dump "UNSPEC_CRC32" "dfinish"} } */
> > +/* { dg-final { scan-rtl-dump-times "pmull" 0 "dfinish"} } */
> > diff --git a/gcc/testsuite/gcc.target/aarch64/crc-crc32-data32.c
> b/gcc/testsuite/gcc.target/aarch64/crc-crc32-data32.c
> > new file mode 100644
> > index 00000000000..f0e319b3ab8
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/aarch64/crc-crc32-data32.c
> > @@ -0,0 +1,52 @@
> > +/* { dg-do run } */
> > +/* { dg-options "-march=armv8-a+crc -O2 -fdump-rtl-dfinish
> -fdump-tree-crc-details" } */
> > +/* { dg-skip-if "" { *-*-* } { "-flto"} } */
> > +
> > +#include <stdint.h>
> > +#include <stdlib.h>
> > +__attribute__ ((noinline,optimize(0)))
> > +uint32_t _crc32_O0 (uint32_t crc, uint32_t data) {
> > +  int i;
> > +  crc = crc ^ data;
> > +
> > +  for (i = 0; i < 32; i++) {
> > +      if (crc & 1)
> > +     crc = (crc >> 1) ^ 0x82F63B78;
> > +      else
> > +     crc = (crc >> 1);
> > +    }
> > +
> > +  return crc;
> > +}
> > +
> > +uint32_t _crc32 (uint32_t crc, uint32_t data) {
> > +  int i;
> > +  crc = crc ^ data;
> > +
> > +  for (i = 0; i < 32; i++) {
> > +      if (crc & 1)
> > +     crc = (crc >> 1) ^ 0x82F63B78;
> > +      else
> > +     crc = (crc >> 1);
> > +    }
> > +
> > +  return crc;
> > +}
> > +
> > +int main ()
> > +{
> > +  uint32_t crc = 0x0D800D80;
> > +  for (uint8_t i = 0; i < 0xff; i++)
> > +    {
> > +      uint32_t res1 = _crc32_O0 (crc, i);
> > +      uint32_t res2 = _crc32 (crc, i);
> > +      if (res1 != res2)
> > +     abort ();
> > +      crc = res1;
> > +    }
> > +}
> > +
> > +/* { dg-final { scan-tree-dump "calculates CRC!" "crc"} } */
> > +/* { dg-final { scan-tree-dump-times "Couldn't generate faster CRC
> code." 0 "crc"} } */
> > +/* { dg-final { scan-rtl-dump "UNSPEC_CRC32" "dfinish"} } */
> > +/* { dg-final { scan-rtl-dump-times "pmull" 0 "dfinish"} } */
> > diff --git a/gcc/testsuite/gcc.target/aarch64/crc-crc32-data8.c
> b/gcc/testsuite/gcc.target/aarch64/crc-crc32-data8.c
> > new file mode 100644
> > index 00000000000..95ffde6a9d2
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/aarch64/crc-crc32-data8.c
> > @@ -0,0 +1,53 @@
> > +/* { dg-do run } */
> > +/* { dg-options "-march=armv8-a+crc -O2 -fdump-rtl-dfinish
> -fdump-tree-crc-details" } */
> > +/* { dg-skip-if "" { *-*-* } { "-flto"} } */
> > +
> > +#include <stdint.h>
> > +#include <stdlib.h>
> > +
> > +__attribute__ ((noinline,optimize(0)))
> > +uint32_t _crc32_O0 (uint32_t crc, uint8_t data) {
> > +  int i;
> > +  crc = crc ^ data;
> > +
> > +  for (i = 0; i < 8; i++) {
> > +      if (crc & 1)
> > +     crc = (crc >> 1) ^ 0x82F63B78;
> > +      else
> > +     crc = (crc >> 1);
> > +    }
> > +
> > +  return crc;
> > +}
> > +
> > +uint32_t _crc32 (uint32_t crc, uint8_t data) {
> > +  int i;
> > +  crc = crc ^ data;
> > +
> > +  for (i = 0; i < 8; i++) {
> > +      if (crc & 1)
> > +     crc = (crc >> 1) ^ 0x82F63B78;
> > +      else
> > +     crc = (crc >> 1);
> > +    }
> > +
> > +  return crc;
> > +}
> > +
> > +int main ()
> > +{
> > +  uint32_t crc = 0x0D800D80;
> > +  for (uint8_t i = 0; i < 0xff; i++)
> > +    {
> > +      uint32_t res1 = _crc32_O0 (crc, i);
> > +      uint32_t res2 = _crc32 (crc, i);
> > +      if (res1 != res2)
> > +     abort ();
> > +      crc = res1;
> > +    }
> > +}
> > +
> > +/* { dg-final { scan-tree-dump "calculates CRC!" "crc"} } */
> > +/* { dg-final { scan-tree-dump-times "Couldn't generate faster CRC
> code." 0 "crc"} } */
> > +/* { dg-final { scan-rtl-dump "UNSPEC_CRC32" "dfinish"} } */
> > +/* { dg-final { scan-rtl-dump-times "pmull" 0 "dfinish"} } */
>