* [PATCH v6 0/2] x86: Convert CONST_WIDE_INT/CONST_VECTOR to broadcast
@ 2021-06-28 22:16 H.J. Lu
2021-06-28 22:16 ` [PATCH v6 1/2] " H.J. Lu
2021-06-28 22:16 ` [PATCH v6 2/2] x86: Add vec_duplicate<mode> expander H.J. Lu
0 siblings, 2 replies; 7+ messages in thread
From: H.J. Lu @ 2021-06-28 22:16 UTC (permalink / raw)
To: gcc-patches
Cc: Uros Bizjak, Jakub Jelinek, Hongtao Liu, Richard Sandiford,
Richard Biener
Changes in the v6 patch:
1. Update SI/DI broadcast with AVX.
2. Require non-standard SSE constant integer broadcast with AVX.
3. Use nonimmediate_operand in vec_duplicate<mode> and verify that it
never fails.
Changes in the v5 patch:
1. Allow AVX with SI/DI broadcast.
2. Add a comment for broadcasting to V64QI and V32HI with AVX512F, but
without AVX512BW.
---
1. Update move expanders to convert the CONST_WIDE_INT and CONST_VECTO
operands to vector broadcast from an integer with AVX2.
2. Add ix86_gen_scratch_sse_rtx to return a scratch SSE register which
won't increase stack alignment requirement and blocks transformation by
the combine pass.
A small benchmark:
https://gitlab.com/x86-benchmarks/microbenchmark/-/tree/memset/broadcast
shows that broadcast is a little bit faster on Intel Core i7-8559U:
$ make
gcc -g -I. -O2 -c -o test.o test.c
gcc -g -c -o memory.o memory.S
gcc -g -c -o broadcast.o broadcast.S
gcc -g -c -o vec_dup_sse2.o vec_dup_sse2.S
gcc -o test test.o memory.o broadcast.o vec_dup_sse2.o
./test
memory : 147215
broadcast : 121213
vec_dup_sse2: 171366
$
broadcast is also smaller:
$ size memory.o broadcast.o
text data bss dec hex filename
132 0 0 132 84 memory.o
122 0 0 122 7a broadcast.o
$
3. Update PR 87767 tests to expect integer broadcast instead of broadcast
from memory.
4. Update avx512f_cond_move.c to expect integer broadcast.
A small benchmark:
https://gitlab.com/x86-benchmarks/microbenchmark/-/tree/vpaddd/broadcast
shows that integer broadcast is faster than embedded memory broadcast:
$ make
gcc -g -I. -O2 -march=skylake-avx512 -c -o test.o test.c
gcc -g -c -o memory.o memory.S
gcc -g -c -o broadcast.o broadcast.S
gcc -o test test.o memory.o broadcast.o
./test
memory : 425538
broadcast : 375260
$
Add vec_duplicate<mode> expander for SSE2 if we can move from GPR to SSE
register directly.
H.J. Lu (2):
x86: Convert CONST_WIDE_INT/CONST_VECTOR to broadcast
x86: Add vec_duplicate<mode> expander
gcc/config/i386/i386-expand.c | 193 ++++++++++++++++--
gcc/config/i386/i386-protos.h | 4 +
gcc/config/i386/i386.c | 13 ++
gcc/config/i386/sse.md | 31 +++
.../i386/avx512f-broadcast-pr87767-1.c | 7 +-
.../i386/avx512f-broadcast-pr87767-5.c | 5 +-
.../gcc.target/i386/avx512f_cond_move.c | 4 +-
.../i386/avx512vl-broadcast-pr87767-1.c | 12 +-
.../i386/avx512vl-broadcast-pr87767-5.c | 9 +-
gcc/testsuite/gcc.target/i386/pr100865-1.c | 13 ++
gcc/testsuite/gcc.target/i386/pr100865-10a.c | 33 +++
gcc/testsuite/gcc.target/i386/pr100865-10b.c | 7 +
gcc/testsuite/gcc.target/i386/pr100865-11a.c | 23 +++
gcc/testsuite/gcc.target/i386/pr100865-11b.c | 8 +
gcc/testsuite/gcc.target/i386/pr100865-11c.c | 8 +
gcc/testsuite/gcc.target/i386/pr100865-12a.c | 20 ++
gcc/testsuite/gcc.target/i386/pr100865-12b.c | 8 +
gcc/testsuite/gcc.target/i386/pr100865-12c.c | 8 +
gcc/testsuite/gcc.target/i386/pr100865-2.c | 14 ++
gcc/testsuite/gcc.target/i386/pr100865-3.c | 15 ++
gcc/testsuite/gcc.target/i386/pr100865-4a.c | 16 ++
gcc/testsuite/gcc.target/i386/pr100865-4b.c | 9 +
gcc/testsuite/gcc.target/i386/pr100865-5a.c | 16 ++
gcc/testsuite/gcc.target/i386/pr100865-5b.c | 9 +
gcc/testsuite/gcc.target/i386/pr100865-6a.c | 16 ++
gcc/testsuite/gcc.target/i386/pr100865-6b.c | 9 +
gcc/testsuite/gcc.target/i386/pr100865-6c.c | 16 ++
gcc/testsuite/gcc.target/i386/pr100865-7a.c | 17 ++
gcc/testsuite/gcc.target/i386/pr100865-7b.c | 9 +
gcc/testsuite/gcc.target/i386/pr100865-7c.c | 17 ++
gcc/testsuite/gcc.target/i386/pr100865-8a.c | 24 +++
gcc/testsuite/gcc.target/i386/pr100865-8b.c | 7 +
gcc/testsuite/gcc.target/i386/pr100865-8c.c | 7 +
gcc/testsuite/gcc.target/i386/pr100865-9a.c | 25 +++
gcc/testsuite/gcc.target/i386/pr100865-9b.c | 7 +
gcc/testsuite/gcc.target/i386/pr100865-9c.c | 7 +
36 files changed, 621 insertions(+), 25 deletions(-)
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-1.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-10a.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-10b.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-11a.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-11b.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-11c.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-12a.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-12b.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-12c.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-2.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-3.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-4a.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-4b.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-5a.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-5b.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-6a.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-6b.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-6c.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-7a.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-7b.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-7c.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-8a.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-8b.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-8c.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-9a.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-9b.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-9c.c
--
2.31.1
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH v6 1/2] x86: Convert CONST_WIDE_INT/CONST_VECTOR to broadcast
2021-06-28 22:16 [PATCH v6 0/2] x86: Convert CONST_WIDE_INT/CONST_VECTOR to broadcast H.J. Lu
@ 2021-06-28 22:16 ` H.J. Lu
2021-07-01 8:37 ` Hongtao Liu
2021-06-28 22:16 ` [PATCH v6 2/2] x86: Add vec_duplicate<mode> expander H.J. Lu
1 sibling, 1 reply; 7+ messages in thread
From: H.J. Lu @ 2021-06-28 22:16 UTC (permalink / raw)
To: gcc-patches
Cc: Uros Bizjak, Jakub Jelinek, Hongtao Liu, Richard Sandiford,
Richard Biener
1. Update move expanders to convert the CONST_WIDE_INT and CONST_VECTOR
operands to vector broadcast from an integer with AVX.
2. Add ix86_gen_scratch_sse_rtx to return a scratch SSE register which
won't increase stack alignment requirement and blocks transformation by
the combine pass.
A small benchmark:
https://gitlab.com/x86-benchmarks/microbenchmark/-/tree/memset/broadcast
shows that broadcast is a little bit faster on Intel Core i7-8559U:
$ make
gcc -g -I. -O2 -c -o test.o test.c
gcc -g -c -o memory.o memory.S
gcc -g -c -o broadcast.o broadcast.S
gcc -g -c -o vec_dup_sse2.o vec_dup_sse2.S
gcc -o test test.o memory.o broadcast.o vec_dup_sse2.o
./test
memory : 147215
broadcast : 121213
vec_dup_sse2: 171366
$
broadcast is also smaller:
$ size memory.o broadcast.o
text data bss dec hex filename
132 0 0 132 84 memory.o
122 0 0 122 7a broadcast.o
$
3. Update PR 87767 tests to expect integer broadcast instead of broadcast
from memory.
4. Update avx512f_cond_move.c to expect integer broadcast.
A small benchmark:
https://gitlab.com/x86-benchmarks/microbenchmark/-/tree/vpaddd/broadcast
shows that integer broadcast is faster than embedded memory broadcast:
$ make
gcc -g -I. -O2 -march=skylake-avx512 -c -o test.o test.c
gcc -g -c -o memory.o memory.S
gcc -g -c -o broadcast.o broadcast.S
gcc -o test test.o memory.o broadcast.o
./test
memory : 425538
broadcast : 375260
$
gcc/
PR target/100865
* config/i386/i386-expand.c (ix86_expand_vector_init_duplicate):
New prototype.
(ix86_byte_broadcast): New function.
(ix86_convert_const_wide_int_to_broadcast): Likewise.
(ix86_expand_move): Convert CONST_WIDE_INT to broadcast if mode
size is 16 bytes or bigger.
(ix86_broadcast_from_integer_constant): New function.
(ix86_expand_vector_move): Convert CONST_WIDE_INT and CONST_VECTOR
to broadcast if mode size is 16 bytes or bigger.
* config/i386/i386-protos.h (ix86_gen_scratch_sse_rtx): New
prototype.
* config/i386/i386.c (ix86_gen_scratch_sse_rtx): New function.
gcc/testsuite/
PR target/100865
* gcc.target/i386/avx512f-broadcast-pr87767-1.c: Expect integer
broadcast.
* gcc.target/i386/avx512f-broadcast-pr87767-5.c: Likewise.
* gcc.target/i386/avx512vl-broadcast-pr87767-1.c: Likewise.
* gcc.target/i386/avx512vl-broadcast-pr87767-5.c: Likewise.
* gcc.target/i386/avx512f_cond_move.c: Also pass
-mprefer-vector-width=512 and expect integer broadcast.
* gcc.target/i386/pr100865-1.c: New test.
* gcc.target/i386/pr100865-2.c: Likewise.
* gcc.target/i386/pr100865-3.c: Likewise.
* gcc.target/i386/pr100865-4a.c: Likewise.
* gcc.target/i386/pr100865-4b.c: Likewise.
* gcc.target/i386/pr100865-5a.c: Likewise.
* gcc.target/i386/pr100865-5b.c: Likewise.
* gcc.target/i386/pr100865-6a.c: Likewise.
* gcc.target/i386/pr100865-6b.c: Likewise.
* gcc.target/i386/pr100865-6c.c: Likewise.
* gcc.target/i386/pr100865-7a.c: Likewise.
* gcc.target/i386/pr100865-7b.c: Likewise.
* gcc.target/i386/pr100865-7c.c: Likewise.
* gcc.target/i386/pr100865-8a.c: Likewise.
* gcc.target/i386/pr100865-8b.c: Likewise.
* gcc.target/i386/pr100865-8c.c: Likewise.
* gcc.target/i386/pr100865-9a.c: Likewise.
* gcc.target/i386/pr100865-9b.c: Likewise.
* gcc.target/i386/pr100865-9c.c: Likewise.
* gcc.target/i386/pr100865-10a.c: Likewise.
* gcc.target/i386/pr100865-10b.c: Likewise.
* gcc.target/i386/pr100865-11a.c: Likewise.
* gcc.target/i386/pr100865-11b.c: Likewise.
* gcc.target/i386/pr100865-11c.c: Likewise.
* gcc.target/i386/pr100865-12a.c: Likewise.
* gcc.target/i386/pr100865-12b.c: Likewise.
* gcc.target/i386/pr100865-12c.c: Likewise.
---
gcc/config/i386/i386-expand.c | 194 ++++++++++++++++--
gcc/config/i386/i386-protos.h | 2 +
gcc/config/i386/i386.c | 13 ++
.../i386/avx512f-broadcast-pr87767-1.c | 7 +-
.../i386/avx512f-broadcast-pr87767-5.c | 5 +-
.../gcc.target/i386/avx512f_cond_move.c | 4 +-
.../i386/avx512vl-broadcast-pr87767-1.c | 12 +-
.../i386/avx512vl-broadcast-pr87767-5.c | 9 +-
gcc/testsuite/gcc.target/i386/pr100865-1.c | 13 ++
gcc/testsuite/gcc.target/i386/pr100865-10a.c | 33 +++
gcc/testsuite/gcc.target/i386/pr100865-10b.c | 7 +
gcc/testsuite/gcc.target/i386/pr100865-11a.c | 23 +++
gcc/testsuite/gcc.target/i386/pr100865-11b.c | 8 +
gcc/testsuite/gcc.target/i386/pr100865-11c.c | 8 +
gcc/testsuite/gcc.target/i386/pr100865-12a.c | 20 ++
gcc/testsuite/gcc.target/i386/pr100865-12b.c | 8 +
gcc/testsuite/gcc.target/i386/pr100865-12c.c | 8 +
gcc/testsuite/gcc.target/i386/pr100865-2.c | 14 ++
gcc/testsuite/gcc.target/i386/pr100865-3.c | 15 ++
gcc/testsuite/gcc.target/i386/pr100865-4a.c | 16 ++
gcc/testsuite/gcc.target/i386/pr100865-4b.c | 9 +
gcc/testsuite/gcc.target/i386/pr100865-5a.c | 16 ++
gcc/testsuite/gcc.target/i386/pr100865-5b.c | 9 +
gcc/testsuite/gcc.target/i386/pr100865-6a.c | 16 ++
gcc/testsuite/gcc.target/i386/pr100865-6b.c | 9 +
gcc/testsuite/gcc.target/i386/pr100865-6c.c | 16 ++
gcc/testsuite/gcc.target/i386/pr100865-7a.c | 17 ++
gcc/testsuite/gcc.target/i386/pr100865-7b.c | 9 +
gcc/testsuite/gcc.target/i386/pr100865-7c.c | 17 ++
gcc/testsuite/gcc.target/i386/pr100865-8a.c | 24 +++
gcc/testsuite/gcc.target/i386/pr100865-8b.c | 7 +
gcc/testsuite/gcc.target/i386/pr100865-8c.c | 7 +
gcc/testsuite/gcc.target/i386/pr100865-9a.c | 25 +++
gcc/testsuite/gcc.target/i386/pr100865-9b.c | 7 +
gcc/testsuite/gcc.target/i386/pr100865-9c.c | 7 +
35 files changed, 590 insertions(+), 24 deletions(-)
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-1.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-10a.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-10b.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-11a.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-11b.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-11c.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-12a.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-12b.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-12c.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-2.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-3.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-4a.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-4b.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-5a.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-5b.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-6a.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-6b.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-6c.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-7a.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-7b.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-7c.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-8a.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-8b.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-8c.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-9a.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-9b.c
create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-9c.c
diff --git a/gcc/config/i386/i386-expand.c b/gcc/config/i386/i386-expand.c
index e9763eb5b3e..e0e3ed4d8a4 100644
--- a/gcc/config/i386/i386-expand.c
+++ b/gcc/config/i386/i386-expand.c
@@ -93,6 +93,9 @@ along with GCC; see the file COPYING3. If not see
#include "i386-builtins.h"
#include "i386-expand.h"
+static bool ix86_expand_vector_init_duplicate (bool, machine_mode, rtx,
+ rtx);
+
/* Split one or more double-mode RTL references into pairs of half-mode
references. The RTL can be REG, offsettable MEM, integer constant, or
CONST_DOUBLE. "operands" is a pointer to an array of double-mode RTLs to
@@ -190,6 +193,82 @@ ix86_expand_clear (rtx dest)
emit_insn (tmp);
}
+/* Return true if V can be broadcasted from an integer of WIDTH bits
+ which is returned in VAL_BROADCAST. Otherwise, return false. */
+
+static bool
+ix86_broadcast (HOST_WIDE_INT v, unsigned int width,
+ HOST_WIDE_INT &val_broadcast)
+{
+ wide_int val = wi::uhwi (v, HOST_BITS_PER_WIDE_INT);
+ val_broadcast = wi::extract_uhwi (val, 0, width);
+ for (unsigned int i = width; i < HOST_BITS_PER_WIDE_INT; i += width)
+ {
+ HOST_WIDE_INT each = wi::extract_uhwi (val, i, width);
+ if (val_broadcast != each)
+ return false;
+ }
+ val_broadcast = sext_hwi (val_broadcast, width);
+ return true;
+}
+
+/* Convert the CONST_WIDE_INT operand OP to broadcast in MODE. */
+
+static rtx
+ix86_convert_const_wide_int_to_broadcast (machine_mode mode, rtx op)
+{
+ /* Don't use integer vector broadcast if we can't move from GPR to SSE
+ register directly. */
+ if (!TARGET_INTER_UNIT_MOVES_TO_VEC)
+ return nullptr;
+
+ /* Convert CONST_WIDE_INT to a non-standard SSE constant integer
+ broadcast only if vector broadcast is available. */
+ if (!TARGET_AVX
+ || !CONST_WIDE_INT_P (op)
+ || standard_sse_constant_p (op, mode))
+ return nullptr;
+
+ HOST_WIDE_INT val = CONST_WIDE_INT_ELT (op, 0);
+ HOST_WIDE_INT val_broadcast;
+ scalar_int_mode broadcast_mode;
+ if (TARGET_AVX2
+ && ix86_broadcast (val, GET_MODE_BITSIZE (QImode),
+ val_broadcast))
+ broadcast_mode = QImode;
+ else if (TARGET_AVX2
+ && ix86_broadcast (val, GET_MODE_BITSIZE (HImode),
+ val_broadcast))
+ broadcast_mode = HImode;
+ else if (ix86_broadcast (val, GET_MODE_BITSIZE (SImode),
+ val_broadcast))
+ broadcast_mode = SImode;
+ else if (TARGET_64BIT
+ && ix86_broadcast (val, GET_MODE_BITSIZE (DImode),
+ val_broadcast))
+ broadcast_mode = DImode;
+ else
+ return nullptr;
+
+ /* Check if OP can be broadcasted from VAL. */
+ for (int i = 1; i < CONST_WIDE_INT_NUNITS (op); i++)
+ if (val != CONST_WIDE_INT_ELT (op, i))
+ return nullptr;
+
+ unsigned int nunits = (GET_MODE_SIZE (mode)
+ / GET_MODE_SIZE (broadcast_mode));
+ machine_mode vector_mode;
+ if (!mode_for_vector (broadcast_mode, nunits).exists (&vector_mode))
+ gcc_unreachable ();
+ rtx target = ix86_gen_scratch_sse_rtx (vector_mode);
+ bool ok = ix86_expand_vector_init_duplicate (false, vector_mode,
+ target,
+ GEN_INT (val_broadcast));
+ gcc_assert (ok);
+ target = lowpart_subreg (mode, target, vector_mode);
+ return target;
+}
+
void
ix86_expand_move (machine_mode mode, rtx operands[])
{
@@ -347,20 +426,29 @@ ix86_expand_move (machine_mode mode, rtx operands[])
&& optimize)
op1 = copy_to_mode_reg (mode, op1);
- if (can_create_pseudo_p ()
- && CONST_DOUBLE_P (op1))
+ if (can_create_pseudo_p ())
{
- /* If we are loading a floating point constant to a register,
- force the value to memory now, since we'll get better code
- out the back end. */
+ if (CONST_DOUBLE_P (op1))
+ {
+ /* If we are loading a floating point constant to a
+ register, force the value to memory now, since we'll
+ get better code out the back end. */
- op1 = validize_mem (force_const_mem (mode, op1));
- if (!register_operand (op0, mode))
+ op1 = validize_mem (force_const_mem (mode, op1));
+ if (!register_operand (op0, mode))
+ {
+ rtx temp = gen_reg_rtx (mode);
+ emit_insn (gen_rtx_SET (temp, op1));
+ emit_move_insn (op0, temp);
+ return;
+ }
+ }
+ else if (GET_MODE_SIZE (mode) >= 16)
{
- rtx temp = gen_reg_rtx (mode);
- emit_insn (gen_rtx_SET (temp, op1));
- emit_move_insn (op0, temp);
- return;
+ rtx tmp = ix86_convert_const_wide_int_to_broadcast
+ (GET_MODE (op0), op1);
+ if (tmp != nullptr)
+ op1 = tmp;
}
}
}
@@ -368,6 +456,59 @@ ix86_expand_move (machine_mode mode, rtx operands[])
emit_insn (gen_rtx_SET (op0, op1));
}
+static rtx
+ix86_broadcast_from_integer_constant (machine_mode mode, rtx op)
+{
+ int nunits = GET_MODE_NUNITS (mode);
+ if (nunits < 2)
+ return nullptr;
+
+ /* Don't use integer vector broadcast if we can't move from GPR to SSE
+ register directly. */
+ if (!TARGET_INTER_UNIT_MOVES_TO_VEC)
+ return nullptr;
+
+ /* Convert CONST_VECTOR to a non-standard SSE constant integer
+ broadcast only if vector broadcast is available. */
+ if (!(TARGET_AVX2
+ || (TARGET_AVX
+ && (GET_MODE_INNER (mode) == SImode
+ || GET_MODE_INNER (mode) == DImode)))
+ || standard_sse_constant_p (op, mode))
+ return nullptr;
+
+ /* Don't broadcast from a 64-bit integer constant in 32-bit mode. */
+ if (GET_MODE_INNER (mode) == DImode && !TARGET_64BIT)
+ return nullptr;
+
+ rtx constant = get_pool_constant (XEXP (op, 0));
+ if (GET_CODE (constant) != CONST_VECTOR)
+ return nullptr;
+
+ /* There could be some rtx like
+ (mem/u/c:V16QI (symbol_ref/u:DI ("*.LC1")))
+ but with "*.LC1" refer to V2DI constant vector. */
+ if (GET_MODE (constant) != mode)
+ {
+ constant = simplify_subreg (mode, constant, GET_MODE (constant),
+ 0);
+ if (constant == nullptr || GET_CODE (constant) != CONST_VECTOR)
+ return nullptr;
+ }
+
+ rtx first = XVECEXP (constant, 0, 0);
+
+ for (int i = 1; i < nunits; ++i)
+ {
+ rtx tmp = XVECEXP (constant, 0, i);
+ /* Vector duplicate value. */
+ if (!rtx_equal_p (tmp, first))
+ return nullptr;
+ }
+
+ return first;
+}
+
void
ix86_expand_vector_move (machine_mode mode, rtx operands[])
{
@@ -407,7 +548,36 @@ ix86_expand_vector_move (machine_mode mode, rtx operands[])
op1 = simplify_gen_subreg (mode, r, imode, SUBREG_BYTE (op1));
}
else
- op1 = validize_mem (force_const_mem (mode, op1));
+ {
+ machine_mode mode = GET_MODE (op0);
+ rtx tmp = ix86_convert_const_wide_int_to_broadcast
+ (mode, op1);
+ if (tmp == nullptr)
+ op1 = validize_mem (force_const_mem (mode, op1));
+ else
+ op1 = tmp;
+ }
+ }
+
+ if (can_create_pseudo_p ()
+ && GET_MODE_SIZE (mode) >= 16
+ && GET_MODE_CLASS (mode) == MODE_VECTOR_INT
+ && (MEM_P (op1)
+ && SYMBOL_REF_P (XEXP (op1, 0))
+ && CONSTANT_POOL_ADDRESS_P (XEXP (op1, 0))))
+ {
+ rtx first = ix86_broadcast_from_integer_constant (mode, op1);
+ if (first != nullptr)
+ {
+ /* Broadcast to XMM/YMM/ZMM register from an integer
+ constant. */
+ op1 = ix86_gen_scratch_sse_rtx (mode);
+ bool ok = ix86_expand_vector_init_duplicate (false, mode,
+ op1, first);
+ gcc_assert (ok);
+ emit_move_insn (op0, op1);
+ return;
+ }
}
/* We need to check memory alignment for SSE mode since attribute
diff --git a/gcc/config/i386/i386-protos.h b/gcc/config/i386/i386-protos.h
index 65fc307dc7b..71745b9a1ea 100644
--- a/gcc/config/i386/i386-protos.h
+++ b/gcc/config/i386/i386-protos.h
@@ -50,6 +50,8 @@ extern void ix86_reset_previous_fndecl (void);
extern bool ix86_using_red_zone (void);
+extern rtx ix86_gen_scratch_sse_rtx (machine_mode);
+
extern unsigned int ix86_regmode_natural_size (machine_mode);
#ifdef RTX_CODE
extern int standard_80387_constant_p (rtx);
diff --git a/gcc/config/i386/i386.c b/gcc/config/i386/i386.c
index a93128fa0a4..2fbaae7cd02 100644
--- a/gcc/config/i386/i386.c
+++ b/gcc/config/i386/i386.c
@@ -23163,6 +23163,19 @@ ix86_optab_supported_p (int op, machine_mode mode1, machine_mode,
}
}
+/* Return a scratch register in MODE for vector load and store. */
+
+rtx
+ix86_gen_scratch_sse_rtx (machine_mode mode)
+{
+ if (TARGET_SSE)
+ return gen_rtx_REG (mode, (TARGET_64BIT
+ ? LAST_REX_SSE_REG
+ : LAST_SSE_REG));
+ else
+ return gen_reg_rtx (mode);
+}
+
/* Address space support.
This is not "far pointers" in the 16-bit sense, but an easy way
diff --git a/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-1.c b/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-1.c
index 0563e696316..a2664d87f29 100644
--- a/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-1.c
+++ b/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-1.c
@@ -2,8 +2,11 @@
/* { dg-do compile } */
/* { dg-options "-O2 -mavx512f -mavx512dq" } */
/* { dg-additional-options "-mdynamic-no-pic" { target { *-*-darwin* && ia32 } } }
-/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to8\\\}" 5 } } */
-/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to16\\\}" 5 } } */
+/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to8\\\}" 2 { target { ! ia32 } } } } */
+/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to8\\\}" 5 { target ia32 } } } */
+/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to16\\\}" 2 } } */
+/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %zmm\[0-9\]+" 3 } } */
+/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%r\[^\n\]*, %zmm\[0-9\]+" 3 { target { ! ia32 } } } } */
typedef int v16si __attribute__ ((vector_size (64)));
typedef long long v8di __attribute__ ((vector_size (64)));
diff --git a/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-5.c b/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-5.c
index ffbe95980ca..477f9ca1282 100644
--- a/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-5.c
+++ b/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-5.c
@@ -2,8 +2,9 @@
/* { dg-do compile } */
/* { dg-options "-O2 -mavx512f" } */
/* { dg-additional-options "-mdynamic-no-pic" { target { *-*-darwin* && ia32 } } }
-/* { dg-final { scan-assembler-times "\[^n\n\]*\\\{1to8\\\}" 4 } } */
-/* { dg-final { scan-assembler-times "\[^n\n\]*\\\{1to16\\\}" 4 } } */
+/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to8\\\}" 4 { target ia32 } } } */
+/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %zmm\[0-9\]+" 4 } } */
+/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%r\[^\n\]*, %zmm\[0-9\]+" 4 { target { ! ia32 } } } } */
typedef int v16si __attribute__ ((vector_size (64)));
typedef long long v8di __attribute__ ((vector_size (64)));
diff --git a/gcc/testsuite/gcc.target/i386/avx512f_cond_move.c b/gcc/testsuite/gcc.target/i386/avx512f_cond_move.c
index 99a89f51202..ca49a585232 100644
--- a/gcc/testsuite/gcc.target/i386/avx512f_cond_move.c
+++ b/gcc/testsuite/gcc.target/i386/avx512f_cond_move.c
@@ -1,6 +1,6 @@
/* { dg-do compile } */
-/* { dg-options "-O3 -mavx512f" } */
-/* { dg-final { scan-assembler-times "(?:vpblendmd|vmovdqa32)\[ \\t\]+\[^\{\n\]*%zmm\[0-9\]+\{%k\[1-7\]\}\{z\}(?:\n|\[ \\t\]+#)" 8 } } */
+/* { dg-options "-O3 -mavx512f -mprefer-vector-width=512" } */
+/* { dg-final { scan-assembler-times "(?:vpbroadcastd|vmovdqa32)\[ \\t\]+\[^\{\n\]*%zmm\[0-9\]+\{%k\[1-7\]\}\{z\}(?:\n|\[ \\t\]+#)" 8 } } */
unsigned int x[128];
int y[128];
diff --git a/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-1.c b/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-1.c
index c06369d93fd..f8eb99f0b5f 100644
--- a/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-1.c
+++ b/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-1.c
@@ -2,9 +2,15 @@
/* { dg-do compile } */
/* { dg-options "-O2 -mavx512f -mavx512vl -mavx512dq" } */
/* { dg-additional-options "-mdynamic-no-pic" { target { *-*-darwin* && ia32 } } }
-/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to2\\\}" 5 } } */
-/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to4\\\}" 10 } } */
-/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to8\\\}" 5 } } */
+/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to2\\\}" 2 { target { ! ia32 } } } } */
+/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to4\\\}" 4 { target { ! ia32 } } } } */
+/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to2\\\}" 5 { target ia32 } } } */
+/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to4\\\}" 7 { target ia32 } } } */
+/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to8\\\}" 2 } } */
+/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 3 } } */
+/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %ymm\[0-9\]+" 3 } } */
+/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%r\[^\n\]*, %xmm\[0-9\]+" 3 { target { ! ia32 } } } } */
+/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%r\[^\n\]*, %ymm\[0-9\]+" 3 { target { ! ia32 } } } } */
typedef int v4si __attribute__ ((vector_size (16)));
typedef int v8si __attribute__ ((vector_size (32)));
diff --git a/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-5.c b/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-5.c
index 4998a9b8d51..32f6ac81841 100644
--- a/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-5.c
+++ b/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-5.c
@@ -2,9 +2,12 @@
/* { dg-do compile } */
/* { dg-options "-O2 -mavx512f -mavx512vl" } */
/* { dg-additional-options "-mdynamic-no-pic" { target { *-*-darwin* && ia32 } } }
-/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to2\\\}" 4 } } */
-/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to4\\\}" 8 } } */
-/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to8\\\}" 4 } } */
+/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to2\\\}" 4 { target ia32 } } } */
+/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to4\\\}" 4 { target ia32 } } } */
+/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 4 } } */
+/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %ymm\[0-9\]+" 4 } } */
+/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%r\[^\n\]*, %xmm\[0-9\]+" 4 { target { ! ia32 } } } } */
+/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%r\[^\n\]*, %ymm\[0-9\]+" 4 { target { ! ia32 } } } } */
typedef int v4si __attribute__ ((vector_size (16)));
typedef int v8si __attribute__ ((vector_size (32)));
diff --git a/gcc/testsuite/gcc.target/i386/pr100865-1.c b/gcc/testsuite/gcc.target/i386/pr100865-1.c
new file mode 100644
index 00000000000..6c3097fb2a6
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr100865-1.c
@@ -0,0 +1,13 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-options "-O2 -march=x86-64" } */
+
+extern char *dst;
+
+void
+foo (void)
+{
+ __builtin_memset (dst, 3, 16);
+}
+
+/* { dg-final { scan-assembler-times "movdqa\[ \\t\]+\[^\n\]*%xmm" 1 } } */
+/* { dg-final { scan-assembler-times "movups\[\\t \]%xmm\[0-9\]+, \\(%\[\^,\]+\\)" 1 } } */
diff --git a/gcc/testsuite/gcc.target/i386/pr100865-10a.c b/gcc/testsuite/gcc.target/i386/pr100865-10a.c
new file mode 100644
index 00000000000..7ffc19e56a8
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr100865-10a.c
@@ -0,0 +1,33 @@
+/* { dg-do compile { target int128 } } */
+/* { dg-options "-O3 -march=skylake" } */
+
+extern __int128 array[16];
+
+#define MK_CONST128_BROADCAST(A) \
+ ((((unsigned __int128) (unsigned char) A) << 120) \
+ | (((unsigned __int128) (unsigned char) A) << 112) \
+ | (((unsigned __int128) (unsigned char) A) << 104) \
+ | (((unsigned __int128) (unsigned char) A) << 96) \
+ | (((unsigned __int128) (unsigned char) A) << 88) \
+ | (((unsigned __int128) (unsigned char) A) << 80) \
+ | (((unsigned __int128) (unsigned char) A) << 72) \
+ | (((unsigned __int128) (unsigned char) A) << 64) \
+ | (((unsigned __int128) (unsigned char) A) << 56) \
+ | (((unsigned __int128) (unsigned char) A) << 48) \
+ | (((unsigned __int128) (unsigned char) A) << 40) \
+ | (((unsigned __int128) (unsigned char) A) << 32) \
+ | (((unsigned __int128) (unsigned char) A) << 24) \
+ | (((unsigned __int128) (unsigned char) A) << 16) \
+ | (((unsigned __int128) (unsigned char) A) << 8) \
+ | ((unsigned __int128) (unsigned char) A) )
+
+void
+foo (void)
+{
+ int i;
+ for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
+ array[i] = MK_CONST128_BROADCAST (0x1f);
+}
+
+/* { dg-final { scan-assembler-times "vpbroadcastb\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
+/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
diff --git a/gcc/testsuite/gcc.target/i386/pr100865-10b.c b/gcc/testsuite/gcc.target/i386/pr100865-10b.c
new file mode 100644
index 00000000000..edf52765c60
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr100865-10b.c
@@ -0,0 +1,7 @@
+/* { dg-do compile { target int128 } } */
+/* { dg-options "-O3 -march=skylake-avx512" } */
+
+#include "pr100865-10a.c"
+
+/* { dg-final { scan-assembler-times "vpbroadcastb\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 1 } } */
+/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
diff --git a/gcc/testsuite/gcc.target/i386/pr100865-11a.c b/gcc/testsuite/gcc.target/i386/pr100865-11a.c
new file mode 100644
index 00000000000..04ce1662f3c
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr100865-11a.c
@@ -0,0 +1,23 @@
+/* { dg-do compile { target int128 } } */
+/* { dg-options "-O3 -march=skylake" } */
+
+extern __int128 array[16];
+
+#define MK_CONST128_BROADCAST(A) \
+ ((((unsigned __int128) (unsigned long long) A) << 64) \
+ | ((unsigned __int128) (unsigned long long) A) )
+
+#define MK_CONST128_BROADCAST_SIGNED(A) \
+ ((__int128) MK_CONST128_BROADCAST (A))
+
+void
+foo (void)
+{
+ int i;
+ for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
+ array[i] = MK_CONST128_BROADCAST_SIGNED (-0x1ffffffffLL);
+}
+
+/* { dg-final { scan-assembler-times "movabsq" 1 } } */
+/* { dg-final { scan-assembler-times "(?:vpbroadcastq|vpunpcklqdq)\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
+/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
diff --git a/gcc/testsuite/gcc.target/i386/pr100865-11b.c b/gcc/testsuite/gcc.target/i386/pr100865-11b.c
new file mode 100644
index 00000000000..12d55b9a642
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr100865-11b.c
@@ -0,0 +1,8 @@
+/* { dg-do compile { target int128 } } */
+/* { dg-options "-O3 -march=skylake-avx512" } */
+
+#include "pr100865-11a.c"
+
+/* { dg-final { scan-assembler-times "movabsq" 1 } } */
+/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 1 } } */
+/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
diff --git a/gcc/testsuite/gcc.target/i386/pr100865-11c.c b/gcc/testsuite/gcc.target/i386/pr100865-11c.c
new file mode 100644
index 00000000000..de56c84b9ca
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr100865-11c.c
@@ -0,0 +1,8 @@
+/* { dg-do compile { target int128 } } */
+/* { dg-options "-O3 -march=skylake -mno-avx2" } */
+
+#include "pr100865-11a.c"
+
+/* { dg-final { scan-assembler-times "movabsq" 1 } } */
+/* { dg-final { scan-assembler-times "vpunpcklqdq\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
+/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
diff --git a/gcc/testsuite/gcc.target/i386/pr100865-12a.c b/gcc/testsuite/gcc.target/i386/pr100865-12a.c
new file mode 100644
index 00000000000..d4833d44475
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr100865-12a.c
@@ -0,0 +1,20 @@
+/* { dg-do compile { target int128 } } */
+/* { dg-options "-O3 -march=skylake" } */
+
+extern __int128 array[16];
+
+#define MK_CONST128_BROADCAST(A) \
+ ((((unsigned __int128) (unsigned long long) A) << 64) \
+ | ((unsigned __int128) (unsigned long long) A) )
+
+void
+foo (void)
+{
+ int i;
+ for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
+ array[i] = MK_CONST128_BROADCAST (0x1ffffffffLL);
+}
+
+/* { dg-final { scan-assembler-times "movabsq" 1 } } */
+/* { dg-final { scan-assembler-times "(?:vpbroadcastq|vpunpcklqdq)\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
+/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
diff --git a/gcc/testsuite/gcc.target/i386/pr100865-12b.c b/gcc/testsuite/gcc.target/i386/pr100865-12b.c
new file mode 100644
index 00000000000..63a5629b90c
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr100865-12b.c
@@ -0,0 +1,8 @@
+/* { dg-do compile { target int128 } } */
+/* { dg-options "-O3 -march=skylake-avx512" } */
+
+#include "pr100865-12a.c"
+
+/* { dg-final { scan-assembler-times "movabsq" 1 } } */
+/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 1 } } */
+/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
diff --git a/gcc/testsuite/gcc.target/i386/pr100865-12c.c b/gcc/testsuite/gcc.target/i386/pr100865-12c.c
new file mode 100644
index 00000000000..77415f22c97
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr100865-12c.c
@@ -0,0 +1,8 @@
+/* { dg-do compile { target int128 } } */
+/* { dg-options "-O3 -march=skylake -mno-avx2" } */
+
+#include "pr100865-12a.c"
+
+/* { dg-final { scan-assembler-times "movabsq" 1 } } */
+/* { dg-final { scan-assembler-times "vpunpcklqdq\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
+/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
diff --git a/gcc/testsuite/gcc.target/i386/pr100865-2.c b/gcc/testsuite/gcc.target/i386/pr100865-2.c
new file mode 100644
index 00000000000..17efe2d72a3
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr100865-2.c
@@ -0,0 +1,14 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-options "-O2 -march=skylake" } */
+
+extern char *dst;
+
+void
+foo (void)
+{
+ __builtin_memset (dst, 3, 16);
+}
+
+/* { dg-final { scan-assembler-times "vpbroadcastb\[\\t \]+%xmm\[0-9\]+, %xmm\[0-9\]+" 1 } } */
+/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%xmm\[0-9\]+, \\(%\[\^,\]+\\)" 1 } } */
+/* { dg-final { scan-assembler-not "vmovdqa" } } */
diff --git a/gcc/testsuite/gcc.target/i386/pr100865-3.c b/gcc/testsuite/gcc.target/i386/pr100865-3.c
new file mode 100644
index 00000000000..b6dbcf7809b
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr100865-3.c
@@ -0,0 +1,15 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-options "-O2 -march=skylake-avx512" } */
+
+extern char *dst;
+
+void
+foo (void)
+{
+ __builtin_memset (dst, 3, 16);
+}
+
+/* { dg-final { scan-assembler-times "vpbroadcastb\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 1 } } */
+/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%xmm\[0-9\]+, \\(%\[\^,\]+\\)" 1 } } */
+/* { dg-final { scan-assembler-not "vpbroadcastb\[\\t \]+%xmm\[0-9\]+, %xmm\[0-9\]+" } } */
+/* { dg-final { scan-assembler-not "vmovdqa" } } */
diff --git a/gcc/testsuite/gcc.target/i386/pr100865-4a.c b/gcc/testsuite/gcc.target/i386/pr100865-4a.c
new file mode 100644
index 00000000000..f55883598f9
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr100865-4a.c
@@ -0,0 +1,16 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-options "-O2 -march=skylake" } */
+
+extern char array[64];
+
+void
+foo (void)
+{
+ int i;
+ for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
+ array[i] = -45;
+}
+
+/* { dg-final { scan-assembler-times "vpbroadcastb\[\\t \]+%xmm\[0-9\]+, %xmm\[0-9\]+" 1 } } */
+/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%xmm\[0-9\]+, " 4 } } */
+/* { dg-final { scan-assembler-not "vmovdqa" } } */
diff --git a/gcc/testsuite/gcc.target/i386/pr100865-4b.c b/gcc/testsuite/gcc.target/i386/pr100865-4b.c
new file mode 100644
index 00000000000..f41e6147b4c
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr100865-4b.c
@@ -0,0 +1,9 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-options "-O2 -march=skylake-avx512" } */
+
+#include "pr100865-4a.c"
+
+/* { dg-final { scan-assembler-times "vpbroadcastb\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 1 } } */
+/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%xmm\[0-9\]+, " 4 } } */
+/* { dg-final { scan-assembler-not "vpbroadcastb\[\\t \]+%xmm\[0-9\]+, %xmm\[0-9\]+" } } */
+/* { dg-final { scan-assembler-not "vmovdqa" } } */
diff --git a/gcc/testsuite/gcc.target/i386/pr100865-5a.c b/gcc/testsuite/gcc.target/i386/pr100865-5a.c
new file mode 100644
index 00000000000..4149797fe81
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr100865-5a.c
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-options "-O3 -march=skylake" } */
+
+extern short array[64];
+
+void
+foo (void)
+{
+ int i;
+ for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
+ array[i] = -45;
+}
+
+/* { dg-final { scan-assembler-times "vpbroadcastw\[\\t \]+%xmm\[0-9\]+, %ymm\[0-9\]+" 1 } } */
+/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%ymm\[0-9\]+, " 4 } } */
+/* { dg-final { scan-assembler-not "vmovdqa" } } */
diff --git a/gcc/testsuite/gcc.target/i386/pr100865-5b.c b/gcc/testsuite/gcc.target/i386/pr100865-5b.c
new file mode 100644
index 00000000000..ded41b680d3
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr100865-5b.c
@@ -0,0 +1,9 @@
+/* { dg-do compile } */
+/* { dg-options "-O3 -march=skylake-avx512" } */
+
+#include "pr100865-5a.c"
+
+/* { dg-final { scan-assembler-times "vpbroadcastw\[\\t \]+%(?:r|e)\[^\n\]*, %ymm\[0-9\]+" 1 } } */
+/* { dg-final { scan-assembler-times "vmovdqu16\[\\t \]%ymm\[0-9\]+, " 4 } } */
+/* { dg-final { scan-assembler-not "vpbroadcastw\[\\t \]+%xmm\[0-9\]+, %ymm\[0-9\]+" } } */
+/* { dg-final { scan-assembler-not "vmovdqa" } } */
diff --git a/gcc/testsuite/gcc.target/i386/pr100865-6a.c b/gcc/testsuite/gcc.target/i386/pr100865-6a.c
new file mode 100644
index 00000000000..3fde549a10d
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr100865-6a.c
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-options "-O3 -march=skylake" } */
+
+extern int array[64];
+
+void
+foo (void)
+{
+ int i;
+ for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
+ array[i] = -45;
+}
+
+/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%xmm\[0-9\]+, %ymm\[0-9\]+" 1 } } */
+/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%ymm\[0-9\]+, " 8 } } */
+/* { dg-final { scan-assembler-not "vmovdqa" } } */
diff --git a/gcc/testsuite/gcc.target/i386/pr100865-6b.c b/gcc/testsuite/gcc.target/i386/pr100865-6b.c
new file mode 100644
index 00000000000..44e74c64e55
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr100865-6b.c
@@ -0,0 +1,9 @@
+/* { dg-do compile } */
+/* { dg-options "-O3 -march=skylake-avx512" } */
+
+#include "pr100865-6a.c"
+
+/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %ymm\[0-9\]+" 1 } } */
+/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%ymm\[0-9\]+, " 8 } } */
+/* { dg-final { scan-assembler-not "vpbroadcastd\[\\t \]+%xmm\[0-9\]+, %ymm\[0-9\]+" } } */
+/* { dg-final { scan-assembler-not "vmovdqa" } } */
diff --git a/gcc/testsuite/gcc.target/i386/pr100865-6c.c b/gcc/testsuite/gcc.target/i386/pr100865-6c.c
new file mode 100644
index 00000000000..46d31030ce8
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr100865-6c.c
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-options "-O3 -march=skylake -mno-avx2" } */
+
+extern int array[64];
+
+void
+foo (void)
+{
+ int i;
+ for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
+ array[i] = -45;
+}
+
+/* { dg-final { scan-assembler-times "vbroadcastss" 1 } } */
+/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%ymm\[0-9\]+, " 8 } } */
+/* { dg-final { scan-assembler-not "vmovdqa" } } */
diff --git a/gcc/testsuite/gcc.target/i386/pr100865-7a.c b/gcc/testsuite/gcc.target/i386/pr100865-7a.c
new file mode 100644
index 00000000000..f6f2be91120
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr100865-7a.c
@@ -0,0 +1,17 @@
+/* { dg-do compile } */
+/* { dg-options "-O3 -march=skylake" } */
+
+extern long long int array[64];
+
+void
+foo (void)
+{
+ int i;
+ for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
+ array[i] = -45;
+}
+
+/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+\[^\n\]*, %ymm\[0-9\]+" 1 { target { ! ia32 } } } } */
+/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%ymm\[0-9\]+, " 16 } } */
+/* { dg-final { scan-assembler-not "vpbroadcastq" { target ia32 } } } */
+/* { dg-final { scan-assembler-not "vmovdqa" { target { ! ia32 } } } } */
diff --git a/gcc/testsuite/gcc.target/i386/pr100865-7b.c b/gcc/testsuite/gcc.target/i386/pr100865-7b.c
new file mode 100644
index 00000000000..0a68820aa32
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr100865-7b.c
@@ -0,0 +1,9 @@
+/* { dg-do compile } */
+/* { dg-options "-O3 -march=skylake-avx512" } */
+
+#include "pr100865-7a.c"
+
+/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%r\[^\n\]*, %ymm\[0-9\]+" 1 { target { ! ia32 } } } } */
+/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+\[^\n\]*, %ymm\[0-9\]+" 1 { target ia32 } } } */
+/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%ymm\[0-9\]+, " 16 } } */
+/* { dg-final { scan-assembler-not "vmovdqa" } } */
diff --git a/gcc/testsuite/gcc.target/i386/pr100865-7c.c b/gcc/testsuite/gcc.target/i386/pr100865-7c.c
new file mode 100644
index 00000000000..4d50bb7e2f6
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr100865-7c.c
@@ -0,0 +1,17 @@
+/* { dg-do compile } */
+/* { dg-options "-O3 -march=skylake -mno-avx2" } */
+
+extern long long int array[64];
+
+void
+foo (void)
+{
+ int i;
+ for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
+ array[i] = -45;
+}
+
+/* { dg-final { scan-assembler-times "vbroadcastsd" 1 { target { ! ia32 } } } } */
+/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%ymm\[0-9\]+, " 16 } } */
+/* { dg-final { scan-assembler-not "vbroadcastsd" { target ia32 } } } */
+/* { dg-final { scan-assembler-not "vmovdqa" { target { ! ia32 } } } } */
diff --git a/gcc/testsuite/gcc.target/i386/pr100865-8a.c b/gcc/testsuite/gcc.target/i386/pr100865-8a.c
new file mode 100644
index 00000000000..544a14db6f7
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr100865-8a.c
@@ -0,0 +1,24 @@
+/* { dg-do compile { target int128 } } */
+/* { dg-options "-O3 -march=skylake" } */
+
+extern __int128 array[16];
+
+#define MK_CONST128_BROADCAST(A) \
+ ((((unsigned __int128) (unsigned int) A) << 96) \
+ | (((unsigned __int128) (unsigned int) A) << 64) \
+ | (((unsigned __int128) (unsigned int) A) << 32) \
+ | ((unsigned __int128) (unsigned int) A) )
+
+#define MK_CONST128_BROADCAST_SIGNED(A) \
+ ((__int128) MK_CONST128_BROADCAST (A))
+
+void
+foo (void)
+{
+ int i;
+ for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
+ array[i] = MK_CONST128_BROADCAST_SIGNED (-45);
+}
+
+/* { dg-final { scan-assembler-times "(?:vpbroadcastd|vpshufd)\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
+/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
diff --git a/gcc/testsuite/gcc.target/i386/pr100865-8b.c b/gcc/testsuite/gcc.target/i386/pr100865-8b.c
new file mode 100644
index 00000000000..99a10ad83bd
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr100865-8b.c
@@ -0,0 +1,7 @@
+/* { dg-do compile { target int128 } } */
+/* { dg-options "-O3 -march=skylake-avx512" } */
+
+#include "pr100865-8a.c"
+
+/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 1 } } */
+/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
diff --git a/gcc/testsuite/gcc.target/i386/pr100865-8c.c b/gcc/testsuite/gcc.target/i386/pr100865-8c.c
new file mode 100644
index 00000000000..efee0488614
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr100865-8c.c
@@ -0,0 +1,7 @@
+/* { dg-do compile { target int128 } } */
+/* { dg-options "-O3 -march=skylake -mno-avx2" } */
+
+#include "pr100865-8a.c"
+
+/* { dg-final { scan-assembler-times "vpshufd\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
+/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
diff --git a/gcc/testsuite/gcc.target/i386/pr100865-9a.c b/gcc/testsuite/gcc.target/i386/pr100865-9a.c
new file mode 100644
index 00000000000..45d0e0d0e2e
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr100865-9a.c
@@ -0,0 +1,25 @@
+/* { dg-do compile { target int128 } } */
+/* { dg-options "-O3 -march=skylake" } */
+
+extern __int128 array[16];
+
+#define MK_CONST128_BROADCAST(A) \
+ ((((unsigned __int128) (unsigned short) A) << 112) \
+ | (((unsigned __int128) (unsigned short) A) << 96) \
+ | (((unsigned __int128) (unsigned short) A) << 80) \
+ | (((unsigned __int128) (unsigned short) A) << 64) \
+ | (((unsigned __int128) (unsigned short) A) << 48) \
+ | (((unsigned __int128) (unsigned short) A) << 32) \
+ | (((unsigned __int128) (unsigned short) A) << 16) \
+ | ((unsigned __int128) (unsigned short) A) )
+
+void
+foo (void)
+{
+ int i;
+ for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
+ array[i] = MK_CONST128_BROADCAST (0x1fff);
+}
+
+/* { dg-final { scan-assembler-times "vpbroadcastw\[\\t \]+%xmm\[0-9\]+, %xmm\[0-9\]+" 1 } } */
+/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
diff --git a/gcc/testsuite/gcc.target/i386/pr100865-9b.c b/gcc/testsuite/gcc.target/i386/pr100865-9b.c
new file mode 100644
index 00000000000..14696248525
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr100865-9b.c
@@ -0,0 +1,7 @@
+/* { dg-do compile { target int128 } } */
+/* { dg-options "-O3 -march=skylake-avx512" } */
+
+#include "pr100865-9a.c"
+
+/* { dg-final { scan-assembler-times "vpbroadcastw\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 1 } } */
+/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
diff --git a/gcc/testsuite/gcc.target/i386/pr100865-9c.c b/gcc/testsuite/gcc.target/i386/pr100865-9c.c
new file mode 100644
index 00000000000..e6f25902c1d
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr100865-9c.c
@@ -0,0 +1,7 @@
+/* { dg-do compile { target int128 } } */
+/* { dg-options "-O3 -march=skylake -mno-avx2" } */
+
+#include "pr100865-9a.c"
+
+/* { dg-final { scan-assembler-times "vpshufd\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
+/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
--
2.31.1
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH v6 2/2] x86: Add vec_duplicate<mode> expander
2021-06-28 22:16 [PATCH v6 0/2] x86: Convert CONST_WIDE_INT/CONST_VECTOR to broadcast H.J. Lu
2021-06-28 22:16 ` [PATCH v6 1/2] " H.J. Lu
@ 2021-06-28 22:16 ` H.J. Lu
2021-07-01 8:30 ` Hongtao Liu
1 sibling, 1 reply; 7+ messages in thread
From: H.J. Lu @ 2021-06-28 22:16 UTC (permalink / raw)
To: gcc-patches
Cc: Uros Bizjak, Jakub Jelinek, Hongtao Liu, Richard Sandiford,
Richard Biener
Add vec_duplicate<mode> expander for SSE2 if we can move from GPR to SSE
register directly.
* config/i386/i386-expand.c (ix86_expand_vector_init_duplicate):
Make it global.
* config/i386/i386-protos.h (ix86_expand_vector_init_duplicate):
New prototype.
* config/i386/sse.md (INT_BROADCAST_MODE): New mode iterator.
(vec_duplicate<mode>): New expander.
---
gcc/config/i386/i386-expand.c | 5 +----
gcc/config/i386/i386-protos.h | 2 ++
gcc/config/i386/sse.md | 31 +++++++++++++++++++++++++++++++
3 files changed, 34 insertions(+), 4 deletions(-)
diff --git a/gcc/config/i386/i386-expand.c b/gcc/config/i386/i386-expand.c
index e0e3ed4d8a4..e04019c4b79 100644
--- a/gcc/config/i386/i386-expand.c
+++ b/gcc/config/i386/i386-expand.c
@@ -93,9 +93,6 @@ along with GCC; see the file COPYING3. If not see
#include "i386-builtins.h"
#include "i386-expand.h"
-static bool ix86_expand_vector_init_duplicate (bool, machine_mode, rtx,
- rtx);
-
/* Split one or more double-mode RTL references into pairs of half-mode
references. The RTL can be REG, offsettable MEM, integer constant, or
CONST_DOUBLE. "operands" is a pointer to an array of double-mode RTLs to
@@ -13909,7 +13906,7 @@ static bool expand_vec_perm_1 (struct expand_vec_perm_d *d);
/* A subroutine of ix86_expand_vector_init. Store into TARGET a vector
with all elements equal to VAR. Return true if successful. */
-static bool
+bool
ix86_expand_vector_init_duplicate (bool mmx_ok, machine_mode mode,
rtx target, rtx val)
{
diff --git a/gcc/config/i386/i386-protos.h b/gcc/config/i386/i386-protos.h
index 71745b9a1ea..51376fcc454 100644
--- a/gcc/config/i386/i386-protos.h
+++ b/gcc/config/i386/i386-protos.h
@@ -258,6 +258,8 @@ extern void ix86_expand_mul_widen_hilo (rtx, rtx, rtx, bool, bool);
extern void ix86_expand_sse2_mulv4si3 (rtx, rtx, rtx);
extern void ix86_expand_sse2_mulvxdi3 (rtx, rtx, rtx);
extern void ix86_expand_sse2_abs (rtx, rtx);
+extern bool ix86_expand_vector_init_duplicate (bool, machine_mode, rtx,
+ rtx);
/* In i386-c.c */
extern void ix86_target_macros (void);
diff --git a/gcc/config/i386/sse.md b/gcc/config/i386/sse.md
index ffcc0c81964..5ededaedac7 100644
--- a/gcc/config/i386/sse.md
+++ b/gcc/config/i386/sse.md
@@ -24814,3 +24814,34 @@ (define_insn "*aes<aeswideklvariant>u8"
"TARGET_WIDEKL"
"aes<aeswideklvariant>\t{%0}"
[(set_attr "type" "other")])
+
+;; Modes handled by broadcast patterns. NB: Allow V64QI and V32HI with
+;; TARGET_AVX512F since ix86_expand_vector_init_duplicate can expand
+;; without TARGET_AVX512BW which is used by memset vector broadcast
+;; expander to XI with:
+;; vmovd %edi, %xmm15
+;; vpbroadcastb %xmm15, %ymm15
+;; vinserti64x4 $0x1, %ymm15, %zmm15, %zmm15
+
+(define_mode_iterator INT_BROADCAST_MODE
+ [(V64QI "TARGET_AVX512F") (V32QI "TARGET_AVX") V16QI
+ (V32HI "TARGET_AVX512F") (V16HI "TARGET_AVX") V8HI
+ (V16SI "TARGET_AVX512F") (V8SI "TARGET_AVX") V4SI
+ (V8DI "TARGET_AVX512F && TARGET_64BIT")
+ (V4DI "TARGET_AVX && TARGET_64BIT") (V2DI "TARGET_64BIT")])
+
+;; Broadcast from an integer. NB: Enable broadcast only if we can move
+;; from GPR to SSE register directly.
+(define_expand "vec_duplicate<mode>"
+ [(set (match_operand:INT_BROADCAST_MODE 0 "register_operand")
+ (vec_duplicate:INT_BROADCAST_MODE
+ (match_operand:<ssescalarmode> 1 "nonimmediate_operand")))]
+ "TARGET_SSE2 && TARGET_INTER_UNIT_MOVES_TO_VEC"
+{
+ if (!ix86_expand_vector_init_duplicate (false,
+ GET_MODE (operands[0]),
+ operands[0],
+ operands[1]))
+ gcc_unreachable ();
+ DONE;
+})
--
2.31.1
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v6 2/2] x86: Add vec_duplicate<mode> expander
2021-06-28 22:16 ` [PATCH v6 2/2] x86: Add vec_duplicate<mode> expander H.J. Lu
@ 2021-07-01 8:30 ` Hongtao Liu
0 siblings, 0 replies; 7+ messages in thread
From: Hongtao Liu @ 2021-07-01 8:30 UTC (permalink / raw)
To: H.J. Lu
Cc: GCC Patches, Uros Bizjak, Jakub Jelinek, Richard Sandiford,
Richard Biener
On Tue, Jun 29, 2021 at 6:16 AM H.J. Lu <hjl.tools@gmail.com> wrote:
>
> Add vec_duplicate<mode> expander for SSE2 if we can move from GPR to SSE
> register directly.
>
> * config/i386/i386-expand.c (ix86_expand_vector_init_duplicate):
> Make it global.
> * config/i386/i386-protos.h (ix86_expand_vector_init_duplicate):
> New prototype.
> * config/i386/sse.md (INT_BROADCAST_MODE): New mode iterator.
> (vec_duplicate<mode>): New expander.
> ---
> gcc/config/i386/i386-expand.c | 5 +----
> gcc/config/i386/i386-protos.h | 2 ++
> gcc/config/i386/sse.md | 31 +++++++++++++++++++++++++++++++
> 3 files changed, 34 insertions(+), 4 deletions(-)
>
> diff --git a/gcc/config/i386/i386-expand.c b/gcc/config/i386/i386-expand.c
> index e0e3ed4d8a4..e04019c4b79 100644
> --- a/gcc/config/i386/i386-expand.c
> +++ b/gcc/config/i386/i386-expand.c
> @@ -93,9 +93,6 @@ along with GCC; see the file COPYING3. If not see
> #include "i386-builtins.h"
> #include "i386-expand.h"
>
> -static bool ix86_expand_vector_init_duplicate (bool, machine_mode, rtx,
> - rtx);
> -
> /* Split one or more double-mode RTL references into pairs of half-mode
> references. The RTL can be REG, offsettable MEM, integer constant, or
> CONST_DOUBLE. "operands" is a pointer to an array of double-mode RTLs to
> @@ -13909,7 +13906,7 @@ static bool expand_vec_perm_1 (struct expand_vec_perm_d *d);
> /* A subroutine of ix86_expand_vector_init. Store into TARGET a vector
> with all elements equal to VAR. Return true if successful. */
>
> -static bool
> +bool
> ix86_expand_vector_init_duplicate (bool mmx_ok, machine_mode mode,
> rtx target, rtx val)
> {
> diff --git a/gcc/config/i386/i386-protos.h b/gcc/config/i386/i386-protos.h
> index 71745b9a1ea..51376fcc454 100644
> --- a/gcc/config/i386/i386-protos.h
> +++ b/gcc/config/i386/i386-protos.h
> @@ -258,6 +258,8 @@ extern void ix86_expand_mul_widen_hilo (rtx, rtx, rtx, bool, bool);
> extern void ix86_expand_sse2_mulv4si3 (rtx, rtx, rtx);
> extern void ix86_expand_sse2_mulvxdi3 (rtx, rtx, rtx);
> extern void ix86_expand_sse2_abs (rtx, rtx);
> +extern bool ix86_expand_vector_init_duplicate (bool, machine_mode, rtx,
> + rtx);
>
> /* In i386-c.c */
> extern void ix86_target_macros (void);
> diff --git a/gcc/config/i386/sse.md b/gcc/config/i386/sse.md
> index ffcc0c81964..5ededaedac7 100644
> --- a/gcc/config/i386/sse.md
> +++ b/gcc/config/i386/sse.md
> @@ -24814,3 +24814,34 @@ (define_insn "*aes<aeswideklvariant>u8"
> "TARGET_WIDEKL"
> "aes<aeswideklvariant>\t{%0}"
> [(set_attr "type" "other")])
> +
> +;; Modes handled by broadcast patterns. NB: Allow V64QI and V32HI with
> +;; TARGET_AVX512F since ix86_expand_vector_init_duplicate can expand
> +;; without TARGET_AVX512BW which is used by memset vector broadcast
> +;; expander to XI with:
> +;; vmovd %edi, %xmm15
> +;; vpbroadcastb %xmm15, %ymm15
> +;; vinserti64x4 $0x1, %ymm15, %zmm15, %zmm15
> +
> +(define_mode_iterator INT_BROADCAST_MODE
> + [(V64QI "TARGET_AVX512F") (V32QI "TARGET_AVX") V16QI
> + (V32HI "TARGET_AVX512F") (V16HI "TARGET_AVX") V8HI
> + (V16SI "TARGET_AVX512F") (V8SI "TARGET_AVX") V4SI
> + (V8DI "TARGET_AVX512F && TARGET_64BIT")
> + (V4DI "TARGET_AVX && TARGET_64BIT") (V2DI "TARGET_64BIT")])
> +
> +;; Broadcast from an integer. NB: Enable broadcast only if we can move
> +;; from GPR to SSE register directly.
> +(define_expand "vec_duplicate<mode>"
> + [(set (match_operand:INT_BROADCAST_MODE 0 "register_operand")
> + (vec_duplicate:INT_BROADCAST_MODE
> + (match_operand:<ssescalarmode> 1 "nonimmediate_operand")))]
> + "TARGET_SSE2 && TARGET_INTER_UNIT_MOVES_TO_VEC"
Although we don't have broadcast instructions for sse2,
ix86_expand_vector_init_duplicate can still handle them by equivalent
instructions.
So the patch LGTM.
> +{
> + if (!ix86_expand_vector_init_duplicate (false,
> + GET_MODE (operands[0]),
> + operands[0],
> + operands[1]))
> + gcc_unreachable ();
> + DONE;
> +})
> --
> 2.31.1
>
--
BR,
Hongtao
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v6 1/2] x86: Convert CONST_WIDE_INT/CONST_VECTOR to broadcast
2021-06-28 22:16 ` [PATCH v6 1/2] " H.J. Lu
@ 2021-07-01 8:37 ` Hongtao Liu
2021-07-01 12:41 ` H.J. Lu
0 siblings, 1 reply; 7+ messages in thread
From: Hongtao Liu @ 2021-07-01 8:37 UTC (permalink / raw)
To: H.J. Lu
Cc: GCC Patches, Uros Bizjak, Jakub Jelinek, Richard Sandiford,
Richard Biener
On Tue, Jun 29, 2021 at 6:16 AM H.J. Lu <hjl.tools@gmail.com> wrote:
>
> 1. Update move expanders to convert the CONST_WIDE_INT and CONST_VECTOR
> operands to vector broadcast from an integer with AVX.
> 2. Add ix86_gen_scratch_sse_rtx to return a scratch SSE register which
> won't increase stack alignment requirement and blocks transformation by
> the combine pass.
>
> A small benchmark:
>
> https://gitlab.com/x86-benchmarks/microbenchmark/-/tree/memset/broadcast
>
> shows that broadcast is a little bit faster on Intel Core i7-8559U:
>
> $ make
> gcc -g -I. -O2 -c -o test.o test.c
> gcc -g -c -o memory.o memory.S
> gcc -g -c -o broadcast.o broadcast.S
> gcc -g -c -o vec_dup_sse2.o vec_dup_sse2.S
> gcc -o test test.o memory.o broadcast.o vec_dup_sse2.o
> ./test
> memory : 147215
> broadcast : 121213
> vec_dup_sse2: 171366
> $
>
> broadcast is also smaller:
>
> $ size memory.o broadcast.o
> text data bss dec hex filename
> 132 0 0 132 84 memory.o
> 122 0 0 122 7a broadcast.o
> $
>
> 3. Update PR 87767 tests to expect integer broadcast instead of broadcast
> from memory.
> 4. Update avx512f_cond_move.c to expect integer broadcast.
>
> A small benchmark:
>
> https://gitlab.com/x86-benchmarks/microbenchmark/-/tree/vpaddd/broadcast
>
> shows that integer broadcast is faster than embedded memory broadcast:
>
> $ make
> gcc -g -I. -O2 -march=skylake-avx512 -c -o test.o test.c
> gcc -g -c -o memory.o memory.S
> gcc -g -c -o broadcast.o broadcast.S
> gcc -o test test.o memory.o broadcast.o
> ./test
> memory : 425538
> broadcast : 375260
> $
>
> gcc/
>
> PR target/100865
> * config/i386/i386-expand.c (ix86_expand_vector_init_duplicate):
> New prototype.
> (ix86_byte_broadcast): New function.
> (ix86_convert_const_wide_int_to_broadcast): Likewise.
> (ix86_expand_move): Convert CONST_WIDE_INT to broadcast if mode
> size is 16 bytes or bigger.
> (ix86_broadcast_from_integer_constant): New function.
> (ix86_expand_vector_move): Convert CONST_WIDE_INT and CONST_VECTOR
> to broadcast if mode size is 16 bytes or bigger.
> * config/i386/i386-protos.h (ix86_gen_scratch_sse_rtx): New
> prototype.
> * config/i386/i386.c (ix86_gen_scratch_sse_rtx): New function.
>
> gcc/testsuite/
>
> PR target/100865
> * gcc.target/i386/avx512f-broadcast-pr87767-1.c: Expect integer
> broadcast.
> * gcc.target/i386/avx512f-broadcast-pr87767-5.c: Likewise.
> * gcc.target/i386/avx512vl-broadcast-pr87767-1.c: Likewise.
> * gcc.target/i386/avx512vl-broadcast-pr87767-5.c: Likewise.
> * gcc.target/i386/avx512f_cond_move.c: Also pass
> -mprefer-vector-width=512 and expect integer broadcast.
Those tests change LGTM since it's faster than the older version, and
I remember uros has already reviewed other parts, right?
> * gcc.target/i386/pr100865-1.c: New test.
> * gcc.target/i386/pr100865-2.c: Likewise.
> * gcc.target/i386/pr100865-3.c: Likewise.
> * gcc.target/i386/pr100865-4a.c: Likewise.
> * gcc.target/i386/pr100865-4b.c: Likewise.
> * gcc.target/i386/pr100865-5a.c: Likewise.
> * gcc.target/i386/pr100865-5b.c: Likewise.
> * gcc.target/i386/pr100865-6a.c: Likewise.
> * gcc.target/i386/pr100865-6b.c: Likewise.
> * gcc.target/i386/pr100865-6c.c: Likewise.
> * gcc.target/i386/pr100865-7a.c: Likewise.
> * gcc.target/i386/pr100865-7b.c: Likewise.
> * gcc.target/i386/pr100865-7c.c: Likewise.
> * gcc.target/i386/pr100865-8a.c: Likewise.
> * gcc.target/i386/pr100865-8b.c: Likewise.
> * gcc.target/i386/pr100865-8c.c: Likewise.
> * gcc.target/i386/pr100865-9a.c: Likewise.
> * gcc.target/i386/pr100865-9b.c: Likewise.
> * gcc.target/i386/pr100865-9c.c: Likewise.
> * gcc.target/i386/pr100865-10a.c: Likewise.
> * gcc.target/i386/pr100865-10b.c: Likewise.
> * gcc.target/i386/pr100865-11a.c: Likewise.
> * gcc.target/i386/pr100865-11b.c: Likewise.
> * gcc.target/i386/pr100865-11c.c: Likewise.
> * gcc.target/i386/pr100865-12a.c: Likewise.
> * gcc.target/i386/pr100865-12b.c: Likewise.
> * gcc.target/i386/pr100865-12c.c: Likewise.
> ---
> gcc/config/i386/i386-expand.c | 194 ++++++++++++++++--
> gcc/config/i386/i386-protos.h | 2 +
> gcc/config/i386/i386.c | 13 ++
> .../i386/avx512f-broadcast-pr87767-1.c | 7 +-
> .../i386/avx512f-broadcast-pr87767-5.c | 5 +-
> .../gcc.target/i386/avx512f_cond_move.c | 4 +-
> .../i386/avx512vl-broadcast-pr87767-1.c | 12 +-
> .../i386/avx512vl-broadcast-pr87767-5.c | 9 +-
> gcc/testsuite/gcc.target/i386/pr100865-1.c | 13 ++
> gcc/testsuite/gcc.target/i386/pr100865-10a.c | 33 +++
> gcc/testsuite/gcc.target/i386/pr100865-10b.c | 7 +
> gcc/testsuite/gcc.target/i386/pr100865-11a.c | 23 +++
> gcc/testsuite/gcc.target/i386/pr100865-11b.c | 8 +
> gcc/testsuite/gcc.target/i386/pr100865-11c.c | 8 +
> gcc/testsuite/gcc.target/i386/pr100865-12a.c | 20 ++
> gcc/testsuite/gcc.target/i386/pr100865-12b.c | 8 +
> gcc/testsuite/gcc.target/i386/pr100865-12c.c | 8 +
> gcc/testsuite/gcc.target/i386/pr100865-2.c | 14 ++
> gcc/testsuite/gcc.target/i386/pr100865-3.c | 15 ++
> gcc/testsuite/gcc.target/i386/pr100865-4a.c | 16 ++
> gcc/testsuite/gcc.target/i386/pr100865-4b.c | 9 +
> gcc/testsuite/gcc.target/i386/pr100865-5a.c | 16 ++
> gcc/testsuite/gcc.target/i386/pr100865-5b.c | 9 +
> gcc/testsuite/gcc.target/i386/pr100865-6a.c | 16 ++
> gcc/testsuite/gcc.target/i386/pr100865-6b.c | 9 +
> gcc/testsuite/gcc.target/i386/pr100865-6c.c | 16 ++
> gcc/testsuite/gcc.target/i386/pr100865-7a.c | 17 ++
> gcc/testsuite/gcc.target/i386/pr100865-7b.c | 9 +
> gcc/testsuite/gcc.target/i386/pr100865-7c.c | 17 ++
> gcc/testsuite/gcc.target/i386/pr100865-8a.c | 24 +++
> gcc/testsuite/gcc.target/i386/pr100865-8b.c | 7 +
> gcc/testsuite/gcc.target/i386/pr100865-8c.c | 7 +
> gcc/testsuite/gcc.target/i386/pr100865-9a.c | 25 +++
> gcc/testsuite/gcc.target/i386/pr100865-9b.c | 7 +
> gcc/testsuite/gcc.target/i386/pr100865-9c.c | 7 +
> 35 files changed, 590 insertions(+), 24 deletions(-)
> create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-1.c
> create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-10a.c
> create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-10b.c
> create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-11a.c
> create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-11b.c
> create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-11c.c
> create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-12a.c
> create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-12b.c
> create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-12c.c
> create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-2.c
> create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-3.c
> create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-4a.c
> create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-4b.c
> create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-5a.c
> create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-5b.c
> create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-6a.c
> create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-6b.c
> create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-6c.c
> create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-7a.c
> create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-7b.c
> create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-7c.c
> create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-8a.c
> create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-8b.c
> create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-8c.c
> create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-9a.c
> create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-9b.c
> create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-9c.c
>
> diff --git a/gcc/config/i386/i386-expand.c b/gcc/config/i386/i386-expand.c
> index e9763eb5b3e..e0e3ed4d8a4 100644
> --- a/gcc/config/i386/i386-expand.c
> +++ b/gcc/config/i386/i386-expand.c
> @@ -93,6 +93,9 @@ along with GCC; see the file COPYING3. If not see
> #include "i386-builtins.h"
> #include "i386-expand.h"
>
> +static bool ix86_expand_vector_init_duplicate (bool, machine_mode, rtx,
> + rtx);
> +
> /* Split one or more double-mode RTL references into pairs of half-mode
> references. The RTL can be REG, offsettable MEM, integer constant, or
> CONST_DOUBLE. "operands" is a pointer to an array of double-mode RTLs to
> @@ -190,6 +193,82 @@ ix86_expand_clear (rtx dest)
> emit_insn (tmp);
> }
>
> +/* Return true if V can be broadcasted from an integer of WIDTH bits
> + which is returned in VAL_BROADCAST. Otherwise, return false. */
> +
> +static bool
> +ix86_broadcast (HOST_WIDE_INT v, unsigned int width,
> + HOST_WIDE_INT &val_broadcast)
> +{
> + wide_int val = wi::uhwi (v, HOST_BITS_PER_WIDE_INT);
> + val_broadcast = wi::extract_uhwi (val, 0, width);
> + for (unsigned int i = width; i < HOST_BITS_PER_WIDE_INT; i += width)
> + {
> + HOST_WIDE_INT each = wi::extract_uhwi (val, i, width);
> + if (val_broadcast != each)
> + return false;
> + }
> + val_broadcast = sext_hwi (val_broadcast, width);
> + return true;
> +}
> +
> +/* Convert the CONST_WIDE_INT operand OP to broadcast in MODE. */
> +
> +static rtx
> +ix86_convert_const_wide_int_to_broadcast (machine_mode mode, rtx op)
> +{
> + /* Don't use integer vector broadcast if we can't move from GPR to SSE
> + register directly. */
> + if (!TARGET_INTER_UNIT_MOVES_TO_VEC)
> + return nullptr;
> +
> + /* Convert CONST_WIDE_INT to a non-standard SSE constant integer
> + broadcast only if vector broadcast is available. */
> + if (!TARGET_AVX
> + || !CONST_WIDE_INT_P (op)
> + || standard_sse_constant_p (op, mode))
> + return nullptr;
> +
> + HOST_WIDE_INT val = CONST_WIDE_INT_ELT (op, 0);
> + HOST_WIDE_INT val_broadcast;
> + scalar_int_mode broadcast_mode;
> + if (TARGET_AVX2
> + && ix86_broadcast (val, GET_MODE_BITSIZE (QImode),
> + val_broadcast))
> + broadcast_mode = QImode;
> + else if (TARGET_AVX2
> + && ix86_broadcast (val, GET_MODE_BITSIZE (HImode),
> + val_broadcast))
> + broadcast_mode = HImode;
> + else if (ix86_broadcast (val, GET_MODE_BITSIZE (SImode),
> + val_broadcast))
> + broadcast_mode = SImode;
> + else if (TARGET_64BIT
> + && ix86_broadcast (val, GET_MODE_BITSIZE (DImode),
> + val_broadcast))
> + broadcast_mode = DImode;
> + else
> + return nullptr;
> +
> + /* Check if OP can be broadcasted from VAL. */
> + for (int i = 1; i < CONST_WIDE_INT_NUNITS (op); i++)
> + if (val != CONST_WIDE_INT_ELT (op, i))
> + return nullptr;
> +
> + unsigned int nunits = (GET_MODE_SIZE (mode)
> + / GET_MODE_SIZE (broadcast_mode));
> + machine_mode vector_mode;
> + if (!mode_for_vector (broadcast_mode, nunits).exists (&vector_mode))
> + gcc_unreachable ();
> + rtx target = ix86_gen_scratch_sse_rtx (vector_mode);
> + bool ok = ix86_expand_vector_init_duplicate (false, vector_mode,
> + target,
> + GEN_INT (val_broadcast));
> + gcc_assert (ok);
> + target = lowpart_subreg (mode, target, vector_mode);
> + return target;
> +}
> +
> void
> ix86_expand_move (machine_mode mode, rtx operands[])
> {
> @@ -347,20 +426,29 @@ ix86_expand_move (machine_mode mode, rtx operands[])
> && optimize)
> op1 = copy_to_mode_reg (mode, op1);
>
> - if (can_create_pseudo_p ()
> - && CONST_DOUBLE_P (op1))
> + if (can_create_pseudo_p ())
> {
> - /* If we are loading a floating point constant to a register,
> - force the value to memory now, since we'll get better code
> - out the back end. */
> + if (CONST_DOUBLE_P (op1))
> + {
> + /* If we are loading a floating point constant to a
> + register, force the value to memory now, since we'll
> + get better code out the back end. */
>
> - op1 = validize_mem (force_const_mem (mode, op1));
> - if (!register_operand (op0, mode))
> + op1 = validize_mem (force_const_mem (mode, op1));
> + if (!register_operand (op0, mode))
> + {
> + rtx temp = gen_reg_rtx (mode);
> + emit_insn (gen_rtx_SET (temp, op1));
> + emit_move_insn (op0, temp);
> + return;
> + }
> + }
> + else if (GET_MODE_SIZE (mode) >= 16)
> {
> - rtx temp = gen_reg_rtx (mode);
> - emit_insn (gen_rtx_SET (temp, op1));
> - emit_move_insn (op0, temp);
> - return;
> + rtx tmp = ix86_convert_const_wide_int_to_broadcast
> + (GET_MODE (op0), op1);
> + if (tmp != nullptr)
> + op1 = tmp;
> }
> }
> }
> @@ -368,6 +456,59 @@ ix86_expand_move (machine_mode mode, rtx operands[])
> emit_insn (gen_rtx_SET (op0, op1));
> }
>
> +static rtx
> +ix86_broadcast_from_integer_constant (machine_mode mode, rtx op)
> +{
> + int nunits = GET_MODE_NUNITS (mode);
> + if (nunits < 2)
> + return nullptr;
> +
> + /* Don't use integer vector broadcast if we can't move from GPR to SSE
> + register directly. */
> + if (!TARGET_INTER_UNIT_MOVES_TO_VEC)
> + return nullptr;
> +
> + /* Convert CONST_VECTOR to a non-standard SSE constant integer
> + broadcast only if vector broadcast is available. */
> + if (!(TARGET_AVX2
> + || (TARGET_AVX
> + && (GET_MODE_INNER (mode) == SImode
> + || GET_MODE_INNER (mode) == DImode)))
> + || standard_sse_constant_p (op, mode))
> + return nullptr;
> +
> + /* Don't broadcast from a 64-bit integer constant in 32-bit mode. */
> + if (GET_MODE_INNER (mode) == DImode && !TARGET_64BIT)
> + return nullptr;
> +
> + rtx constant = get_pool_constant (XEXP (op, 0));
> + if (GET_CODE (constant) != CONST_VECTOR)
> + return nullptr;
> +
> + /* There could be some rtx like
> + (mem/u/c:V16QI (symbol_ref/u:DI ("*.LC1")))
> + but with "*.LC1" refer to V2DI constant vector. */
> + if (GET_MODE (constant) != mode)
> + {
> + constant = simplify_subreg (mode, constant, GET_MODE (constant),
> + 0);
> + if (constant == nullptr || GET_CODE (constant) != CONST_VECTOR)
> + return nullptr;
> + }
> +
> + rtx first = XVECEXP (constant, 0, 0);
> +
> + for (int i = 1; i < nunits; ++i)
> + {
> + rtx tmp = XVECEXP (constant, 0, i);
> + /* Vector duplicate value. */
> + if (!rtx_equal_p (tmp, first))
> + return nullptr;
> + }
> +
> + return first;
> +}
> +
> void
> ix86_expand_vector_move (machine_mode mode, rtx operands[])
> {
> @@ -407,7 +548,36 @@ ix86_expand_vector_move (machine_mode mode, rtx operands[])
> op1 = simplify_gen_subreg (mode, r, imode, SUBREG_BYTE (op1));
> }
> else
> - op1 = validize_mem (force_const_mem (mode, op1));
> + {
> + machine_mode mode = GET_MODE (op0);
> + rtx tmp = ix86_convert_const_wide_int_to_broadcast
> + (mode, op1);
> + if (tmp == nullptr)
> + op1 = validize_mem (force_const_mem (mode, op1));
> + else
> + op1 = tmp;
> + }
> + }
> +
> + if (can_create_pseudo_p ()
> + && GET_MODE_SIZE (mode) >= 16
> + && GET_MODE_CLASS (mode) == MODE_VECTOR_INT
> + && (MEM_P (op1)
> + && SYMBOL_REF_P (XEXP (op1, 0))
> + && CONSTANT_POOL_ADDRESS_P (XEXP (op1, 0))))
> + {
> + rtx first = ix86_broadcast_from_integer_constant (mode, op1);
> + if (first != nullptr)
> + {
> + /* Broadcast to XMM/YMM/ZMM register from an integer
> + constant. */
> + op1 = ix86_gen_scratch_sse_rtx (mode);
> + bool ok = ix86_expand_vector_init_duplicate (false, mode,
> + op1, first);
> + gcc_assert (ok);
> + emit_move_insn (op0, op1);
> + return;
> + }
> }
>
> /* We need to check memory alignment for SSE mode since attribute
> diff --git a/gcc/config/i386/i386-protos.h b/gcc/config/i386/i386-protos.h
> index 65fc307dc7b..71745b9a1ea 100644
> --- a/gcc/config/i386/i386-protos.h
> +++ b/gcc/config/i386/i386-protos.h
> @@ -50,6 +50,8 @@ extern void ix86_reset_previous_fndecl (void);
>
> extern bool ix86_using_red_zone (void);
>
> +extern rtx ix86_gen_scratch_sse_rtx (machine_mode);
> +
> extern unsigned int ix86_regmode_natural_size (machine_mode);
> #ifdef RTX_CODE
> extern int standard_80387_constant_p (rtx);
> diff --git a/gcc/config/i386/i386.c b/gcc/config/i386/i386.c
> index a93128fa0a4..2fbaae7cd02 100644
> --- a/gcc/config/i386/i386.c
> +++ b/gcc/config/i386/i386.c
> @@ -23163,6 +23163,19 @@ ix86_optab_supported_p (int op, machine_mode mode1, machine_mode,
> }
> }
>
> +/* Return a scratch register in MODE for vector load and store. */
> +
> +rtx
> +ix86_gen_scratch_sse_rtx (machine_mode mode)
> +{
> + if (TARGET_SSE)
> + return gen_rtx_REG (mode, (TARGET_64BIT
> + ? LAST_REX_SSE_REG
> + : LAST_SSE_REG));
> + else
> + return gen_reg_rtx (mode);
> +}
> +
> /* Address space support.
>
> This is not "far pointers" in the 16-bit sense, but an easy way
> diff --git a/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-1.c b/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-1.c
> index 0563e696316..a2664d87f29 100644
> --- a/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-1.c
> +++ b/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-1.c
> @@ -2,8 +2,11 @@
> /* { dg-do compile } */
> /* { dg-options "-O2 -mavx512f -mavx512dq" } */
> /* { dg-additional-options "-mdynamic-no-pic" { target { *-*-darwin* && ia32 } } }
> -/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to8\\\}" 5 } } */
> -/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to16\\\}" 5 } } */
> +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to8\\\}" 2 { target { ! ia32 } } } } */
> +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to8\\\}" 5 { target ia32 } } } */
> +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to16\\\}" 2 } } */
> +/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %zmm\[0-9\]+" 3 } } */
> +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%r\[^\n\]*, %zmm\[0-9\]+" 3 { target { ! ia32 } } } } */
>
> typedef int v16si __attribute__ ((vector_size (64)));
> typedef long long v8di __attribute__ ((vector_size (64)));
> diff --git a/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-5.c b/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-5.c
> index ffbe95980ca..477f9ca1282 100644
> --- a/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-5.c
> +++ b/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-5.c
> @@ -2,8 +2,9 @@
> /* { dg-do compile } */
> /* { dg-options "-O2 -mavx512f" } */
> /* { dg-additional-options "-mdynamic-no-pic" { target { *-*-darwin* && ia32 } } }
> -/* { dg-final { scan-assembler-times "\[^n\n\]*\\\{1to8\\\}" 4 } } */
> -/* { dg-final { scan-assembler-times "\[^n\n\]*\\\{1to16\\\}" 4 } } */
> +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to8\\\}" 4 { target ia32 } } } */
> +/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %zmm\[0-9\]+" 4 } } */
> +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%r\[^\n\]*, %zmm\[0-9\]+" 4 { target { ! ia32 } } } } */
>
> typedef int v16si __attribute__ ((vector_size (64)));
> typedef long long v8di __attribute__ ((vector_size (64)));
> diff --git a/gcc/testsuite/gcc.target/i386/avx512f_cond_move.c b/gcc/testsuite/gcc.target/i386/avx512f_cond_move.c
> index 99a89f51202..ca49a585232 100644
> --- a/gcc/testsuite/gcc.target/i386/avx512f_cond_move.c
> +++ b/gcc/testsuite/gcc.target/i386/avx512f_cond_move.c
> @@ -1,6 +1,6 @@
> /* { dg-do compile } */
> -/* { dg-options "-O3 -mavx512f" } */
> -/* { dg-final { scan-assembler-times "(?:vpblendmd|vmovdqa32)\[ \\t\]+\[^\{\n\]*%zmm\[0-9\]+\{%k\[1-7\]\}\{z\}(?:\n|\[ \\t\]+#)" 8 } } */
> +/* { dg-options "-O3 -mavx512f -mprefer-vector-width=512" } */
> +/* { dg-final { scan-assembler-times "(?:vpbroadcastd|vmovdqa32)\[ \\t\]+\[^\{\n\]*%zmm\[0-9\]+\{%k\[1-7\]\}\{z\}(?:\n|\[ \\t\]+#)" 8 } } */
>
> unsigned int x[128];
> int y[128];
> diff --git a/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-1.c b/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-1.c
> index c06369d93fd..f8eb99f0b5f 100644
> --- a/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-1.c
> +++ b/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-1.c
> @@ -2,9 +2,15 @@
> /* { dg-do compile } */
> /* { dg-options "-O2 -mavx512f -mavx512vl -mavx512dq" } */
> /* { dg-additional-options "-mdynamic-no-pic" { target { *-*-darwin* && ia32 } } }
> -/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to2\\\}" 5 } } */
> -/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to4\\\}" 10 } } */
> -/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to8\\\}" 5 } } */
> +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to2\\\}" 2 { target { ! ia32 } } } } */
> +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to4\\\}" 4 { target { ! ia32 } } } } */
> +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to2\\\}" 5 { target ia32 } } } */
> +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to4\\\}" 7 { target ia32 } } } */
> +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to8\\\}" 2 } } */
> +/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 3 } } */
> +/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %ymm\[0-9\]+" 3 } } */
> +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%r\[^\n\]*, %xmm\[0-9\]+" 3 { target { ! ia32 } } } } */
> +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%r\[^\n\]*, %ymm\[0-9\]+" 3 { target { ! ia32 } } } } */
>
> typedef int v4si __attribute__ ((vector_size (16)));
> typedef int v8si __attribute__ ((vector_size (32)));
> diff --git a/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-5.c b/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-5.c
> index 4998a9b8d51..32f6ac81841 100644
> --- a/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-5.c
> +++ b/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-5.c
> @@ -2,9 +2,12 @@
> /* { dg-do compile } */
> /* { dg-options "-O2 -mavx512f -mavx512vl" } */
> /* { dg-additional-options "-mdynamic-no-pic" { target { *-*-darwin* && ia32 } } }
> -/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to2\\\}" 4 } } */
> -/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to4\\\}" 8 } } */
> -/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to8\\\}" 4 } } */
> +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to2\\\}" 4 { target ia32 } } } */
> +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to4\\\}" 4 { target ia32 } } } */
> +/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 4 } } */
> +/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %ymm\[0-9\]+" 4 } } */
> +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%r\[^\n\]*, %xmm\[0-9\]+" 4 { target { ! ia32 } } } } */
> +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%r\[^\n\]*, %ymm\[0-9\]+" 4 { target { ! ia32 } } } } */
>
> typedef int v4si __attribute__ ((vector_size (16)));
> typedef int v8si __attribute__ ((vector_size (32)));
> diff --git a/gcc/testsuite/gcc.target/i386/pr100865-1.c b/gcc/testsuite/gcc.target/i386/pr100865-1.c
> new file mode 100644
> index 00000000000..6c3097fb2a6
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/i386/pr100865-1.c
> @@ -0,0 +1,13 @@
> +/* { dg-do compile { target { ! ia32 } } } */
> +/* { dg-options "-O2 -march=x86-64" } */
> +
> +extern char *dst;
> +
> +void
> +foo (void)
> +{
> + __builtin_memset (dst, 3, 16);
> +}
> +
> +/* { dg-final { scan-assembler-times "movdqa\[ \\t\]+\[^\n\]*%xmm" 1 } } */
> +/* { dg-final { scan-assembler-times "movups\[\\t \]%xmm\[0-9\]+, \\(%\[\^,\]+\\)" 1 } } */
> diff --git a/gcc/testsuite/gcc.target/i386/pr100865-10a.c b/gcc/testsuite/gcc.target/i386/pr100865-10a.c
> new file mode 100644
> index 00000000000..7ffc19e56a8
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/i386/pr100865-10a.c
> @@ -0,0 +1,33 @@
> +/* { dg-do compile { target int128 } } */
> +/* { dg-options "-O3 -march=skylake" } */
> +
> +extern __int128 array[16];
> +
> +#define MK_CONST128_BROADCAST(A) \
> + ((((unsigned __int128) (unsigned char) A) << 120) \
> + | (((unsigned __int128) (unsigned char) A) << 112) \
> + | (((unsigned __int128) (unsigned char) A) << 104) \
> + | (((unsigned __int128) (unsigned char) A) << 96) \
> + | (((unsigned __int128) (unsigned char) A) << 88) \
> + | (((unsigned __int128) (unsigned char) A) << 80) \
> + | (((unsigned __int128) (unsigned char) A) << 72) \
> + | (((unsigned __int128) (unsigned char) A) << 64) \
> + | (((unsigned __int128) (unsigned char) A) << 56) \
> + | (((unsigned __int128) (unsigned char) A) << 48) \
> + | (((unsigned __int128) (unsigned char) A) << 40) \
> + | (((unsigned __int128) (unsigned char) A) << 32) \
> + | (((unsigned __int128) (unsigned char) A) << 24) \
> + | (((unsigned __int128) (unsigned char) A) << 16) \
> + | (((unsigned __int128) (unsigned char) A) << 8) \
> + | ((unsigned __int128) (unsigned char) A) )
> +
> +void
> +foo (void)
> +{
> + int i;
> + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> + array[i] = MK_CONST128_BROADCAST (0x1f);
> +}
> +
> +/* { dg-final { scan-assembler-times "vpbroadcastb\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> diff --git a/gcc/testsuite/gcc.target/i386/pr100865-10b.c b/gcc/testsuite/gcc.target/i386/pr100865-10b.c
> new file mode 100644
> index 00000000000..edf52765c60
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/i386/pr100865-10b.c
> @@ -0,0 +1,7 @@
> +/* { dg-do compile { target int128 } } */
> +/* { dg-options "-O3 -march=skylake-avx512" } */
> +
> +#include "pr100865-10a.c"
> +
> +/* { dg-final { scan-assembler-times "vpbroadcastb\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> diff --git a/gcc/testsuite/gcc.target/i386/pr100865-11a.c b/gcc/testsuite/gcc.target/i386/pr100865-11a.c
> new file mode 100644
> index 00000000000..04ce1662f3c
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/i386/pr100865-11a.c
> @@ -0,0 +1,23 @@
> +/* { dg-do compile { target int128 } } */
> +/* { dg-options "-O3 -march=skylake" } */
> +
> +extern __int128 array[16];
> +
> +#define MK_CONST128_BROADCAST(A) \
> + ((((unsigned __int128) (unsigned long long) A) << 64) \
> + | ((unsigned __int128) (unsigned long long) A) )
> +
> +#define MK_CONST128_BROADCAST_SIGNED(A) \
> + ((__int128) MK_CONST128_BROADCAST (A))
> +
> +void
> +foo (void)
> +{
> + int i;
> + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> + array[i] = MK_CONST128_BROADCAST_SIGNED (-0x1ffffffffLL);
> +}
> +
> +/* { dg-final { scan-assembler-times "movabsq" 1 } } */
> +/* { dg-final { scan-assembler-times "(?:vpbroadcastq|vpunpcklqdq)\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> diff --git a/gcc/testsuite/gcc.target/i386/pr100865-11b.c b/gcc/testsuite/gcc.target/i386/pr100865-11b.c
> new file mode 100644
> index 00000000000..12d55b9a642
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/i386/pr100865-11b.c
> @@ -0,0 +1,8 @@
> +/* { dg-do compile { target int128 } } */
> +/* { dg-options "-O3 -march=skylake-avx512" } */
> +
> +#include "pr100865-11a.c"
> +
> +/* { dg-final { scan-assembler-times "movabsq" 1 } } */
> +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> diff --git a/gcc/testsuite/gcc.target/i386/pr100865-11c.c b/gcc/testsuite/gcc.target/i386/pr100865-11c.c
> new file mode 100644
> index 00000000000..de56c84b9ca
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/i386/pr100865-11c.c
> @@ -0,0 +1,8 @@
> +/* { dg-do compile { target int128 } } */
> +/* { dg-options "-O3 -march=skylake -mno-avx2" } */
> +
> +#include "pr100865-11a.c"
> +
> +/* { dg-final { scan-assembler-times "movabsq" 1 } } */
> +/* { dg-final { scan-assembler-times "vpunpcklqdq\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> diff --git a/gcc/testsuite/gcc.target/i386/pr100865-12a.c b/gcc/testsuite/gcc.target/i386/pr100865-12a.c
> new file mode 100644
> index 00000000000..d4833d44475
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/i386/pr100865-12a.c
> @@ -0,0 +1,20 @@
> +/* { dg-do compile { target int128 } } */
> +/* { dg-options "-O3 -march=skylake" } */
> +
> +extern __int128 array[16];
> +
> +#define MK_CONST128_BROADCAST(A) \
> + ((((unsigned __int128) (unsigned long long) A) << 64) \
> + | ((unsigned __int128) (unsigned long long) A) )
> +
> +void
> +foo (void)
> +{
> + int i;
> + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> + array[i] = MK_CONST128_BROADCAST (0x1ffffffffLL);
> +}
> +
> +/* { dg-final { scan-assembler-times "movabsq" 1 } } */
> +/* { dg-final { scan-assembler-times "(?:vpbroadcastq|vpunpcklqdq)\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> diff --git a/gcc/testsuite/gcc.target/i386/pr100865-12b.c b/gcc/testsuite/gcc.target/i386/pr100865-12b.c
> new file mode 100644
> index 00000000000..63a5629b90c
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/i386/pr100865-12b.c
> @@ -0,0 +1,8 @@
> +/* { dg-do compile { target int128 } } */
> +/* { dg-options "-O3 -march=skylake-avx512" } */
> +
> +#include "pr100865-12a.c"
> +
> +/* { dg-final { scan-assembler-times "movabsq" 1 } } */
> +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> diff --git a/gcc/testsuite/gcc.target/i386/pr100865-12c.c b/gcc/testsuite/gcc.target/i386/pr100865-12c.c
> new file mode 100644
> index 00000000000..77415f22c97
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/i386/pr100865-12c.c
> @@ -0,0 +1,8 @@
> +/* { dg-do compile { target int128 } } */
> +/* { dg-options "-O3 -march=skylake -mno-avx2" } */
> +
> +#include "pr100865-12a.c"
> +
> +/* { dg-final { scan-assembler-times "movabsq" 1 } } */
> +/* { dg-final { scan-assembler-times "vpunpcklqdq\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> diff --git a/gcc/testsuite/gcc.target/i386/pr100865-2.c b/gcc/testsuite/gcc.target/i386/pr100865-2.c
> new file mode 100644
> index 00000000000..17efe2d72a3
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/i386/pr100865-2.c
> @@ -0,0 +1,14 @@
> +/* { dg-do compile { target { ! ia32 } } } */
> +/* { dg-options "-O2 -march=skylake" } */
> +
> +extern char *dst;
> +
> +void
> +foo (void)
> +{
> + __builtin_memset (dst, 3, 16);
> +}
> +
> +/* { dg-final { scan-assembler-times "vpbroadcastb\[\\t \]+%xmm\[0-9\]+, %xmm\[0-9\]+" 1 } } */
> +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%xmm\[0-9\]+, \\(%\[\^,\]+\\)" 1 } } */
> +/* { dg-final { scan-assembler-not "vmovdqa" } } */
> diff --git a/gcc/testsuite/gcc.target/i386/pr100865-3.c b/gcc/testsuite/gcc.target/i386/pr100865-3.c
> new file mode 100644
> index 00000000000..b6dbcf7809b
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/i386/pr100865-3.c
> @@ -0,0 +1,15 @@
> +/* { dg-do compile { target { ! ia32 } } } */
> +/* { dg-options "-O2 -march=skylake-avx512" } */
> +
> +extern char *dst;
> +
> +void
> +foo (void)
> +{
> + __builtin_memset (dst, 3, 16);
> +}
> +
> +/* { dg-final { scan-assembler-times "vpbroadcastb\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%xmm\[0-9\]+, \\(%\[\^,\]+\\)" 1 } } */
> +/* { dg-final { scan-assembler-not "vpbroadcastb\[\\t \]+%xmm\[0-9\]+, %xmm\[0-9\]+" } } */
> +/* { dg-final { scan-assembler-not "vmovdqa" } } */
> diff --git a/gcc/testsuite/gcc.target/i386/pr100865-4a.c b/gcc/testsuite/gcc.target/i386/pr100865-4a.c
> new file mode 100644
> index 00000000000..f55883598f9
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/i386/pr100865-4a.c
> @@ -0,0 +1,16 @@
> +/* { dg-do compile { target { ! ia32 } } } */
> +/* { dg-options "-O2 -march=skylake" } */
> +
> +extern char array[64];
> +
> +void
> +foo (void)
> +{
> + int i;
> + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> + array[i] = -45;
> +}
> +
> +/* { dg-final { scan-assembler-times "vpbroadcastb\[\\t \]+%xmm\[0-9\]+, %xmm\[0-9\]+" 1 } } */
> +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%xmm\[0-9\]+, " 4 } } */
> +/* { dg-final { scan-assembler-not "vmovdqa" } } */
> diff --git a/gcc/testsuite/gcc.target/i386/pr100865-4b.c b/gcc/testsuite/gcc.target/i386/pr100865-4b.c
> new file mode 100644
> index 00000000000..f41e6147b4c
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/i386/pr100865-4b.c
> @@ -0,0 +1,9 @@
> +/* { dg-do compile { target { ! ia32 } } } */
> +/* { dg-options "-O2 -march=skylake-avx512" } */
> +
> +#include "pr100865-4a.c"
> +
> +/* { dg-final { scan-assembler-times "vpbroadcastb\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%xmm\[0-9\]+, " 4 } } */
> +/* { dg-final { scan-assembler-not "vpbroadcastb\[\\t \]+%xmm\[0-9\]+, %xmm\[0-9\]+" } } */
> +/* { dg-final { scan-assembler-not "vmovdqa" } } */
> diff --git a/gcc/testsuite/gcc.target/i386/pr100865-5a.c b/gcc/testsuite/gcc.target/i386/pr100865-5a.c
> new file mode 100644
> index 00000000000..4149797fe81
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/i386/pr100865-5a.c
> @@ -0,0 +1,16 @@
> +/* { dg-do compile } */
> +/* { dg-options "-O3 -march=skylake" } */
> +
> +extern short array[64];
> +
> +void
> +foo (void)
> +{
> + int i;
> + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> + array[i] = -45;
> +}
> +
> +/* { dg-final { scan-assembler-times "vpbroadcastw\[\\t \]+%xmm\[0-9\]+, %ymm\[0-9\]+" 1 } } */
> +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%ymm\[0-9\]+, " 4 } } */
> +/* { dg-final { scan-assembler-not "vmovdqa" } } */
> diff --git a/gcc/testsuite/gcc.target/i386/pr100865-5b.c b/gcc/testsuite/gcc.target/i386/pr100865-5b.c
> new file mode 100644
> index 00000000000..ded41b680d3
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/i386/pr100865-5b.c
> @@ -0,0 +1,9 @@
> +/* { dg-do compile } */
> +/* { dg-options "-O3 -march=skylake-avx512" } */
> +
> +#include "pr100865-5a.c"
> +
> +/* { dg-final { scan-assembler-times "vpbroadcastw\[\\t \]+%(?:r|e)\[^\n\]*, %ymm\[0-9\]+" 1 } } */
> +/* { dg-final { scan-assembler-times "vmovdqu16\[\\t \]%ymm\[0-9\]+, " 4 } } */
> +/* { dg-final { scan-assembler-not "vpbroadcastw\[\\t \]+%xmm\[0-9\]+, %ymm\[0-9\]+" } } */
> +/* { dg-final { scan-assembler-not "vmovdqa" } } */
> diff --git a/gcc/testsuite/gcc.target/i386/pr100865-6a.c b/gcc/testsuite/gcc.target/i386/pr100865-6a.c
> new file mode 100644
> index 00000000000..3fde549a10d
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/i386/pr100865-6a.c
> @@ -0,0 +1,16 @@
> +/* { dg-do compile } */
> +/* { dg-options "-O3 -march=skylake" } */
> +
> +extern int array[64];
> +
> +void
> +foo (void)
> +{
> + int i;
> + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> + array[i] = -45;
> +}
> +
> +/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%xmm\[0-9\]+, %ymm\[0-9\]+" 1 } } */
> +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%ymm\[0-9\]+, " 8 } } */
> +/* { dg-final { scan-assembler-not "vmovdqa" } } */
> diff --git a/gcc/testsuite/gcc.target/i386/pr100865-6b.c b/gcc/testsuite/gcc.target/i386/pr100865-6b.c
> new file mode 100644
> index 00000000000..44e74c64e55
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/i386/pr100865-6b.c
> @@ -0,0 +1,9 @@
> +/* { dg-do compile } */
> +/* { dg-options "-O3 -march=skylake-avx512" } */
> +
> +#include "pr100865-6a.c"
> +
> +/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %ymm\[0-9\]+" 1 } } */
> +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%ymm\[0-9\]+, " 8 } } */
> +/* { dg-final { scan-assembler-not "vpbroadcastd\[\\t \]+%xmm\[0-9\]+, %ymm\[0-9\]+" } } */
> +/* { dg-final { scan-assembler-not "vmovdqa" } } */
> diff --git a/gcc/testsuite/gcc.target/i386/pr100865-6c.c b/gcc/testsuite/gcc.target/i386/pr100865-6c.c
> new file mode 100644
> index 00000000000..46d31030ce8
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/i386/pr100865-6c.c
> @@ -0,0 +1,16 @@
> +/* { dg-do compile } */
> +/* { dg-options "-O3 -march=skylake -mno-avx2" } */
> +
> +extern int array[64];
> +
> +void
> +foo (void)
> +{
> + int i;
> + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> + array[i] = -45;
> +}
> +
> +/* { dg-final { scan-assembler-times "vbroadcastss" 1 } } */
> +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%ymm\[0-9\]+, " 8 } } */
> +/* { dg-final { scan-assembler-not "vmovdqa" } } */
> diff --git a/gcc/testsuite/gcc.target/i386/pr100865-7a.c b/gcc/testsuite/gcc.target/i386/pr100865-7a.c
> new file mode 100644
> index 00000000000..f6f2be91120
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/i386/pr100865-7a.c
> @@ -0,0 +1,17 @@
> +/* { dg-do compile } */
> +/* { dg-options "-O3 -march=skylake" } */
> +
> +extern long long int array[64];
> +
> +void
> +foo (void)
> +{
> + int i;
> + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> + array[i] = -45;
> +}
> +
> +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+\[^\n\]*, %ymm\[0-9\]+" 1 { target { ! ia32 } } } } */
> +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%ymm\[0-9\]+, " 16 } } */
> +/* { dg-final { scan-assembler-not "vpbroadcastq" { target ia32 } } } */
> +/* { dg-final { scan-assembler-not "vmovdqa" { target { ! ia32 } } } } */
> diff --git a/gcc/testsuite/gcc.target/i386/pr100865-7b.c b/gcc/testsuite/gcc.target/i386/pr100865-7b.c
> new file mode 100644
> index 00000000000..0a68820aa32
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/i386/pr100865-7b.c
> @@ -0,0 +1,9 @@
> +/* { dg-do compile } */
> +/* { dg-options "-O3 -march=skylake-avx512" } */
> +
> +#include "pr100865-7a.c"
> +
> +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%r\[^\n\]*, %ymm\[0-9\]+" 1 { target { ! ia32 } } } } */
> +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+\[^\n\]*, %ymm\[0-9\]+" 1 { target ia32 } } } */
> +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%ymm\[0-9\]+, " 16 } } */
> +/* { dg-final { scan-assembler-not "vmovdqa" } } */
> diff --git a/gcc/testsuite/gcc.target/i386/pr100865-7c.c b/gcc/testsuite/gcc.target/i386/pr100865-7c.c
> new file mode 100644
> index 00000000000..4d50bb7e2f6
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/i386/pr100865-7c.c
> @@ -0,0 +1,17 @@
> +/* { dg-do compile } */
> +/* { dg-options "-O3 -march=skylake -mno-avx2" } */
> +
> +extern long long int array[64];
> +
> +void
> +foo (void)
> +{
> + int i;
> + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> + array[i] = -45;
> +}
> +
> +/* { dg-final { scan-assembler-times "vbroadcastsd" 1 { target { ! ia32 } } } } */
> +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%ymm\[0-9\]+, " 16 } } */
> +/* { dg-final { scan-assembler-not "vbroadcastsd" { target ia32 } } } */
> +/* { dg-final { scan-assembler-not "vmovdqa" { target { ! ia32 } } } } */
> diff --git a/gcc/testsuite/gcc.target/i386/pr100865-8a.c b/gcc/testsuite/gcc.target/i386/pr100865-8a.c
> new file mode 100644
> index 00000000000..544a14db6f7
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/i386/pr100865-8a.c
> @@ -0,0 +1,24 @@
> +/* { dg-do compile { target int128 } } */
> +/* { dg-options "-O3 -march=skylake" } */
> +
> +extern __int128 array[16];
> +
> +#define MK_CONST128_BROADCAST(A) \
> + ((((unsigned __int128) (unsigned int) A) << 96) \
> + | (((unsigned __int128) (unsigned int) A) << 64) \
> + | (((unsigned __int128) (unsigned int) A) << 32) \
> + | ((unsigned __int128) (unsigned int) A) )
> +
> +#define MK_CONST128_BROADCAST_SIGNED(A) \
> + ((__int128) MK_CONST128_BROADCAST (A))
> +
> +void
> +foo (void)
> +{
> + int i;
> + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> + array[i] = MK_CONST128_BROADCAST_SIGNED (-45);
> +}
> +
> +/* { dg-final { scan-assembler-times "(?:vpbroadcastd|vpshufd)\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> diff --git a/gcc/testsuite/gcc.target/i386/pr100865-8b.c b/gcc/testsuite/gcc.target/i386/pr100865-8b.c
> new file mode 100644
> index 00000000000..99a10ad83bd
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/i386/pr100865-8b.c
> @@ -0,0 +1,7 @@
> +/* { dg-do compile { target int128 } } */
> +/* { dg-options "-O3 -march=skylake-avx512" } */
> +
> +#include "pr100865-8a.c"
> +
> +/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> diff --git a/gcc/testsuite/gcc.target/i386/pr100865-8c.c b/gcc/testsuite/gcc.target/i386/pr100865-8c.c
> new file mode 100644
> index 00000000000..efee0488614
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/i386/pr100865-8c.c
> @@ -0,0 +1,7 @@
> +/* { dg-do compile { target int128 } } */
> +/* { dg-options "-O3 -march=skylake -mno-avx2" } */
> +
> +#include "pr100865-8a.c"
> +
> +/* { dg-final { scan-assembler-times "vpshufd\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> diff --git a/gcc/testsuite/gcc.target/i386/pr100865-9a.c b/gcc/testsuite/gcc.target/i386/pr100865-9a.c
> new file mode 100644
> index 00000000000..45d0e0d0e2e
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/i386/pr100865-9a.c
> @@ -0,0 +1,25 @@
> +/* { dg-do compile { target int128 } } */
> +/* { dg-options "-O3 -march=skylake" } */
> +
> +extern __int128 array[16];
> +
> +#define MK_CONST128_BROADCAST(A) \
> + ((((unsigned __int128) (unsigned short) A) << 112) \
> + | (((unsigned __int128) (unsigned short) A) << 96) \
> + | (((unsigned __int128) (unsigned short) A) << 80) \
> + | (((unsigned __int128) (unsigned short) A) << 64) \
> + | (((unsigned __int128) (unsigned short) A) << 48) \
> + | (((unsigned __int128) (unsigned short) A) << 32) \
> + | (((unsigned __int128) (unsigned short) A) << 16) \
> + | ((unsigned __int128) (unsigned short) A) )
> +
> +void
> +foo (void)
> +{
> + int i;
> + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> + array[i] = MK_CONST128_BROADCAST (0x1fff);
> +}
> +
> +/* { dg-final { scan-assembler-times "vpbroadcastw\[\\t \]+%xmm\[0-9\]+, %xmm\[0-9\]+" 1 } } */
> +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> diff --git a/gcc/testsuite/gcc.target/i386/pr100865-9b.c b/gcc/testsuite/gcc.target/i386/pr100865-9b.c
> new file mode 100644
> index 00000000000..14696248525
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/i386/pr100865-9b.c
> @@ -0,0 +1,7 @@
> +/* { dg-do compile { target int128 } } */
> +/* { dg-options "-O3 -march=skylake-avx512" } */
> +
> +#include "pr100865-9a.c"
> +
> +/* { dg-final { scan-assembler-times "vpbroadcastw\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> diff --git a/gcc/testsuite/gcc.target/i386/pr100865-9c.c b/gcc/testsuite/gcc.target/i386/pr100865-9c.c
> new file mode 100644
> index 00000000000..e6f25902c1d
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/i386/pr100865-9c.c
> @@ -0,0 +1,7 @@
> +/* { dg-do compile { target int128 } } */
> +/* { dg-options "-O3 -march=skylake -mno-avx2" } */
> +
> +#include "pr100865-9a.c"
> +
> +/* { dg-final { scan-assembler-times "vpshufd\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> --
> 2.31.1
>
--
BR,
Hongtao
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v6 1/2] x86: Convert CONST_WIDE_INT/CONST_VECTOR to broadcast
2021-07-01 8:37 ` Hongtao Liu
@ 2021-07-01 12:41 ` H.J. Lu
2021-07-01 14:12 ` Uros Bizjak
0 siblings, 1 reply; 7+ messages in thread
From: H.J. Lu @ 2021-07-01 12:41 UTC (permalink / raw)
To: Hongtao Liu
Cc: GCC Patches, Uros Bizjak, Jakub Jelinek, Richard Sandiford,
Richard Biener
Hi Uros,
On Thu, Jul 1, 2021 at 1:32 AM Hongtao Liu <crazylht@gmail.com> wrote:
>
> On Tue, Jun 29, 2021 at 6:16 AM H.J. Lu <hjl.tools@gmail.com> wrote:
> >
> > 1. Update move expanders to convert the CONST_WIDE_INT and CONST_VECTOR
> > operands to vector broadcast from an integer with AVX.
> > 2. Add ix86_gen_scratch_sse_rtx to return a scratch SSE register which
> > won't increase stack alignment requirement and blocks transformation by
> > the combine pass.
> >
> > A small benchmark:
> >
> > https://gitlab.com/x86-benchmarks/microbenchmark/-/tree/memset/broadcast
> >
> > shows that broadcast is a little bit faster on Intel Core i7-8559U:
> >
> > $ make
> > gcc -g -I. -O2 -c -o test.o test.c
> > gcc -g -c -o memory.o memory.S
> > gcc -g -c -o broadcast.o broadcast.S
> > gcc -g -c -o vec_dup_sse2.o vec_dup_sse2.S
> > gcc -o test test.o memory.o broadcast.o vec_dup_sse2.o
> > ./test
> > memory : 147215
> > broadcast : 121213
> > vec_dup_sse2: 171366
> > $
> >
> > broadcast is also smaller:
> >
> > $ size memory.o broadcast.o
> > text data bss dec hex filename
> > 132 0 0 132 84 memory.o
> > 122 0 0 122 7a broadcast.o
> > $
> >
> > 3. Update PR 87767 tests to expect integer broadcast instead of broadcast
> > from memory.
> > 4. Update avx512f_cond_move.c to expect integer broadcast.
> >
> > A small benchmark:
> >
> > https://gitlab.com/x86-benchmarks/microbenchmark/-/tree/vpaddd/broadcast
> >
> > shows that integer broadcast is faster than embedded memory broadcast:
> >
> > $ make
> > gcc -g -I. -O2 -march=skylake-avx512 -c -o test.o test.c
> > gcc -g -c -o memory.o memory.S
> > gcc -g -c -o broadcast.o broadcast.S
> > gcc -o test test.o memory.o broadcast.o
> > ./test
> > memory : 425538
> > broadcast : 375260
> > $
> >
> > gcc/
> >
> > PR target/100865
> > * config/i386/i386-expand.c (ix86_expand_vector_init_duplicate):
> > New prototype.
> > (ix86_byte_broadcast): New function.
> > (ix86_convert_const_wide_int_to_broadcast): Likewise.
> > (ix86_expand_move): Convert CONST_WIDE_INT to broadcast if mode
> > size is 16 bytes or bigger.
> > (ix86_broadcast_from_integer_constant): New function.
> > (ix86_expand_vector_move): Convert CONST_WIDE_INT and CONST_VECTOR
> > to broadcast if mode size is 16 bytes or bigger.
> > * config/i386/i386-protos.h (ix86_gen_scratch_sse_rtx): New
> > prototype.
> > * config/i386/i386.c (ix86_gen_scratch_sse_rtx): New function.
> >
> > gcc/testsuite/
> >
> > PR target/100865
> > * gcc.target/i386/avx512f-broadcast-pr87767-1.c: Expect integer
> > broadcast.
> > * gcc.target/i386/avx512f-broadcast-pr87767-5.c: Likewise.
> > * gcc.target/i386/avx512vl-broadcast-pr87767-1.c: Likewise.
> > * gcc.target/i386/avx512vl-broadcast-pr87767-5.c: Likewise.
> > * gcc.target/i386/avx512f_cond_move.c: Also pass
> > -mprefer-vector-width=512 and expect integer broadcast.
> Those tests change LGTM since it's faster than the older version, and
> I remember uros has already reviewed other parts, right?
Uros, does this version look OK?
Thanks.
> > * gcc.target/i386/pr100865-1.c: New test.
> > * gcc.target/i386/pr100865-2.c: Likewise.
> > * gcc.target/i386/pr100865-3.c: Likewise.
> > * gcc.target/i386/pr100865-4a.c: Likewise.
> > * gcc.target/i386/pr100865-4b.c: Likewise.
> > * gcc.target/i386/pr100865-5a.c: Likewise.
> > * gcc.target/i386/pr100865-5b.c: Likewise.
> > * gcc.target/i386/pr100865-6a.c: Likewise.
> > * gcc.target/i386/pr100865-6b.c: Likewise.
> > * gcc.target/i386/pr100865-6c.c: Likewise.
> > * gcc.target/i386/pr100865-7a.c: Likewise.
> > * gcc.target/i386/pr100865-7b.c: Likewise.
> > * gcc.target/i386/pr100865-7c.c: Likewise.
> > * gcc.target/i386/pr100865-8a.c: Likewise.
> > * gcc.target/i386/pr100865-8b.c: Likewise.
> > * gcc.target/i386/pr100865-8c.c: Likewise.
> > * gcc.target/i386/pr100865-9a.c: Likewise.
> > * gcc.target/i386/pr100865-9b.c: Likewise.
> > * gcc.target/i386/pr100865-9c.c: Likewise.
> > * gcc.target/i386/pr100865-10a.c: Likewise.
> > * gcc.target/i386/pr100865-10b.c: Likewise.
> > * gcc.target/i386/pr100865-11a.c: Likewise.
> > * gcc.target/i386/pr100865-11b.c: Likewise.
> > * gcc.target/i386/pr100865-11c.c: Likewise.
> > * gcc.target/i386/pr100865-12a.c: Likewise.
> > * gcc.target/i386/pr100865-12b.c: Likewise.
> > * gcc.target/i386/pr100865-12c.c: Likewise.
> > ---
> > gcc/config/i386/i386-expand.c | 194 ++++++++++++++++--
> > gcc/config/i386/i386-protos.h | 2 +
> > gcc/config/i386/i386.c | 13 ++
> > .../i386/avx512f-broadcast-pr87767-1.c | 7 +-
> > .../i386/avx512f-broadcast-pr87767-5.c | 5 +-
> > .../gcc.target/i386/avx512f_cond_move.c | 4 +-
> > .../i386/avx512vl-broadcast-pr87767-1.c | 12 +-
> > .../i386/avx512vl-broadcast-pr87767-5.c | 9 +-
> > gcc/testsuite/gcc.target/i386/pr100865-1.c | 13 ++
> > gcc/testsuite/gcc.target/i386/pr100865-10a.c | 33 +++
> > gcc/testsuite/gcc.target/i386/pr100865-10b.c | 7 +
> > gcc/testsuite/gcc.target/i386/pr100865-11a.c | 23 +++
> > gcc/testsuite/gcc.target/i386/pr100865-11b.c | 8 +
> > gcc/testsuite/gcc.target/i386/pr100865-11c.c | 8 +
> > gcc/testsuite/gcc.target/i386/pr100865-12a.c | 20 ++
> > gcc/testsuite/gcc.target/i386/pr100865-12b.c | 8 +
> > gcc/testsuite/gcc.target/i386/pr100865-12c.c | 8 +
> > gcc/testsuite/gcc.target/i386/pr100865-2.c | 14 ++
> > gcc/testsuite/gcc.target/i386/pr100865-3.c | 15 ++
> > gcc/testsuite/gcc.target/i386/pr100865-4a.c | 16 ++
> > gcc/testsuite/gcc.target/i386/pr100865-4b.c | 9 +
> > gcc/testsuite/gcc.target/i386/pr100865-5a.c | 16 ++
> > gcc/testsuite/gcc.target/i386/pr100865-5b.c | 9 +
> > gcc/testsuite/gcc.target/i386/pr100865-6a.c | 16 ++
> > gcc/testsuite/gcc.target/i386/pr100865-6b.c | 9 +
> > gcc/testsuite/gcc.target/i386/pr100865-6c.c | 16 ++
> > gcc/testsuite/gcc.target/i386/pr100865-7a.c | 17 ++
> > gcc/testsuite/gcc.target/i386/pr100865-7b.c | 9 +
> > gcc/testsuite/gcc.target/i386/pr100865-7c.c | 17 ++
> > gcc/testsuite/gcc.target/i386/pr100865-8a.c | 24 +++
> > gcc/testsuite/gcc.target/i386/pr100865-8b.c | 7 +
> > gcc/testsuite/gcc.target/i386/pr100865-8c.c | 7 +
> > gcc/testsuite/gcc.target/i386/pr100865-9a.c | 25 +++
> > gcc/testsuite/gcc.target/i386/pr100865-9b.c | 7 +
> > gcc/testsuite/gcc.target/i386/pr100865-9c.c | 7 +
> > 35 files changed, 590 insertions(+), 24 deletions(-)
> > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-1.c
> > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-10a.c
> > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-10b.c
> > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-11a.c
> > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-11b.c
> > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-11c.c
> > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-12a.c
> > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-12b.c
> > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-12c.c
> > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-2.c
> > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-3.c
> > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-4a.c
> > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-4b.c
> > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-5a.c
> > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-5b.c
> > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-6a.c
> > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-6b.c
> > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-6c.c
> > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-7a.c
> > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-7b.c
> > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-7c.c
> > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-8a.c
> > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-8b.c
> > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-8c.c
> > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-9a.c
> > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-9b.c
> > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-9c.c
> >
> > diff --git a/gcc/config/i386/i386-expand.c b/gcc/config/i386/i386-expand.c
> > index e9763eb5b3e..e0e3ed4d8a4 100644
> > --- a/gcc/config/i386/i386-expand.c
> > +++ b/gcc/config/i386/i386-expand.c
> > @@ -93,6 +93,9 @@ along with GCC; see the file COPYING3. If not see
> > #include "i386-builtins.h"
> > #include "i386-expand.h"
> >
> > +static bool ix86_expand_vector_init_duplicate (bool, machine_mode, rtx,
> > + rtx);
> > +
> > /* Split one or more double-mode RTL references into pairs of half-mode
> > references. The RTL can be REG, offsettable MEM, integer constant, or
> > CONST_DOUBLE. "operands" is a pointer to an array of double-mode RTLs to
> > @@ -190,6 +193,82 @@ ix86_expand_clear (rtx dest)
> > emit_insn (tmp);
> > }
> >
> > +/* Return true if V can be broadcasted from an integer of WIDTH bits
> > + which is returned in VAL_BROADCAST. Otherwise, return false. */
> > +
> > +static bool
> > +ix86_broadcast (HOST_WIDE_INT v, unsigned int width,
> > + HOST_WIDE_INT &val_broadcast)
> > +{
> > + wide_int val = wi::uhwi (v, HOST_BITS_PER_WIDE_INT);
> > + val_broadcast = wi::extract_uhwi (val, 0, width);
> > + for (unsigned int i = width; i < HOST_BITS_PER_WIDE_INT; i += width)
> > + {
> > + HOST_WIDE_INT each = wi::extract_uhwi (val, i, width);
> > + if (val_broadcast != each)
> > + return false;
> > + }
> > + val_broadcast = sext_hwi (val_broadcast, width);
> > + return true;
> > +}
> > +
> > +/* Convert the CONST_WIDE_INT operand OP to broadcast in MODE. */
> > +
> > +static rtx
> > +ix86_convert_const_wide_int_to_broadcast (machine_mode mode, rtx op)
> > +{
> > + /* Don't use integer vector broadcast if we can't move from GPR to SSE
> > + register directly. */
> > + if (!TARGET_INTER_UNIT_MOVES_TO_VEC)
> > + return nullptr;
> > +
> > + /* Convert CONST_WIDE_INT to a non-standard SSE constant integer
> > + broadcast only if vector broadcast is available. */
> > + if (!TARGET_AVX
> > + || !CONST_WIDE_INT_P (op)
> > + || standard_sse_constant_p (op, mode))
> > + return nullptr;
> > +
> > + HOST_WIDE_INT val = CONST_WIDE_INT_ELT (op, 0);
> > + HOST_WIDE_INT val_broadcast;
> > + scalar_int_mode broadcast_mode;
> > + if (TARGET_AVX2
> > + && ix86_broadcast (val, GET_MODE_BITSIZE (QImode),
> > + val_broadcast))
> > + broadcast_mode = QImode;
> > + else if (TARGET_AVX2
> > + && ix86_broadcast (val, GET_MODE_BITSIZE (HImode),
> > + val_broadcast))
> > + broadcast_mode = HImode;
> > + else if (ix86_broadcast (val, GET_MODE_BITSIZE (SImode),
> > + val_broadcast))
> > + broadcast_mode = SImode;
> > + else if (TARGET_64BIT
> > + && ix86_broadcast (val, GET_MODE_BITSIZE (DImode),
> > + val_broadcast))
> > + broadcast_mode = DImode;
> > + else
> > + return nullptr;
> > +
> > + /* Check if OP can be broadcasted from VAL. */
> > + for (int i = 1; i < CONST_WIDE_INT_NUNITS (op); i++)
> > + if (val != CONST_WIDE_INT_ELT (op, i))
> > + return nullptr;
> > +
> > + unsigned int nunits = (GET_MODE_SIZE (mode)
> > + / GET_MODE_SIZE (broadcast_mode));
> > + machine_mode vector_mode;
> > + if (!mode_for_vector (broadcast_mode, nunits).exists (&vector_mode))
> > + gcc_unreachable ();
> > + rtx target = ix86_gen_scratch_sse_rtx (vector_mode);
> > + bool ok = ix86_expand_vector_init_duplicate (false, vector_mode,
> > + target,
> > + GEN_INT (val_broadcast));
> > + gcc_assert (ok);
> > + target = lowpart_subreg (mode, target, vector_mode);
> > + return target;
> > +}
> > +
> > void
> > ix86_expand_move (machine_mode mode, rtx operands[])
> > {
> > @@ -347,20 +426,29 @@ ix86_expand_move (machine_mode mode, rtx operands[])
> > && optimize)
> > op1 = copy_to_mode_reg (mode, op1);
> >
> > - if (can_create_pseudo_p ()
> > - && CONST_DOUBLE_P (op1))
> > + if (can_create_pseudo_p ())
> > {
> > - /* If we are loading a floating point constant to a register,
> > - force the value to memory now, since we'll get better code
> > - out the back end. */
> > + if (CONST_DOUBLE_P (op1))
> > + {
> > + /* If we are loading a floating point constant to a
> > + register, force the value to memory now, since we'll
> > + get better code out the back end. */
> >
> > - op1 = validize_mem (force_const_mem (mode, op1));
> > - if (!register_operand (op0, mode))
> > + op1 = validize_mem (force_const_mem (mode, op1));
> > + if (!register_operand (op0, mode))
> > + {
> > + rtx temp = gen_reg_rtx (mode);
> > + emit_insn (gen_rtx_SET (temp, op1));
> > + emit_move_insn (op0, temp);
> > + return;
> > + }
> > + }
> > + else if (GET_MODE_SIZE (mode) >= 16)
> > {
> > - rtx temp = gen_reg_rtx (mode);
> > - emit_insn (gen_rtx_SET (temp, op1));
> > - emit_move_insn (op0, temp);
> > - return;
> > + rtx tmp = ix86_convert_const_wide_int_to_broadcast
> > + (GET_MODE (op0), op1);
> > + if (tmp != nullptr)
> > + op1 = tmp;
> > }
> > }
> > }
> > @@ -368,6 +456,59 @@ ix86_expand_move (machine_mode mode, rtx operands[])
> > emit_insn (gen_rtx_SET (op0, op1));
> > }
> >
> > +static rtx
> > +ix86_broadcast_from_integer_constant (machine_mode mode, rtx op)
> > +{
> > + int nunits = GET_MODE_NUNITS (mode);
> > + if (nunits < 2)
> > + return nullptr;
> > +
> > + /* Don't use integer vector broadcast if we can't move from GPR to SSE
> > + register directly. */
> > + if (!TARGET_INTER_UNIT_MOVES_TO_VEC)
> > + return nullptr;
> > +
> > + /* Convert CONST_VECTOR to a non-standard SSE constant integer
> > + broadcast only if vector broadcast is available. */
> > + if (!(TARGET_AVX2
> > + || (TARGET_AVX
> > + && (GET_MODE_INNER (mode) == SImode
> > + || GET_MODE_INNER (mode) == DImode)))
> > + || standard_sse_constant_p (op, mode))
> > + return nullptr;
> > +
> > + /* Don't broadcast from a 64-bit integer constant in 32-bit mode. */
> > + if (GET_MODE_INNER (mode) == DImode && !TARGET_64BIT)
> > + return nullptr;
> > +
> > + rtx constant = get_pool_constant (XEXP (op, 0));
> > + if (GET_CODE (constant) != CONST_VECTOR)
> > + return nullptr;
> > +
> > + /* There could be some rtx like
> > + (mem/u/c:V16QI (symbol_ref/u:DI ("*.LC1")))
> > + but with "*.LC1" refer to V2DI constant vector. */
> > + if (GET_MODE (constant) != mode)
> > + {
> > + constant = simplify_subreg (mode, constant, GET_MODE (constant),
> > + 0);
> > + if (constant == nullptr || GET_CODE (constant) != CONST_VECTOR)
> > + return nullptr;
> > + }
> > +
> > + rtx first = XVECEXP (constant, 0, 0);
> > +
> > + for (int i = 1; i < nunits; ++i)
> > + {
> > + rtx tmp = XVECEXP (constant, 0, i);
> > + /* Vector duplicate value. */
> > + if (!rtx_equal_p (tmp, first))
> > + return nullptr;
> > + }
> > +
> > + return first;
> > +}
> > +
> > void
> > ix86_expand_vector_move (machine_mode mode, rtx operands[])
> > {
> > @@ -407,7 +548,36 @@ ix86_expand_vector_move (machine_mode mode, rtx operands[])
> > op1 = simplify_gen_subreg (mode, r, imode, SUBREG_BYTE (op1));
> > }
> > else
> > - op1 = validize_mem (force_const_mem (mode, op1));
> > + {
> > + machine_mode mode = GET_MODE (op0);
> > + rtx tmp = ix86_convert_const_wide_int_to_broadcast
> > + (mode, op1);
> > + if (tmp == nullptr)
> > + op1 = validize_mem (force_const_mem (mode, op1));
> > + else
> > + op1 = tmp;
> > + }
> > + }
> > +
> > + if (can_create_pseudo_p ()
> > + && GET_MODE_SIZE (mode) >= 16
> > + && GET_MODE_CLASS (mode) == MODE_VECTOR_INT
> > + && (MEM_P (op1)
> > + && SYMBOL_REF_P (XEXP (op1, 0))
> > + && CONSTANT_POOL_ADDRESS_P (XEXP (op1, 0))))
> > + {
> > + rtx first = ix86_broadcast_from_integer_constant (mode, op1);
> > + if (first != nullptr)
> > + {
> > + /* Broadcast to XMM/YMM/ZMM register from an integer
> > + constant. */
> > + op1 = ix86_gen_scratch_sse_rtx (mode);
> > + bool ok = ix86_expand_vector_init_duplicate (false, mode,
> > + op1, first);
> > + gcc_assert (ok);
> > + emit_move_insn (op0, op1);
> > + return;
> > + }
> > }
> >
> > /* We need to check memory alignment for SSE mode since attribute
> > diff --git a/gcc/config/i386/i386-protos.h b/gcc/config/i386/i386-protos.h
> > index 65fc307dc7b..71745b9a1ea 100644
> > --- a/gcc/config/i386/i386-protos.h
> > +++ b/gcc/config/i386/i386-protos.h
> > @@ -50,6 +50,8 @@ extern void ix86_reset_previous_fndecl (void);
> >
> > extern bool ix86_using_red_zone (void);
> >
> > +extern rtx ix86_gen_scratch_sse_rtx (machine_mode);
> > +
> > extern unsigned int ix86_regmode_natural_size (machine_mode);
> > #ifdef RTX_CODE
> > extern int standard_80387_constant_p (rtx);
> > diff --git a/gcc/config/i386/i386.c b/gcc/config/i386/i386.c
> > index a93128fa0a4..2fbaae7cd02 100644
> > --- a/gcc/config/i386/i386.c
> > +++ b/gcc/config/i386/i386.c
> > @@ -23163,6 +23163,19 @@ ix86_optab_supported_p (int op, machine_mode mode1, machine_mode,
> > }
> > }
> >
> > +/* Return a scratch register in MODE for vector load and store. */
> > +
> > +rtx
> > +ix86_gen_scratch_sse_rtx (machine_mode mode)
> > +{
> > + if (TARGET_SSE)
> > + return gen_rtx_REG (mode, (TARGET_64BIT
> > + ? LAST_REX_SSE_REG
> > + : LAST_SSE_REG));
> > + else
> > + return gen_reg_rtx (mode);
> > +}
> > +
> > /* Address space support.
> >
> > This is not "far pointers" in the 16-bit sense, but an easy way
> > diff --git a/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-1.c b/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-1.c
> > index 0563e696316..a2664d87f29 100644
> > --- a/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-1.c
> > +++ b/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-1.c
> > @@ -2,8 +2,11 @@
> > /* { dg-do compile } */
> > /* { dg-options "-O2 -mavx512f -mavx512dq" } */
> > /* { dg-additional-options "-mdynamic-no-pic" { target { *-*-darwin* && ia32 } } }
> > -/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to8\\\}" 5 } } */
> > -/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to16\\\}" 5 } } */
> > +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to8\\\}" 2 { target { ! ia32 } } } } */
> > +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to8\\\}" 5 { target ia32 } } } */
> > +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to16\\\}" 2 } } */
> > +/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %zmm\[0-9\]+" 3 } } */
> > +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%r\[^\n\]*, %zmm\[0-9\]+" 3 { target { ! ia32 } } } } */
> >
> > typedef int v16si __attribute__ ((vector_size (64)));
> > typedef long long v8di __attribute__ ((vector_size (64)));
> > diff --git a/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-5.c b/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-5.c
> > index ffbe95980ca..477f9ca1282 100644
> > --- a/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-5.c
> > +++ b/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-5.c
> > @@ -2,8 +2,9 @@
> > /* { dg-do compile } */
> > /* { dg-options "-O2 -mavx512f" } */
> > /* { dg-additional-options "-mdynamic-no-pic" { target { *-*-darwin* && ia32 } } }
> > -/* { dg-final { scan-assembler-times "\[^n\n\]*\\\{1to8\\\}" 4 } } */
> > -/* { dg-final { scan-assembler-times "\[^n\n\]*\\\{1to16\\\}" 4 } } */
> > +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to8\\\}" 4 { target ia32 } } } */
> > +/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %zmm\[0-9\]+" 4 } } */
> > +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%r\[^\n\]*, %zmm\[0-9\]+" 4 { target { ! ia32 } } } } */
> >
> > typedef int v16si __attribute__ ((vector_size (64)));
> > typedef long long v8di __attribute__ ((vector_size (64)));
> > diff --git a/gcc/testsuite/gcc.target/i386/avx512f_cond_move.c b/gcc/testsuite/gcc.target/i386/avx512f_cond_move.c
> > index 99a89f51202..ca49a585232 100644
> > --- a/gcc/testsuite/gcc.target/i386/avx512f_cond_move.c
> > +++ b/gcc/testsuite/gcc.target/i386/avx512f_cond_move.c
> > @@ -1,6 +1,6 @@
> > /* { dg-do compile } */
> > -/* { dg-options "-O3 -mavx512f" } */
> > -/* { dg-final { scan-assembler-times "(?:vpblendmd|vmovdqa32)\[ \\t\]+\[^\{\n\]*%zmm\[0-9\]+\{%k\[1-7\]\}\{z\}(?:\n|\[ \\t\]+#)" 8 } } */
> > +/* { dg-options "-O3 -mavx512f -mprefer-vector-width=512" } */
> > +/* { dg-final { scan-assembler-times "(?:vpbroadcastd|vmovdqa32)\[ \\t\]+\[^\{\n\]*%zmm\[0-9\]+\{%k\[1-7\]\}\{z\}(?:\n|\[ \\t\]+#)" 8 } } */
> >
> > unsigned int x[128];
> > int y[128];
> > diff --git a/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-1.c b/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-1.c
> > index c06369d93fd..f8eb99f0b5f 100644
> > --- a/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-1.c
> > +++ b/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-1.c
> > @@ -2,9 +2,15 @@
> > /* { dg-do compile } */
> > /* { dg-options "-O2 -mavx512f -mavx512vl -mavx512dq" } */
> > /* { dg-additional-options "-mdynamic-no-pic" { target { *-*-darwin* && ia32 } } }
> > -/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to2\\\}" 5 } } */
> > -/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to4\\\}" 10 } } */
> > -/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to8\\\}" 5 } } */
> > +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to2\\\}" 2 { target { ! ia32 } } } } */
> > +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to4\\\}" 4 { target { ! ia32 } } } } */
> > +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to2\\\}" 5 { target ia32 } } } */
> > +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to4\\\}" 7 { target ia32 } } } */
> > +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to8\\\}" 2 } } */
> > +/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 3 } } */
> > +/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %ymm\[0-9\]+" 3 } } */
> > +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%r\[^\n\]*, %xmm\[0-9\]+" 3 { target { ! ia32 } } } } */
> > +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%r\[^\n\]*, %ymm\[0-9\]+" 3 { target { ! ia32 } } } } */
> >
> > typedef int v4si __attribute__ ((vector_size (16)));
> > typedef int v8si __attribute__ ((vector_size (32)));
> > diff --git a/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-5.c b/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-5.c
> > index 4998a9b8d51..32f6ac81841 100644
> > --- a/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-5.c
> > +++ b/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-5.c
> > @@ -2,9 +2,12 @@
> > /* { dg-do compile } */
> > /* { dg-options "-O2 -mavx512f -mavx512vl" } */
> > /* { dg-additional-options "-mdynamic-no-pic" { target { *-*-darwin* && ia32 } } }
> > -/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to2\\\}" 4 } } */
> > -/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to4\\\}" 8 } } */
> > -/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to8\\\}" 4 } } */
> > +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to2\\\}" 4 { target ia32 } } } */
> > +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to4\\\}" 4 { target ia32 } } } */
> > +/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 4 } } */
> > +/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %ymm\[0-9\]+" 4 } } */
> > +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%r\[^\n\]*, %xmm\[0-9\]+" 4 { target { ! ia32 } } } } */
> > +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%r\[^\n\]*, %ymm\[0-9\]+" 4 { target { ! ia32 } } } } */
> >
> > typedef int v4si __attribute__ ((vector_size (16)));
> > typedef int v8si __attribute__ ((vector_size (32)));
> > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-1.c b/gcc/testsuite/gcc.target/i386/pr100865-1.c
> > new file mode 100644
> > index 00000000000..6c3097fb2a6
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/i386/pr100865-1.c
> > @@ -0,0 +1,13 @@
> > +/* { dg-do compile { target { ! ia32 } } } */
> > +/* { dg-options "-O2 -march=x86-64" } */
> > +
> > +extern char *dst;
> > +
> > +void
> > +foo (void)
> > +{
> > + __builtin_memset (dst, 3, 16);
> > +}
> > +
> > +/* { dg-final { scan-assembler-times "movdqa\[ \\t\]+\[^\n\]*%xmm" 1 } } */
> > +/* { dg-final { scan-assembler-times "movups\[\\t \]%xmm\[0-9\]+, \\(%\[\^,\]+\\)" 1 } } */
> > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-10a.c b/gcc/testsuite/gcc.target/i386/pr100865-10a.c
> > new file mode 100644
> > index 00000000000..7ffc19e56a8
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/i386/pr100865-10a.c
> > @@ -0,0 +1,33 @@
> > +/* { dg-do compile { target int128 } } */
> > +/* { dg-options "-O3 -march=skylake" } */
> > +
> > +extern __int128 array[16];
> > +
> > +#define MK_CONST128_BROADCAST(A) \
> > + ((((unsigned __int128) (unsigned char) A) << 120) \
> > + | (((unsigned __int128) (unsigned char) A) << 112) \
> > + | (((unsigned __int128) (unsigned char) A) << 104) \
> > + | (((unsigned __int128) (unsigned char) A) << 96) \
> > + | (((unsigned __int128) (unsigned char) A) << 88) \
> > + | (((unsigned __int128) (unsigned char) A) << 80) \
> > + | (((unsigned __int128) (unsigned char) A) << 72) \
> > + | (((unsigned __int128) (unsigned char) A) << 64) \
> > + | (((unsigned __int128) (unsigned char) A) << 56) \
> > + | (((unsigned __int128) (unsigned char) A) << 48) \
> > + | (((unsigned __int128) (unsigned char) A) << 40) \
> > + | (((unsigned __int128) (unsigned char) A) << 32) \
> > + | (((unsigned __int128) (unsigned char) A) << 24) \
> > + | (((unsigned __int128) (unsigned char) A) << 16) \
> > + | (((unsigned __int128) (unsigned char) A) << 8) \
> > + | ((unsigned __int128) (unsigned char) A) )
> > +
> > +void
> > +foo (void)
> > +{
> > + int i;
> > + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> > + array[i] = MK_CONST128_BROADCAST (0x1f);
> > +}
> > +
> > +/* { dg-final { scan-assembler-times "vpbroadcastb\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> > +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-10b.c b/gcc/testsuite/gcc.target/i386/pr100865-10b.c
> > new file mode 100644
> > index 00000000000..edf52765c60
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/i386/pr100865-10b.c
> > @@ -0,0 +1,7 @@
> > +/* { dg-do compile { target int128 } } */
> > +/* { dg-options "-O3 -march=skylake-avx512" } */
> > +
> > +#include "pr100865-10a.c"
> > +
> > +/* { dg-final { scan-assembler-times "vpbroadcastb\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> > +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-11a.c b/gcc/testsuite/gcc.target/i386/pr100865-11a.c
> > new file mode 100644
> > index 00000000000..04ce1662f3c
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/i386/pr100865-11a.c
> > @@ -0,0 +1,23 @@
> > +/* { dg-do compile { target int128 } } */
> > +/* { dg-options "-O3 -march=skylake" } */
> > +
> > +extern __int128 array[16];
> > +
> > +#define MK_CONST128_BROADCAST(A) \
> > + ((((unsigned __int128) (unsigned long long) A) << 64) \
> > + | ((unsigned __int128) (unsigned long long) A) )
> > +
> > +#define MK_CONST128_BROADCAST_SIGNED(A) \
> > + ((__int128) MK_CONST128_BROADCAST (A))
> > +
> > +void
> > +foo (void)
> > +{
> > + int i;
> > + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> > + array[i] = MK_CONST128_BROADCAST_SIGNED (-0x1ffffffffLL);
> > +}
> > +
> > +/* { dg-final { scan-assembler-times "movabsq" 1 } } */
> > +/* { dg-final { scan-assembler-times "(?:vpbroadcastq|vpunpcklqdq)\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> > +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-11b.c b/gcc/testsuite/gcc.target/i386/pr100865-11b.c
> > new file mode 100644
> > index 00000000000..12d55b9a642
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/i386/pr100865-11b.c
> > @@ -0,0 +1,8 @@
> > +/* { dg-do compile { target int128 } } */
> > +/* { dg-options "-O3 -march=skylake-avx512" } */
> > +
> > +#include "pr100865-11a.c"
> > +
> > +/* { dg-final { scan-assembler-times "movabsq" 1 } } */
> > +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> > +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-11c.c b/gcc/testsuite/gcc.target/i386/pr100865-11c.c
> > new file mode 100644
> > index 00000000000..de56c84b9ca
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/i386/pr100865-11c.c
> > @@ -0,0 +1,8 @@
> > +/* { dg-do compile { target int128 } } */
> > +/* { dg-options "-O3 -march=skylake -mno-avx2" } */
> > +
> > +#include "pr100865-11a.c"
> > +
> > +/* { dg-final { scan-assembler-times "movabsq" 1 } } */
> > +/* { dg-final { scan-assembler-times "vpunpcklqdq\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> > +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-12a.c b/gcc/testsuite/gcc.target/i386/pr100865-12a.c
> > new file mode 100644
> > index 00000000000..d4833d44475
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/i386/pr100865-12a.c
> > @@ -0,0 +1,20 @@
> > +/* { dg-do compile { target int128 } } */
> > +/* { dg-options "-O3 -march=skylake" } */
> > +
> > +extern __int128 array[16];
> > +
> > +#define MK_CONST128_BROADCAST(A) \
> > + ((((unsigned __int128) (unsigned long long) A) << 64) \
> > + | ((unsigned __int128) (unsigned long long) A) )
> > +
> > +void
> > +foo (void)
> > +{
> > + int i;
> > + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> > + array[i] = MK_CONST128_BROADCAST (0x1ffffffffLL);
> > +}
> > +
> > +/* { dg-final { scan-assembler-times "movabsq" 1 } } */
> > +/* { dg-final { scan-assembler-times "(?:vpbroadcastq|vpunpcklqdq)\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> > +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-12b.c b/gcc/testsuite/gcc.target/i386/pr100865-12b.c
> > new file mode 100644
> > index 00000000000..63a5629b90c
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/i386/pr100865-12b.c
> > @@ -0,0 +1,8 @@
> > +/* { dg-do compile { target int128 } } */
> > +/* { dg-options "-O3 -march=skylake-avx512" } */
> > +
> > +#include "pr100865-12a.c"
> > +
> > +/* { dg-final { scan-assembler-times "movabsq" 1 } } */
> > +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> > +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-12c.c b/gcc/testsuite/gcc.target/i386/pr100865-12c.c
> > new file mode 100644
> > index 00000000000..77415f22c97
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/i386/pr100865-12c.c
> > @@ -0,0 +1,8 @@
> > +/* { dg-do compile { target int128 } } */
> > +/* { dg-options "-O3 -march=skylake -mno-avx2" } */
> > +
> > +#include "pr100865-12a.c"
> > +
> > +/* { dg-final { scan-assembler-times "movabsq" 1 } } */
> > +/* { dg-final { scan-assembler-times "vpunpcklqdq\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> > +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-2.c b/gcc/testsuite/gcc.target/i386/pr100865-2.c
> > new file mode 100644
> > index 00000000000..17efe2d72a3
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/i386/pr100865-2.c
> > @@ -0,0 +1,14 @@
> > +/* { dg-do compile { target { ! ia32 } } } */
> > +/* { dg-options "-O2 -march=skylake" } */
> > +
> > +extern char *dst;
> > +
> > +void
> > +foo (void)
> > +{
> > + __builtin_memset (dst, 3, 16);
> > +}
> > +
> > +/* { dg-final { scan-assembler-times "vpbroadcastb\[\\t \]+%xmm\[0-9\]+, %xmm\[0-9\]+" 1 } } */
> > +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%xmm\[0-9\]+, \\(%\[\^,\]+\\)" 1 } } */
> > +/* { dg-final { scan-assembler-not "vmovdqa" } } */
> > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-3.c b/gcc/testsuite/gcc.target/i386/pr100865-3.c
> > new file mode 100644
> > index 00000000000..b6dbcf7809b
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/i386/pr100865-3.c
> > @@ -0,0 +1,15 @@
> > +/* { dg-do compile { target { ! ia32 } } } */
> > +/* { dg-options "-O2 -march=skylake-avx512" } */
> > +
> > +extern char *dst;
> > +
> > +void
> > +foo (void)
> > +{
> > + __builtin_memset (dst, 3, 16);
> > +}
> > +
> > +/* { dg-final { scan-assembler-times "vpbroadcastb\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> > +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%xmm\[0-9\]+, \\(%\[\^,\]+\\)" 1 } } */
> > +/* { dg-final { scan-assembler-not "vpbroadcastb\[\\t \]+%xmm\[0-9\]+, %xmm\[0-9\]+" } } */
> > +/* { dg-final { scan-assembler-not "vmovdqa" } } */
> > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-4a.c b/gcc/testsuite/gcc.target/i386/pr100865-4a.c
> > new file mode 100644
> > index 00000000000..f55883598f9
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/i386/pr100865-4a.c
> > @@ -0,0 +1,16 @@
> > +/* { dg-do compile { target { ! ia32 } } } */
> > +/* { dg-options "-O2 -march=skylake" } */
> > +
> > +extern char array[64];
> > +
> > +void
> > +foo (void)
> > +{
> > + int i;
> > + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> > + array[i] = -45;
> > +}
> > +
> > +/* { dg-final { scan-assembler-times "vpbroadcastb\[\\t \]+%xmm\[0-9\]+, %xmm\[0-9\]+" 1 } } */
> > +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%xmm\[0-9\]+, " 4 } } */
> > +/* { dg-final { scan-assembler-not "vmovdqa" } } */
> > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-4b.c b/gcc/testsuite/gcc.target/i386/pr100865-4b.c
> > new file mode 100644
> > index 00000000000..f41e6147b4c
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/i386/pr100865-4b.c
> > @@ -0,0 +1,9 @@
> > +/* { dg-do compile { target { ! ia32 } } } */
> > +/* { dg-options "-O2 -march=skylake-avx512" } */
> > +
> > +#include "pr100865-4a.c"
> > +
> > +/* { dg-final { scan-assembler-times "vpbroadcastb\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> > +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%xmm\[0-9\]+, " 4 } } */
> > +/* { dg-final { scan-assembler-not "vpbroadcastb\[\\t \]+%xmm\[0-9\]+, %xmm\[0-9\]+" } } */
> > +/* { dg-final { scan-assembler-not "vmovdqa" } } */
> > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-5a.c b/gcc/testsuite/gcc.target/i386/pr100865-5a.c
> > new file mode 100644
> > index 00000000000..4149797fe81
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/i386/pr100865-5a.c
> > @@ -0,0 +1,16 @@
> > +/* { dg-do compile } */
> > +/* { dg-options "-O3 -march=skylake" } */
> > +
> > +extern short array[64];
> > +
> > +void
> > +foo (void)
> > +{
> > + int i;
> > + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> > + array[i] = -45;
> > +}
> > +
> > +/* { dg-final { scan-assembler-times "vpbroadcastw\[\\t \]+%xmm\[0-9\]+, %ymm\[0-9\]+" 1 } } */
> > +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%ymm\[0-9\]+, " 4 } } */
> > +/* { dg-final { scan-assembler-not "vmovdqa" } } */
> > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-5b.c b/gcc/testsuite/gcc.target/i386/pr100865-5b.c
> > new file mode 100644
> > index 00000000000..ded41b680d3
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/i386/pr100865-5b.c
> > @@ -0,0 +1,9 @@
> > +/* { dg-do compile } */
> > +/* { dg-options "-O3 -march=skylake-avx512" } */
> > +
> > +#include "pr100865-5a.c"
> > +
> > +/* { dg-final { scan-assembler-times "vpbroadcastw\[\\t \]+%(?:r|e)\[^\n\]*, %ymm\[0-9\]+" 1 } } */
> > +/* { dg-final { scan-assembler-times "vmovdqu16\[\\t \]%ymm\[0-9\]+, " 4 } } */
> > +/* { dg-final { scan-assembler-not "vpbroadcastw\[\\t \]+%xmm\[0-9\]+, %ymm\[0-9\]+" } } */
> > +/* { dg-final { scan-assembler-not "vmovdqa" } } */
> > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-6a.c b/gcc/testsuite/gcc.target/i386/pr100865-6a.c
> > new file mode 100644
> > index 00000000000..3fde549a10d
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/i386/pr100865-6a.c
> > @@ -0,0 +1,16 @@
> > +/* { dg-do compile } */
> > +/* { dg-options "-O3 -march=skylake" } */
> > +
> > +extern int array[64];
> > +
> > +void
> > +foo (void)
> > +{
> > + int i;
> > + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> > + array[i] = -45;
> > +}
> > +
> > +/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%xmm\[0-9\]+, %ymm\[0-9\]+" 1 } } */
> > +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%ymm\[0-9\]+, " 8 } } */
> > +/* { dg-final { scan-assembler-not "vmovdqa" } } */
> > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-6b.c b/gcc/testsuite/gcc.target/i386/pr100865-6b.c
> > new file mode 100644
> > index 00000000000..44e74c64e55
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/i386/pr100865-6b.c
> > @@ -0,0 +1,9 @@
> > +/* { dg-do compile } */
> > +/* { dg-options "-O3 -march=skylake-avx512" } */
> > +
> > +#include "pr100865-6a.c"
> > +
> > +/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %ymm\[0-9\]+" 1 } } */
> > +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%ymm\[0-9\]+, " 8 } } */
> > +/* { dg-final { scan-assembler-not "vpbroadcastd\[\\t \]+%xmm\[0-9\]+, %ymm\[0-9\]+" } } */
> > +/* { dg-final { scan-assembler-not "vmovdqa" } } */
> > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-6c.c b/gcc/testsuite/gcc.target/i386/pr100865-6c.c
> > new file mode 100644
> > index 00000000000..46d31030ce8
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/i386/pr100865-6c.c
> > @@ -0,0 +1,16 @@
> > +/* { dg-do compile } */
> > +/* { dg-options "-O3 -march=skylake -mno-avx2" } */
> > +
> > +extern int array[64];
> > +
> > +void
> > +foo (void)
> > +{
> > + int i;
> > + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> > + array[i] = -45;
> > +}
> > +
> > +/* { dg-final { scan-assembler-times "vbroadcastss" 1 } } */
> > +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%ymm\[0-9\]+, " 8 } } */
> > +/* { dg-final { scan-assembler-not "vmovdqa" } } */
> > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-7a.c b/gcc/testsuite/gcc.target/i386/pr100865-7a.c
> > new file mode 100644
> > index 00000000000..f6f2be91120
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/i386/pr100865-7a.c
> > @@ -0,0 +1,17 @@
> > +/* { dg-do compile } */
> > +/* { dg-options "-O3 -march=skylake" } */
> > +
> > +extern long long int array[64];
> > +
> > +void
> > +foo (void)
> > +{
> > + int i;
> > + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> > + array[i] = -45;
> > +}
> > +
> > +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+\[^\n\]*, %ymm\[0-9\]+" 1 { target { ! ia32 } } } } */
> > +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%ymm\[0-9\]+, " 16 } } */
> > +/* { dg-final { scan-assembler-not "vpbroadcastq" { target ia32 } } } */
> > +/* { dg-final { scan-assembler-not "vmovdqa" { target { ! ia32 } } } } */
> > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-7b.c b/gcc/testsuite/gcc.target/i386/pr100865-7b.c
> > new file mode 100644
> > index 00000000000..0a68820aa32
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/i386/pr100865-7b.c
> > @@ -0,0 +1,9 @@
> > +/* { dg-do compile } */
> > +/* { dg-options "-O3 -march=skylake-avx512" } */
> > +
> > +#include "pr100865-7a.c"
> > +
> > +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%r\[^\n\]*, %ymm\[0-9\]+" 1 { target { ! ia32 } } } } */
> > +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+\[^\n\]*, %ymm\[0-9\]+" 1 { target ia32 } } } */
> > +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%ymm\[0-9\]+, " 16 } } */
> > +/* { dg-final { scan-assembler-not "vmovdqa" } } */
> > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-7c.c b/gcc/testsuite/gcc.target/i386/pr100865-7c.c
> > new file mode 100644
> > index 00000000000..4d50bb7e2f6
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/i386/pr100865-7c.c
> > @@ -0,0 +1,17 @@
> > +/* { dg-do compile } */
> > +/* { dg-options "-O3 -march=skylake -mno-avx2" } */
> > +
> > +extern long long int array[64];
> > +
> > +void
> > +foo (void)
> > +{
> > + int i;
> > + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> > + array[i] = -45;
> > +}
> > +
> > +/* { dg-final { scan-assembler-times "vbroadcastsd" 1 { target { ! ia32 } } } } */
> > +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%ymm\[0-9\]+, " 16 } } */
> > +/* { dg-final { scan-assembler-not "vbroadcastsd" { target ia32 } } } */
> > +/* { dg-final { scan-assembler-not "vmovdqa" { target { ! ia32 } } } } */
> > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-8a.c b/gcc/testsuite/gcc.target/i386/pr100865-8a.c
> > new file mode 100644
> > index 00000000000..544a14db6f7
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/i386/pr100865-8a.c
> > @@ -0,0 +1,24 @@
> > +/* { dg-do compile { target int128 } } */
> > +/* { dg-options "-O3 -march=skylake" } */
> > +
> > +extern __int128 array[16];
> > +
> > +#define MK_CONST128_BROADCAST(A) \
> > + ((((unsigned __int128) (unsigned int) A) << 96) \
> > + | (((unsigned __int128) (unsigned int) A) << 64) \
> > + | (((unsigned __int128) (unsigned int) A) << 32) \
> > + | ((unsigned __int128) (unsigned int) A) )
> > +
> > +#define MK_CONST128_BROADCAST_SIGNED(A) \
> > + ((__int128) MK_CONST128_BROADCAST (A))
> > +
> > +void
> > +foo (void)
> > +{
> > + int i;
> > + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> > + array[i] = MK_CONST128_BROADCAST_SIGNED (-45);
> > +}
> > +
> > +/* { dg-final { scan-assembler-times "(?:vpbroadcastd|vpshufd)\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> > +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-8b.c b/gcc/testsuite/gcc.target/i386/pr100865-8b.c
> > new file mode 100644
> > index 00000000000..99a10ad83bd
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/i386/pr100865-8b.c
> > @@ -0,0 +1,7 @@
> > +/* { dg-do compile { target int128 } } */
> > +/* { dg-options "-O3 -march=skylake-avx512" } */
> > +
> > +#include "pr100865-8a.c"
> > +
> > +/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> > +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-8c.c b/gcc/testsuite/gcc.target/i386/pr100865-8c.c
> > new file mode 100644
> > index 00000000000..efee0488614
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/i386/pr100865-8c.c
> > @@ -0,0 +1,7 @@
> > +/* { dg-do compile { target int128 } } */
> > +/* { dg-options "-O3 -march=skylake -mno-avx2" } */
> > +
> > +#include "pr100865-8a.c"
> > +
> > +/* { dg-final { scan-assembler-times "vpshufd\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> > +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-9a.c b/gcc/testsuite/gcc.target/i386/pr100865-9a.c
> > new file mode 100644
> > index 00000000000..45d0e0d0e2e
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/i386/pr100865-9a.c
> > @@ -0,0 +1,25 @@
> > +/* { dg-do compile { target int128 } } */
> > +/* { dg-options "-O3 -march=skylake" } */
> > +
> > +extern __int128 array[16];
> > +
> > +#define MK_CONST128_BROADCAST(A) \
> > + ((((unsigned __int128) (unsigned short) A) << 112) \
> > + | (((unsigned __int128) (unsigned short) A) << 96) \
> > + | (((unsigned __int128) (unsigned short) A) << 80) \
> > + | (((unsigned __int128) (unsigned short) A) << 64) \
> > + | (((unsigned __int128) (unsigned short) A) << 48) \
> > + | (((unsigned __int128) (unsigned short) A) << 32) \
> > + | (((unsigned __int128) (unsigned short) A) << 16) \
> > + | ((unsigned __int128) (unsigned short) A) )
> > +
> > +void
> > +foo (void)
> > +{
> > + int i;
> > + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> > + array[i] = MK_CONST128_BROADCAST (0x1fff);
> > +}
> > +
> > +/* { dg-final { scan-assembler-times "vpbroadcastw\[\\t \]+%xmm\[0-9\]+, %xmm\[0-9\]+" 1 } } */
> > +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-9b.c b/gcc/testsuite/gcc.target/i386/pr100865-9b.c
> > new file mode 100644
> > index 00000000000..14696248525
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/i386/pr100865-9b.c
> > @@ -0,0 +1,7 @@
> > +/* { dg-do compile { target int128 } } */
> > +/* { dg-options "-O3 -march=skylake-avx512" } */
> > +
> > +#include "pr100865-9a.c"
> > +
> > +/* { dg-final { scan-assembler-times "vpbroadcastw\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> > +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-9c.c b/gcc/testsuite/gcc.target/i386/pr100865-9c.c
> > new file mode 100644
> > index 00000000000..e6f25902c1d
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.target/i386/pr100865-9c.c
> > @@ -0,0 +1,7 @@
> > +/* { dg-do compile { target int128 } } */
> > +/* { dg-options "-O3 -march=skylake -mno-avx2" } */
> > +
> > +#include "pr100865-9a.c"
> > +
> > +/* { dg-final { scan-assembler-times "vpshufd\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> > +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> > --
> > 2.31.1
> >
>
>
> --
> BR,
> Hongtao
--
H.J.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v6 1/2] x86: Convert CONST_WIDE_INT/CONST_VECTOR to broadcast
2021-07-01 12:41 ` H.J. Lu
@ 2021-07-01 14:12 ` Uros Bizjak
0 siblings, 0 replies; 7+ messages in thread
From: Uros Bizjak @ 2021-07-01 14:12 UTC (permalink / raw)
To: H.J. Lu
Cc: Hongtao Liu, GCC Patches, Jakub Jelinek, Richard Sandiford,
Richard Biener
On Thu, Jul 1, 2021 at 2:42 PM H.J. Lu <hjl.tools@gmail.com> wrote:
>
> Hi Uros,
>
> On Thu, Jul 1, 2021 at 1:32 AM Hongtao Liu <crazylht@gmail.com> wrote:
> >
> > On Tue, Jun 29, 2021 at 6:16 AM H.J. Lu <hjl.tools@gmail.com> wrote:
> > >
> > > 1. Update move expanders to convert the CONST_WIDE_INT and CONST_VECTOR
> > > operands to vector broadcast from an integer with AVX.
> > > 2. Add ix86_gen_scratch_sse_rtx to return a scratch SSE register which
> > > won't increase stack alignment requirement and blocks transformation by
> > > the combine pass.
> > >
> > > A small benchmark:
> > >
> > > https://gitlab.com/x86-benchmarks/microbenchmark/-/tree/memset/broadcast
> > >
> > > shows that broadcast is a little bit faster on Intel Core i7-8559U:
> > >
> > > $ make
> > > gcc -g -I. -O2 -c -o test.o test.c
> > > gcc -g -c -o memory.o memory.S
> > > gcc -g -c -o broadcast.o broadcast.S
> > > gcc -g -c -o vec_dup_sse2.o vec_dup_sse2.S
> > > gcc -o test test.o memory.o broadcast.o vec_dup_sse2.o
> > > ./test
> > > memory : 147215
> > > broadcast : 121213
> > > vec_dup_sse2: 171366
> > > $
> > >
> > > broadcast is also smaller:
> > >
> > > $ size memory.o broadcast.o
> > > text data bss dec hex filename
> > > 132 0 0 132 84 memory.o
> > > 122 0 0 122 7a broadcast.o
> > > $
> > >
> > > 3. Update PR 87767 tests to expect integer broadcast instead of broadcast
> > > from memory.
> > > 4. Update avx512f_cond_move.c to expect integer broadcast.
> > >
> > > A small benchmark:
> > >
> > > https://gitlab.com/x86-benchmarks/microbenchmark/-/tree/vpaddd/broadcast
> > >
> > > shows that integer broadcast is faster than embedded memory broadcast:
> > >
> > > $ make
> > > gcc -g -I. -O2 -march=skylake-avx512 -c -o test.o test.c
> > > gcc -g -c -o memory.o memory.S
> > > gcc -g -c -o broadcast.o broadcast.S
> > > gcc -o test test.o memory.o broadcast.o
> > > ./test
> > > memory : 425538
> > > broadcast : 375260
> > > $
> > >
> > > gcc/
> > >
> > > PR target/100865
> > > * config/i386/i386-expand.c (ix86_expand_vector_init_duplicate):
> > > New prototype.
> > > (ix86_byte_broadcast): New function.
> > > (ix86_convert_const_wide_int_to_broadcast): Likewise.
> > > (ix86_expand_move): Convert CONST_WIDE_INT to broadcast if mode
> > > size is 16 bytes or bigger.
> > > (ix86_broadcast_from_integer_constant): New function.
> > > (ix86_expand_vector_move): Convert CONST_WIDE_INT and CONST_VECTOR
> > > to broadcast if mode size is 16 bytes or bigger.
> > > * config/i386/i386-protos.h (ix86_gen_scratch_sse_rtx): New
> > > prototype.
> > > * config/i386/i386.c (ix86_gen_scratch_sse_rtx): New function.
> > >
> > > gcc/testsuite/
> > >
> > > PR target/100865
> > > * gcc.target/i386/avx512f-broadcast-pr87767-1.c: Expect integer
> > > broadcast.
> > > * gcc.target/i386/avx512f-broadcast-pr87767-5.c: Likewise.
> > > * gcc.target/i386/avx512vl-broadcast-pr87767-1.c: Likewise.
> > > * gcc.target/i386/avx512vl-broadcast-pr87767-5.c: Likewise.
> > > * gcc.target/i386/avx512f_cond_move.c: Also pass
> > > -mprefer-vector-width=512 and expect integer broadcast.
> > Those tests change LGTM since it's faster than the older version, and
> > I remember uros has already reviewed other parts, right?
>
> Uros, does this version look OK?
Yes, LGTM.
My main concern was around embedded broadcast, but as shown above,
using temporary reg is faster.
Uros.
> Thanks.
>
> > > * gcc.target/i386/pr100865-1.c: New test.
> > > * gcc.target/i386/pr100865-2.c: Likewise.
> > > * gcc.target/i386/pr100865-3.c: Likewise.
> > > * gcc.target/i386/pr100865-4a.c: Likewise.
> > > * gcc.target/i386/pr100865-4b.c: Likewise.
> > > * gcc.target/i386/pr100865-5a.c: Likewise.
> > > * gcc.target/i386/pr100865-5b.c: Likewise.
> > > * gcc.target/i386/pr100865-6a.c: Likewise.
> > > * gcc.target/i386/pr100865-6b.c: Likewise.
> > > * gcc.target/i386/pr100865-6c.c: Likewise.
> > > * gcc.target/i386/pr100865-7a.c: Likewise.
> > > * gcc.target/i386/pr100865-7b.c: Likewise.
> > > * gcc.target/i386/pr100865-7c.c: Likewise.
> > > * gcc.target/i386/pr100865-8a.c: Likewise.
> > > * gcc.target/i386/pr100865-8b.c: Likewise.
> > > * gcc.target/i386/pr100865-8c.c: Likewise.
> > > * gcc.target/i386/pr100865-9a.c: Likewise.
> > > * gcc.target/i386/pr100865-9b.c: Likewise.
> > > * gcc.target/i386/pr100865-9c.c: Likewise.
> > > * gcc.target/i386/pr100865-10a.c: Likewise.
> > > * gcc.target/i386/pr100865-10b.c: Likewise.
> > > * gcc.target/i386/pr100865-11a.c: Likewise.
> > > * gcc.target/i386/pr100865-11b.c: Likewise.
> > > * gcc.target/i386/pr100865-11c.c: Likewise.
> > > * gcc.target/i386/pr100865-12a.c: Likewise.
> > > * gcc.target/i386/pr100865-12b.c: Likewise.
> > > * gcc.target/i386/pr100865-12c.c: Likewise.
> > > ---
> > > gcc/config/i386/i386-expand.c | 194 ++++++++++++++++--
> > > gcc/config/i386/i386-protos.h | 2 +
> > > gcc/config/i386/i386.c | 13 ++
> > > .../i386/avx512f-broadcast-pr87767-1.c | 7 +-
> > > .../i386/avx512f-broadcast-pr87767-5.c | 5 +-
> > > .../gcc.target/i386/avx512f_cond_move.c | 4 +-
> > > .../i386/avx512vl-broadcast-pr87767-1.c | 12 +-
> > > .../i386/avx512vl-broadcast-pr87767-5.c | 9 +-
> > > gcc/testsuite/gcc.target/i386/pr100865-1.c | 13 ++
> > > gcc/testsuite/gcc.target/i386/pr100865-10a.c | 33 +++
> > > gcc/testsuite/gcc.target/i386/pr100865-10b.c | 7 +
> > > gcc/testsuite/gcc.target/i386/pr100865-11a.c | 23 +++
> > > gcc/testsuite/gcc.target/i386/pr100865-11b.c | 8 +
> > > gcc/testsuite/gcc.target/i386/pr100865-11c.c | 8 +
> > > gcc/testsuite/gcc.target/i386/pr100865-12a.c | 20 ++
> > > gcc/testsuite/gcc.target/i386/pr100865-12b.c | 8 +
> > > gcc/testsuite/gcc.target/i386/pr100865-12c.c | 8 +
> > > gcc/testsuite/gcc.target/i386/pr100865-2.c | 14 ++
> > > gcc/testsuite/gcc.target/i386/pr100865-3.c | 15 ++
> > > gcc/testsuite/gcc.target/i386/pr100865-4a.c | 16 ++
> > > gcc/testsuite/gcc.target/i386/pr100865-4b.c | 9 +
> > > gcc/testsuite/gcc.target/i386/pr100865-5a.c | 16 ++
> > > gcc/testsuite/gcc.target/i386/pr100865-5b.c | 9 +
> > > gcc/testsuite/gcc.target/i386/pr100865-6a.c | 16 ++
> > > gcc/testsuite/gcc.target/i386/pr100865-6b.c | 9 +
> > > gcc/testsuite/gcc.target/i386/pr100865-6c.c | 16 ++
> > > gcc/testsuite/gcc.target/i386/pr100865-7a.c | 17 ++
> > > gcc/testsuite/gcc.target/i386/pr100865-7b.c | 9 +
> > > gcc/testsuite/gcc.target/i386/pr100865-7c.c | 17 ++
> > > gcc/testsuite/gcc.target/i386/pr100865-8a.c | 24 +++
> > > gcc/testsuite/gcc.target/i386/pr100865-8b.c | 7 +
> > > gcc/testsuite/gcc.target/i386/pr100865-8c.c | 7 +
> > > gcc/testsuite/gcc.target/i386/pr100865-9a.c | 25 +++
> > > gcc/testsuite/gcc.target/i386/pr100865-9b.c | 7 +
> > > gcc/testsuite/gcc.target/i386/pr100865-9c.c | 7 +
> > > 35 files changed, 590 insertions(+), 24 deletions(-)
> > > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-1.c
> > > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-10a.c
> > > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-10b.c
> > > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-11a.c
> > > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-11b.c
> > > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-11c.c
> > > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-12a.c
> > > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-12b.c
> > > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-12c.c
> > > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-2.c
> > > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-3.c
> > > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-4a.c
> > > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-4b.c
> > > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-5a.c
> > > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-5b.c
> > > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-6a.c
> > > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-6b.c
> > > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-6c.c
> > > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-7a.c
> > > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-7b.c
> > > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-7c.c
> > > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-8a.c
> > > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-8b.c
> > > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-8c.c
> > > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-9a.c
> > > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-9b.c
> > > create mode 100644 gcc/testsuite/gcc.target/i386/pr100865-9c.c
> > >
> > > diff --git a/gcc/config/i386/i386-expand.c b/gcc/config/i386/i386-expand.c
> > > index e9763eb5b3e..e0e3ed4d8a4 100644
> > > --- a/gcc/config/i386/i386-expand.c
> > > +++ b/gcc/config/i386/i386-expand.c
> > > @@ -93,6 +93,9 @@ along with GCC; see the file COPYING3. If not see
> > > #include "i386-builtins.h"
> > > #include "i386-expand.h"
> > >
> > > +static bool ix86_expand_vector_init_duplicate (bool, machine_mode, rtx,
> > > + rtx);
> > > +
> > > /* Split one or more double-mode RTL references into pairs of half-mode
> > > references. The RTL can be REG, offsettable MEM, integer constant, or
> > > CONST_DOUBLE. "operands" is a pointer to an array of double-mode RTLs to
> > > @@ -190,6 +193,82 @@ ix86_expand_clear (rtx dest)
> > > emit_insn (tmp);
> > > }
> > >
> > > +/* Return true if V can be broadcasted from an integer of WIDTH bits
> > > + which is returned in VAL_BROADCAST. Otherwise, return false. */
> > > +
> > > +static bool
> > > +ix86_broadcast (HOST_WIDE_INT v, unsigned int width,
> > > + HOST_WIDE_INT &val_broadcast)
> > > +{
> > > + wide_int val = wi::uhwi (v, HOST_BITS_PER_WIDE_INT);
> > > + val_broadcast = wi::extract_uhwi (val, 0, width);
> > > + for (unsigned int i = width; i < HOST_BITS_PER_WIDE_INT; i += width)
> > > + {
> > > + HOST_WIDE_INT each = wi::extract_uhwi (val, i, width);
> > > + if (val_broadcast != each)
> > > + return false;
> > > + }
> > > + val_broadcast = sext_hwi (val_broadcast, width);
> > > + return true;
> > > +}
> > > +
> > > +/* Convert the CONST_WIDE_INT operand OP to broadcast in MODE. */
> > > +
> > > +static rtx
> > > +ix86_convert_const_wide_int_to_broadcast (machine_mode mode, rtx op)
> > > +{
> > > + /* Don't use integer vector broadcast if we can't move from GPR to SSE
> > > + register directly. */
> > > + if (!TARGET_INTER_UNIT_MOVES_TO_VEC)
> > > + return nullptr;
> > > +
> > > + /* Convert CONST_WIDE_INT to a non-standard SSE constant integer
> > > + broadcast only if vector broadcast is available. */
> > > + if (!TARGET_AVX
> > > + || !CONST_WIDE_INT_P (op)
> > > + || standard_sse_constant_p (op, mode))
> > > + return nullptr;
> > > +
> > > + HOST_WIDE_INT val = CONST_WIDE_INT_ELT (op, 0);
> > > + HOST_WIDE_INT val_broadcast;
> > > + scalar_int_mode broadcast_mode;
> > > + if (TARGET_AVX2
> > > + && ix86_broadcast (val, GET_MODE_BITSIZE (QImode),
> > > + val_broadcast))
> > > + broadcast_mode = QImode;
> > > + else if (TARGET_AVX2
> > > + && ix86_broadcast (val, GET_MODE_BITSIZE (HImode),
> > > + val_broadcast))
> > > + broadcast_mode = HImode;
> > > + else if (ix86_broadcast (val, GET_MODE_BITSIZE (SImode),
> > > + val_broadcast))
> > > + broadcast_mode = SImode;
> > > + else if (TARGET_64BIT
> > > + && ix86_broadcast (val, GET_MODE_BITSIZE (DImode),
> > > + val_broadcast))
> > > + broadcast_mode = DImode;
> > > + else
> > > + return nullptr;
> > > +
> > > + /* Check if OP can be broadcasted from VAL. */
> > > + for (int i = 1; i < CONST_WIDE_INT_NUNITS (op); i++)
> > > + if (val != CONST_WIDE_INT_ELT (op, i))
> > > + return nullptr;
> > > +
> > > + unsigned int nunits = (GET_MODE_SIZE (mode)
> > > + / GET_MODE_SIZE (broadcast_mode));
> > > + machine_mode vector_mode;
> > > + if (!mode_for_vector (broadcast_mode, nunits).exists (&vector_mode))
> > > + gcc_unreachable ();
> > > + rtx target = ix86_gen_scratch_sse_rtx (vector_mode);
> > > + bool ok = ix86_expand_vector_init_duplicate (false, vector_mode,
> > > + target,
> > > + GEN_INT (val_broadcast));
> > > + gcc_assert (ok);
> > > + target = lowpart_subreg (mode, target, vector_mode);
> > > + return target;
> > > +}
> > > +
> > > void
> > > ix86_expand_move (machine_mode mode, rtx operands[])
> > > {
> > > @@ -347,20 +426,29 @@ ix86_expand_move (machine_mode mode, rtx operands[])
> > > && optimize)
> > > op1 = copy_to_mode_reg (mode, op1);
> > >
> > > - if (can_create_pseudo_p ()
> > > - && CONST_DOUBLE_P (op1))
> > > + if (can_create_pseudo_p ())
> > > {
> > > - /* If we are loading a floating point constant to a register,
> > > - force the value to memory now, since we'll get better code
> > > - out the back end. */
> > > + if (CONST_DOUBLE_P (op1))
> > > + {
> > > + /* If we are loading a floating point constant to a
> > > + register, force the value to memory now, since we'll
> > > + get better code out the back end. */
> > >
> > > - op1 = validize_mem (force_const_mem (mode, op1));
> > > - if (!register_operand (op0, mode))
> > > + op1 = validize_mem (force_const_mem (mode, op1));
> > > + if (!register_operand (op0, mode))
> > > + {
> > > + rtx temp = gen_reg_rtx (mode);
> > > + emit_insn (gen_rtx_SET (temp, op1));
> > > + emit_move_insn (op0, temp);
> > > + return;
> > > + }
> > > + }
> > > + else if (GET_MODE_SIZE (mode) >= 16)
> > > {
> > > - rtx temp = gen_reg_rtx (mode);
> > > - emit_insn (gen_rtx_SET (temp, op1));
> > > - emit_move_insn (op0, temp);
> > > - return;
> > > + rtx tmp = ix86_convert_const_wide_int_to_broadcast
> > > + (GET_MODE (op0), op1);
> > > + if (tmp != nullptr)
> > > + op1 = tmp;
> > > }
> > > }
> > > }
> > > @@ -368,6 +456,59 @@ ix86_expand_move (machine_mode mode, rtx operands[])
> > > emit_insn (gen_rtx_SET (op0, op1));
> > > }
> > >
> > > +static rtx
> > > +ix86_broadcast_from_integer_constant (machine_mode mode, rtx op)
> > > +{
> > > + int nunits = GET_MODE_NUNITS (mode);
> > > + if (nunits < 2)
> > > + return nullptr;
> > > +
> > > + /* Don't use integer vector broadcast if we can't move from GPR to SSE
> > > + register directly. */
> > > + if (!TARGET_INTER_UNIT_MOVES_TO_VEC)
> > > + return nullptr;
> > > +
> > > + /* Convert CONST_VECTOR to a non-standard SSE constant integer
> > > + broadcast only if vector broadcast is available. */
> > > + if (!(TARGET_AVX2
> > > + || (TARGET_AVX
> > > + && (GET_MODE_INNER (mode) == SImode
> > > + || GET_MODE_INNER (mode) == DImode)))
> > > + || standard_sse_constant_p (op, mode))
> > > + return nullptr;
> > > +
> > > + /* Don't broadcast from a 64-bit integer constant in 32-bit mode. */
> > > + if (GET_MODE_INNER (mode) == DImode && !TARGET_64BIT)
> > > + return nullptr;
> > > +
> > > + rtx constant = get_pool_constant (XEXP (op, 0));
> > > + if (GET_CODE (constant) != CONST_VECTOR)
> > > + return nullptr;
> > > +
> > > + /* There could be some rtx like
> > > + (mem/u/c:V16QI (symbol_ref/u:DI ("*.LC1")))
> > > + but with "*.LC1" refer to V2DI constant vector. */
> > > + if (GET_MODE (constant) != mode)
> > > + {
> > > + constant = simplify_subreg (mode, constant, GET_MODE (constant),
> > > + 0);
> > > + if (constant == nullptr || GET_CODE (constant) != CONST_VECTOR)
> > > + return nullptr;
> > > + }
> > > +
> > > + rtx first = XVECEXP (constant, 0, 0);
> > > +
> > > + for (int i = 1; i < nunits; ++i)
> > > + {
> > > + rtx tmp = XVECEXP (constant, 0, i);
> > > + /* Vector duplicate value. */
> > > + if (!rtx_equal_p (tmp, first))
> > > + return nullptr;
> > > + }
> > > +
> > > + return first;
> > > +}
> > > +
> > > void
> > > ix86_expand_vector_move (machine_mode mode, rtx operands[])
> > > {
> > > @@ -407,7 +548,36 @@ ix86_expand_vector_move (machine_mode mode, rtx operands[])
> > > op1 = simplify_gen_subreg (mode, r, imode, SUBREG_BYTE (op1));
> > > }
> > > else
> > > - op1 = validize_mem (force_const_mem (mode, op1));
> > > + {
> > > + machine_mode mode = GET_MODE (op0);
> > > + rtx tmp = ix86_convert_const_wide_int_to_broadcast
> > > + (mode, op1);
> > > + if (tmp == nullptr)
> > > + op1 = validize_mem (force_const_mem (mode, op1));
> > > + else
> > > + op1 = tmp;
> > > + }
> > > + }
> > > +
> > > + if (can_create_pseudo_p ()
> > > + && GET_MODE_SIZE (mode) >= 16
> > > + && GET_MODE_CLASS (mode) == MODE_VECTOR_INT
> > > + && (MEM_P (op1)
> > > + && SYMBOL_REF_P (XEXP (op1, 0))
> > > + && CONSTANT_POOL_ADDRESS_P (XEXP (op1, 0))))
> > > + {
> > > + rtx first = ix86_broadcast_from_integer_constant (mode, op1);
> > > + if (first != nullptr)
> > > + {
> > > + /* Broadcast to XMM/YMM/ZMM register from an integer
> > > + constant. */
> > > + op1 = ix86_gen_scratch_sse_rtx (mode);
> > > + bool ok = ix86_expand_vector_init_duplicate (false, mode,
> > > + op1, first);
> > > + gcc_assert (ok);
> > > + emit_move_insn (op0, op1);
> > > + return;
> > > + }
> > > }
> > >
> > > /* We need to check memory alignment for SSE mode since attribute
> > > diff --git a/gcc/config/i386/i386-protos.h b/gcc/config/i386/i386-protos.h
> > > index 65fc307dc7b..71745b9a1ea 100644
> > > --- a/gcc/config/i386/i386-protos.h
> > > +++ b/gcc/config/i386/i386-protos.h
> > > @@ -50,6 +50,8 @@ extern void ix86_reset_previous_fndecl (void);
> > >
> > > extern bool ix86_using_red_zone (void);
> > >
> > > +extern rtx ix86_gen_scratch_sse_rtx (machine_mode);
> > > +
> > > extern unsigned int ix86_regmode_natural_size (machine_mode);
> > > #ifdef RTX_CODE
> > > extern int standard_80387_constant_p (rtx);
> > > diff --git a/gcc/config/i386/i386.c b/gcc/config/i386/i386.c
> > > index a93128fa0a4..2fbaae7cd02 100644
> > > --- a/gcc/config/i386/i386.c
> > > +++ b/gcc/config/i386/i386.c
> > > @@ -23163,6 +23163,19 @@ ix86_optab_supported_p (int op, machine_mode mode1, machine_mode,
> > > }
> > > }
> > >
> > > +/* Return a scratch register in MODE for vector load and store. */
> > > +
> > > +rtx
> > > +ix86_gen_scratch_sse_rtx (machine_mode mode)
> > > +{
> > > + if (TARGET_SSE)
> > > + return gen_rtx_REG (mode, (TARGET_64BIT
> > > + ? LAST_REX_SSE_REG
> > > + : LAST_SSE_REG));
> > > + else
> > > + return gen_reg_rtx (mode);
> > > +}
> > > +
> > > /* Address space support.
> > >
> > > This is not "far pointers" in the 16-bit sense, but an easy way
> > > diff --git a/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-1.c b/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-1.c
> > > index 0563e696316..a2664d87f29 100644
> > > --- a/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-1.c
> > > +++ b/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-1.c
> > > @@ -2,8 +2,11 @@
> > > /* { dg-do compile } */
> > > /* { dg-options "-O2 -mavx512f -mavx512dq" } */
> > > /* { dg-additional-options "-mdynamic-no-pic" { target { *-*-darwin* && ia32 } } }
> > > -/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to8\\\}" 5 } } */
> > > -/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to16\\\}" 5 } } */
> > > +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to8\\\}" 2 { target { ! ia32 } } } } */
> > > +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to8\\\}" 5 { target ia32 } } } */
> > > +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to16\\\}" 2 } } */
> > > +/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %zmm\[0-9\]+" 3 } } */
> > > +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%r\[^\n\]*, %zmm\[0-9\]+" 3 { target { ! ia32 } } } } */
> > >
> > > typedef int v16si __attribute__ ((vector_size (64)));
> > > typedef long long v8di __attribute__ ((vector_size (64)));
> > > diff --git a/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-5.c b/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-5.c
> > > index ffbe95980ca..477f9ca1282 100644
> > > --- a/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-5.c
> > > +++ b/gcc/testsuite/gcc.target/i386/avx512f-broadcast-pr87767-5.c
> > > @@ -2,8 +2,9 @@
> > > /* { dg-do compile } */
> > > /* { dg-options "-O2 -mavx512f" } */
> > > /* { dg-additional-options "-mdynamic-no-pic" { target { *-*-darwin* && ia32 } } }
> > > -/* { dg-final { scan-assembler-times "\[^n\n\]*\\\{1to8\\\}" 4 } } */
> > > -/* { dg-final { scan-assembler-times "\[^n\n\]*\\\{1to16\\\}" 4 } } */
> > > +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to8\\\}" 4 { target ia32 } } } */
> > > +/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %zmm\[0-9\]+" 4 } } */
> > > +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%r\[^\n\]*, %zmm\[0-9\]+" 4 { target { ! ia32 } } } } */
> > >
> > > typedef int v16si __attribute__ ((vector_size (64)));
> > > typedef long long v8di __attribute__ ((vector_size (64)));
> > > diff --git a/gcc/testsuite/gcc.target/i386/avx512f_cond_move.c b/gcc/testsuite/gcc.target/i386/avx512f_cond_move.c
> > > index 99a89f51202..ca49a585232 100644
> > > --- a/gcc/testsuite/gcc.target/i386/avx512f_cond_move.c
> > > +++ b/gcc/testsuite/gcc.target/i386/avx512f_cond_move.c
> > > @@ -1,6 +1,6 @@
> > > /* { dg-do compile } */
> > > -/* { dg-options "-O3 -mavx512f" } */
> > > -/* { dg-final { scan-assembler-times "(?:vpblendmd|vmovdqa32)\[ \\t\]+\[^\{\n\]*%zmm\[0-9\]+\{%k\[1-7\]\}\{z\}(?:\n|\[ \\t\]+#)" 8 } } */
> > > +/* { dg-options "-O3 -mavx512f -mprefer-vector-width=512" } */
> > > +/* { dg-final { scan-assembler-times "(?:vpbroadcastd|vmovdqa32)\[ \\t\]+\[^\{\n\]*%zmm\[0-9\]+\{%k\[1-7\]\}\{z\}(?:\n|\[ \\t\]+#)" 8 } } */
> > >
> > > unsigned int x[128];
> > > int y[128];
> > > diff --git a/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-1.c b/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-1.c
> > > index c06369d93fd..f8eb99f0b5f 100644
> > > --- a/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-1.c
> > > +++ b/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-1.c
> > > @@ -2,9 +2,15 @@
> > > /* { dg-do compile } */
> > > /* { dg-options "-O2 -mavx512f -mavx512vl -mavx512dq" } */
> > > /* { dg-additional-options "-mdynamic-no-pic" { target { *-*-darwin* && ia32 } } }
> > > -/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to2\\\}" 5 } } */
> > > -/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to4\\\}" 10 } } */
> > > -/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to8\\\}" 5 } } */
> > > +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to2\\\}" 2 { target { ! ia32 } } } } */
> > > +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to4\\\}" 4 { target { ! ia32 } } } } */
> > > +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to2\\\}" 5 { target ia32 } } } */
> > > +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to4\\\}" 7 { target ia32 } } } */
> > > +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to8\\\}" 2 } } */
> > > +/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 3 } } */
> > > +/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %ymm\[0-9\]+" 3 } } */
> > > +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%r\[^\n\]*, %xmm\[0-9\]+" 3 { target { ! ia32 } } } } */
> > > +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%r\[^\n\]*, %ymm\[0-9\]+" 3 { target { ! ia32 } } } } */
> > >
> > > typedef int v4si __attribute__ ((vector_size (16)));
> > > typedef int v8si __attribute__ ((vector_size (32)));
> > > diff --git a/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-5.c b/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-5.c
> > > index 4998a9b8d51..32f6ac81841 100644
> > > --- a/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-5.c
> > > +++ b/gcc/testsuite/gcc.target/i386/avx512vl-broadcast-pr87767-5.c
> > > @@ -2,9 +2,12 @@
> > > /* { dg-do compile } */
> > > /* { dg-options "-O2 -mavx512f -mavx512vl" } */
> > > /* { dg-additional-options "-mdynamic-no-pic" { target { *-*-darwin* && ia32 } } }
> > > -/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to2\\\}" 4 } } */
> > > -/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to4\\\}" 8 } } */
> > > -/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to8\\\}" 4 } } */
> > > +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to2\\\}" 4 { target ia32 } } } */
> > > +/* { dg-final { scan-assembler-times "\[^\n\]*\\\{1to4\\\}" 4 { target ia32 } } } */
> > > +/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 4 } } */
> > > +/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %ymm\[0-9\]+" 4 } } */
> > > +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%r\[^\n\]*, %xmm\[0-9\]+" 4 { target { ! ia32 } } } } */
> > > +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%r\[^\n\]*, %ymm\[0-9\]+" 4 { target { ! ia32 } } } } */
> > >
> > > typedef int v4si __attribute__ ((vector_size (16)));
> > > typedef int v8si __attribute__ ((vector_size (32)));
> > > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-1.c b/gcc/testsuite/gcc.target/i386/pr100865-1.c
> > > new file mode 100644
> > > index 00000000000..6c3097fb2a6
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/i386/pr100865-1.c
> > > @@ -0,0 +1,13 @@
> > > +/* { dg-do compile { target { ! ia32 } } } */
> > > +/* { dg-options "-O2 -march=x86-64" } */
> > > +
> > > +extern char *dst;
> > > +
> > > +void
> > > +foo (void)
> > > +{
> > > + __builtin_memset (dst, 3, 16);
> > > +}
> > > +
> > > +/* { dg-final { scan-assembler-times "movdqa\[ \\t\]+\[^\n\]*%xmm" 1 } } */
> > > +/* { dg-final { scan-assembler-times "movups\[\\t \]%xmm\[0-9\]+, \\(%\[\^,\]+\\)" 1 } } */
> > > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-10a.c b/gcc/testsuite/gcc.target/i386/pr100865-10a.c
> > > new file mode 100644
> > > index 00000000000..7ffc19e56a8
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/i386/pr100865-10a.c
> > > @@ -0,0 +1,33 @@
> > > +/* { dg-do compile { target int128 } } */
> > > +/* { dg-options "-O3 -march=skylake" } */
> > > +
> > > +extern __int128 array[16];
> > > +
> > > +#define MK_CONST128_BROADCAST(A) \
> > > + ((((unsigned __int128) (unsigned char) A) << 120) \
> > > + | (((unsigned __int128) (unsigned char) A) << 112) \
> > > + | (((unsigned __int128) (unsigned char) A) << 104) \
> > > + | (((unsigned __int128) (unsigned char) A) << 96) \
> > > + | (((unsigned __int128) (unsigned char) A) << 88) \
> > > + | (((unsigned __int128) (unsigned char) A) << 80) \
> > > + | (((unsigned __int128) (unsigned char) A) << 72) \
> > > + | (((unsigned __int128) (unsigned char) A) << 64) \
> > > + | (((unsigned __int128) (unsigned char) A) << 56) \
> > > + | (((unsigned __int128) (unsigned char) A) << 48) \
> > > + | (((unsigned __int128) (unsigned char) A) << 40) \
> > > + | (((unsigned __int128) (unsigned char) A) << 32) \
> > > + | (((unsigned __int128) (unsigned char) A) << 24) \
> > > + | (((unsigned __int128) (unsigned char) A) << 16) \
> > > + | (((unsigned __int128) (unsigned char) A) << 8) \
> > > + | ((unsigned __int128) (unsigned char) A) )
> > > +
> > > +void
> > > +foo (void)
> > > +{
> > > + int i;
> > > + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> > > + array[i] = MK_CONST128_BROADCAST (0x1f);
> > > +}
> > > +
> > > +/* { dg-final { scan-assembler-times "vpbroadcastb\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> > > +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> > > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-10b.c b/gcc/testsuite/gcc.target/i386/pr100865-10b.c
> > > new file mode 100644
> > > index 00000000000..edf52765c60
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/i386/pr100865-10b.c
> > > @@ -0,0 +1,7 @@
> > > +/* { dg-do compile { target int128 } } */
> > > +/* { dg-options "-O3 -march=skylake-avx512" } */
> > > +
> > > +#include "pr100865-10a.c"
> > > +
> > > +/* { dg-final { scan-assembler-times "vpbroadcastb\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> > > +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> > > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-11a.c b/gcc/testsuite/gcc.target/i386/pr100865-11a.c
> > > new file mode 100644
> > > index 00000000000..04ce1662f3c
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/i386/pr100865-11a.c
> > > @@ -0,0 +1,23 @@
> > > +/* { dg-do compile { target int128 } } */
> > > +/* { dg-options "-O3 -march=skylake" } */
> > > +
> > > +extern __int128 array[16];
> > > +
> > > +#define MK_CONST128_BROADCAST(A) \
> > > + ((((unsigned __int128) (unsigned long long) A) << 64) \
> > > + | ((unsigned __int128) (unsigned long long) A) )
> > > +
> > > +#define MK_CONST128_BROADCAST_SIGNED(A) \
> > > + ((__int128) MK_CONST128_BROADCAST (A))
> > > +
> > > +void
> > > +foo (void)
> > > +{
> > > + int i;
> > > + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> > > + array[i] = MK_CONST128_BROADCAST_SIGNED (-0x1ffffffffLL);
> > > +}
> > > +
> > > +/* { dg-final { scan-assembler-times "movabsq" 1 } } */
> > > +/* { dg-final { scan-assembler-times "(?:vpbroadcastq|vpunpcklqdq)\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> > > +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> > > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-11b.c b/gcc/testsuite/gcc.target/i386/pr100865-11b.c
> > > new file mode 100644
> > > index 00000000000..12d55b9a642
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/i386/pr100865-11b.c
> > > @@ -0,0 +1,8 @@
> > > +/* { dg-do compile { target int128 } } */
> > > +/* { dg-options "-O3 -march=skylake-avx512" } */
> > > +
> > > +#include "pr100865-11a.c"
> > > +
> > > +/* { dg-final { scan-assembler-times "movabsq" 1 } } */
> > > +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> > > +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> > > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-11c.c b/gcc/testsuite/gcc.target/i386/pr100865-11c.c
> > > new file mode 100644
> > > index 00000000000..de56c84b9ca
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/i386/pr100865-11c.c
> > > @@ -0,0 +1,8 @@
> > > +/* { dg-do compile { target int128 } } */
> > > +/* { dg-options "-O3 -march=skylake -mno-avx2" } */
> > > +
> > > +#include "pr100865-11a.c"
> > > +
> > > +/* { dg-final { scan-assembler-times "movabsq" 1 } } */
> > > +/* { dg-final { scan-assembler-times "vpunpcklqdq\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> > > +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> > > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-12a.c b/gcc/testsuite/gcc.target/i386/pr100865-12a.c
> > > new file mode 100644
> > > index 00000000000..d4833d44475
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/i386/pr100865-12a.c
> > > @@ -0,0 +1,20 @@
> > > +/* { dg-do compile { target int128 } } */
> > > +/* { dg-options "-O3 -march=skylake" } */
> > > +
> > > +extern __int128 array[16];
> > > +
> > > +#define MK_CONST128_BROADCAST(A) \
> > > + ((((unsigned __int128) (unsigned long long) A) << 64) \
> > > + | ((unsigned __int128) (unsigned long long) A) )
> > > +
> > > +void
> > > +foo (void)
> > > +{
> > > + int i;
> > > + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> > > + array[i] = MK_CONST128_BROADCAST (0x1ffffffffLL);
> > > +}
> > > +
> > > +/* { dg-final { scan-assembler-times "movabsq" 1 } } */
> > > +/* { dg-final { scan-assembler-times "(?:vpbroadcastq|vpunpcklqdq)\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> > > +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> > > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-12b.c b/gcc/testsuite/gcc.target/i386/pr100865-12b.c
> > > new file mode 100644
> > > index 00000000000..63a5629b90c
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/i386/pr100865-12b.c
> > > @@ -0,0 +1,8 @@
> > > +/* { dg-do compile { target int128 } } */
> > > +/* { dg-options "-O3 -march=skylake-avx512" } */
> > > +
> > > +#include "pr100865-12a.c"
> > > +
> > > +/* { dg-final { scan-assembler-times "movabsq" 1 } } */
> > > +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> > > +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> > > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-12c.c b/gcc/testsuite/gcc.target/i386/pr100865-12c.c
> > > new file mode 100644
> > > index 00000000000..77415f22c97
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/i386/pr100865-12c.c
> > > @@ -0,0 +1,8 @@
> > > +/* { dg-do compile { target int128 } } */
> > > +/* { dg-options "-O3 -march=skylake -mno-avx2" } */
> > > +
> > > +#include "pr100865-12a.c"
> > > +
> > > +/* { dg-final { scan-assembler-times "movabsq" 1 } } */
> > > +/* { dg-final { scan-assembler-times "vpunpcklqdq\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> > > +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> > > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-2.c b/gcc/testsuite/gcc.target/i386/pr100865-2.c
> > > new file mode 100644
> > > index 00000000000..17efe2d72a3
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/i386/pr100865-2.c
> > > @@ -0,0 +1,14 @@
> > > +/* { dg-do compile { target { ! ia32 } } } */
> > > +/* { dg-options "-O2 -march=skylake" } */
> > > +
> > > +extern char *dst;
> > > +
> > > +void
> > > +foo (void)
> > > +{
> > > + __builtin_memset (dst, 3, 16);
> > > +}
> > > +
> > > +/* { dg-final { scan-assembler-times "vpbroadcastb\[\\t \]+%xmm\[0-9\]+, %xmm\[0-9\]+" 1 } } */
> > > +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%xmm\[0-9\]+, \\(%\[\^,\]+\\)" 1 } } */
> > > +/* { dg-final { scan-assembler-not "vmovdqa" } } */
> > > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-3.c b/gcc/testsuite/gcc.target/i386/pr100865-3.c
> > > new file mode 100644
> > > index 00000000000..b6dbcf7809b
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/i386/pr100865-3.c
> > > @@ -0,0 +1,15 @@
> > > +/* { dg-do compile { target { ! ia32 } } } */
> > > +/* { dg-options "-O2 -march=skylake-avx512" } */
> > > +
> > > +extern char *dst;
> > > +
> > > +void
> > > +foo (void)
> > > +{
> > > + __builtin_memset (dst, 3, 16);
> > > +}
> > > +
> > > +/* { dg-final { scan-assembler-times "vpbroadcastb\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> > > +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%xmm\[0-9\]+, \\(%\[\^,\]+\\)" 1 } } */
> > > +/* { dg-final { scan-assembler-not "vpbroadcastb\[\\t \]+%xmm\[0-9\]+, %xmm\[0-9\]+" } } */
> > > +/* { dg-final { scan-assembler-not "vmovdqa" } } */
> > > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-4a.c b/gcc/testsuite/gcc.target/i386/pr100865-4a.c
> > > new file mode 100644
> > > index 00000000000..f55883598f9
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/i386/pr100865-4a.c
> > > @@ -0,0 +1,16 @@
> > > +/* { dg-do compile { target { ! ia32 } } } */
> > > +/* { dg-options "-O2 -march=skylake" } */
> > > +
> > > +extern char array[64];
> > > +
> > > +void
> > > +foo (void)
> > > +{
> > > + int i;
> > > + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> > > + array[i] = -45;
> > > +}
> > > +
> > > +/* { dg-final { scan-assembler-times "vpbroadcastb\[\\t \]+%xmm\[0-9\]+, %xmm\[0-9\]+" 1 } } */
> > > +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%xmm\[0-9\]+, " 4 } } */
> > > +/* { dg-final { scan-assembler-not "vmovdqa" } } */
> > > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-4b.c b/gcc/testsuite/gcc.target/i386/pr100865-4b.c
> > > new file mode 100644
> > > index 00000000000..f41e6147b4c
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/i386/pr100865-4b.c
> > > @@ -0,0 +1,9 @@
> > > +/* { dg-do compile { target { ! ia32 } } } */
> > > +/* { dg-options "-O2 -march=skylake-avx512" } */
> > > +
> > > +#include "pr100865-4a.c"
> > > +
> > > +/* { dg-final { scan-assembler-times "vpbroadcastb\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> > > +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%xmm\[0-9\]+, " 4 } } */
> > > +/* { dg-final { scan-assembler-not "vpbroadcastb\[\\t \]+%xmm\[0-9\]+, %xmm\[0-9\]+" } } */
> > > +/* { dg-final { scan-assembler-not "vmovdqa" } } */
> > > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-5a.c b/gcc/testsuite/gcc.target/i386/pr100865-5a.c
> > > new file mode 100644
> > > index 00000000000..4149797fe81
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/i386/pr100865-5a.c
> > > @@ -0,0 +1,16 @@
> > > +/* { dg-do compile } */
> > > +/* { dg-options "-O3 -march=skylake" } */
> > > +
> > > +extern short array[64];
> > > +
> > > +void
> > > +foo (void)
> > > +{
> > > + int i;
> > > + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> > > + array[i] = -45;
> > > +}
> > > +
> > > +/* { dg-final { scan-assembler-times "vpbroadcastw\[\\t \]+%xmm\[0-9\]+, %ymm\[0-9\]+" 1 } } */
> > > +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%ymm\[0-9\]+, " 4 } } */
> > > +/* { dg-final { scan-assembler-not "vmovdqa" } } */
> > > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-5b.c b/gcc/testsuite/gcc.target/i386/pr100865-5b.c
> > > new file mode 100644
> > > index 00000000000..ded41b680d3
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/i386/pr100865-5b.c
> > > @@ -0,0 +1,9 @@
> > > +/* { dg-do compile } */
> > > +/* { dg-options "-O3 -march=skylake-avx512" } */
> > > +
> > > +#include "pr100865-5a.c"
> > > +
> > > +/* { dg-final { scan-assembler-times "vpbroadcastw\[\\t \]+%(?:r|e)\[^\n\]*, %ymm\[0-9\]+" 1 } } */
> > > +/* { dg-final { scan-assembler-times "vmovdqu16\[\\t \]%ymm\[0-9\]+, " 4 } } */
> > > +/* { dg-final { scan-assembler-not "vpbroadcastw\[\\t \]+%xmm\[0-9\]+, %ymm\[0-9\]+" } } */
> > > +/* { dg-final { scan-assembler-not "vmovdqa" } } */
> > > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-6a.c b/gcc/testsuite/gcc.target/i386/pr100865-6a.c
> > > new file mode 100644
> > > index 00000000000..3fde549a10d
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/i386/pr100865-6a.c
> > > @@ -0,0 +1,16 @@
> > > +/* { dg-do compile } */
> > > +/* { dg-options "-O3 -march=skylake" } */
> > > +
> > > +extern int array[64];
> > > +
> > > +void
> > > +foo (void)
> > > +{
> > > + int i;
> > > + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> > > + array[i] = -45;
> > > +}
> > > +
> > > +/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%xmm\[0-9\]+, %ymm\[0-9\]+" 1 } } */
> > > +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%ymm\[0-9\]+, " 8 } } */
> > > +/* { dg-final { scan-assembler-not "vmovdqa" } } */
> > > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-6b.c b/gcc/testsuite/gcc.target/i386/pr100865-6b.c
> > > new file mode 100644
> > > index 00000000000..44e74c64e55
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/i386/pr100865-6b.c
> > > @@ -0,0 +1,9 @@
> > > +/* { dg-do compile } */
> > > +/* { dg-options "-O3 -march=skylake-avx512" } */
> > > +
> > > +#include "pr100865-6a.c"
> > > +
> > > +/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %ymm\[0-9\]+" 1 } } */
> > > +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%ymm\[0-9\]+, " 8 } } */
> > > +/* { dg-final { scan-assembler-not "vpbroadcastd\[\\t \]+%xmm\[0-9\]+, %ymm\[0-9\]+" } } */
> > > +/* { dg-final { scan-assembler-not "vmovdqa" } } */
> > > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-6c.c b/gcc/testsuite/gcc.target/i386/pr100865-6c.c
> > > new file mode 100644
> > > index 00000000000..46d31030ce8
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/i386/pr100865-6c.c
> > > @@ -0,0 +1,16 @@
> > > +/* { dg-do compile } */
> > > +/* { dg-options "-O3 -march=skylake -mno-avx2" } */
> > > +
> > > +extern int array[64];
> > > +
> > > +void
> > > +foo (void)
> > > +{
> > > + int i;
> > > + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> > > + array[i] = -45;
> > > +}
> > > +
> > > +/* { dg-final { scan-assembler-times "vbroadcastss" 1 } } */
> > > +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%ymm\[0-9\]+, " 8 } } */
> > > +/* { dg-final { scan-assembler-not "vmovdqa" } } */
> > > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-7a.c b/gcc/testsuite/gcc.target/i386/pr100865-7a.c
> > > new file mode 100644
> > > index 00000000000..f6f2be91120
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/i386/pr100865-7a.c
> > > @@ -0,0 +1,17 @@
> > > +/* { dg-do compile } */
> > > +/* { dg-options "-O3 -march=skylake" } */
> > > +
> > > +extern long long int array[64];
> > > +
> > > +void
> > > +foo (void)
> > > +{
> > > + int i;
> > > + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> > > + array[i] = -45;
> > > +}
> > > +
> > > +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+\[^\n\]*, %ymm\[0-9\]+" 1 { target { ! ia32 } } } } */
> > > +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%ymm\[0-9\]+, " 16 } } */
> > > +/* { dg-final { scan-assembler-not "vpbroadcastq" { target ia32 } } } */
> > > +/* { dg-final { scan-assembler-not "vmovdqa" { target { ! ia32 } } } } */
> > > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-7b.c b/gcc/testsuite/gcc.target/i386/pr100865-7b.c
> > > new file mode 100644
> > > index 00000000000..0a68820aa32
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/i386/pr100865-7b.c
> > > @@ -0,0 +1,9 @@
> > > +/* { dg-do compile } */
> > > +/* { dg-options "-O3 -march=skylake-avx512" } */
> > > +
> > > +#include "pr100865-7a.c"
> > > +
> > > +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+%r\[^\n\]*, %ymm\[0-9\]+" 1 { target { ! ia32 } } } } */
> > > +/* { dg-final { scan-assembler-times "vpbroadcastq\[\\t \]+\[^\n\]*, %ymm\[0-9\]+" 1 { target ia32 } } } */
> > > +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%ymm\[0-9\]+, " 16 } } */
> > > +/* { dg-final { scan-assembler-not "vmovdqa" } } */
> > > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-7c.c b/gcc/testsuite/gcc.target/i386/pr100865-7c.c
> > > new file mode 100644
> > > index 00000000000..4d50bb7e2f6
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/i386/pr100865-7c.c
> > > @@ -0,0 +1,17 @@
> > > +/* { dg-do compile } */
> > > +/* { dg-options "-O3 -march=skylake -mno-avx2" } */
> > > +
> > > +extern long long int array[64];
> > > +
> > > +void
> > > +foo (void)
> > > +{
> > > + int i;
> > > + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> > > + array[i] = -45;
> > > +}
> > > +
> > > +/* { dg-final { scan-assembler-times "vbroadcastsd" 1 { target { ! ia32 } } } } */
> > > +/* { dg-final { scan-assembler-times "vmovdqu\[\\t \]%ymm\[0-9\]+, " 16 } } */
> > > +/* { dg-final { scan-assembler-not "vbroadcastsd" { target ia32 } } } */
> > > +/* { dg-final { scan-assembler-not "vmovdqa" { target { ! ia32 } } } } */
> > > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-8a.c b/gcc/testsuite/gcc.target/i386/pr100865-8a.c
> > > new file mode 100644
> > > index 00000000000..544a14db6f7
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/i386/pr100865-8a.c
> > > @@ -0,0 +1,24 @@
> > > +/* { dg-do compile { target int128 } } */
> > > +/* { dg-options "-O3 -march=skylake" } */
> > > +
> > > +extern __int128 array[16];
> > > +
> > > +#define MK_CONST128_BROADCAST(A) \
> > > + ((((unsigned __int128) (unsigned int) A) << 96) \
> > > + | (((unsigned __int128) (unsigned int) A) << 64) \
> > > + | (((unsigned __int128) (unsigned int) A) << 32) \
> > > + | ((unsigned __int128) (unsigned int) A) )
> > > +
> > > +#define MK_CONST128_BROADCAST_SIGNED(A) \
> > > + ((__int128) MK_CONST128_BROADCAST (A))
> > > +
> > > +void
> > > +foo (void)
> > > +{
> > > + int i;
> > > + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> > > + array[i] = MK_CONST128_BROADCAST_SIGNED (-45);
> > > +}
> > > +
> > > +/* { dg-final { scan-assembler-times "(?:vpbroadcastd|vpshufd)\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> > > +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> > > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-8b.c b/gcc/testsuite/gcc.target/i386/pr100865-8b.c
> > > new file mode 100644
> > > index 00000000000..99a10ad83bd
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/i386/pr100865-8b.c
> > > @@ -0,0 +1,7 @@
> > > +/* { dg-do compile { target int128 } } */
> > > +/* { dg-options "-O3 -march=skylake-avx512" } */
> > > +
> > > +#include "pr100865-8a.c"
> > > +
> > > +/* { dg-final { scan-assembler-times "vpbroadcastd\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> > > +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> > > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-8c.c b/gcc/testsuite/gcc.target/i386/pr100865-8c.c
> > > new file mode 100644
> > > index 00000000000..efee0488614
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/i386/pr100865-8c.c
> > > @@ -0,0 +1,7 @@
> > > +/* { dg-do compile { target int128 } } */
> > > +/* { dg-options "-O3 -march=skylake -mno-avx2" } */
> > > +
> > > +#include "pr100865-8a.c"
> > > +
> > > +/* { dg-final { scan-assembler-times "vpshufd\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> > > +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> > > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-9a.c b/gcc/testsuite/gcc.target/i386/pr100865-9a.c
> > > new file mode 100644
> > > index 00000000000..45d0e0d0e2e
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/i386/pr100865-9a.c
> > > @@ -0,0 +1,25 @@
> > > +/* { dg-do compile { target int128 } } */
> > > +/* { dg-options "-O3 -march=skylake" } */
> > > +
> > > +extern __int128 array[16];
> > > +
> > > +#define MK_CONST128_BROADCAST(A) \
> > > + ((((unsigned __int128) (unsigned short) A) << 112) \
> > > + | (((unsigned __int128) (unsigned short) A) << 96) \
> > > + | (((unsigned __int128) (unsigned short) A) << 80) \
> > > + | (((unsigned __int128) (unsigned short) A) << 64) \
> > > + | (((unsigned __int128) (unsigned short) A) << 48) \
> > > + | (((unsigned __int128) (unsigned short) A) << 32) \
> > > + | (((unsigned __int128) (unsigned short) A) << 16) \
> > > + | ((unsigned __int128) (unsigned short) A) )
> > > +
> > > +void
> > > +foo (void)
> > > +{
> > > + int i;
> > > + for (i = 0; i < sizeof (array) / sizeof (array[0]); i++)
> > > + array[i] = MK_CONST128_BROADCAST (0x1fff);
> > > +}
> > > +
> > > +/* { dg-final { scan-assembler-times "vpbroadcastw\[\\t \]+%xmm\[0-9\]+, %xmm\[0-9\]+" 1 } } */
> > > +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> > > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-9b.c b/gcc/testsuite/gcc.target/i386/pr100865-9b.c
> > > new file mode 100644
> > > index 00000000000..14696248525
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/i386/pr100865-9b.c
> > > @@ -0,0 +1,7 @@
> > > +/* { dg-do compile { target int128 } } */
> > > +/* { dg-options "-O3 -march=skylake-avx512" } */
> > > +
> > > +#include "pr100865-9a.c"
> > > +
> > > +/* { dg-final { scan-assembler-times "vpbroadcastw\[\\t \]+%(?:r|e)\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> > > +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> > > diff --git a/gcc/testsuite/gcc.target/i386/pr100865-9c.c b/gcc/testsuite/gcc.target/i386/pr100865-9c.c
> > > new file mode 100644
> > > index 00000000000..e6f25902c1d
> > > --- /dev/null
> > > +++ b/gcc/testsuite/gcc.target/i386/pr100865-9c.c
> > > @@ -0,0 +1,7 @@
> > > +/* { dg-do compile { target int128 } } */
> > > +/* { dg-options "-O3 -march=skylake -mno-avx2" } */
> > > +
> > > +#include "pr100865-9a.c"
> > > +
> > > +/* { dg-final { scan-assembler-times "vpshufd\[\\t \]+\[^\n\]*, %xmm\[0-9\]+" 1 } } */
> > > +/* { dg-final { scan-assembler-times "vmovdqa\[\\t \]%xmm\[0-9\]+, " 16 } } */
> > > --
> > > 2.31.1
> > >
> >
> >
> > --
> > BR,
> > Hongtao
>
>
>
> --
> H.J.
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2021-07-01 14:13 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-28 22:16 [PATCH v6 0/2] x86: Convert CONST_WIDE_INT/CONST_VECTOR to broadcast H.J. Lu
2021-06-28 22:16 ` [PATCH v6 1/2] " H.J. Lu
2021-07-01 8:37 ` Hongtao Liu
2021-07-01 12:41 ` H.J. Lu
2021-07-01 14:12 ` Uros Bizjak
2021-06-28 22:16 ` [PATCH v6 2/2] x86: Add vec_duplicate<mode> expander H.J. Lu
2021-07-01 8:30 ` Hongtao Liu
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).