* [PATCH 0/7] AVX512FP16: Support bunch of expanders for HFmode and vector HFmodes
@ 2021-09-23 5:46 liuhongt
2021-09-23 5:46 ` [PATCH 1/7] AVX512FP16: Add expander for rint/nearbyinthf2 liuhongt
` (6 more replies)
0 siblings, 7 replies; 8+ messages in thread
From: liuhongt @ 2021-09-23 5:46 UTC (permalink / raw)
To: gcc-patches
xfail are added for testcases related to truncmn2/extendmn2 expanders since V2HF/V4HFmode
are not supported yet, they should be removed later.
Bootstrapped and regtested on x86_64-pc-linux-gnu{-m32,}.
Newly added runtime testcases passed on sde{-m32,}.
Hongyu Wang (5):
AVX512FP16: Add expander for smin/maxhf3.
AVX512FP16: Add fix(uns)?_truncmn2 for HF scalar and vector modes
AVX512FP16: Add float(uns)?mn2 expander
AVX512FP16: add truncmn2/extendmn2 expanders
AVX512FP16: Enable vec_cmpmn/vcondmn expanders for HF modes.
liuhongt (2):
AVX512FP16: Add expander for rint/nearbyinthf2.
AVX512FP16: Add expander for fmahf4
gcc/config/i386/i386-expand.c | 2 +
gcc/config/i386/i386.md | 62 +++++
gcc/config/i386/sse.md | 259 +++++++++++++++---
.../i386/avx512fp16-vcondmn-minmax.C | 25 ++
.../g++.target/i386/avx512fp16-vcondmn-vec.C | 70 +++++
.../i386/avx512fp16-builtin-minmax-1.c | 35 +++
.../i386/avx512fp16-builtin-round-1.c | 14 +
.../gcc.target/i386/avx512fp16-floatvnhf.c | 61 +++++
.../gcc.target/i386/avx512fp16-fma-1.c | 69 +++++
.../i386/avx512fp16-trunc-extendvnhf.c | 55 ++++
.../gcc.target/i386/avx512fp16-trunchf.c | 59 ++++
.../gcc.target/i386/avx512fp16-truncvnhf.c | 61 +++++
.../i386/avx512fp16-vcondmn-loop-1.c | 70 +++++
.../i386/avx512fp16-vcondmn-loop-2.c | 143 ++++++++++
.../gcc.target/i386/avx512fp16-vec_cmpmn.c | 32 +++
.../gcc.target/i386/avx512fp16vl-fma-1.c | 70 +++++
.../i386/avx512fp16vl-fma-vectorize-1.c | 45 +++
17 files changed, 1100 insertions(+), 32 deletions(-)
create mode 100644 gcc/testsuite/g++.target/i386/avx512fp16-vcondmn-minmax.C
create mode 100644 gcc/testsuite/g++.target/i386/avx512fp16-vcondmn-vec.C
create mode 100644 gcc/testsuite/gcc.target/i386/avx512fp16-builtin-minmax-1.c
create mode 100644 gcc/testsuite/gcc.target/i386/avx512fp16-floatvnhf.c
create mode 100644 gcc/testsuite/gcc.target/i386/avx512fp16-fma-1.c
create mode 100644 gcc/testsuite/gcc.target/i386/avx512fp16-trunc-extendvnhf.c
create mode 100644 gcc/testsuite/gcc.target/i386/avx512fp16-trunchf.c
create mode 100644 gcc/testsuite/gcc.target/i386/avx512fp16-truncvnhf.c
create mode 100644 gcc/testsuite/gcc.target/i386/avx512fp16-vcondmn-loop-1.c
create mode 100644 gcc/testsuite/gcc.target/i386/avx512fp16-vcondmn-loop-2.c
create mode 100644 gcc/testsuite/gcc.target/i386/avx512fp16-vec_cmpmn.c
create mode 100644 gcc/testsuite/gcc.target/i386/avx512fp16vl-fma-1.c
create mode 100644 gcc/testsuite/gcc.target/i386/avx512fp16vl-fma-vectorize-1.c
--
2.27.0
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH 1/7] AVX512FP16: Add expander for rint/nearbyinthf2.
2021-09-23 5:46 [PATCH 0/7] AVX512FP16: Support bunch of expanders for HFmode and vector HFmodes liuhongt
@ 2021-09-23 5:46 ` liuhongt
2021-09-23 5:46 ` [PATCH 2/7] AVX512FP16: Add expander for fmahf4 liuhongt
` (5 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: liuhongt @ 2021-09-23 5:46 UTC (permalink / raw)
To: gcc-patches
gcc/ChangeLog:
* config/i386/i386.md (rinthf2): New expander.
(nearbyinthf2): New expander.
gcc/testsuite/ChangeLog:
* gcc.target/i386/avx512fp16-builtin-round-1.c: Add new testcase.
---
gcc/config/i386/i386.md | 22 +++++++++++++++++++
.../i386/avx512fp16-builtin-round-1.c | 14 ++++++++++++
2 files changed, 36 insertions(+)
diff --git a/gcc/config/i386/i386.md b/gcc/config/i386/i386.md
index 60d877668d5..4b13a59be82 100644
--- a/gcc/config/i386/i386.md
+++ b/gcc/config/i386/i386.md
@@ -18287,6 +18287,17 @@ (define_insn "rintxf2"
(set_attr "znver1_decode" "vector")
(set_attr "mode" "XF")])
+(define_expand "rinthf2"
+ [(match_operand:HF 0 "register_operand")
+ (match_operand:HF 1 "nonimmediate_operand")]
+ "TARGET_AVX512FP16"
+{
+ emit_insn (gen_sse4_1_roundhf2 (operands[0],
+ operands[1],
+ GEN_INT (ROUND_MXCSR)));
+ DONE;
+})
+
(define_expand "rint<mode>2"
[(use (match_operand:MODEF 0 "register_operand"))
(use (match_operand:MODEF 1 "nonimmediate_operand"))]
@@ -18320,6 +18331,17 @@ (define_expand "nearbyintxf2"
"TARGET_USE_FANCY_MATH_387
&& !flag_trapping_math")
+(define_expand "nearbyinthf2"
+ [(match_operand:HF 0 "register_operand")
+ (match_operand:HF 1 "nonimmediate_operand")]
+ "TARGET_AVX512FP16"
+{
+ emit_insn (gen_sse4_1_roundhf2 (operands[0],
+ operands[1],
+ GEN_INT (ROUND_MXCSR | ROUND_NO_EXC)));
+ DONE;
+})
+
(define_expand "nearbyint<mode>2"
[(use (match_operand:MODEF 0 "register_operand"))
(use (match_operand:MODEF 1 "nonimmediate_operand"))]
diff --git a/gcc/testsuite/gcc.target/i386/avx512fp16-builtin-round-1.c b/gcc/testsuite/gcc.target/i386/avx512fp16-builtin-round-1.c
index 3cab1526967..a1c6636e354 100644
--- a/gcc/testsuite/gcc.target/i386/avx512fp16-builtin-round-1.c
+++ b/gcc/testsuite/gcc.target/i386/avx512fp16-builtin-round-1.c
@@ -25,7 +25,21 @@ f4 (_Float16 x)
return __builtin_roundevenf16 (x);
}
+_Float16
+f5 (_Float16 x)
+{
+ return __builtin_rintf16 (x);
+}
+
+_Float16
+f6 (_Float16 x)
+{
+ return __builtin_nearbyintf16 (x);
+}
+
/* { dg-final { scan-assembler-times "vrndscalesh\[ \\t\]+\\\$11\[^\n\r\]*xmm\[0-9\]" 1 } } */
/* { dg-final { scan-assembler-times "vrndscalesh\[ \\t\]+\\\$10\[^\n\r\]*xmm\[0-9\]" 1 } } */
/* { dg-final { scan-assembler-times "vrndscalesh\[ \\t\]+\\\$9\[^\n\r\]*xmm\[0-9\]" 1 } } */
/* { dg-final { scan-assembler-times "vrndscalesh\[ \\t\]+\\\$8\[^\n\r\]*xmm\[0-9\]" 1 } } */
+/* { dg-final { scan-assembler-times "vrndscalesh\[ \\t\]+\\\$4\[^\n\r\]*xmm\[0-9\]" 1 } } */
+/* { dg-final { scan-assembler-times "vrndscalesh\[ \\t\]+\\\$12\[^\n\r\]*xmm\[0-9\]" 1 } } */
--
2.27.0
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH 2/7] AVX512FP16: Add expander for fmahf4
2021-09-23 5:46 [PATCH 0/7] AVX512FP16: Support bunch of expanders for HFmode and vector HFmodes liuhongt
2021-09-23 5:46 ` [PATCH 1/7] AVX512FP16: Add expander for rint/nearbyinthf2 liuhongt
@ 2021-09-23 5:46 ` liuhongt
2021-09-23 5:46 ` [PATCH 3/7] AVX512FP16: Add expander for smin/maxhf3 liuhongt
` (4 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: liuhongt @ 2021-09-23 5:46 UTC (permalink / raw)
To: gcc-patches
gcc/ChangeLog:
* config/i386/sse.md (FMAMODEM): extend to handle FP16.
(VFH_SF_AVX512VL): Extend to handle HFmode.
(VF_SF_AVX512VL): Deleted.
gcc/testsuite/ChangeLog:
* gcc.target/i386/avx512fp16-fma-1.c: New test.
* gcc.target/i386/avx512fp16vl-fma-1.c: New test.
* gcc.target/i386/avx512fp16vl-fma-vectorize-1.c: New test.
---
gcc/config/i386/sse.md | 11 +--
.../gcc.target/i386/avx512fp16-fma-1.c | 69 ++++++++++++++++++
.../gcc.target/i386/avx512fp16vl-fma-1.c | 70 +++++++++++++++++++
.../i386/avx512fp16vl-fma-vectorize-1.c | 45 ++++++++++++
4 files changed, 190 insertions(+), 5 deletions(-)
create mode 100644 gcc/testsuite/gcc.target/i386/avx512fp16-fma-1.c
create mode 100644 gcc/testsuite/gcc.target/i386/avx512fp16vl-fma-1.c
create mode 100644 gcc/testsuite/gcc.target/i386/avx512fp16vl-fma-vectorize-1.c
diff --git a/gcc/config/i386/sse.md b/gcc/config/i386/sse.md
index 9079613e829..1ca95984afc 100644
--- a/gcc/config/i386/sse.md
+++ b/gcc/config/i386/sse.md
@@ -4650,7 +4650,11 @@ (define_mode_iterator FMAMODEM
(V8SF "TARGET_FMA || TARGET_FMA4 || TARGET_AVX512VL")
(V4DF "TARGET_FMA || TARGET_FMA4 || TARGET_AVX512VL")
(V16SF "TARGET_AVX512F")
- (V8DF "TARGET_AVX512F")])
+ (V8DF "TARGET_AVX512F")
+ (HF "TARGET_AVX512FP16")
+ (V8HF "TARGET_AVX512FP16 && TARGET_AVX512VL")
+ (V16HF "TARGET_AVX512FP16 && TARGET_AVX512VL")
+ (V32HF "TARGET_AVX512FP16")])
(define_expand "fma<mode>4"
[(set (match_operand:FMAMODEM 0 "register_operand")
@@ -4758,14 +4762,11 @@ (define_insn "*fma_fmadd_<mode>"
(set_attr "mode" "<MODE>")])
;; Suppose AVX-512F as baseline
-(define_mode_iterator VF_SF_AVX512VL
- [SF V16SF (V8SF "TARGET_AVX512VL") (V4SF "TARGET_AVX512VL")
- DF V8DF (V4DF "TARGET_AVX512VL") (V2DF "TARGET_AVX512VL")])
-
(define_mode_iterator VFH_SF_AVX512VL
[(V32HF "TARGET_AVX512FP16")
(V16HF "TARGET_AVX512FP16 && TARGET_AVX512VL")
(V8HF "TARGET_AVX512FP16 && TARGET_AVX512VL")
+ (HF "TARGET_AVX512FP16")
SF V16SF (V8SF "TARGET_AVX512VL") (V4SF "TARGET_AVX512VL")
DF V8DF (V4DF "TARGET_AVX512VL") (V2DF "TARGET_AVX512VL")])
diff --git a/gcc/testsuite/gcc.target/i386/avx512fp16-fma-1.c b/gcc/testsuite/gcc.target/i386/avx512fp16-fma-1.c
new file mode 100644
index 00000000000..d78d7629838
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/avx512fp16-fma-1.c
@@ -0,0 +1,69 @@
+/* { dg-do compile } */
+/* { dg-options "-Ofast -mavx512fp16" } */
+
+typedef _Float16 v32hf __attribute__ ((__vector_size__ (64)));
+
+_Float16
+foo1 (_Float16 a, _Float16 b, _Float16 c)
+{
+ return a * b + c;
+}
+
+/* { dg-final { scan-assembler-times "vfmadd132sh\[^\n\r\]*xmm\[0-9\]" 1 } } */
+
+_Float16
+foo2 (_Float16 a, _Float16 b, _Float16 c)
+{
+ return -a * b + c;
+}
+
+/* { dg-final { scan-assembler-times "vfnmadd132sh\[^\n\r\]*xmm\[0-9\]" 1 } } */
+
+_Float16
+foo3 (_Float16 a, _Float16 b, _Float16 c)
+{
+ return a * b - c;
+}
+
+/* { dg-final { scan-assembler-times "vfmsub132sh\[^\n\r\]*xmm\[0-9\]" 1 } } */
+
+_Float16
+foo4 (_Float16 a, _Float16 b, _Float16 c)
+{
+ return -a * b - c;
+}
+
+/* { dg-final { scan-assembler-times "vfnmsub132sh\[^\n\r\]*xmm\[0-9\]" 1 } } */
+
+v32hf
+foo5 (v32hf a, v32hf b, v32hf c)
+{
+ return a * b + c;
+}
+
+/* { dg-final { scan-assembler-times "vfmadd132ph\[^\n\r\]*zmm\[0-9\]" 1 } } */
+
+v32hf
+foo6 (v32hf a, v32hf b, v32hf c)
+{
+ return -a * b + c;
+}
+
+/* { dg-final { scan-assembler-times "vfnmadd132ph\[^\n\r\]*zmm\[0-9\]" 1 } } */
+
+v32hf
+foo7 (v32hf a, v32hf b, v32hf c)
+{
+ return a * b - c;
+}
+
+/* { dg-final { scan-assembler-times "vfmsub132ph\[^\n\r\]*zmm\[0-9\]" 1 } } */
+
+v32hf
+foo8 (v32hf a, v32hf b, v32hf c)
+{
+ return -a * b - c;
+}
+
+/* { dg-final { scan-assembler-times "vfnmsub132ph\[^\n\r\]*zmm\[0-9\]" 1 } } */
+
diff --git a/gcc/testsuite/gcc.target/i386/avx512fp16vl-fma-1.c b/gcc/testsuite/gcc.target/i386/avx512fp16vl-fma-1.c
new file mode 100644
index 00000000000..1a832f37d6c
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/avx512fp16vl-fma-1.c
@@ -0,0 +1,70 @@
+/* { dg-do compile } */
+/* { dg-options "-Ofast -mavx512fp16 -mavx512vl" } */
+
+typedef _Float16 v8hf __attribute__ ((__vector_size__ (16)));
+typedef _Float16 v16hf __attribute__ ((__vector_size__ (32)));
+
+v8hf
+foo1 (v8hf a, v8hf b, v8hf c)
+{
+ return a * b + c;
+}
+
+/* { dg-final { scan-assembler-times "vfmadd132ph\[^\n\r\]*xmm\[0-9\]" 1 } } */
+
+v8hf
+foo2 (v8hf a, v8hf b, v8hf c)
+{
+ return -a * b + c;
+}
+
+/* { dg-final { scan-assembler-times "vfnmadd132ph\[^\n\r\]*xmm\[0-9\]" 1 } } */
+
+v8hf
+foo3 (v8hf a, v8hf b, v8hf c)
+{
+ return a * b - c;
+}
+
+/* { dg-final { scan-assembler-times "vfmsub132ph\[^\n\r\]*xmm\[0-9\]" 1 } } */
+
+v8hf
+foo4 (v8hf a, v8hf b, v8hf c)
+{
+ return -a * b - c;
+}
+
+/* { dg-final { scan-assembler-times "vfnmsub132ph\[^\n\r\]*xmm\[0-9\]" 1 } } */
+
+v16hf
+foo5 (v16hf a, v16hf b, v16hf c)
+{
+ return a * b + c;
+}
+
+/* { dg-final { scan-assembler-times "vfmadd132ph\[^\n\r\]*ymm\[0-9\]" 1 } } */
+
+v16hf
+foo6 (v16hf a, v16hf b, v16hf c)
+{
+ return -a * b + c;
+}
+
+/* { dg-final { scan-assembler-times "vfnmadd132ph\[^\n\r\]*ymm\[0-9\]" 1 } } */
+
+v16hf
+foo7 (v16hf a, v16hf b, v16hf c)
+{
+ return a * b - c;
+}
+
+/* { dg-final { scan-assembler-times "vfmsub132ph\[^\n\r\]*ymm\[0-9\]" 1 } } */
+
+v16hf
+foo8 (v16hf a, v16hf b, v16hf c)
+{
+ return -a * b - c;
+}
+
+/* { dg-final { scan-assembler-times "vfnmsub132ph\[^\n\r\]*ymm\[0-9\]" 1 } } */
+
diff --git a/gcc/testsuite/gcc.target/i386/avx512fp16vl-fma-vectorize-1.c b/gcc/testsuite/gcc.target/i386/avx512fp16vl-fma-vectorize-1.c
new file mode 100644
index 00000000000..d0b8bec34f1
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/avx512fp16vl-fma-vectorize-1.c
@@ -0,0 +1,45 @@
+/* { dg-do compile } */
+/* { dg-options "-Ofast -mavx512fp16 -mavx512vl" } */
+
+typedef _Float16 v8hf __attribute__ ((__vector_size__ (16)));
+typedef _Float16 v16hf __attribute__ ((__vector_size__ (32)));
+
+void
+foo1 (_Float16* __restrict pa, _Float16* __restrict pb,
+ _Float16* __restrict pc, _Float16* __restrict pd)
+{
+ for (int i = 0; i != 8; i++)
+ pd[i] = pa[i] * pb[i] + pc[i];
+}
+
+/* { dg-final { scan-assembler-times "vfmadd132ph\[^\n\r\]*xmm\[0-9\]" 1 } } */
+
+void
+foo2 (_Float16* __restrict pa, _Float16* __restrict pb,
+ _Float16* __restrict pc, _Float16* __restrict pd)
+{
+ for (int i = 0; i != 8; i++)
+ pd[i] = -pa[i] * pb[i] + pc[i];
+}
+
+/* { dg-final { scan-assembler-times "vfnmadd132ph\[^\n\r\]*xmm\[0-9\]" 1 } } */
+
+void
+foo3 (_Float16* __restrict pa, _Float16* __restrict pb,
+ _Float16* __restrict pc, _Float16* __restrict pd)
+{
+ for (int i = 0; i != 8; i++)
+ pd[i] = pa[i] * pb[i] - pc[i];
+}
+
+/* { dg-final { scan-assembler-times "vfmsub132ph\[^\n\r\]*xmm\[0-9\]" 1 } } */
+
+void
+foo4 (_Float16* __restrict pa, _Float16* __restrict pb,
+ _Float16* __restrict pc, _Float16* __restrict pd)
+{
+ for (int i = 0; i != 8; i++)
+ pd[i] = -pa[i] * pb[i] - pc[i];
+}
+
+/* { dg-final { scan-assembler-times "vfnmsub132ph\[^\n\r\]*xmm\[0-9\]" 1 } } */
--
2.27.0
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH 3/7] AVX512FP16: Add expander for smin/maxhf3.
2021-09-23 5:46 [PATCH 0/7] AVX512FP16: Support bunch of expanders for HFmode and vector HFmodes liuhongt
2021-09-23 5:46 ` [PATCH 1/7] AVX512FP16: Add expander for rint/nearbyinthf2 liuhongt
2021-09-23 5:46 ` [PATCH 2/7] AVX512FP16: Add expander for fmahf4 liuhongt
@ 2021-09-23 5:46 ` liuhongt
2021-09-23 5:46 ` [PATCH 4/7] AVX512FP16: Add fix(uns)?_truncmn2 for HF scalar and vector modes liuhongt
` (3 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: liuhongt @ 2021-09-23 5:46 UTC (permalink / raw)
To: gcc-patches; +Cc: crazylht, hjl.tools, Hongyu Wang
From: Hongyu Wang <hongyu.wang@intel.com>
gcc/ChangeLog:
* config/i386/i386.md (<code>hf3): New expander.
gcc/testsuite/ChangeLog:
* gcc.target/i386/avx512fp16-builtin-minmax-1.c: New test.
---
gcc/config/i386/i386.md | 11 ++++++
.../i386/avx512fp16-builtin-minmax-1.c | 35 +++++++++++++++++++
2 files changed, 46 insertions(+)
create mode 100644 gcc/testsuite/gcc.target/i386/avx512fp16-builtin-minmax-1.c
diff --git a/gcc/config/i386/i386.md b/gcc/config/i386/i386.md
index 4b13a59be82..a087e557d7f 100644
--- a/gcc/config/i386/i386.md
+++ b/gcc/config/i386/i386.md
@@ -19946,6 +19946,17 @@ (define_insn "<code><mode>3"
(set_attr "type" "sseadd")
(set_attr "mode" "<MODE>")])
+(define_insn "<code>hf3"
+ [(set (match_operand:HF 0 "register_operand" "=v")
+ (smaxmin:HF
+ (match_operand:HF 1 "nonimmediate_operand" "%v")
+ (match_operand:HF 2 "nonimmediate_operand" "vm")))]
+ "TARGET_AVX512FP16"
+ "v<maxmin_float>sh\t{%2, %1, %0|%0, %1, %2}"
+ [(set_attr "prefix" "evex")
+ (set_attr "type" "sseadd")
+ (set_attr "mode" "HF")])
+
;; These versions of the min/max patterns implement exactly the operations
;; min = (op1 < op2 ? op1 : op2)
;; max = (!(op1 < op2) ? op1 : op2)
diff --git a/gcc/testsuite/gcc.target/i386/avx512fp16-builtin-minmax-1.c b/gcc/testsuite/gcc.target/i386/avx512fp16-builtin-minmax-1.c
new file mode 100644
index 00000000000..90080e44216
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/avx512fp16-builtin-minmax-1.c
@@ -0,0 +1,35 @@
+/* { dg-do compile } */
+/* { dg-options "-Ofast -mavx512fp16 -mprefer-vector-width=512" } */
+
+_Float16
+minf1 (_Float16 a, _Float16 b)
+{
+ return __builtin_fminf16 (a, b);
+}
+
+void
+minf2 (_Float16* __restrict psrc1, _Float16* __restrict psrc2,
+ _Float16* __restrict pdst)
+{
+ for (int i = 0; i != 32; i++)
+ pdst[i] = __builtin_fminf16 (psrc1[i], psrc2[i]);
+}
+
+_Float16
+maxf1 (_Float16 a, _Float16 b)
+{
+ return __builtin_fmaxf16 (a, b);
+}
+
+void
+maxf2 (_Float16* __restrict psrc1, _Float16* __restrict psrc2,
+ _Float16* __restrict pdst)
+{
+ for (int i = 0; i != 32; i++)
+ pdst[i] = __builtin_fmaxf16 (psrc1[i], psrc2[i]);
+}
+
+/* { dg-final { scan-assembler-times "vmaxsh\[^\n\r\]*xmm\[0-9\]" 1 } } */
+/* { dg-final { scan-assembler-times "vmaxph\[^\n\r\]*zmm\[0-9\]" 1 } } */
+/* { dg-final { scan-assembler-times "vminsh\[^\n\r\]*xmm\[0-9\]" 1 } } */
+/* { dg-final { scan-assembler-times "vminph\[^\n\r\]*zmm\[0-9\]" 1 } } */
--
2.27.0
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH 4/7] AVX512FP16: Add fix(uns)?_truncmn2 for HF scalar and vector modes
2021-09-23 5:46 [PATCH 0/7] AVX512FP16: Support bunch of expanders for HFmode and vector HFmodes liuhongt
` (2 preceding siblings ...)
2021-09-23 5:46 ` [PATCH 3/7] AVX512FP16: Add expander for smin/maxhf3 liuhongt
@ 2021-09-23 5:46 ` liuhongt
2021-09-23 5:46 ` [PATCH 5/7] AVX512FP16: Add float(uns)?mn2 expander liuhongt
` (2 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: liuhongt @ 2021-09-23 5:46 UTC (permalink / raw)
To: gcc-patches; +Cc: crazylht, hjl.tools, Hongyu Wang
From: Hongyu Wang <hongyu.wang@intel.com>
NB: 64bit/32bit vectorize for HFmode is not supported for now, will
adjust this patch when V2HF/V4HF operations supported.
gcc/ChangeLog:
* config/i386/i386.md (fix<fixunssuffix>_trunchf<mode>2): New expander.
(fixuns_trunchfhi2): Likewise.
(*fixuns_trunchfsi2zext): New define_insn.
* config/i386/sse.md (ssePHmodelower): New mode_attr.
(fix<fixunssuffix>_trunc<ssePHmodelower><mode>2):
New expander for same element vector fix_truncate.
(fix<fixunssuffix>_trunc<ssePHmodelower><mode>2):
Likewise for V4HF to V4SI/V4DI fix_truncate.
(fix<fixunssuffix>_truncv2hfv2di2):
Likeise for V2HF to V2DI fix_truncate.
gcc/testsuite/ChangeLog:
* gcc.target/i386/avx512fp16-trunchf.c: New test.
* gcc.target/i386/avx512fp16-truncvnhf.c: Ditto.
---
gcc/config/i386/i386.md | 29 +++++++++
gcc/config/i386/sse.md | 43 +++++++++++++
.../gcc.target/i386/avx512fp16-trunchf.c | 59 ++++++++++++++++++
.../gcc.target/i386/avx512fp16-truncvnhf.c | 61 +++++++++++++++++++
4 files changed, 192 insertions(+)
create mode 100644 gcc/testsuite/gcc.target/i386/avx512fp16-trunchf.c
create mode 100644 gcc/testsuite/gcc.target/i386/avx512fp16-truncvnhf.c
diff --git a/gcc/config/i386/i386.md b/gcc/config/i386/i386.md
index a087e557d7f..c6279e620c9 100644
--- a/gcc/config/i386/i386.md
+++ b/gcc/config/i386/i386.md
@@ -4810,6 +4810,16 @@ (define_expand "fix_trunc<mode>di2"
}
})
+(define_insn "fix<fixunssuffix>_trunchf<mode>2"
+ [(set (match_operand:SWI48 0 "register_operand" "=r")
+ (any_fix:SWI48
+ (match_operand:HF 1 "nonimmediate_operand" "vm")))]
+ "TARGET_AVX512FP16"
+ "vcvttsh2<fixsuffix>si\t{%1, %0|%0, %1}"
+ [(set_attr "type" "sseicvt")
+ (set_attr "prefix" "evex")
+ (set_attr "mode" "<MODE>")])
+
;; Signed conversion to SImode.
(define_expand "fix_truncxfsi2"
@@ -4917,6 +4927,17 @@ (define_insn "fixuns_trunc<mode>si2_avx512f"
(set_attr "prefix" "evex")
(set_attr "mode" "SI")])
+(define_insn "*fixuns_trunchfsi2zext"
+ [(set (match_operand:DI 0 "register_operand" "=r")
+ (zero_extend:DI
+ (unsigned_fix:SI
+ (match_operand:HF 1 "nonimmediate_operand" "vm"))))]
+ "TARGET_64BIT && TARGET_AVX512FP16"
+ "vcvttsh2usi\t{%1, %k0|%k0, %1}"
+ [(set_attr "type" "sseicvt")
+ (set_attr "prefix" "evex")
+ (set_attr "mode" "SI")])
+
(define_insn "*fixuns_trunc<mode>si2_avx512f_zext"
[(set (match_operand:DI 0 "register_operand" "=r")
(zero_extend:DI
@@ -4949,6 +4970,14 @@ (define_insn_and_split "*fixuns_trunc<mode>_1"
;; Without these patterns, we'll try the unsigned SI conversion which
;; is complex for SSE, rather than the signed SI conversion, which isn't.
+(define_expand "fixuns_trunchfhi2"
+ [(set (match_dup 2)
+ (fix:SI (match_operand:HF 1 "nonimmediate_operand")))
+ (set (match_operand:HI 0 "nonimmediate_operand")
+ (subreg:HI (match_dup 2) 0))]
+ "TARGET_AVX512FP16"
+ "operands[2] = gen_reg_rtx (SImode);")
+
(define_expand "fixuns_trunc<mode>hi2"
[(set (match_dup 2)
(fix:SI (match_operand:MODEF 1 "nonimmediate_operand")))
diff --git a/gcc/config/i386/sse.md b/gcc/config/i386/sse.md
index 1ca95984afc..f8a5f197f3c 100644
--- a/gcc/config/i386/sse.md
+++ b/gcc/config/i386/sse.md
@@ -1034,6 +1034,13 @@ (define_mode_attr ssePHmode
(V8DI "V8HF") (V4DI "V8HF") (V2DI "V8HF")
(V8DF "V8HF") (V16SF "V16HF") (V8SF "V8HF")])
+;; Mapping of vector modes to vector hf modes of same element.
+(define_mode_attr ssePHmodelower
+ [(V32HI "v32hf") (V16HI "v16hf") (V8HI "v8hf")
+ (V16SI "v16hf") (V8SI "v8hf") (V4SI "v4hf")
+ (V8DI "v8hf") (V4DI "v4hf") (V2DI "v2hf")
+ (V8DF "v8hf") (V16SF "v16hf") (V8SF "v8hf")])
+
;; Mapping of vector modes to packed single mode of the same size
(define_mode_attr ssePSmode
[(V16SI "V16SF") (V8DF "V16SF")
@@ -6175,6 +6182,12 @@ (define_insn "avx512fp16_vcvt<floatsuffix>si2sh<rex64namesuffix><round_name>"
(set_attr "prefix" "evex")
(set_attr "mode" "HF")])
+(define_expand "fix<fixunssuffix>_trunc<ssePHmodelower><mode>2"
+ [(set (match_operand:VI2H_AVX512VL 0 "register_operand")
+ (any_fix:VI2H_AVX512VL
+ (match_operand:<ssePHmode> 1 "nonimmediate_operand")))]
+ "TARGET_AVX512FP16")
+
(define_insn "avx512fp16_fix<fixunssuffix>_trunc<mode>2<mask_name><round_saeonly_name>"
[(set (match_operand:VI2H_AVX512VL 0 "register_operand" "=v")
(any_fix:VI2H_AVX512VL
@@ -6185,6 +6198,21 @@ (define_insn "avx512fp16_fix<fixunssuffix>_trunc<mode>2<mask_name><round_saeonly
(set_attr "prefix" "evex")
(set_attr "mode" "<sseinsnmode>")])
+(define_expand "fix<fixunssuffix>_truncv4hf<mode>2"
+ [(set (match_operand:VI4_128_8_256 0 "register_operand")
+ (any_fix:VI4_128_8_256
+ (match_operand:V4HF 1 "nonimmediate_operand")))]
+ "TARGET_AVX512FP16 && TARGET_AVX512VL"
+{
+ if (!MEM_P (operands[1]))
+ {
+ operands[1] = lowpart_subreg (V8HFmode, operands[1], V4HFmode);
+ emit_insn (gen_avx512fp16_fix<fixunssuffix>_trunc<mode>2 (operands[0],
+ operands[1]));
+ DONE;
+ }
+})
+
(define_insn "avx512fp16_fix<fixunssuffix>_trunc<mode>2<mask_name>"
[(set (match_operand:VI4_128_8_256 0 "register_operand" "=v")
(any_fix:VI4_128_8_256
@@ -6207,6 +6235,21 @@ (define_insn "*avx512fp16_fix<fixunssuffix>_trunc<mode>2_load<mask_name>"
(set_attr "prefix" "evex")
(set_attr "mode" "<sseinsnmode>")])
+(define_expand "fix<fixunssuffix>_truncv2hfv2di2"
+ [(set (match_operand:V2DI 0 "register_operand")
+ (any_fix:V2DI
+ (match_operand:V2HF 1 "nonimmediate_operand")))]
+ "TARGET_AVX512FP16 && TARGET_AVX512VL"
+{
+ if (!MEM_P (operands[1]))
+ {
+ operands[1] = lowpart_subreg (V8HFmode, operands[1], V2HFmode);
+ emit_insn (gen_avx512fp16_fix<fixunssuffix>_truncv2di2 (operands[0],
+ operands[1]));
+ DONE;
+ }
+})
+
(define_insn "avx512fp16_fix<fixunssuffix>_truncv2di2<mask_name>"
[(set (match_operand:V2DI 0 "register_operand" "=v")
(any_fix:V2DI
diff --git a/gcc/testsuite/gcc.target/i386/avx512fp16-trunchf.c b/gcc/testsuite/gcc.target/i386/avx512fp16-trunchf.c
new file mode 100644
index 00000000000..2c025b7803c
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/avx512fp16-trunchf.c
@@ -0,0 +1,59 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -mavx512fp16" } */
+/* { dg-final { scan-assembler-times "vcvttsh2si\[ \\t\]+\[^\{\n\]*(?:%xmm\[0-9\]|\\(%esp\\))+, %eax(?:\n|\[ \\t\]+#)" 3 } } */
+/* { dg-final { scan-assembler-times "vcvttsh2usi\[ \\t\]+\[^\{\n\]*(?:%xmm\[0-9\]|\\(%esp\\))+, %eax(?:\n|\[ \\t\]+#)" 2 } } */
+/* { dg-final { scan-assembler-times "vcvttsh2si\[ \\t\]+\[^\{\n\]*%xmm\[0-9\]+, %rax(?:\n|\[ \\t\]+#)" 1 { target { ! ia32 } } } } */
+/* { dg-final { scan-assembler-times "vcvttsh2usi\[ \\t\]+\[^\{\n\]*%xmm\[0-9\]+, %rax(?:\n|\[ \\t\]+#)" 1 { target { ! ia32 } } } } */
+/* { dg-final { scan-assembler "xorl\[ \\t\]+%edx, %edx" { target ia32 } } } */
+
+#include <immintrin.h>
+
+short
+__attribute__ ((noinline, noclone))
+trunc_f16_to_si16 (_Float16 f)
+{
+ return f;
+}
+
+unsigned short
+__attribute__ ((noinline, noclone))
+trunc_f16_to_su16 (_Float16 f)
+{
+ return f;
+}
+
+int
+__attribute__ ((noinline, noclone))
+trunc_f16_to_si32 (_Float16 f)
+{
+ return f;
+}
+
+unsigned int
+__attribute__ ((noinline, noclone))
+trunc_f16_to_su32 (_Float16 f)
+{
+ return f;
+}
+
+long long
+__attribute__ ((noinline, noclone))
+trunc_f16_to_si64 (_Float16 f)
+{
+ return f;
+}
+
+unsigned long long
+__attribute__ ((noinline, noclone))
+trunc_f16_to_su64 (_Float16 f)
+{
+ return f;
+}
+
+unsigned long long
+__attribute__ ((noinline, noclone))
+trunc_f16_to_su64_zext (_Float16 f)
+{
+ return (unsigned int) f;
+}
+
diff --git a/gcc/testsuite/gcc.target/i386/avx512fp16-truncvnhf.c b/gcc/testsuite/gcc.target/i386/avx512fp16-truncvnhf.c
new file mode 100644
index 00000000000..ee55cd12300
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/avx512fp16-truncvnhf.c
@@ -0,0 +1,61 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -mavx512fp16 -mavx512vl -ftree-slp-vectorize -mprefer-vector-width=512" } */
+
+extern long long di[8];
+extern unsigned long long udi[8];
+extern int si[16];
+extern unsigned int usi[16];
+extern short hi[32];
+extern unsigned short uhi[32];
+extern _Float16 hf[32];
+
+#define DO_PRAGMA(X) _Pragma(#X)
+
+#define FIX_TRUNCHFVV(size, mode) \
+ void __attribute__ ((noinline, noclone)) \
+fix_trunc##size##hf##v##size##mode () \
+{\
+ int i; \
+ DO_PRAGMA (GCC unroll size) \
+ for (i = 0; i < size; i++) \
+ mode[i] = hf[i]; \
+}
+
+FIX_TRUNCHFVV(32, hi)
+FIX_TRUNCHFVV(16, hi)
+FIX_TRUNCHFVV(8, hi)
+FIX_TRUNCHFVV(16, si)
+FIX_TRUNCHFVV(8, si)
+FIX_TRUNCHFVV(4, si)
+FIX_TRUNCHFVV(8, di)
+FIX_TRUNCHFVV(4, di)
+FIX_TRUNCHFVV(2, di)
+
+FIX_TRUNCHFVV(32, uhi)
+FIX_TRUNCHFVV(16, uhi)
+FIX_TRUNCHFVV(8, uhi)
+FIX_TRUNCHFVV(16, usi)
+FIX_TRUNCHFVV(8, usi)
+FIX_TRUNCHFVV(4, usi)
+FIX_TRUNCHFVV(8, udi)
+FIX_TRUNCHFVV(4, udi)
+FIX_TRUNCHFVV(2, udi)
+
+/* { dg-final { scan-assembler-times "vcvttph2qq\[ \\t\]+\[^\{\n\]*\[^\n\r]*%zmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 } } */
+/* { dg-final { scan-assembler-times "vcvttph2uqq\[ \\t\]+\[^\{\n\]*\[^\n\r]*%zmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 } } */
+/* { dg-final { scan-assembler-times "vcvttph2qq\[ \\t\]+\[^\{\n\]*\[^\n\r]*%ymm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 { xfail *-*-* } } } */
+/* { dg-final { scan-assembler-times "vcvttph2uqq\[ \\t\]+\[^\{\n\]*\[^\n\r]*%ymm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 { xfail *-*-* } } } */
+/* { dg-final { scan-assembler-times "vcvttph2qq\[ \\t\]+\[^\{\n\]*\[^\n\r]*%xmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 { xfail *-*-* } } } */
+/* { dg-final { scan-assembler-times "vcvttph2uqq\[ \\t\]+\[^\{\n\]*\[^\n\r]*%xmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 { xfail *-*-* } } } */
+/* { dg-final { scan-assembler-times "vcvttph2dq\[ \\t\]+\[^\{\n\]*\[^\n\r]*%zmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 } } */
+/* { dg-final { scan-assembler-times "vcvttph2udq\[ \\t\]+\[^\{\n\]*\[^\n\r]*%zmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 } } */
+/* { dg-final { scan-assembler-times "vcvttph2dq\[ \\t\]+\[^\{\n\]*\[^\n\r]*%ymm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 } } */
+/* { dg-final { scan-assembler-times "vcvttph2udq\[ \\t\]+\[^\{\n\]*\[^\n\r]*%ymm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 } } */
+/* { dg-final { scan-assembler-times "vcvttph2dq\[ \\t\]+\[^\{\n\]*\[^\n\r]*%xmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 { xfail *-*-* } } } */
+/* { dg-final { scan-assembler-times "vcvttph2udq\[ \\t\]+\[^\{\n\]*\[^\n\r]*%xmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 { xfail *-*-* } } } */
+/* { dg-final { scan-assembler-times "vcvttph2w\[ \\t\]+\[^\{\n\]*\[^\n\r]*%zmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 } } */
+/* { dg-final { scan-assembler-times "vcvttph2uw\[ \\t\]+\[^\{\n\]*\[^\n\r]*%zmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 } } */
+/* { dg-final { scan-assembler-times "vcvttph2w\[ \\t\]+\[^\{\n\]*\[^\n\r]*%ymm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 } } */
+/* { dg-final { scan-assembler-times "vcvttph2uw\[ \\t\]+\[^\{\n\]*\[^\n\r]*%ymm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 } } */
+/* { dg-final { scan-assembler-times "vcvttph2w\[ \\t\]+\[^\{\n\]*\[^\n\r]*%xmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 } } */
+/* { dg-final { scan-assembler-times "vcvttph2uw\[ \\t\]+\[^\{\n\]*\[^\n\r]*%xmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 } } */
--
2.27.0
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH 5/7] AVX512FP16: Add float(uns)?mn2 expander
2021-09-23 5:46 [PATCH 0/7] AVX512FP16: Support bunch of expanders for HFmode and vector HFmodes liuhongt
` (3 preceding siblings ...)
2021-09-23 5:46 ` [PATCH 4/7] AVX512FP16: Add fix(uns)?_truncmn2 for HF scalar and vector modes liuhongt
@ 2021-09-23 5:46 ` liuhongt
2021-09-23 5:46 ` [PATCH 6/7] AVX512FP16: add truncmn2/extendmn2 expanders liuhongt
2021-09-23 5:46 ` [PATCH 7/7] AVX512FP16: Enable vec_cmpmn/vcondmn expanders for HF modes liuhongt
6 siblings, 0 replies; 8+ messages in thread
From: liuhongt @ 2021-09-23 5:46 UTC (permalink / raw)
To: gcc-patches; +Cc: crazylht, hjl.tools, Hongyu Wang
From: Hongyu Wang <hongyu.wang@intel.com>
gcc/ChangeLog:
* config/i386/sse.md (float<floatunssuffix><mode><ssePHmodelower>2):
New expander.
(avx512fp16_vcvt<floatsuffix><sseintconvert>2ph_<mode>):
Rename to ...
(float<floatunssuffix><mode>v4hf2): ... this, and drop constraints.
(avx512fp16_vcvt<floatsuffix>qq2ph_v2di): Rename to ...
(float<floatunssuffix>v2div2hf2): ... this, and likewise.
gcc/testsuite/ChangeLog:
* gcc.target/i386/avx512fp16-floatvnhf.c: New test.
---
gcc/config/i386/sse.md | 46 +++++++++++---
.../gcc.target/i386/avx512fp16-floatvnhf.c | 61 +++++++++++++++++++
2 files changed, 99 insertions(+), 8 deletions(-)
create mode 100644 gcc/testsuite/gcc.target/i386/avx512fp16-floatvnhf.c
diff --git a/gcc/config/i386/sse.md b/gcc/config/i386/sse.md
index f8a5f197f3c..66062dc3bcf 100644
--- a/gcc/config/i386/sse.md
+++ b/gcc/config/i386/sse.md
@@ -6006,6 +6006,12 @@ (define_insn "avx512fp16_vcvtph2<sseintconvertsignprefix><sseintconvert>_<mode><
(set_attr "prefix" "evex")
(set_attr "mode" "<sseinsnmode>")])
+(define_expand "float<floatunssuffix><mode><ssePHmodelower>2"
+ [(set (match_operand:<ssePHmode> 0 "register_operand")
+ (any_float:<ssePHmode>
+ (match_operand:VI2H_AVX512VL 1 "nonimmediate_operand")))]
+ "TARGET_AVX512FP16")
+
(define_insn "avx512fp16_vcvt<floatsuffix><sseintconvert>2ph_<mode><mask_name><round_name>"
[(set (match_operand:<ssePHmode> 0 "register_operand" "=v")
(any_float:<ssePHmode>
@@ -6016,11 +6022,23 @@ (define_insn "avx512fp16_vcvt<floatsuffix><sseintconvert>2ph_<mode><mask_name><r
(set_attr "prefix" "evex")
(set_attr "mode" "<sseinsnmode>")])
-(define_expand "avx512fp16_vcvt<floatsuffix><sseintconvert>2ph_<mode>"
- [(set (match_operand:V8HF 0 "register_operand" "=v")
+(define_expand "float<floatunssuffix><mode>v4hf2"
+ [(set (match_operand:V4HF 0 "register_operand")
+ (any_float:V4HF
+ (match_operand:VI4_128_8_256 1 "vector_operand")))]
+ "TARGET_AVX512FP16 && TARGET_AVX512VL"
+{
+ operands[0] = lowpart_subreg (V8HFmode, operands[0], V4HFmode);
+ emit_insn (gen_avx512fp16_float<floatunssuffix><mode>v4hf2 (operands[0],
+ operands[1]));
+ DONE;
+})
+
+(define_expand "avx512fp16_float<floatunssuffix><mode>v4hf2"
+ [(set (match_operand:V8HF 0 "register_operand")
(vec_concat:V8HF
- (any_float:V4HF (match_operand:VI4_128_8_256 1 "vector_operand" "vm"))
- (match_dup 2)))]
+ (any_float:V4HF (match_operand:VI4_128_8_256 1 "vector_operand"))
+ (match_dup 2)))]
"TARGET_AVX512FP16 && TARGET_AVX512VL"
"operands[2] = CONST0_RTX (V4HFmode);")
@@ -6079,11 +6097,23 @@ (define_insn "*avx512fp16_vcvt<floatsuffix><sseintconvert>2ph_<mode>_mask_1"
(set_attr "prefix" "evex")
(set_attr "mode" "<sseinsnmode>")])
-(define_expand "avx512fp16_vcvt<floatsuffix>qq2ph_v2di"
- [(set (match_operand:V8HF 0 "register_operand" "=v")
+(define_expand "float<floatunssuffix>v2div2hf2"
+ [(set (match_operand:V2HF 0 "register_operand")
+ (any_float:V2HF
+ (match_operand:V2DI 1 "vector_operand")))]
+ "TARGET_AVX512FP16 && TARGET_AVX512VL"
+{
+ operands[0] = lowpart_subreg (V8HFmode, operands[0], V2HFmode);
+ emit_insn (gen_avx512fp16_float<floatunssuffix>v2div2hf2 (operands[0],
+ operands[1]));
+ DONE;
+})
+
+(define_expand "avx512fp16_float<floatunssuffix>v2div2hf2"
+ [(set (match_operand:V8HF 0 "register_operand")
(vec_concat:V8HF
- (any_float:V2HF (match_operand:V2DI 1 "vector_operand" "vm"))
- (match_dup 2)))]
+ (any_float:V2HF (match_operand:V2DI 1 "vector_operand"))
+ (match_dup 2)))]
"TARGET_AVX512FP16 && TARGET_AVX512VL"
"operands[2] = CONST0_RTX (V6HFmode);")
diff --git a/gcc/testsuite/gcc.target/i386/avx512fp16-floatvnhf.c b/gcc/testsuite/gcc.target/i386/avx512fp16-floatvnhf.c
new file mode 100644
index 00000000000..112ac3e74d5
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/avx512fp16-floatvnhf.c
@@ -0,0 +1,61 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -mavx512fp16 -mavx512vl -ftree-slp-vectorize -mprefer-vector-width=512" } */
+
+extern long long di[8];
+extern unsigned long long udi[8];
+extern int si[16];
+extern unsigned int usi[16];
+extern short hi[32];
+extern unsigned short uhi[32];
+extern _Float16 hf[32];
+
+#define DO_PRAGMA(X) _Pragma(#X)
+
+#define FLOATHFVV(size, mode) \
+ void __attribute__ ((noinline, noclone)) \
+float##v##size##mode##v##size##hf () \
+{\
+ int i; \
+ DO_PRAGMA (GCC unroll size) \
+ for (i = 0; i < size; i++) \
+ hf[i] = (_Float16) mode[i]; \
+}
+
+FLOATHFVV(32, hi)
+FLOATHFVV(16, hi)
+FLOATHFVV(8, hi)
+FLOATHFVV(16, si)
+FLOATHFVV(8, si)
+FLOATHFVV(4, si)
+FLOATHFVV(8, di)
+FLOATHFVV(4, di)
+FLOATHFVV(2, di)
+
+FLOATHFVV(32, uhi)
+FLOATHFVV(16, uhi)
+FLOATHFVV(8, uhi)
+FLOATHFVV(16, usi)
+FLOATHFVV(8, usi)
+FLOATHFVV(4, usi)
+FLOATHFVV(8, udi)
+FLOATHFVV(4, udi)
+FLOATHFVV(2, udi)
+
+/* { dg-final { scan-assembler-times "vcvtqq2phz\[ \\t\]+\[^\{\n\]*\[^\n\r]*%xmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 } } */
+/* { dg-final { scan-assembler-times "vcvtuqq2phz\[ \\t\]+\[^\{\n\]*\[^\n\r]*%xmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 } } */
+/* { dg-final { scan-assembler-times "vcvtqq2phy\[ \\t\]+\[^\{\n\]*\[^\n\r]*%xmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 { xfail *-*-* } } } */
+/* { dg-final { scan-assembler-times "vcvtuqq2phy\[ \\t\]+\[^\{\n\]*\[^\n\r]*%xmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 { xfail *-*-* } } } */
+/* { dg-final { scan-assembler-times "vcvtqq2phx\[ \\t\]+\[^\{\n\]*\[^\n\r]*%xmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 { xfail *-*-* } } } */
+/* { dg-final { scan-assembler-times "vcvtuqq2phx\[ \\t\]+\[^\{\n\]*\[^\n\r]*%xmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 { xfail *-*-* } } } */
+/* { dg-final { scan-assembler-times "vcvtdq2ph\[ \\t\]+\[^\{\n\]*\[^\n\r]*%ymm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 } } */
+/* { dg-final { scan-assembler-times "vcvtudq2ph\[ \\t\]+\[^\{\n\]*\[^\n\r]*%ymm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 } } */
+/* { dg-final { scan-assembler-times "vcvtdq2phy\[ \\t\]+\[^\{\n\]*\[^\n\r]*%xmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 } } */
+/* { dg-final { scan-assembler-times "vcvtudq2phy\[ \\t\]+\[^\{\n\]*\[^\n\r]*%xmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 } } */
+/* { dg-final { scan-assembler-times "vcvtdq2phx\[ \\t\]+\[^\{\n\]*\[^\n\r]*%xmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 { xfail *-*-* } } } */
+/* { dg-final { scan-assembler-times "vcvtudq2phx\[ \\t\]+\[^\{\n\]*\[^\n\r]*%xmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 { xfail *-*-* } } } */
+/* { dg-final { scan-assembler-times "vcvtw2ph\[ \\t\]+\[^\{\n\]*\[^\n\r]*%zmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 } } */
+/* { dg-final { scan-assembler-times "vcvtuw2ph\[ \\t\]+\[^\{\n\]*\[^\n\r]*%zmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 } } */
+/* { dg-final { scan-assembler-times "vcvtw2ph\[ \\t\]+\[^\{\n\]*\[^\n\r]*%ymm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 } } */
+/* { dg-final { scan-assembler-times "vcvtuw2ph\[ \\t\]+\[^\{\n\]*\[^\n\r]*%ymm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 } } */
+/* { dg-final { scan-assembler-times "vcvtw2ph\[ \\t\]+\[^\{\n\]*\[^\n\r]*%xmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 } } */
+/* { dg-final { scan-assembler-times "vcvtuw2ph\[ \\t\]+\[^\{\n\]*\[^\n\r]*%xmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 } } */
--
2.27.0
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH 6/7] AVX512FP16: add truncmn2/extendmn2 expanders
2021-09-23 5:46 [PATCH 0/7] AVX512FP16: Support bunch of expanders for HFmode and vector HFmodes liuhongt
` (4 preceding siblings ...)
2021-09-23 5:46 ` [PATCH 5/7] AVX512FP16: Add float(uns)?mn2 expander liuhongt
@ 2021-09-23 5:46 ` liuhongt
2021-09-23 5:46 ` [PATCH 7/7] AVX512FP16: Enable vec_cmpmn/vcondmn expanders for HF modes liuhongt
6 siblings, 0 replies; 8+ messages in thread
From: liuhongt @ 2021-09-23 5:46 UTC (permalink / raw)
To: gcc-patches; +Cc: crazylht, hjl.tools, Hongyu Wang
From: Hongyu Wang <hongyu.wang@intel.com>
gcc/ChangeLog:
* config/i386/sse.md (extend<ssePHmodelower><mode>2):
New expander.
(extendv4hf<mode>2): Likewise.
(extendv2hfv2df2): Likewise.
(trunc<mode><ssePHmodelower>2): Likewise.
(avx512fp16_vcvt<castmode>2ph_<mode>): Rename to ...
(trunc<mode>v4hf2): ... this, and drop constraints.
(avx512fp16_vcvtpd2ph_v2df): Rename to ...
(truncv2dfv2hf2): ... this, and likewise.
gcc/testsuite/ChangeLog:
* gcc.target/i386/avx512fp16-trunc-extendvnhf.c: New test.
---
gcc/config/i386/sse.md | 75 +++++++++++++++++--
.../i386/avx512fp16-trunc-extendvnhf.c | 55 ++++++++++++++
2 files changed, 123 insertions(+), 7 deletions(-)
create mode 100644 gcc/testsuite/gcc.target/i386/avx512fp16-trunc-extendvnhf.c
diff --git a/gcc/config/i386/sse.md b/gcc/config/i386/sse.md
index 66062dc3bcf..a48c8e8bede 100644
--- a/gcc/config/i386/sse.md
+++ b/gcc/config/i386/sse.md
@@ -6328,6 +6328,12 @@ (define_mode_attr ph2pssuffix
[(V16SF "x") (V8SF "x") (V4SF "x")
(V8DF "") (V4DF "") (V2DF "")])
+(define_expand "extend<ssePHmodelower><mode>2"
+ [(set (match_operand:VF48H_AVX512VL 0 "register_operand")
+ (float_extend:VF48H_AVX512VL
+ (match_operand:<ssePHmode> 1 "nonimmediate_operand")))]
+ "TARGET_AVX512FP16")
+
(define_insn "avx512fp16_float_extend_ph<mode>2<mask_name><round_saeonly_name>"
[(set (match_operand:VF48H_AVX512VL 0 "register_operand" "=v")
(float_extend:VF48H_AVX512VL
@@ -6338,6 +6344,21 @@ (define_insn "avx512fp16_float_extend_ph<mode>2<mask_name><round_saeonly_name>"
(set_attr "prefix" "evex")
(set_attr "mode" "<sseinsnmode>")])
+(define_expand "extendv4hf<mode>2"
+ [(set (match_operand:VF4_128_8_256 0 "register_operand")
+ (float_extend:VF4_128_8_256
+ (match_operand:V4HF 1 "nonimmediate_operand")))]
+ "TARGET_AVX512FP16 && TARGET_AVX512VL"
+{
+ if (!MEM_P (operands[1]))
+ {
+ operands[1] = lowpart_subreg (V8HFmode, operands[1], V4HFmode);
+ emit_insn (gen_avx512fp16_float_extend_ph<mode>2
+ (operands[0], operands[1]));
+ DONE;
+ }
+})
+
(define_insn "avx512fp16_float_extend_ph<mode>2<mask_name>"
[(set (match_operand:VF4_128_8_256 0 "register_operand" "=v")
(float_extend:VF4_128_8_256
@@ -6360,6 +6381,21 @@ (define_insn "*avx512fp16_float_extend_ph<mode>2_load<mask_name>"
(set_attr "prefix" "evex")
(set_attr "mode" "<sseinsnmode>")])
+(define_expand "extendv2hfv2df2"
+ [(set (match_operand:V2DF 0 "register_operand")
+ (float_extend:V2DF
+ (match_operand:V2HF 1 "nonimmediate_operand")))]
+ "TARGET_AVX512FP16 && TARGET_AVX512VL"
+{
+ if (!MEM_P (operands[1]))
+ {
+ operands[1] = lowpart_subreg (V8HFmode, operands[1], V2HFmode);
+ emit_insn (gen_avx512fp16_float_extend_phv2df2
+ (operands[0], operands[1]));
+ DONE;
+ }
+})
+
(define_insn "avx512fp16_float_extend_phv2df2<mask_name>"
[(set (match_operand:V2DF 0 "register_operand" "=v")
(float_extend:V2DF
@@ -6382,6 +6418,12 @@ (define_insn "*avx512fp16_float_extend_phv2df2_load<mask_name>"
(set_attr "prefix" "evex")
(set_attr "mode" "TI")])
+(define_expand "trunc<mode><ssePHmodelower>2"
+ [(set (match_operand:<ssePHmode> 0 "register_operand")
+ (float_truncate:<ssePHmode>
+ (match_operand:VF48H_AVX512VL 1 "nonimmediate_operand")))]
+ "TARGET_AVX512FP16")
+
(define_insn "avx512fp16_vcvt<castmode>2ph_<mode><mask_name><round_name>"
[(set (match_operand:<ssePHmode> 0 "register_operand" "=v")
(float_truncate:<ssePHmode>
@@ -6392,11 +6434,21 @@ (define_insn "avx512fp16_vcvt<castmode>2ph_<mode><mask_name><round_name>"
(set_attr "prefix" "evex")
(set_attr "mode" "<sseinsnmode>")])
-(define_expand "avx512fp16_vcvt<castmode>2ph_<mode>"
- [(set (match_operand:V8HF 0 "register_operand" "=v")
+(define_expand "trunc<mode>v4hf2"
+ [(set (match_operand:V4HF 0 "register_operand")
+ (float_truncate:V4HF (match_operand:VF4_128_8_256 1 "vector_operand")))]
+ "TARGET_AVX512FP16 && TARGET_AVX512VL"
+{
+ operands[0] = lowpart_subreg (V8HFmode, operands[0], V4HFmode);
+ emit_insn (gen_avx512fp16_trunc<mode>v4hf2 (operands[0], operands[1]));
+ DONE;
+})
+
+(define_expand "avx512fp16_trunc<mode>v4hf2"
+ [(set (match_operand:V8HF 0 "register_operand")
(vec_concat:V8HF
(float_truncate:V4HF
- (match_operand:VF4_128_8_256 1 "vector_operand" "vm"))
+ (match_operand:VF4_128_8_256 1 "vector_operand"))
(match_dup 2)))]
"TARGET_AVX512FP16 && TARGET_AVX512VL"
"operands[2] = CONST0_RTX (V4HFmode);")
@@ -6461,11 +6513,20 @@ (define_insn "*avx512fp16_vcvt<castmode>2ph_<mode>_mask_1"
(set_attr "prefix" "evex")
(set_attr "mode" "<sseinsnmode>")])
-(define_expand "avx512fp16_vcvtpd2ph_v2df"
- [(set (match_operand:V8HF 0 "register_operand" "=v")
+(define_expand "truncv2dfv2hf2"
+ [(set (match_operand:V2HF 0 "register_operand")
+ (float_truncate:V2HF (match_operand:V2DF 1 "vector_operand")))]
+ "TARGET_AVX512FP16 && TARGET_AVX512VL"
+{
+ operands[0] = lowpart_subreg (V8HFmode, operands[0], V2HFmode);
+ emit_insn (gen_avx512fp16_truncv2dfv2hf2 (operands[0], operands[1]));
+ DONE;
+})
+
+(define_expand "avx512fp16_truncv2dfv2hf2"
+ [(set (match_operand:V8HF 0 "register_operand")
(vec_concat:V8HF
- (float_truncate:V2HF
- (match_operand:V2DF 1 "vector_operand" "vm"))
+ (float_truncate:V2HF (match_operand:V2DF 1 "vector_operand"))
(match_dup 2)))]
"TARGET_AVX512FP16 && TARGET_AVX512VL"
"operands[2] = CONST0_RTX (V6HFmode);")
diff --git a/gcc/testsuite/gcc.target/i386/avx512fp16-trunc-extendvnhf.c b/gcc/testsuite/gcc.target/i386/avx512fp16-trunc-extendvnhf.c
new file mode 100644
index 00000000000..286ea9f2624
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/avx512fp16-trunc-extendvnhf.c
@@ -0,0 +1,55 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -mavx512fp16 -mavx512vl -ftree-slp-vectorize -mprefer-vector-width=512" } */
+
+extern double df[8];
+extern float sf[16];
+extern _Float16 hf[32];
+
+#define DO_PRAGMA(X) _Pragma(#X)
+
+#define TRUNCHFVV(size, mode) \
+ void __attribute__ ((noinline, noclone)) \
+truncv##size##mode##v##size##hf () \
+{\
+ int i; \
+ DO_PRAGMA (GCC unroll size) \
+ for (i = 0; i < size; i++) \
+ hf[i] = mode[i]; \
+}
+
+#define EXTENDHFVV(size, mode) \
+ void __attribute__ ((noinline, noclone)) \
+extendv##size##hf##v##size##mode () \
+{\
+ int i; \
+ DO_PRAGMA (GCC unroll size) \
+ for (i = 0; i < size; i++) \
+ mode[i] = hf[i]; \
+}
+
+TRUNCHFVV(8, df)
+TRUNCHFVV(4, df)
+TRUNCHFVV(2, df)
+TRUNCHFVV(16, sf)
+TRUNCHFVV(8, sf)
+TRUNCHFVV(4, sf)
+EXTENDHFVV(8, df)
+EXTENDHFVV(4, df)
+EXTENDHFVV(2, df)
+EXTENDHFVV(16, sf)
+EXTENDHFVV(8, sf)
+EXTENDHFVV(4, sf)
+
+/* { dg-final { scan-assembler-times "vcvtpd2phz\[ \\t\]+\[^\{\n\]*\[^\n\r]*%xmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 } } */
+/* { dg-final { scan-assembler-times "vcvtpd2phy\[ \\t\]+\[^\{\n\]*\[^\n\r]*%xmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 { xfail *-*-* } } } */
+/* { dg-final { scan-assembler-times "vcvtpd2phx\[ \\t\]+\[^\{\n\]*\[^\n\r]*%xmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 { xfail *-*-* } } } */
+/* { dg-final { scan-assembler-times "vcvtps2phx\[ \\t\]+\[^\{\n\]*\[^\n\r]*%ymm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 } } */
+/* { dg-final { scan-assembler-times "vcvtps2phxy\[ \\t\]+\[^\{\n\]*\[^\n\r]*%xmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 } } */
+/* { dg-final { scan-assembler-times "vcvtps2phxx\[ \\t\]+\[^\{\n\]*\[^\n\r]*%xmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 { xfail *-*-* } } } */
+/* { dg-final { scan-assembler-times "vcvtph2pd\[ \\t\]+\[^\{\n\]*\[^\n\r]*%zmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 } } */
+/* { dg-final { scan-assembler-times "vcvtph2pd\[ \\t\]+\[^\{\n\]*\[^\n\r]*%ymm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 { xfail *-*-* } } } */
+/* { dg-final { scan-assembler-times "vcvtph2pd\[ \\t\]+\[^\{\n\]*\[^\n\r]*%xmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 { xfail *-*-* } } } */
+/* { dg-final { scan-assembler-times "vcvtph2psx\[ \\t\]+\[^\{\n\]*\[^\n\r]*%zmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 } } */
+/* { dg-final { scan-assembler-times "vcvtph2psx\[ \\t\]+\[^\{\n\]*\[^\n\r]*%ymm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 } } */
+/* { dg-final { scan-assembler-times "vcvtph2psx\[ \\t\]+\[^\{\n\]*\[^\n\r]*%xmm\[0-9\]+(?:\n|\[ \\t\]+#)" 1 { xfail *-*-* } } } */
+
--
2.27.0
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH 7/7] AVX512FP16: Enable vec_cmpmn/vcondmn expanders for HF modes.
2021-09-23 5:46 [PATCH 0/7] AVX512FP16: Support bunch of expanders for HFmode and vector HFmodes liuhongt
` (5 preceding siblings ...)
2021-09-23 5:46 ` [PATCH 6/7] AVX512FP16: add truncmn2/extendmn2 expanders liuhongt
@ 2021-09-23 5:46 ` liuhongt
6 siblings, 0 replies; 8+ messages in thread
From: liuhongt @ 2021-09-23 5:46 UTC (permalink / raw)
To: gcc-patches; +Cc: crazylht, hjl.tools, Hongyu Wang
From: Hongyu Wang <hongyu.wang@intel.com>
gcc/ChangeLog:
* config/i386/i386-expand.c (ix86_use_mask_cmp_p): Enable
HFmode mask_cmp.
* config/i386/sse.md (sseintvecmodelower): Add HF vector modes.
(<avx512>_store<mode>_mask): Extend to support HF vector modes.
(vec_cmp<mode><avx512fmaskmodelower>): Likewise.
(vcond_mask_<mode><avx512fmaskmodelower>): Likewise.
(vcond<mode><mode>): New expander.
(vcond<mode><sseintvecmodelower>): Likewise.
(vcond<sseintvecmodelower><mode>): Likewise.
(vcondu<mode><sseintvecmodelower>): Likewise.
gcc/testsuite/ChangeLog:
* g++.target/i386/avx512fp16-vcondmn-vec.C: New test.
* g++.target/i386/avx512fp16-vcondmn-minmax.C: Ditto.
* gcc.target/i386/avx512fp16-vcondmn-loop-1.c: Ditto.
* gcc.target/i386/avx512fp16-vcondmn-loop-2.c: Ditto.
* gcc.target/i386/avx512fp16-vec_cmpmn.c: Ditto.
---
gcc/config/i386/i386-expand.c | 2 +
gcc/config/i386/sse.md | 84 ++++++++--
.../i386/avx512fp16-vcondmn-minmax.C | 25 +++
.../g++.target/i386/avx512fp16-vcondmn-vec.C | 70 +++++++++
.../i386/avx512fp16-vcondmn-loop-1.c | 70 +++++++++
.../i386/avx512fp16-vcondmn-loop-2.c | 143 ++++++++++++++++++
.../gcc.target/i386/avx512fp16-vec_cmpmn.c | 32 ++++
7 files changed, 414 insertions(+), 12 deletions(-)
create mode 100644 gcc/testsuite/g++.target/i386/avx512fp16-vcondmn-minmax.C
create mode 100644 gcc/testsuite/g++.target/i386/avx512fp16-vcondmn-vec.C
create mode 100644 gcc/testsuite/gcc.target/i386/avx512fp16-vcondmn-loop-1.c
create mode 100644 gcc/testsuite/gcc.target/i386/avx512fp16-vcondmn-loop-2.c
create mode 100644 gcc/testsuite/gcc.target/i386/avx512fp16-vec_cmpmn.c
diff --git a/gcc/config/i386/i386-expand.c b/gcc/config/i386/i386-expand.c
index dbbf5e34656..94ac303585e 100644
--- a/gcc/config/i386/i386-expand.c
+++ b/gcc/config/i386/i386-expand.c
@@ -3638,6 +3638,8 @@ ix86_use_mask_cmp_p (machine_mode mode, machine_mode cmp_mode,
return false;
else if (vector_size == 64)
return true;
+ else if (GET_MODE_INNER (cmp_mode) == HFmode)
+ return true;
/* When op_true is NULL, op_false must be NULL, or vice versa. */
gcc_assert (!op_true == !op_false);
diff --git a/gcc/config/i386/sse.md b/gcc/config/i386/sse.md
index a48c8e8bede..084fc7f4693 100644
--- a/gcc/config/i386/sse.md
+++ b/gcc/config/i386/sse.md
@@ -989,9 +989,9 @@ (define_mode_attr sseintvecmode2
(V16HF "OI") (V8HF "TI")])
(define_mode_attr sseintvecmodelower
- [(V16SF "v16si") (V8DF "v8di")
- (V8SF "v8si") (V4DF "v4di")
- (V4SF "v4si") (V2DF "v2di")
+ [(V32HF "v32hi") (V16SF "v16si") (V8DF "v8di")
+ (V16HF "v16hi") (V8SF "v8si") (V4DF "v4di")
+ (V8HF "v8hi") (V4SF "v4si") (V2DF "v2di")
(V8SI "v8si") (V4DI "v4di")
(V4SI "v4si") (V2DI "v2di")
(V16HI "v16hi") (V8HI "v8hi")
@@ -1568,9 +1568,9 @@ (define_insn "<avx512>_store<mode>_mask"
(set_attr "mode" "<sseinsnmode>")])
(define_insn "<avx512>_store<mode>_mask"
- [(set (match_operand:VI12_AVX512VL 0 "memory_operand" "=m")
- (vec_merge:VI12_AVX512VL
- (match_operand:VI12_AVX512VL 1 "register_operand" "v")
+ [(set (match_operand:VI12HF_AVX512VL 0 "memory_operand" "=m")
+ (vec_merge:VI12HF_AVX512VL
+ (match_operand:VI12HF_AVX512VL 1 "register_operand" "v")
(match_dup 0)
(match_operand:<avx512fmaskmode> 2 "register_operand" "Yk")))]
"TARGET_AVX512BW"
@@ -3810,8 +3810,8 @@ (define_insn "<sse>_<unord>comi<round_saeonly_name>"
(define_expand "vec_cmp<mode><avx512fmaskmodelower>"
[(set (match_operand:<avx512fmaskmode> 0 "register_operand")
(match_operator:<avx512fmaskmode> 1 ""
- [(match_operand:V48_AVX512VL 2 "register_operand")
- (match_operand:V48_AVX512VL 3 "nonimmediate_operand")]))]
+ [(match_operand:V48H_AVX512VL 2 "register_operand")
+ (match_operand:V48H_AVX512VL 3 "nonimmediate_operand")]))]
"TARGET_AVX512F"
{
bool ok = ix86_expand_mask_vec_cmp (operands[0], GET_CODE (operands[1]),
@@ -4018,6 +4018,51 @@ (define_expand "vcond<V_128:mode><VF_128:mode>"
DONE;
})
+(define_expand "vcond<mode><mode>"
+ [(set (match_operand:VF_AVX512FP16VL 0 "register_operand")
+ (if_then_else:VF_AVX512FP16VL
+ (match_operator 3 ""
+ [(match_operand:VF_AVX512FP16VL 4 "vector_operand")
+ (match_operand:VF_AVX512FP16VL 5 "vector_operand")])
+ (match_operand:VF_AVX512FP16VL 1 "general_operand")
+ (match_operand:VF_AVX512FP16VL 2 "general_operand")))]
+ "TARGET_AVX512FP16"
+{
+ bool ok = ix86_expand_fp_vcond (operands);
+ gcc_assert (ok);
+ DONE;
+})
+
+(define_expand "vcond<mode><sseintvecmodelower>"
+ [(set (match_operand:VF_AVX512FP16VL 0 "register_operand")
+ (if_then_else:VF_AVX512FP16VL
+ (match_operator 3 ""
+ [(match_operand:<sseintvecmode> 4 "vector_operand")
+ (match_operand:<sseintvecmode> 5 "vector_operand")])
+ (match_operand:VF_AVX512FP16VL 1 "general_operand")
+ (match_operand:VF_AVX512FP16VL 2 "general_operand")))]
+ "TARGET_AVX512FP16"
+{
+ bool ok = ix86_expand_int_vcond (operands);
+ gcc_assert (ok);
+ DONE;
+})
+
+(define_expand "vcond<sseintvecmodelower><mode>"
+ [(set (match_operand:<sseintvecmode> 0 "register_operand")
+ (if_then_else:<sseintvecmode>
+ (match_operator 3 ""
+ [(match_operand:VF_AVX512FP16VL 4 "vector_operand")
+ (match_operand:VF_AVX512FP16VL 5 "vector_operand")])
+ (match_operand:<sseintvecmode> 1 "general_operand")
+ (match_operand:<sseintvecmode> 2 "general_operand")))]
+ "TARGET_AVX512FP16"
+{
+ bool ok = ix86_expand_fp_vcond (operands);
+ gcc_assert (ok);
+ DONE;
+})
+
(define_expand "vcond_mask_<mode><avx512fmaskmodelower>"
[(set (match_operand:V48_AVX512VL 0 "register_operand")
(vec_merge:V48_AVX512VL
@@ -4027,10 +4072,10 @@ (define_expand "vcond_mask_<mode><avx512fmaskmodelower>"
"TARGET_AVX512F")
(define_expand "vcond_mask_<mode><avx512fmaskmodelower>"
- [(set (match_operand:VI12_AVX512VL 0 "register_operand")
- (vec_merge:VI12_AVX512VL
- (match_operand:VI12_AVX512VL 1 "nonimmediate_operand")
- (match_operand:VI12_AVX512VL 2 "nonimm_or_0_operand")
+ [(set (match_operand:VI12HF_AVX512VL 0 "register_operand")
+ (vec_merge:VI12HF_AVX512VL
+ (match_operand:VI12HF_AVX512VL 1 "nonimmediate_operand")
+ (match_operand:VI12HF_AVX512VL 2 "nonimm_or_0_operand")
(match_operand:<avx512fmaskmode> 3 "register_operand")))]
"TARGET_AVX512BW")
@@ -15538,6 +15583,21 @@ (define_expand "vcondu<VI8F_128:mode>v2di"
DONE;
})
+(define_expand "vcondu<mode><sseintvecmodelower>"
+ [(set (match_operand:VF_AVX512FP16VL 0 "register_operand")
+ (if_then_else:VF_AVX512FP16VL
+ (match_operator 3 ""
+ [(match_operand:<sseintvecmode> 4 "vector_operand")
+ (match_operand:<sseintvecmode> 5 "vector_operand")])
+ (match_operand:VF_AVX512FP16VL 1 "general_operand")
+ (match_operand:VF_AVX512FP16VL 2 "general_operand")))]
+ "TARGET_AVX512FP16"
+{
+ bool ok = ix86_expand_int_vcond (operands);
+ gcc_assert (ok);
+ DONE;
+})
+
(define_expand "vcondeq<VI8F_128:mode>v2di"
[(set (match_operand:VI8F_128 0 "register_operand")
(if_then_else:VI8F_128
diff --git a/gcc/testsuite/g++.target/i386/avx512fp16-vcondmn-minmax.C b/gcc/testsuite/g++.target/i386/avx512fp16-vcondmn-minmax.C
new file mode 100644
index 00000000000..6d50f4974c5
--- /dev/null
+++ b/gcc/testsuite/g++.target/i386/avx512fp16-vcondmn-minmax.C
@@ -0,0 +1,25 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -mavx512fp16 -mavx512vl" } */
+
+/* { dg-final { scan-assembler-times "vminph" 3 } } */
+/* { dg-final { scan-assembler-times "vmaxph" 3 } } */
+
+typedef _Float16 v8hf __attribute__ ((vector_size (16)));
+typedef _Float16 v16hf __attribute__ ((vector_size (32)));
+typedef _Float16 v32hf __attribute__ ((vector_size (64)));
+
+#define VCONDMINMAX(size, op, name) \
+v##size##hf \
+__attribute__ ((noinline, noclone)) \
+vminmax_##v##size##hf##v##size##hf##name (v##size##hf a, v##size##hf b) \
+{ \
+ return (a op b) ? a : b; \
+}
+
+VCONDMINMAX (8, <, min)
+VCONDMINMAX (8, >, max)
+VCONDMINMAX (16, <, min)
+VCONDMINMAX (16, >, max)
+VCONDMINMAX (32, <, min)
+VCONDMINMAX (32, >, max)
+
diff --git a/gcc/testsuite/g++.target/i386/avx512fp16-vcondmn-vec.C b/gcc/testsuite/g++.target/i386/avx512fp16-vcondmn-vec.C
new file mode 100644
index 00000000000..de93e2c5c86
--- /dev/null
+++ b/gcc/testsuite/g++.target/i386/avx512fp16-vcondmn-vec.C
@@ -0,0 +1,70 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -mavx512fp16 -mavx512vl" } */
+
+/* { dg-final { scan-assembler-times "vcmpph" 45 } } */
+/* { dg-final { scan-assembler-times "vpcmpuw" 12 } } */
+/* { dg-final { scan-assembler-times "vpcmpw" 18 } } */
+/* { dg-final { scan-assembler-times "(?:vpblendmw|vmovdqu16\[^\{\n\]+\{%k\[1-7\]\})" 75 } } */
+
+typedef _Float16 v8hf __attribute__ ((vector_size (16)));
+typedef _Float16 v16hf __attribute__ ((vector_size (32)));
+typedef _Float16 v32hf __attribute__ ((vector_size (64)));
+typedef short v8hi __attribute__ ((vector_size (16)));
+typedef short v16hi __attribute__ ((vector_size (32)));
+typedef short v32hi __attribute__ ((vector_size (64)));
+typedef unsigned short v8uhi __attribute__ ((vector_size (16)));
+typedef unsigned short v16uhi __attribute__ ((vector_size (32)));
+typedef unsigned short v32uhi __attribute__ ((vector_size (64)));
+
+#define VCONDMOV(size, op, name) \
+v##size##hf \
+__attribute__ ((noinline, noclone)) \
+vcond_##v##size##hf##v##size##hf##name (v##size##hf a, v##size##hf b, \
+ v##size##hf c, v##size##hf d) \
+{ \
+ return (a op b) ? c : d; \
+}\
+v##size##hf \
+__attribute__ ((noinline, noclone)) \
+vcond_##v##size##hi##v##size##hf##name (v##size##hi a, v##size##hi b, \
+ v##size##hf c, v##size##hf d) \
+{ \
+ return (a op b) ? c : d; \
+}\
+v##size##hi \
+__attribute__ ((noinline, noclone)) \
+vcond_##v##size##hf##v##size##hi##name (v##size##hi a, v##size##hi b, \
+ v##size##hf c, v##size##hf d) \
+{ \
+ return (c op d) ? a : b; \
+} \
+v##size##hf \
+__attribute__ ((noinline, noclone)) \
+vcond_##v##size##uhi##v##size##hf##name (v##size##uhi a, v##size##uhi b, \
+ v##size##hf c, v##size##hf d) \
+{ \
+ return (a op b) ? c : d; \
+}\
+v##size##uhi \
+__attribute__ ((noinline, noclone)) \
+vcond_##v##size##hf##v##size##uhi##name (v##size##uhi a, v##size##uhi b, \
+ v##size##hf c, v##size##hf d) \
+{ \
+ return (c op d) ? a : b; \
+} \
+
+VCONDMOV (8, <, lt)
+VCONDMOV (8, >, gt)
+VCONDMOV (8, ==, eq)
+VCONDMOV (8, <=, le)
+VCONDMOV (8, >=, ge)
+VCONDMOV (16, <, lt)
+VCONDMOV (16, >, gt)
+VCONDMOV (16, <=, le)
+VCONDMOV (16, >=, ge)
+VCONDMOV (16, ==, eq)
+VCONDMOV (32, <, lt)
+VCONDMOV (32, >, gt)
+VCONDMOV (32, <=, le)
+VCONDMOV (32, >=, ge)
+VCONDMOV (32, ==, eq)
diff --git a/gcc/testsuite/gcc.target/i386/avx512fp16-vcondmn-loop-1.c b/gcc/testsuite/gcc.target/i386/avx512fp16-vcondmn-loop-1.c
new file mode 100644
index 00000000000..e8745aba64e
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/avx512fp16-vcondmn-loop-1.c
@@ -0,0 +1,70 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -mavx512fp16 -mavx512vl -mprefer-vector-width=512" } */
+
+/* { dg-final { scan-assembler-times "vcmpph" 27 } } */
+/* { dg-final { scan-assembler-times "(?:vpcmpw|vpcmpeqw)" 12 } } */
+/* { dg-final { scan-assembler-times "vpcmpuw" 6 } } */
+
+typedef unsigned short u16;
+typedef short s16;
+
+#define CONDMOV_LOOP(size, type, ptype, op, name) \
+void \
+__attribute__ ((noinline, noclone, optimize("tree-vectorize"))) \
+loop_cond_##size##ptype##type##name ( \
+ ptype * restrict a, ptype * restrict b, \
+ type * restrict c, type * restrict d) \
+{ \
+ int i; \
+ for (i = 0; i < size; i++) \
+ { \
+ if (a[i] op b[i]) \
+ d[i] = c[i]; \
+ } \
+}
+
+CONDMOV_LOOP (32, _Float16, _Float16, <, lt)
+CONDMOV_LOOP (32, _Float16, _Float16, >, gt)
+CONDMOV_LOOP (32, _Float16, _Float16, ==, eq)
+CONDMOV_LOOP (16, _Float16, _Float16, <, lt)
+CONDMOV_LOOP (16, _Float16, _Float16, >, gt)
+CONDMOV_LOOP (16, _Float16, _Float16, ==, eq)
+CONDMOV_LOOP (8, _Float16, _Float16, <, lt)
+CONDMOV_LOOP (8, _Float16, _Float16, >, gt)
+CONDMOV_LOOP (8, _Float16, _Float16, ==, eq)
+CONDMOV_LOOP (32, _Float16, s16, <, lt)
+CONDMOV_LOOP (32, _Float16, s16, >, gt)
+CONDMOV_LOOP (32, _Float16, s16, ==, eq)
+CONDMOV_LOOP (16, _Float16, s16, <, lt)
+CONDMOV_LOOP (16, _Float16, s16, >, gt)
+CONDMOV_LOOP (16, _Float16, s16, ==, eq)
+CONDMOV_LOOP (8, _Float16, s16, <, lt)
+CONDMOV_LOOP (8, _Float16, s16, >, gt)
+CONDMOV_LOOP (8, _Float16, s16, ==, eq)
+CONDMOV_LOOP (32, s16, _Float16, <, lt)
+CONDMOV_LOOP (32, s16, _Float16, >, gt)
+CONDMOV_LOOP (32, s16, _Float16, ==, eq)
+CONDMOV_LOOP (16, s16, _Float16, <, lt)
+CONDMOV_LOOP (16, s16, _Float16, >, gt)
+CONDMOV_LOOP (16, s16, _Float16, ==, eq)
+CONDMOV_LOOP (8, s16, _Float16, <, lt)
+CONDMOV_LOOP (8, s16, _Float16, >, gt)
+CONDMOV_LOOP (8, s16, _Float16, ==, eq)
+CONDMOV_LOOP (32, _Float16, u16, <, lt)
+CONDMOV_LOOP (32, _Float16, u16, >, gt)
+CONDMOV_LOOP (32, _Float16, u16, ==, eq)
+CONDMOV_LOOP (16, _Float16, u16, <, lt)
+CONDMOV_LOOP (16, _Float16, u16, >, gt)
+CONDMOV_LOOP (16, _Float16, u16, ==, eq)
+CONDMOV_LOOP (8, _Float16, u16, <, lt)
+CONDMOV_LOOP (8, _Float16, u16, >, gt)
+CONDMOV_LOOP (8, _Float16, u16, ==, eq)
+CONDMOV_LOOP (32, u16, _Float16, <, lt)
+CONDMOV_LOOP (32, u16, _Float16, >, gt)
+CONDMOV_LOOP (32, u16, _Float16, ==, eq)
+CONDMOV_LOOP (16, u16, _Float16, <, lt)
+CONDMOV_LOOP (16, u16, _Float16, >, gt)
+CONDMOV_LOOP (16, u16, _Float16, ==, eq)
+CONDMOV_LOOP (8, u16, _Float16, <, lt)
+CONDMOV_LOOP (8, u16, _Float16, >, gt)
+CONDMOV_LOOP (8, u16, _Float16, ==, eq)
diff --git a/gcc/testsuite/gcc.target/i386/avx512fp16-vcondmn-loop-2.c b/gcc/testsuite/gcc.target/i386/avx512fp16-vcondmn-loop-2.c
new file mode 100644
index 00000000000..a0d5f988088
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/avx512fp16-vcondmn-loop-2.c
@@ -0,0 +1,143 @@
+/* { dg-do run { target avx512fp16 } } */
+/* { dg-options "-O2 -mavx512fp16 -mavx512vl -mprefer-vector-width=512" } */
+
+static void condmov_test (void);
+#define DO_TEST condmov_test
+#define AVX512FP16
+#define AVX512VL
+#include "avx512f-check.h"
+#include "avx512fp16-vcondmn-loop-1.c"
+
+_Float16 a[32], b[32], c[32], fexp[32], fref[32];
+s16 sa[32], sb[32], sc[32], sexp[32], sref[32];
+u16 ua[32], ub[32], uc[32], uexp[32], uref[32];
+
+#define EMULATE_CONDMOV_LOOP(size, type, ptype, op, name) \
+void \
+__attribute__ ((noinline, noclone)) \
+scalar_cond_##size##ptype##type##name ( \
+ ptype * restrict a, ptype * restrict b, \
+ type * restrict c, type * restrict d) \
+{ \
+ int i; \
+ for (i = 0; i < size; i++) \
+ { \
+ if (a[i] op b[i]) \
+ d[i] = c[i]; \
+ } \
+}
+
+EMULATE_CONDMOV_LOOP (32, _Float16, _Float16, <, lt)
+EMULATE_CONDMOV_LOOP (32, _Float16, _Float16, >, gt)
+EMULATE_CONDMOV_LOOP (32, _Float16, _Float16, ==, eq)
+EMULATE_CONDMOV_LOOP (16, _Float16, _Float16, <, lt)
+EMULATE_CONDMOV_LOOP (16, _Float16, _Float16, >, gt)
+EMULATE_CONDMOV_LOOP (16, _Float16, _Float16, ==, eq)
+EMULATE_CONDMOV_LOOP (8, _Float16, _Float16, <, lt)
+EMULATE_CONDMOV_LOOP (8, _Float16, _Float16, >, gt)
+EMULATE_CONDMOV_LOOP (8, _Float16, _Float16, ==, eq)
+EMULATE_CONDMOV_LOOP (32, _Float16, s16, <, lt)
+EMULATE_CONDMOV_LOOP (32, _Float16, s16, >, gt)
+EMULATE_CONDMOV_LOOP (32, _Float16, s16, ==, eq)
+EMULATE_CONDMOV_LOOP (16, _Float16, s16, <, lt)
+EMULATE_CONDMOV_LOOP (16, _Float16, s16, >, gt)
+EMULATE_CONDMOV_LOOP (16, _Float16, s16, ==, eq)
+EMULATE_CONDMOV_LOOP (8, _Float16, s16, <, lt)
+EMULATE_CONDMOV_LOOP (8, _Float16, s16, >, gt)
+EMULATE_CONDMOV_LOOP (8, _Float16, s16, ==, eq)
+EMULATE_CONDMOV_LOOP (32, s16, _Float16, <, lt)
+EMULATE_CONDMOV_LOOP (32, s16, _Float16, >, gt)
+EMULATE_CONDMOV_LOOP (32, s16, _Float16, ==, eq)
+EMULATE_CONDMOV_LOOP (16, s16, _Float16, <, lt)
+EMULATE_CONDMOV_LOOP (16, s16, _Float16, >, gt)
+EMULATE_CONDMOV_LOOP (16, s16, _Float16, ==, eq)
+EMULATE_CONDMOV_LOOP (8, s16, _Float16, <, lt)
+EMULATE_CONDMOV_LOOP (8, s16, _Float16, >, gt)
+EMULATE_CONDMOV_LOOP (8, s16, _Float16, ==, eq)
+EMULATE_CONDMOV_LOOP (32, _Float16, u16, <, lt)
+EMULATE_CONDMOV_LOOP (32, _Float16, u16, >, gt)
+EMULATE_CONDMOV_LOOP (32, _Float16, u16, ==, eq)
+EMULATE_CONDMOV_LOOP (16, _Float16, u16, <, lt)
+EMULATE_CONDMOV_LOOP (16, _Float16, u16, >, gt)
+EMULATE_CONDMOV_LOOP (16, _Float16, u16, ==, eq)
+EMULATE_CONDMOV_LOOP (8, _Float16, u16, <, lt)
+EMULATE_CONDMOV_LOOP (8, _Float16, u16, >, gt)
+EMULATE_CONDMOV_LOOP (8, _Float16, u16, ==, eq)
+EMULATE_CONDMOV_LOOP (32, u16, _Float16, <, lt)
+EMULATE_CONDMOV_LOOP (32, u16, _Float16, >, gt)
+EMULATE_CONDMOV_LOOP (32, u16, _Float16, ==, eq)
+EMULATE_CONDMOV_LOOP (16, u16, _Float16, <, lt)
+EMULATE_CONDMOV_LOOP (16, u16, _Float16, >, gt)
+EMULATE_CONDMOV_LOOP (16, u16, _Float16, ==, eq)
+EMULATE_CONDMOV_LOOP (8, u16, _Float16, <, lt)
+EMULATE_CONDMOV_LOOP (8, u16, _Float16, >, gt)
+EMULATE_CONDMOV_LOOP (8, u16, _Float16, ==, eq)
+
+void init()
+{
+ int i;
+ for (i = 0; i < 32; i++)
+ {
+ ua[i] = sa[i] = a[i] = i;
+ ub[i] = sb[i] = b[i] = i;
+ uc[i] = sc[i] = c[i] = (32 - i) * 2;
+ uexp[i] = sexp[i] = fexp[i] = -1;
+ uref[i] = sref[i] = fref[i] = -1;
+ }
+}
+
+int check_cond(void *a, void *b, int size)
+{
+ int i;
+ u16 *pa = (u16 *)a, *pb = (u16 *)b;
+ for (i = 0; i < size; i++)
+ if (pa[i] != pb[i])
+ return 0;
+ return 1;
+}
+
+#define TEST_CONDMOV_LOOP(size, name) \
+{ \
+ init (); \
+ scalar_cond_##size##_Float16_Float16##name (a, b, c, fexp); \
+ loop_cond_##size##_Float16_Float16##name (a, b, c, fref); \
+ if (!check_cond ((void *)fexp, (void *)fref, size)) \
+ abort(); \
+ \
+ init (); \
+ scalar_cond_##size##_Float16s16##name (a, b, sc, sexp); \
+ loop_cond_##size##_Float16s16##name (a, b, sc, sref); \
+ if (!check_cond ((void *)sexp, (void *)sref, size)) \
+ abort(); \
+ \
+ init (); \
+ scalar_cond_##size##s16_Float16##name (sa, sb, c, fexp); \
+ loop_cond_##size##s16_Float16##name (sa, sb, c, fref); \
+ if (!check_cond ((void *)fexp, (void *)fref, size)) \
+ abort(); \
+ \
+ init (); \
+ scalar_cond_##size##_Float16u16##name (a, b, uc, uexp); \
+ loop_cond_##size##_Float16u16##name (a, b, uc, uref); \
+ if (!check_cond ((void *)uexp, (void *)uref, size)) \
+ abort(); \
+ \
+ init (); \
+ scalar_cond_##size##u16_Float16##name (ua, ub, c, fexp); \
+ loop_cond_##size##u16_Float16##name (ua, ub, c, fref); \
+ if (!check_cond ((void *)fexp, (void *)fref, size)) \
+ abort(); \
+}
+
+static void condmov_test()
+{
+ TEST_CONDMOV_LOOP (32, lt)
+ TEST_CONDMOV_LOOP (32, gt)
+ TEST_CONDMOV_LOOP (32, eq)
+ TEST_CONDMOV_LOOP (16, lt)
+ TEST_CONDMOV_LOOP (16, gt)
+ TEST_CONDMOV_LOOP (16, eq)
+ TEST_CONDMOV_LOOP (8, lt)
+ TEST_CONDMOV_LOOP (8, gt)
+ TEST_CONDMOV_LOOP (8, eq)
+}
diff --git a/gcc/testsuite/gcc.target/i386/avx512fp16-vec_cmpmn.c b/gcc/testsuite/gcc.target/i386/avx512fp16-vec_cmpmn.c
new file mode 100644
index 00000000000..ef9f85373f9
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/avx512fp16-vec_cmpmn.c
@@ -0,0 +1,32 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -mavx512fp16 -mavx512vl" } */
+
+/* { dg-final { scan-assembler-times "vcmpph" 15 } } */
+
+typedef _Float16 v8hf __attribute__ ((vector_size (16)));
+typedef _Float16 v16hf __attribute__ ((vector_size (32)));
+typedef _Float16 v32hf __attribute__ ((vector_size (64)));
+
+#define VCMPMN(type, op, name) \
+type \
+__attribute__ ((noinline, noclone)) \
+vec_cmp_##type##type##name (type a, type b) \
+{ \
+ return a op b; \
+}
+
+VCMPMN (v8hf, <, lt)
+VCMPMN (v16hf, <, lt)
+VCMPMN (v32hf, <, lt)
+VCMPMN (v8hf, <=, le)
+VCMPMN (v16hf, <=, le)
+VCMPMN (v32hf, <=, le)
+VCMPMN (v8hf, >, gt)
+VCMPMN (v16hf, >, gt)
+VCMPMN (v32hf, >, gt)
+VCMPMN (v8hf, >=, ge)
+VCMPMN (v16hf, >=, ge)
+VCMPMN (v32hf, >=, ge)
+VCMPMN (v8hf, ==, eq)
+VCMPMN (v16hf, ==, eq)
+VCMPMN (v32hf, ==, eq)
--
2.27.0
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2021-09-23 5:46 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-23 5:46 [PATCH 0/7] AVX512FP16: Support bunch of expanders for HFmode and vector HFmodes liuhongt
2021-09-23 5:46 ` [PATCH 1/7] AVX512FP16: Add expander for rint/nearbyinthf2 liuhongt
2021-09-23 5:46 ` [PATCH 2/7] AVX512FP16: Add expander for fmahf4 liuhongt
2021-09-23 5:46 ` [PATCH 3/7] AVX512FP16: Add expander for smin/maxhf3 liuhongt
2021-09-23 5:46 ` [PATCH 4/7] AVX512FP16: Add fix(uns)?_truncmn2 for HF scalar and vector modes liuhongt
2021-09-23 5:46 ` [PATCH 5/7] AVX512FP16: Add float(uns)?mn2 expander liuhongt
2021-09-23 5:46 ` [PATCH 6/7] AVX512FP16: add truncmn2/extendmn2 expanders liuhongt
2021-09-23 5:46 ` [PATCH 7/7] AVX512FP16: Enable vec_cmpmn/vcondmn expanders for HF modes liuhongt
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).