From: Tamar Christina <Tamar.Christina@arm.com>
To: Richard Biener <rguenther@suse.de>,
Richard Sandiford <Richard.Sandiford@arm.com>
Cc: "gcc-patches@gcc.gnu.org" <gcc-patches@gcc.gnu.org>,
nd <nd@arm.com>, "jlaw@ventanamicro.com" <jlaw@ventanamicro.com>
Subject: RE: [PATCH 9/21]middle-end: implement vectorizable_early_exit for codegen of exit code
Date: Wed, 13 Dec 2023 14:13:11 +0000 [thread overview]
Message-ID: <VI1PR08MB5325688E37E6A409E577E00EFF8DA@VI1PR08MB5325.eurprd08.prod.outlook.com> (raw)
In-Reply-To: <nn4ponq7-6n41-592n-7p0o-r95s58r47ro5@fhfr.qr>
[-- Attachment #1: Type: text/plain, Size: 25396 bytes --]
> > > else if (vect_use_mask_type_p (stmt_info))
> > > {
> > > unsigned int precision = stmt_info->mask_precision;
> > > scalar_type = build_nonstandard_integer_type (precision, 1);
> > > vectype = get_mask_type_for_scalar_type (vinfo, scalar_type,
> > > group_size);
> > > if (!vectype)
> > > return opt_result::failure_at (stmt, "not vectorized: unsupported"
> > > " data-type %T\n", scalar_type);
> > >
> > > Richard, do you have any advice here? I suppose vect_determine_precisions
> > > needs to handle the gcond case with bool != 0 somehow and for the
> > > extra mask producer we add here we have to emulate what it would have
> > > done, right?
> >
> > How about handling gconds directly in vect_determine_mask_precision?
> > In a sense it's not needed, since gconds are always roots, and so we
> > could calculate their precision on the fly instead. But handling it in
> > vect_determine_mask_precision feels like it should reduce the number
> > of special cases.
>
> Yeah, that sounds worth trying.
>
> Richard.
So here's a respin with this suggestion and the other issues fixed.
Note that the testcases still need to be updated with the right stanzas.
The patch is much smaller, I still have a small change to
vect_get_vector_types_for_stmt in case we get there on a gcond where
vect_recog_gcond_pattern couldn't apply due to the target missing an
appropriate vectype. The change only gracefully rejects the gcond.
Since patterns cannot apply to the same root twice I've had to also do
the split of the condition out of the gcond in bitfield lowering.
Bootstrapped Regtested on aarch64-none-linux-gnu, x86_64-pc-linux-gnu and no issues.
Ok for master?
Thanks,
Tamar
gcc/ChangeLog:
* tree-vect-patterns.cc (vect_init_pattern_stmt): Support gcond
(vect_recog_bitfield_ref_pattern): Update to split out bool.
(vect_recog_gcond_pattern): New.
(possible_vector_mask_operation_p): Support gcond.
(vect_determine_mask_precision): Likewise.
* tree-vect-stmts.cc (vectorizable_comparison_1): Support stmts without
lhs.
(vectorizable_early_exit): New.
(vect_analyze_stmt, vect_transform_stmt): Use it.
(vect_get_vector_types_for_stmt): Rejects gcond if not lowered by
vect_recog_gcond_pattern.
gcc/testsuite/ChangeLog:
* gcc.dg/vect/vect-early-break_84.c: New test.
* gcc.dg/vect/vect-early-break_85.c: New test.
* gcc.dg/vect/vect-early-break_86.c: New test.
* gcc.dg/vect/vect-early-break_87.c: New test.
* gcc.dg/vect/vect-early-break_88.c: New test.
--- inline copy of patch ---
diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break_84.c b/gcc/testsuite/gcc.dg/vect/vect-early-break_84.c
new file mode 100644
index 0000000000000000000000000000000000000000..0622339491d333b07c2ce895785b5216713097a9
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-early-break_84.c
@@ -0,0 +1,39 @@
+/* { dg-require-effective-target vect_early_break } */
+/* { dg-require-effective-target vect_int } */
+
+/* { dg-final { scan-tree-dump "LOOP VECTORIZED" "vect" } } */
+
+#include <stdbool.h>
+
+#ifndef N
+#define N 17
+#endif
+bool vect_a[N] = { false, false, true, false, false, false,
+ false, false, false, false, false, false,
+ false, false, false, false, false };
+unsigned vect_b[N] = { 0 };
+
+__attribute__ ((noinline, noipa))
+unsigned test4(bool x)
+{
+ unsigned ret = 0;
+ for (int i = 0; i < N; i++)
+ {
+ if (vect_a[i] == x)
+ return 1;
+ vect_a[i] = x;
+
+ }
+ return ret;
+}
+
+extern void abort ();
+
+int main ()
+{
+ if (test4 (true) != 1)
+ abort ();
+
+ if (vect_b[2] != 0 && vect_b[1] == 0)
+ abort ();
+}
diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break_85.c b/gcc/testsuite/gcc.dg/vect/vect-early-break_85.c
new file mode 100644
index 0000000000000000000000000000000000000000..39b3d9bad8681a2d15d7fc7de86bdd3ce0f0bd4e
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-early-break_85.c
@@ -0,0 +1,35 @@
+/* { dg-require-effective-target vect_early_break } */
+/* { dg-require-effective-target vect_int } */
+
+/* { dg-final { scan-tree-dump "LOOP VECTORIZED" "vect" } } */
+
+#ifndef N
+#define N 5
+#endif
+int vect_a[N] = { 5, 4, 8, 4, 6 };
+unsigned vect_b[N] = { 0 };
+
+__attribute__ ((noinline, noipa))
+unsigned test4(int x)
+{
+ unsigned ret = 0;
+ for (int i = 0; i < N; i++)
+ {
+ if (vect_a[i] > x)
+ return 1;
+ vect_a[i] = x;
+
+ }
+ return ret;
+}
+
+extern void abort ();
+
+int main ()
+{
+ if (test4 (7) != 1)
+ abort ();
+
+ if (vect_b[2] != 0 && vect_b[1] == 0)
+ abort ();
+}
diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break_86.c b/gcc/testsuite/gcc.dg/vect/vect-early-break_86.c
new file mode 100644
index 0000000000000000000000000000000000000000..66eb570f4028bca4b631329d7af50c646d3c0cb3
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-early-break_86.c
@@ -0,0 +1,21 @@
+/* { dg-additional-options "-std=gnu89" } */
+/* { dg-require-effective-target vect_early_break } */
+/* { dg-require-effective-target vect_int } */
+
+/* { dg-final { scan-tree-dump "LOOP VECTORIZED" "vect" } } */
+
+extern void abort ();
+extern void exit (int);
+
+__attribute__((noinline, noipa))
+int f(x) {
+ int i;
+ for (i = 0; i < 8 && (x & 1) == 1; i++)
+ x >>= 1;
+ return i;
+}
+main() {
+ if (f(4) != 0)
+ abort();
+ exit(0);
+}
diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break_87.c b/gcc/testsuite/gcc.dg/vect/vect-early-break_87.c
new file mode 100644
index 0000000000000000000000000000000000000000..67be67da0583ba7feda3bed09c42fa735da9b98e
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-early-break_87.c
@@ -0,0 +1,21 @@
+/* { dg-additional-options "-std=gnu89" } */
+/* { dg-require-effective-target vect_early_break } */
+/* { dg-require-effective-target vect_int } */
+
+/* { dg-final { scan-tree-dump "LOOP VECTORIZED" "vect" } } */
+
+extern void abort ();
+extern void exit (int);
+
+__attribute__((noinline, noipa))
+int f(x) {
+ int i;
+ for (i = 0; i < 8 && (x & 1) == 0; i++)
+ x >>= 1;
+ return i;
+}
+main() {
+ if (f(4) != 2)
+ abort();
+ exit(0);
+}
diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break_88.c b/gcc/testsuite/gcc.dg/vect/vect-early-break_88.c
new file mode 100644
index 0000000000000000000000000000000000000000..b64becd588973f58601196bfcb15afbe4bab60f2
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-early-break_88.c
@@ -0,0 +1,36 @@
+/* { dg-require-effective-target vect_early_break } */
+/* { dg-require-effective-target vect_int } */
+
+/* { dg-additional-options "-Ofast --param vect-partial-vector-usage=2" } */
+/* { dg-final { scan-tree-dump "LOOP VECTORIZED" "vect" } } */
+
+#ifndef N
+#define N 5
+#endif
+float vect_a[N] = { 5.1f, 4.2f, 8.0f, 4.25f, 6.5f };
+unsigned vect_b[N] = { 0 };
+
+__attribute__ ((noinline, noipa))
+unsigned test4(double x)
+{
+ unsigned ret = 0;
+ for (int i = 0; i < N; i++)
+ {
+ if (vect_a[i] > x)
+ break;
+ vect_a[i] = x;
+
+ }
+ return ret;
+}
+
+extern void abort ();
+
+int main ()
+{
+ if (test4 (7.0) != 0)
+ abort ();
+
+ if (vect_b[2] != 0 && vect_b[1] == 0)
+ abort ();
+}
diff --git a/gcc/tree-vect-patterns.cc b/gcc/tree-vect-patterns.cc
index 7debe7f0731673cd1bf25cd39d55e23990a73d0e..f6ce27a7c45aa6ce72c402987958ee395c045a14 100644
--- a/gcc/tree-vect-patterns.cc
+++ b/gcc/tree-vect-patterns.cc
@@ -132,6 +132,7 @@ vect_init_pattern_stmt (vec_info *vinfo, gimple *pattern_stmt,
if (!STMT_VINFO_VECTYPE (pattern_stmt_info))
{
gcc_assert (!vectype
+ || is_a <gcond *> (pattern_stmt)
|| (VECTOR_BOOLEAN_TYPE_P (vectype)
== vect_use_mask_type_p (orig_stmt_info)));
STMT_VINFO_VECTYPE (pattern_stmt_info) = vectype;
@@ -2786,15 +2787,24 @@ vect_recog_bitfield_ref_pattern (vec_info *vinfo, stmt_vec_info stmt_info,
if (!lhs)
{
+ if (!vectype)
+ return NULL;
+
append_pattern_def_seq (vinfo, stmt_info, pattern_stmt, vectype);
+ vectype = truth_type_for (vectype);
+
+ tree new_lhs = vect_recog_temp_ssa_var (boolean_type_node, NULL);
gcond *cond_stmt = dyn_cast <gcond *> (stmt_info->stmt);
tree cond_cst = gimple_cond_rhs (cond_stmt);
+ gimple *new_stmt
+ = gimple_build_assign (new_lhs, gimple_cond_code (cond_stmt),
+ gimple_get_lhs (pattern_stmt),
+ fold_convert (container_type, cond_cst));
+ append_pattern_def_seq (vinfo, stmt_info, new_stmt, vectype, container_type);
pattern_stmt
- = gimple_build_cond (gimple_cond_code (cond_stmt),
- gimple_get_lhs (pattern_stmt),
- fold_convert (ret_type, cond_cst),
- gimple_cond_true_label (cond_stmt),
- gimple_cond_false_label (cond_stmt));
+ = gimple_build_cond (NE_EXPR, new_lhs,
+ build_zero_cst (TREE_TYPE (new_lhs)),
+ NULL_TREE, NULL_TREE);
}
*type_out = STMT_VINFO_VECTYPE (stmt_info);
@@ -5553,6 +5563,72 @@ integer_type_for_mask (tree var, vec_info *vinfo)
return build_nonstandard_integer_type (def_stmt_info->mask_precision, 1);
}
+/* Function vect_recog_gcond_pattern
+
+ Try to find pattern like following:
+
+ if (a op b)
+
+ where operator 'op' is not != and convert it to an adjusted boolean pattern
+
+ mask = a op b
+ if (mask != 0)
+
+ and set the mask type on MASK.
+
+ Input:
+
+ * STMT_VINFO: The stmt at the end from which the pattern
+ search begins, i.e. cast of a bool to
+ an integer type.
+
+ Output:
+
+ * TYPE_OUT: The type of the output of this pattern.
+
+ * Return value: A new stmt that will be used to replace the pattern. */
+
+static gimple *
+vect_recog_gcond_pattern (vec_info *vinfo,
+ stmt_vec_info stmt_vinfo, tree *type_out)
+{
+ gimple *last_stmt = STMT_VINFO_STMT (stmt_vinfo);
+ gcond* cond = NULL;
+ if (!(cond = dyn_cast <gcond *> (last_stmt)))
+ return NULL;
+
+ auto lhs = gimple_cond_lhs (cond);
+ auto rhs = gimple_cond_rhs (cond);
+ auto code = gimple_cond_code (cond);
+
+ tree scalar_type = TREE_TYPE (lhs);
+ if (VECTOR_TYPE_P (scalar_type))
+ return NULL;
+
+ if (code == NE_EXPR
+ && zerop (rhs)
+ && VECT_SCALAR_BOOLEAN_TYPE_P (scalar_type))
+ return NULL;
+
+ tree vecitype = get_vectype_for_scalar_type (vinfo, scalar_type);
+ if (vecitype == NULL_TREE)
+ return NULL;
+
+ tree vectype = truth_type_for (vecitype);
+
+ tree new_lhs = vect_recog_temp_ssa_var (boolean_type_node, NULL);
+ gimple *new_stmt = gimple_build_assign (new_lhs, code, lhs, rhs);
+ append_pattern_def_seq (vinfo, stmt_vinfo, new_stmt, vectype, scalar_type);
+
+ gimple *pattern_stmt
+ = gimple_build_cond (NE_EXPR, new_lhs,
+ build_int_cst (TREE_TYPE (new_lhs), 0),
+ NULL_TREE, NULL_TREE);
+ *type_out = vectype;
+ vect_pattern_detected ("vect_recog_gcond_pattern", last_stmt);
+ return pattern_stmt;
+}
+
/* Function vect_recog_bool_pattern
Try to find pattern like following:
@@ -6581,15 +6657,26 @@ static bool
possible_vector_mask_operation_p (stmt_vec_info stmt_info)
{
tree lhs = gimple_get_lhs (stmt_info->stmt);
+ tree_code code = ERROR_MARK;
+ gassign *assign = NULL;
+ gcond *cond = NULL;
+
+ if ((assign = dyn_cast <gassign *> (stmt_info->stmt)))
+ code = gimple_assign_rhs_code (assign);
+ else if ((cond = dyn_cast <gcond *> (stmt_info->stmt)))
+ {
+ lhs = gimple_cond_lhs (cond);
+ code = gimple_cond_code (cond);
+ }
+
if (!lhs
|| TREE_CODE (lhs) != SSA_NAME
|| !VECT_SCALAR_BOOLEAN_TYPE_P (TREE_TYPE (lhs)))
return false;
- if (gassign *assign = dyn_cast <gassign *> (stmt_info->stmt))
+ if (code != ERROR_MARK)
{
- tree_code rhs_code = gimple_assign_rhs_code (assign);
- switch (rhs_code)
+ switch (code)
{
CASE_CONVERT:
case SSA_NAME:
@@ -6600,7 +6687,7 @@ possible_vector_mask_operation_p (stmt_vec_info stmt_info)
return true;
default:
- return TREE_CODE_CLASS (rhs_code) == tcc_comparison;
+ return TREE_CODE_CLASS (code) == tcc_comparison;
}
}
else if (is_a <gphi *> (stmt_info->stmt))
@@ -6647,12 +6734,35 @@ vect_determine_mask_precision (vec_info *vinfo, stmt_vec_info stmt_info)
The number of operations are equal, but M16 would have given
a shorter dependency chain and allowed more ILP. */
unsigned int precision = ~0U;
- if (gassign *assign = dyn_cast <gassign *> (stmt_info->stmt))
+ gimple *stmt = STMT_VINFO_STMT (stmt_info);
+
+ /* If the statement compares two values that shouldn't use vector masks,
+ try comparing the values as normal scalars instead. */
+ tree_code code = ERROR_MARK;
+ tree op0_type;
+ unsigned int nops = -1;
+ unsigned int ops_start = 0;
+
+ if (gassign *assign = dyn_cast <gassign *> (stmt))
+ {
+ code = gimple_assign_rhs_code (assign);
+ op0_type = TREE_TYPE (gimple_assign_rhs1 (assign));
+ nops = gimple_num_ops (assign);
+ ops_start = 1;
+ }
+ else if (gcond *cond = dyn_cast <gcond *> (stmt))
+ {
+ code = gimple_cond_code (cond);
+ op0_type = TREE_TYPE (gimple_cond_lhs (cond));
+ nops = 2;
+ ops_start = 0;
+ }
+
+ if (code != ERROR_MARK)
{
- unsigned int nops = gimple_num_ops (assign);
- for (unsigned int i = 1; i < nops; ++i)
+ for (unsigned int i = ops_start; i < nops; ++i)
{
- tree rhs = gimple_op (assign, i);
+ tree rhs = gimple_op (stmt, i);
if (!VECT_SCALAR_BOOLEAN_TYPE_P (TREE_TYPE (rhs)))
continue;
@@ -6669,19 +6779,15 @@ vect_determine_mask_precision (vec_info *vinfo, stmt_vec_info stmt_info)
}
}
- /* If the statement compares two values that shouldn't use vector masks,
- try comparing the values as normal scalars instead. */
- tree_code rhs_code = gimple_assign_rhs_code (assign);
if (precision == ~0U
- && TREE_CODE_CLASS (rhs_code) == tcc_comparison)
+ && TREE_CODE_CLASS (code) == tcc_comparison)
{
- tree rhs1_type = TREE_TYPE (gimple_assign_rhs1 (assign));
scalar_mode mode;
tree vectype, mask_type;
- if (is_a <scalar_mode> (TYPE_MODE (rhs1_type), &mode)
- && (vectype = get_vectype_for_scalar_type (vinfo, rhs1_type))
- && (mask_type = get_mask_type_for_scalar_type (vinfo, rhs1_type))
- && expand_vec_cmp_expr_p (vectype, mask_type, rhs_code))
+ if (is_a <scalar_mode> (TYPE_MODE (op0_type), &mode)
+ && (vectype = get_vectype_for_scalar_type (vinfo, op0_type))
+ && (mask_type = get_mask_type_for_scalar_type (vinfo, op0_type))
+ && expand_vec_cmp_expr_p (vectype, mask_type, code))
precision = GET_MODE_BITSIZE (mode);
}
}
@@ -6860,6 +6966,7 @@ static vect_recog_func vect_vect_recog_func_ptrs[] = {
{ vect_recog_divmod_pattern, "divmod" },
{ vect_recog_mult_pattern, "mult" },
{ vect_recog_mixed_size_cond_pattern, "mixed_size_cond" },
+ { vect_recog_gcond_pattern, "gcond" },
{ vect_recog_bool_pattern, "bool" },
/* This must come before mask conversion, and includes the parts
of mask conversion that are needed for gather and scatter
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index 582c5e678fad802d6e76300fe3c939b9f2978f17..766450cd85b55ce4dfd45878c5dc44cd09c68681 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -12489,7 +12489,7 @@ vectorizable_comparison_1 (vec_info *vinfo, tree vectype,
vec<tree> vec_oprnds0 = vNULL;
vec<tree> vec_oprnds1 = vNULL;
tree mask_type;
- tree mask;
+ tree mask = NULL_TREE;
if (!STMT_VINFO_RELEVANT_P (stmt_info) && !bb_vinfo)
return false;
@@ -12629,8 +12629,9 @@ vectorizable_comparison_1 (vec_info *vinfo, tree vectype,
/* Transform. */
/* Handle def. */
- lhs = gimple_assign_lhs (STMT_VINFO_STMT (stmt_info));
- mask = vect_create_destination_var (lhs, mask_type);
+ lhs = gimple_get_lhs (STMT_VINFO_STMT (stmt_info));
+ if (lhs)
+ mask = vect_create_destination_var (lhs, mask_type);
vect_get_vec_defs (vinfo, stmt_info, slp_node, ncopies,
rhs1, &vec_oprnds0, vectype,
@@ -12644,7 +12645,10 @@ vectorizable_comparison_1 (vec_info *vinfo, tree vectype,
gimple *new_stmt;
vec_rhs2 = vec_oprnds1[i];
- new_temp = make_ssa_name (mask);
+ if (lhs)
+ new_temp = make_ssa_name (mask);
+ else
+ new_temp = make_temp_ssa_name (mask_type, NULL, "cmp");
if (bitop1 == NOP_EXPR)
{
new_stmt = gimple_build_assign (new_temp, code,
@@ -12723,6 +12727,207 @@ vectorizable_comparison (vec_info *vinfo,
return true;
}
+/* Check to see if the current early break given in STMT_INFO is valid for
+ vectorization. */
+
+static bool
+vectorizable_early_exit (vec_info *vinfo, stmt_vec_info stmt_info,
+ gimple_stmt_iterator *gsi, gimple **vec_stmt,
+ slp_tree slp_node, stmt_vector_for_cost *cost_vec)
+{
+ loop_vec_info loop_vinfo = dyn_cast <loop_vec_info> (vinfo);
+ if (!loop_vinfo
+ || !is_a <gcond *> (STMT_VINFO_STMT (stmt_info)))
+ return false;
+
+ if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_condition_def)
+ return false;
+
+ if (!STMT_VINFO_RELEVANT_P (stmt_info))
+ return false;
+
+ DUMP_VECT_SCOPE ("vectorizable_early_exit");
+
+ auto code = gimple_cond_code (STMT_VINFO_STMT (stmt_info));
+
+ tree vectype = NULL_TREE;
+ slp_tree slp_op0;
+ tree op0;
+ enum vect_def_type dt0;
+ if (!vect_is_simple_use (vinfo, stmt_info, slp_node, 0, &op0, &slp_op0, &dt0,
+ &vectype))
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
+ "use not simple.\n");
+ return false;
+ }
+
+ if (!vectype)
+ return false;
+
+ machine_mode mode = TYPE_MODE (vectype);
+ int ncopies;
+
+ if (slp_node)
+ ncopies = 1;
+ else
+ ncopies = vect_get_num_copies (loop_vinfo, vectype);
+
+ vec_loop_masks *masks = &LOOP_VINFO_MASKS (loop_vinfo);
+ bool masked_loop_p = LOOP_VINFO_FULLY_MASKED_P (loop_vinfo);
+
+ /* Analyze only. */
+ if (!vec_stmt)
+ {
+ if (direct_optab_handler (cbranch_optab, mode) == CODE_FOR_nothing)
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
+ "can't vectorize early exit because the "
+ "target doesn't support flag setting vector "
+ "comparisons.\n");
+ return false;
+ }
+
+ if (ncopies > 1
+ && direct_optab_handler (ior_optab, mode) == CODE_FOR_nothing)
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
+ "can't vectorize early exit because the "
+ "target does not support boolean vector OR for "
+ "type %T.\n", vectype);
+ return false;
+ }
+
+ if (!vectorizable_comparison_1 (vinfo, vectype, stmt_info, code, gsi,
+ vec_stmt, slp_node, cost_vec))
+ return false;
+
+ if (LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo))
+ {
+ if (direct_internal_fn_supported_p (IFN_VCOND_MASK_LEN, vectype,
+ OPTIMIZE_FOR_SPEED))
+ return false;
+ else
+ vect_record_loop_mask (loop_vinfo, masks, ncopies, vectype, NULL);
+ }
+
+
+ return true;
+ }
+
+ /* Tranform. */
+
+ tree new_temp = NULL_TREE;
+ gimple *new_stmt = NULL;
+
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, vect_location, "transform early-exit.\n");
+
+ if (!vectorizable_comparison_1 (vinfo, vectype, stmt_info, code, gsi,
+ vec_stmt, slp_node, cost_vec))
+ gcc_unreachable ();
+
+ gimple *stmt = STMT_VINFO_STMT (stmt_info);
+ basic_block cond_bb = gimple_bb (stmt);
+ gimple_stmt_iterator cond_gsi = gsi_last_bb (cond_bb);
+
+ auto_vec<tree> stmts;
+
+ if (slp_node)
+ stmts.safe_splice (SLP_TREE_VEC_DEFS (slp_node));
+ else
+ {
+ auto vec_stmts = STMT_VINFO_VEC_STMTS (stmt_info);
+ stmts.reserve_exact (vec_stmts.length ());
+ for (auto stmt : vec_stmts)
+ stmts.quick_push (gimple_assign_lhs (stmt));
+ }
+
+ /* Determine if we need to reduce the final value. */
+ if (stmts.length () > 1)
+ {
+ /* We build the reductions in a way to maintain as much parallelism as
+ possible. */
+ auto_vec<tree> workset (stmts.length ());
+
+ /* Mask the statements as we queue them up. Normally we loop over
+ vec_num, but since we inspect the exact results of vectorization
+ we don't need to and instead can just use the stmts themselves. */
+ if (masked_loop_p)
+ for (unsigned i = 0; i < stmts.length (); i++)
+ {
+ tree stmt_mask
+ = vect_get_loop_mask (loop_vinfo, gsi, masks, ncopies, vectype,
+ i);
+ stmt_mask
+ = prepare_vec_mask (loop_vinfo, TREE_TYPE (stmt_mask), stmt_mask,
+ stmts[i], &cond_gsi);
+ workset.quick_push (stmt_mask);
+ }
+ else
+ workset.splice (stmts);
+
+ while (workset.length () > 1)
+ {
+ new_temp = make_temp_ssa_name (vectype, NULL, "vexit_reduc");
+ tree arg0 = workset.pop ();
+ tree arg1 = workset.pop ();
+ new_stmt = gimple_build_assign (new_temp, BIT_IOR_EXPR, arg0, arg1);
+ vect_finish_stmt_generation (loop_vinfo, stmt_info, new_stmt,
+ &cond_gsi);
+ workset.quick_insert (0, new_temp);
+ }
+ }
+ else
+ {
+ new_temp = stmts[0];
+ if (masked_loop_p)
+ {
+ tree mask
+ = vect_get_loop_mask (loop_vinfo, gsi, masks, ncopies, vectype, 0);
+ new_temp = prepare_vec_mask (loop_vinfo, TREE_TYPE (mask), mask,
+ new_temp, &cond_gsi);
+ }
+ }
+
+ gcc_assert (new_temp);
+
+ /* Now build the new conditional. Pattern gimple_conds get dropped during
+ codegen so we must replace the original insn. */
+ stmt = STMT_VINFO_STMT (vect_orig_stmt (stmt_info));
+ gcond *cond_stmt = as_a <gcond *>(stmt);
+ /* When vectorizing we assume that if the branch edge is taken that we're
+ exiting the loop. This is not however always the case as the compiler will
+ rewrite conditions to always be a comparison against 0. To do this it
+ sometimes flips the edges. This is fine for scalar, but for vector we
+ then have to flip the test, as we're still assuming that if you take the
+ branch edge that we found the exit condition. */
+ auto new_code = NE_EXPR;
+ tree cst = build_zero_cst (vectype);
+ if (flow_bb_inside_loop_p (LOOP_VINFO_LOOP (loop_vinfo),
+ BRANCH_EDGE (gimple_bb (cond_stmt))->dest))
+ {
+ new_code = EQ_EXPR;
+ cst = build_minus_one_cst (vectype);
+ }
+
+ gimple_cond_set_condition (cond_stmt, new_code, new_temp, cst);
+ update_stmt (stmt);
+
+ if (slp_node)
+ SLP_TREE_VEC_DEFS (slp_node).truncate (0);
+ else
+ STMT_VINFO_VEC_STMTS (stmt_info).truncate (0);
+
+ if (!slp_node)
+ *vec_stmt = stmt;
+
+ return true;
+}
+
/* If SLP_NODE is nonnull, return true if vectorizable_live_operation
can handle all live statements in the node. Otherwise return true
if STMT_INFO is not live or if vectorizable_live_operation can handle it.
@@ -12949,7 +13154,9 @@ vect_analyze_stmt (vec_info *vinfo,
|| vectorizable_lc_phi (as_a <loop_vec_info> (vinfo),
stmt_info, NULL, node)
|| vectorizable_recurr (as_a <loop_vec_info> (vinfo),
- stmt_info, NULL, node, cost_vec));
+ stmt_info, NULL, node, cost_vec)
+ || vectorizable_early_exit (vinfo, stmt_info, NULL, NULL, node,
+ cost_vec));
else
{
if (bb_vinfo)
@@ -12972,7 +13179,10 @@ vect_analyze_stmt (vec_info *vinfo,
NULL, NULL, node, cost_vec)
|| vectorizable_comparison (vinfo, stmt_info, NULL, NULL, node,
cost_vec)
- || vectorizable_phi (vinfo, stmt_info, NULL, node, cost_vec));
+ || vectorizable_phi (vinfo, stmt_info, NULL, node, cost_vec)
+ || vectorizable_early_exit (vinfo, stmt_info, NULL, NULL, node,
+ cost_vec));
+
}
if (node)
@@ -13131,6 +13341,12 @@ vect_transform_stmt (vec_info *vinfo,
gcc_assert (done);
break;
+ case loop_exit_ctrl_vec_info_type:
+ done = vectorizable_early_exit (vinfo, stmt_info, gsi, &vec_stmt,
+ slp_node, NULL);
+ gcc_assert (done);
+ break;
+
default:
if (!STMT_VINFO_LIVE_P (stmt_info))
{
@@ -14321,6 +14537,14 @@ vect_get_vector_types_for_stmt (vec_info *vinfo, stmt_vec_info stmt_info,
}
else
{
+ /* If we got here with a gcond it means that the target had no available vector
+ mode for the scalar type. We can't vectorize so abort. */
+ if (is_a <gcond *> (stmt))
+ return opt_result::failure_at (stmt,
+ "not vectorized:"
+ " unsupported data-type for gcond %T\n",
+ scalar_type);
+
if (data_reference *dr = STMT_VINFO_DATA_REF (stmt_info))
scalar_type = TREE_TYPE (DR_REF (dr));
else if (gimple_call_internal_p (stmt, IFN_MASK_STORE))
[-- Attachment #2: rb17969 (2).patch --]
[-- Type: application/octet-stream, Size: 22031 bytes --]
diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break_84.c b/gcc/testsuite/gcc.dg/vect/vect-early-break_84.c
new file mode 100644
index 0000000000000000000000000000000000000000..0622339491d333b07c2ce895785b5216713097a9
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-early-break_84.c
@@ -0,0 +1,39 @@
+/* { dg-require-effective-target vect_early_break } */
+/* { dg-require-effective-target vect_int } */
+
+/* { dg-final { scan-tree-dump "LOOP VECTORIZED" "vect" } } */
+
+#include <stdbool.h>
+
+#ifndef N
+#define N 17
+#endif
+bool vect_a[N] = { false, false, true, false, false, false,
+ false, false, false, false, false, false,
+ false, false, false, false, false };
+unsigned vect_b[N] = { 0 };
+
+__attribute__ ((noinline, noipa))
+unsigned test4(bool x)
+{
+ unsigned ret = 0;
+ for (int i = 0; i < N; i++)
+ {
+ if (vect_a[i] == x)
+ return 1;
+ vect_a[i] = x;
+
+ }
+ return ret;
+}
+
+extern void abort ();
+
+int main ()
+{
+ if (test4 (true) != 1)
+ abort ();
+
+ if (vect_b[2] != 0 && vect_b[1] == 0)
+ abort ();
+}
diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break_85.c b/gcc/testsuite/gcc.dg/vect/vect-early-break_85.c
new file mode 100644
index 0000000000000000000000000000000000000000..39b3d9bad8681a2d15d7fc7de86bdd3ce0f0bd4e
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-early-break_85.c
@@ -0,0 +1,35 @@
+/* { dg-require-effective-target vect_early_break } */
+/* { dg-require-effective-target vect_int } */
+
+/* { dg-final { scan-tree-dump "LOOP VECTORIZED" "vect" } } */
+
+#ifndef N
+#define N 5
+#endif
+int vect_a[N] = { 5, 4, 8, 4, 6 };
+unsigned vect_b[N] = { 0 };
+
+__attribute__ ((noinline, noipa))
+unsigned test4(int x)
+{
+ unsigned ret = 0;
+ for (int i = 0; i < N; i++)
+ {
+ if (vect_a[i] > x)
+ return 1;
+ vect_a[i] = x;
+
+ }
+ return ret;
+}
+
+extern void abort ();
+
+int main ()
+{
+ if (test4 (7) != 1)
+ abort ();
+
+ if (vect_b[2] != 0 && vect_b[1] == 0)
+ abort ();
+}
diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break_86.c b/gcc/testsuite/gcc.dg/vect/vect-early-break_86.c
new file mode 100644
index 0000000000000000000000000000000000000000..66eb570f4028bca4b631329d7af50c646d3c0cb3
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-early-break_86.c
@@ -0,0 +1,21 @@
+/* { dg-additional-options "-std=gnu89" } */
+/* { dg-require-effective-target vect_early_break } */
+/* { dg-require-effective-target vect_int } */
+
+/* { dg-final { scan-tree-dump "LOOP VECTORIZED" "vect" } } */
+
+extern void abort ();
+extern void exit (int);
+
+__attribute__((noinline, noipa))
+int f(x) {
+ int i;
+ for (i = 0; i < 8 && (x & 1) == 1; i++)
+ x >>= 1;
+ return i;
+}
+main() {
+ if (f(4) != 0)
+ abort();
+ exit(0);
+}
diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break_87.c b/gcc/testsuite/gcc.dg/vect/vect-early-break_87.c
new file mode 100644
index 0000000000000000000000000000000000000000..67be67da0583ba7feda3bed09c42fa735da9b98e
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-early-break_87.c
@@ -0,0 +1,21 @@
+/* { dg-additional-options "-std=gnu89" } */
+/* { dg-require-effective-target vect_early_break } */
+/* { dg-require-effective-target vect_int } */
+
+/* { dg-final { scan-tree-dump "LOOP VECTORIZED" "vect" } } */
+
+extern void abort ();
+extern void exit (int);
+
+__attribute__((noinline, noipa))
+int f(x) {
+ int i;
+ for (i = 0; i < 8 && (x & 1) == 0; i++)
+ x >>= 1;
+ return i;
+}
+main() {
+ if (f(4) != 2)
+ abort();
+ exit(0);
+}
diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break_88.c b/gcc/testsuite/gcc.dg/vect/vect-early-break_88.c
new file mode 100644
index 0000000000000000000000000000000000000000..b64becd588973f58601196bfcb15afbe4bab60f2
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-early-break_88.c
@@ -0,0 +1,36 @@
+/* { dg-require-effective-target vect_early_break } */
+/* { dg-require-effective-target vect_int } */
+
+/* { dg-additional-options "-Ofast --param vect-partial-vector-usage=2" } */
+/* { dg-final { scan-tree-dump "LOOP VECTORIZED" "vect" } } */
+
+#ifndef N
+#define N 5
+#endif
+float vect_a[N] = { 5.1f, 4.2f, 8.0f, 4.25f, 6.5f };
+unsigned vect_b[N] = { 0 };
+
+__attribute__ ((noinline, noipa))
+unsigned test4(double x)
+{
+ unsigned ret = 0;
+ for (int i = 0; i < N; i++)
+ {
+ if (vect_a[i] > x)
+ break;
+ vect_a[i] = x;
+
+ }
+ return ret;
+}
+
+extern void abort ();
+
+int main ()
+{
+ if (test4 (7.0) != 0)
+ abort ();
+
+ if (vect_b[2] != 0 && vect_b[1] == 0)
+ abort ();
+}
diff --git a/gcc/tree-vect-patterns.cc b/gcc/tree-vect-patterns.cc
index 7debe7f0731673cd1bf25cd39d55e23990a73d0e..f6ce27a7c45aa6ce72c402987958ee395c045a14 100644
--- a/gcc/tree-vect-patterns.cc
+++ b/gcc/tree-vect-patterns.cc
@@ -132,6 +132,7 @@ vect_init_pattern_stmt (vec_info *vinfo, gimple *pattern_stmt,
if (!STMT_VINFO_VECTYPE (pattern_stmt_info))
{
gcc_assert (!vectype
+ || is_a <gcond *> (pattern_stmt)
|| (VECTOR_BOOLEAN_TYPE_P (vectype)
== vect_use_mask_type_p (orig_stmt_info)));
STMT_VINFO_VECTYPE (pattern_stmt_info) = vectype;
@@ -2786,15 +2787,24 @@ vect_recog_bitfield_ref_pattern (vec_info *vinfo, stmt_vec_info stmt_info,
if (!lhs)
{
+ if (!vectype)
+ return NULL;
+
append_pattern_def_seq (vinfo, stmt_info, pattern_stmt, vectype);
+ vectype = truth_type_for (vectype);
+
+ tree new_lhs = vect_recog_temp_ssa_var (boolean_type_node, NULL);
gcond *cond_stmt = dyn_cast <gcond *> (stmt_info->stmt);
tree cond_cst = gimple_cond_rhs (cond_stmt);
+ gimple *new_stmt
+ = gimple_build_assign (new_lhs, gimple_cond_code (cond_stmt),
+ gimple_get_lhs (pattern_stmt),
+ fold_convert (container_type, cond_cst));
+ append_pattern_def_seq (vinfo, stmt_info, new_stmt, vectype, container_type);
pattern_stmt
- = gimple_build_cond (gimple_cond_code (cond_stmt),
- gimple_get_lhs (pattern_stmt),
- fold_convert (ret_type, cond_cst),
- gimple_cond_true_label (cond_stmt),
- gimple_cond_false_label (cond_stmt));
+ = gimple_build_cond (NE_EXPR, new_lhs,
+ build_zero_cst (TREE_TYPE (new_lhs)),
+ NULL_TREE, NULL_TREE);
}
*type_out = STMT_VINFO_VECTYPE (stmt_info);
@@ -5553,6 +5563,72 @@ integer_type_for_mask (tree var, vec_info *vinfo)
return build_nonstandard_integer_type (def_stmt_info->mask_precision, 1);
}
+/* Function vect_recog_gcond_pattern
+
+ Try to find pattern like following:
+
+ if (a op b)
+
+ where operator 'op' is not != and convert it to an adjusted boolean pattern
+
+ mask = a op b
+ if (mask != 0)
+
+ and set the mask type on MASK.
+
+ Input:
+
+ * STMT_VINFO: The stmt at the end from which the pattern
+ search begins, i.e. cast of a bool to
+ an integer type.
+
+ Output:
+
+ * TYPE_OUT: The type of the output of this pattern.
+
+ * Return value: A new stmt that will be used to replace the pattern. */
+
+static gimple *
+vect_recog_gcond_pattern (vec_info *vinfo,
+ stmt_vec_info stmt_vinfo, tree *type_out)
+{
+ gimple *last_stmt = STMT_VINFO_STMT (stmt_vinfo);
+ gcond* cond = NULL;
+ if (!(cond = dyn_cast <gcond *> (last_stmt)))
+ return NULL;
+
+ auto lhs = gimple_cond_lhs (cond);
+ auto rhs = gimple_cond_rhs (cond);
+ auto code = gimple_cond_code (cond);
+
+ tree scalar_type = TREE_TYPE (lhs);
+ if (VECTOR_TYPE_P (scalar_type))
+ return NULL;
+
+ if (code == NE_EXPR
+ && zerop (rhs)
+ && VECT_SCALAR_BOOLEAN_TYPE_P (scalar_type))
+ return NULL;
+
+ tree vecitype = get_vectype_for_scalar_type (vinfo, scalar_type);
+ if (vecitype == NULL_TREE)
+ return NULL;
+
+ tree vectype = truth_type_for (vecitype);
+
+ tree new_lhs = vect_recog_temp_ssa_var (boolean_type_node, NULL);
+ gimple *new_stmt = gimple_build_assign (new_lhs, code, lhs, rhs);
+ append_pattern_def_seq (vinfo, stmt_vinfo, new_stmt, vectype, scalar_type);
+
+ gimple *pattern_stmt
+ = gimple_build_cond (NE_EXPR, new_lhs,
+ build_int_cst (TREE_TYPE (new_lhs), 0),
+ NULL_TREE, NULL_TREE);
+ *type_out = vectype;
+ vect_pattern_detected ("vect_recog_gcond_pattern", last_stmt);
+ return pattern_stmt;
+}
+
/* Function vect_recog_bool_pattern
Try to find pattern like following:
@@ -6581,15 +6657,26 @@ static bool
possible_vector_mask_operation_p (stmt_vec_info stmt_info)
{
tree lhs = gimple_get_lhs (stmt_info->stmt);
+ tree_code code = ERROR_MARK;
+ gassign *assign = NULL;
+ gcond *cond = NULL;
+
+ if ((assign = dyn_cast <gassign *> (stmt_info->stmt)))
+ code = gimple_assign_rhs_code (assign);
+ else if ((cond = dyn_cast <gcond *> (stmt_info->stmt)))
+ {
+ lhs = gimple_cond_lhs (cond);
+ code = gimple_cond_code (cond);
+ }
+
if (!lhs
|| TREE_CODE (lhs) != SSA_NAME
|| !VECT_SCALAR_BOOLEAN_TYPE_P (TREE_TYPE (lhs)))
return false;
- if (gassign *assign = dyn_cast <gassign *> (stmt_info->stmt))
+ if (code != ERROR_MARK)
{
- tree_code rhs_code = gimple_assign_rhs_code (assign);
- switch (rhs_code)
+ switch (code)
{
CASE_CONVERT:
case SSA_NAME:
@@ -6600,7 +6687,7 @@ possible_vector_mask_operation_p (stmt_vec_info stmt_info)
return true;
default:
- return TREE_CODE_CLASS (rhs_code) == tcc_comparison;
+ return TREE_CODE_CLASS (code) == tcc_comparison;
}
}
else if (is_a <gphi *> (stmt_info->stmt))
@@ -6647,12 +6734,35 @@ vect_determine_mask_precision (vec_info *vinfo, stmt_vec_info stmt_info)
The number of operations are equal, but M16 would have given
a shorter dependency chain and allowed more ILP. */
unsigned int precision = ~0U;
- if (gassign *assign = dyn_cast <gassign *> (stmt_info->stmt))
+ gimple *stmt = STMT_VINFO_STMT (stmt_info);
+
+ /* If the statement compares two values that shouldn't use vector masks,
+ try comparing the values as normal scalars instead. */
+ tree_code code = ERROR_MARK;
+ tree op0_type;
+ unsigned int nops = -1;
+ unsigned int ops_start = 0;
+
+ if (gassign *assign = dyn_cast <gassign *> (stmt))
+ {
+ code = gimple_assign_rhs_code (assign);
+ op0_type = TREE_TYPE (gimple_assign_rhs1 (assign));
+ nops = gimple_num_ops (assign);
+ ops_start = 1;
+ }
+ else if (gcond *cond = dyn_cast <gcond *> (stmt))
+ {
+ code = gimple_cond_code (cond);
+ op0_type = TREE_TYPE (gimple_cond_lhs (cond));
+ nops = 2;
+ ops_start = 0;
+ }
+
+ if (code != ERROR_MARK)
{
- unsigned int nops = gimple_num_ops (assign);
- for (unsigned int i = 1; i < nops; ++i)
+ for (unsigned int i = ops_start; i < nops; ++i)
{
- tree rhs = gimple_op (assign, i);
+ tree rhs = gimple_op (stmt, i);
if (!VECT_SCALAR_BOOLEAN_TYPE_P (TREE_TYPE (rhs)))
continue;
@@ -6669,19 +6779,15 @@ vect_determine_mask_precision (vec_info *vinfo, stmt_vec_info stmt_info)
}
}
- /* If the statement compares two values that shouldn't use vector masks,
- try comparing the values as normal scalars instead. */
- tree_code rhs_code = gimple_assign_rhs_code (assign);
if (precision == ~0U
- && TREE_CODE_CLASS (rhs_code) == tcc_comparison)
+ && TREE_CODE_CLASS (code) == tcc_comparison)
{
- tree rhs1_type = TREE_TYPE (gimple_assign_rhs1 (assign));
scalar_mode mode;
tree vectype, mask_type;
- if (is_a <scalar_mode> (TYPE_MODE (rhs1_type), &mode)
- && (vectype = get_vectype_for_scalar_type (vinfo, rhs1_type))
- && (mask_type = get_mask_type_for_scalar_type (vinfo, rhs1_type))
- && expand_vec_cmp_expr_p (vectype, mask_type, rhs_code))
+ if (is_a <scalar_mode> (TYPE_MODE (op0_type), &mode)
+ && (vectype = get_vectype_for_scalar_type (vinfo, op0_type))
+ && (mask_type = get_mask_type_for_scalar_type (vinfo, op0_type))
+ && expand_vec_cmp_expr_p (vectype, mask_type, code))
precision = GET_MODE_BITSIZE (mode);
}
}
@@ -6860,6 +6966,7 @@ static vect_recog_func vect_vect_recog_func_ptrs[] = {
{ vect_recog_divmod_pattern, "divmod" },
{ vect_recog_mult_pattern, "mult" },
{ vect_recog_mixed_size_cond_pattern, "mixed_size_cond" },
+ { vect_recog_gcond_pattern, "gcond" },
{ vect_recog_bool_pattern, "bool" },
/* This must come before mask conversion, and includes the parts
of mask conversion that are needed for gather and scatter
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index 582c5e678fad802d6e76300fe3c939b9f2978f17..766450cd85b55ce4dfd45878c5dc44cd09c68681 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -12489,7 +12489,7 @@ vectorizable_comparison_1 (vec_info *vinfo, tree vectype,
vec<tree> vec_oprnds0 = vNULL;
vec<tree> vec_oprnds1 = vNULL;
tree mask_type;
- tree mask;
+ tree mask = NULL_TREE;
if (!STMT_VINFO_RELEVANT_P (stmt_info) && !bb_vinfo)
return false;
@@ -12629,8 +12629,9 @@ vectorizable_comparison_1 (vec_info *vinfo, tree vectype,
/* Transform. */
/* Handle def. */
- lhs = gimple_assign_lhs (STMT_VINFO_STMT (stmt_info));
- mask = vect_create_destination_var (lhs, mask_type);
+ lhs = gimple_get_lhs (STMT_VINFO_STMT (stmt_info));
+ if (lhs)
+ mask = vect_create_destination_var (lhs, mask_type);
vect_get_vec_defs (vinfo, stmt_info, slp_node, ncopies,
rhs1, &vec_oprnds0, vectype,
@@ -12644,7 +12645,10 @@ vectorizable_comparison_1 (vec_info *vinfo, tree vectype,
gimple *new_stmt;
vec_rhs2 = vec_oprnds1[i];
- new_temp = make_ssa_name (mask);
+ if (lhs)
+ new_temp = make_ssa_name (mask);
+ else
+ new_temp = make_temp_ssa_name (mask_type, NULL, "cmp");
if (bitop1 == NOP_EXPR)
{
new_stmt = gimple_build_assign (new_temp, code,
@@ -12723,6 +12727,207 @@ vectorizable_comparison (vec_info *vinfo,
return true;
}
+/* Check to see if the current early break given in STMT_INFO is valid for
+ vectorization. */
+
+static bool
+vectorizable_early_exit (vec_info *vinfo, stmt_vec_info stmt_info,
+ gimple_stmt_iterator *gsi, gimple **vec_stmt,
+ slp_tree slp_node, stmt_vector_for_cost *cost_vec)
+{
+ loop_vec_info loop_vinfo = dyn_cast <loop_vec_info> (vinfo);
+ if (!loop_vinfo
+ || !is_a <gcond *> (STMT_VINFO_STMT (stmt_info)))
+ return false;
+
+ if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_condition_def)
+ return false;
+
+ if (!STMT_VINFO_RELEVANT_P (stmt_info))
+ return false;
+
+ DUMP_VECT_SCOPE ("vectorizable_early_exit");
+
+ auto code = gimple_cond_code (STMT_VINFO_STMT (stmt_info));
+
+ tree vectype = NULL_TREE;
+ slp_tree slp_op0;
+ tree op0;
+ enum vect_def_type dt0;
+ if (!vect_is_simple_use (vinfo, stmt_info, slp_node, 0, &op0, &slp_op0, &dt0,
+ &vectype))
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
+ "use not simple.\n");
+ return false;
+ }
+
+ if (!vectype)
+ return false;
+
+ machine_mode mode = TYPE_MODE (vectype);
+ int ncopies;
+
+ if (slp_node)
+ ncopies = 1;
+ else
+ ncopies = vect_get_num_copies (loop_vinfo, vectype);
+
+ vec_loop_masks *masks = &LOOP_VINFO_MASKS (loop_vinfo);
+ bool masked_loop_p = LOOP_VINFO_FULLY_MASKED_P (loop_vinfo);
+
+ /* Analyze only. */
+ if (!vec_stmt)
+ {
+ if (direct_optab_handler (cbranch_optab, mode) == CODE_FOR_nothing)
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
+ "can't vectorize early exit because the "
+ "target doesn't support flag setting vector "
+ "comparisons.\n");
+ return false;
+ }
+
+ if (ncopies > 1
+ && direct_optab_handler (ior_optab, mode) == CODE_FOR_nothing)
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
+ "can't vectorize early exit because the "
+ "target does not support boolean vector OR for "
+ "type %T.\n", vectype);
+ return false;
+ }
+
+ if (!vectorizable_comparison_1 (vinfo, vectype, stmt_info, code, gsi,
+ vec_stmt, slp_node, cost_vec))
+ return false;
+
+ if (LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo))
+ {
+ if (direct_internal_fn_supported_p (IFN_VCOND_MASK_LEN, vectype,
+ OPTIMIZE_FOR_SPEED))
+ return false;
+ else
+ vect_record_loop_mask (loop_vinfo, masks, ncopies, vectype, NULL);
+ }
+
+
+ return true;
+ }
+
+ /* Tranform. */
+
+ tree new_temp = NULL_TREE;
+ gimple *new_stmt = NULL;
+
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, vect_location, "transform early-exit.\n");
+
+ if (!vectorizable_comparison_1 (vinfo, vectype, stmt_info, code, gsi,
+ vec_stmt, slp_node, cost_vec))
+ gcc_unreachable ();
+
+ gimple *stmt = STMT_VINFO_STMT (stmt_info);
+ basic_block cond_bb = gimple_bb (stmt);
+ gimple_stmt_iterator cond_gsi = gsi_last_bb (cond_bb);
+
+ auto_vec<tree> stmts;
+
+ if (slp_node)
+ stmts.safe_splice (SLP_TREE_VEC_DEFS (slp_node));
+ else
+ {
+ auto vec_stmts = STMT_VINFO_VEC_STMTS (stmt_info);
+ stmts.reserve_exact (vec_stmts.length ());
+ for (auto stmt : vec_stmts)
+ stmts.quick_push (gimple_assign_lhs (stmt));
+ }
+
+ /* Determine if we need to reduce the final value. */
+ if (stmts.length () > 1)
+ {
+ /* We build the reductions in a way to maintain as much parallelism as
+ possible. */
+ auto_vec<tree> workset (stmts.length ());
+
+ /* Mask the statements as we queue them up. Normally we loop over
+ vec_num, but since we inspect the exact results of vectorization
+ we don't need to and instead can just use the stmts themselves. */
+ if (masked_loop_p)
+ for (unsigned i = 0; i < stmts.length (); i++)
+ {
+ tree stmt_mask
+ = vect_get_loop_mask (loop_vinfo, gsi, masks, ncopies, vectype,
+ i);
+ stmt_mask
+ = prepare_vec_mask (loop_vinfo, TREE_TYPE (stmt_mask), stmt_mask,
+ stmts[i], &cond_gsi);
+ workset.quick_push (stmt_mask);
+ }
+ else
+ workset.splice (stmts);
+
+ while (workset.length () > 1)
+ {
+ new_temp = make_temp_ssa_name (vectype, NULL, "vexit_reduc");
+ tree arg0 = workset.pop ();
+ tree arg1 = workset.pop ();
+ new_stmt = gimple_build_assign (new_temp, BIT_IOR_EXPR, arg0, arg1);
+ vect_finish_stmt_generation (loop_vinfo, stmt_info, new_stmt,
+ &cond_gsi);
+ workset.quick_insert (0, new_temp);
+ }
+ }
+ else
+ {
+ new_temp = stmts[0];
+ if (masked_loop_p)
+ {
+ tree mask
+ = vect_get_loop_mask (loop_vinfo, gsi, masks, ncopies, vectype, 0);
+ new_temp = prepare_vec_mask (loop_vinfo, TREE_TYPE (mask), mask,
+ new_temp, &cond_gsi);
+ }
+ }
+
+ gcc_assert (new_temp);
+
+ /* Now build the new conditional. Pattern gimple_conds get dropped during
+ codegen so we must replace the original insn. */
+ stmt = STMT_VINFO_STMT (vect_orig_stmt (stmt_info));
+ gcond *cond_stmt = as_a <gcond *>(stmt);
+ /* When vectorizing we assume that if the branch edge is taken that we're
+ exiting the loop. This is not however always the case as the compiler will
+ rewrite conditions to always be a comparison against 0. To do this it
+ sometimes flips the edges. This is fine for scalar, but for vector we
+ then have to flip the test, as we're still assuming that if you take the
+ branch edge that we found the exit condition. */
+ auto new_code = NE_EXPR;
+ tree cst = build_zero_cst (vectype);
+ if (flow_bb_inside_loop_p (LOOP_VINFO_LOOP (loop_vinfo),
+ BRANCH_EDGE (gimple_bb (cond_stmt))->dest))
+ {
+ new_code = EQ_EXPR;
+ cst = build_minus_one_cst (vectype);
+ }
+
+ gimple_cond_set_condition (cond_stmt, new_code, new_temp, cst);
+ update_stmt (stmt);
+
+ if (slp_node)
+ SLP_TREE_VEC_DEFS (slp_node).truncate (0);
+ else
+ STMT_VINFO_VEC_STMTS (stmt_info).truncate (0);
+
+ if (!slp_node)
+ *vec_stmt = stmt;
+
+ return true;
+}
+
/* If SLP_NODE is nonnull, return true if vectorizable_live_operation
can handle all live statements in the node. Otherwise return true
if STMT_INFO is not live or if vectorizable_live_operation can handle it.
@@ -12949,7 +13154,9 @@ vect_analyze_stmt (vec_info *vinfo,
|| vectorizable_lc_phi (as_a <loop_vec_info> (vinfo),
stmt_info, NULL, node)
|| vectorizable_recurr (as_a <loop_vec_info> (vinfo),
- stmt_info, NULL, node, cost_vec));
+ stmt_info, NULL, node, cost_vec)
+ || vectorizable_early_exit (vinfo, stmt_info, NULL, NULL, node,
+ cost_vec));
else
{
if (bb_vinfo)
@@ -12972,7 +13179,10 @@ vect_analyze_stmt (vec_info *vinfo,
NULL, NULL, node, cost_vec)
|| vectorizable_comparison (vinfo, stmt_info, NULL, NULL, node,
cost_vec)
- || vectorizable_phi (vinfo, stmt_info, NULL, node, cost_vec));
+ || vectorizable_phi (vinfo, stmt_info, NULL, node, cost_vec)
+ || vectorizable_early_exit (vinfo, stmt_info, NULL, NULL, node,
+ cost_vec));
+
}
if (node)
@@ -13131,6 +13341,12 @@ vect_transform_stmt (vec_info *vinfo,
gcc_assert (done);
break;
+ case loop_exit_ctrl_vec_info_type:
+ done = vectorizable_early_exit (vinfo, stmt_info, gsi, &vec_stmt,
+ slp_node, NULL);
+ gcc_assert (done);
+ break;
+
default:
if (!STMT_VINFO_LIVE_P (stmt_info))
{
@@ -14321,6 +14537,14 @@ vect_get_vector_types_for_stmt (vec_info *vinfo, stmt_vec_info stmt_info,
}
else
{
+ /* If we got here with a gcond it means that the target had no available vector
+ mode for the scalar type. We can't vectorize so abort. */
+ if (is_a <gcond *> (stmt))
+ return opt_result::failure_at (stmt,
+ "not vectorized:"
+ " unsupported data-type for gcond %T\n",
+ scalar_type);
+
if (data_reference *dr = STMT_VINFO_DATA_REF (stmt_info))
scalar_type = TREE_TYPE (DR_REF (dr));
else if (gimple_call_internal_p (stmt, IFN_MASK_STORE))
next prev parent reply other threads:[~2023-12-13 14:13 UTC|newest]
Thread overview: 202+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-06-28 13:40 [PATCH v5 0/19] Support early break/return auto-vectorization Tamar Christina
2023-06-28 13:41 ` [PATCH 1/19]middle-end ifcvt: Support bitfield lowering of multiple-exit loops Tamar Christina
2023-07-04 11:29 ` Richard Biener
2023-06-28 13:41 ` [PATCH 2/19][front-end] C/C++ front-end: add pragma GCC novector Tamar Christina
2023-06-29 22:17 ` Jason Merrill
2023-06-30 16:18 ` Tamar Christina
2023-06-30 16:44 ` Jason Merrill
2023-06-28 13:42 ` [PATCH 3/19]middle-end clean up vect testsuite using pragma novector Tamar Christina
2023-06-28 13:54 ` Tamar Christina
2023-07-04 11:31 ` Richard Biener
2023-06-28 13:43 ` [PATCH 4/19]middle-end: Fix scale_loop_frequencies segfault on multiple-exits Tamar Christina
2023-07-04 11:52 ` Richard Biener
2023-07-04 14:57 ` Jan Hubicka
2023-07-06 14:34 ` Jan Hubicka
2023-07-07 5:59 ` Richard Biener
2023-07-07 12:20 ` Jan Hubicka
2023-07-07 12:27 ` Tamar Christina
2023-07-07 14:10 ` Jan Hubicka
2023-07-10 7:07 ` Richard Biener
2023-07-10 8:33 ` Jan Hubicka
2023-07-10 9:24 ` Richard Biener
2023-07-10 9:23 ` Jan Hubicka
2023-07-10 9:29 ` Richard Biener
2023-07-11 9:28 ` Jan Hubicka
2023-07-11 10:31 ` Richard Biener
2023-07-11 12:40 ` Jan Hubicka
2023-07-11 13:04 ` Richard Biener
2023-06-28 13:43 ` [PATCH 5/19]middle-end: Enable bit-field vectorization to work correctly when we're vectoring inside conds Tamar Christina
2023-07-04 12:05 ` Richard Biener
2023-07-10 15:32 ` Tamar Christina
2023-07-11 11:03 ` Richard Biener
2023-06-28 13:44 ` [PATCH 6/19]middle-end: Don't enter piecewise expansion if VF is not constant Tamar Christina
2023-07-04 12:10 ` Richard Biener
2023-07-06 10:37 ` Tamar Christina
2023-07-06 10:51 ` Richard Biener
2023-06-28 13:44 ` [PATCH 7/19]middle-end: Refactor vectorizer loop conditionals and separate out IV to new variables Tamar Christina
2023-07-13 11:32 ` Richard Biener
2023-07-13 11:54 ` Tamar Christina
2023-07-13 12:10 ` Richard Biener
2023-06-28 13:45 ` [PATCH 8/19]middle-end: updated niters analysis to handle multiple exits Tamar Christina
2023-07-13 11:49 ` Richard Biener
2023-07-13 12:03 ` Tamar Christina
2023-07-14 9:09 ` Richard Biener
2023-06-28 13:45 ` [PATCH 9/19]AArch64 middle-end: refactor vectorizable_comparison to make the main body re-usable Tamar Christina
2023-06-28 13:55 ` [PATCH 9/19] " Tamar Christina
2023-07-13 16:23 ` Richard Biener
2023-06-28 13:46 ` [PATCH 10/19]middle-end: implement vectorizable_early_break Tamar Christina
2023-06-28 13:46 ` [PATCH 11/19]middle-end: implement code motion for early break Tamar Christina
2023-06-28 13:47 ` [PATCH 12/19]middle-end: implement loop peeling and IV updates " Tamar Christina
2023-07-13 17:31 ` Richard Biener
2023-07-13 19:05 ` Tamar Christina
2023-07-14 13:34 ` Richard Biener
2023-07-17 10:56 ` Tamar Christina
2023-07-17 12:48 ` Richard Biener
2023-08-18 11:35 ` Tamar Christina
2023-08-18 12:53 ` Richard Biener
2023-08-18 13:12 ` Tamar Christina
2023-08-18 13:15 ` Richard Biener
2023-10-23 20:21 ` Tamar Christina
2023-06-28 13:47 ` [PATCH 13/19]middle-end testsuite: un-xfail TSVC loops that check for exit control flow vectorization Tamar Christina
2023-06-28 13:47 ` [PATCH 14/19]middle-end testsuite: Add new tests for early break vectorization Tamar Christina
2023-06-28 13:48 ` [PATCH 15/19]AArch64: Add implementation for vector cbranch for Advanced SIMD Tamar Christina
2023-06-28 13:48 ` [PATCH 16/19]AArch64 Add optimization for vector != cbranch fed into compare with 0 " Tamar Christina
2023-06-28 13:48 ` [PATCH 17/19]AArch64 Add optimization for vector cbranch combining SVE and " Tamar Christina
2023-06-28 13:49 ` [PATCH 18/19]Arm: Add Advanced SIMD cbranch implementation Tamar Christina
2023-06-28 13:50 ` [PATCH 19/19]Arm: Add MVE " Tamar Christina
[not found] ` <MW5PR11MB5908414D8B2AB0580A888ECAA924A@MW5PR11MB5908.namprd11.prod.outlook.com>
2023-06-28 14:49 ` FW: [PATCH v5 0/19] Support early break/return auto-vectorization 钟居哲
2023-06-28 16:00 ` Tamar Christina
2023-11-06 7:36 ` [PATCH v6 0/21]middle-end: " Tamar Christina
2023-11-06 7:37 ` [PATCH 1/21]middle-end testsuite: Add more pragma novector to new tests Tamar Christina
2023-11-07 9:46 ` Richard Biener
2023-11-06 7:37 ` [PATCH 2/21]middle-end testsuite: Add tests for early break vectorization Tamar Christina
2023-11-07 9:52 ` Richard Biener
2023-11-16 10:53 ` Richard Biener
2023-11-06 7:37 ` [PATCH 3/21]middle-end: Implement code motion and dependency analysis for early breaks Tamar Christina
2023-11-07 10:53 ` Richard Biener
2023-11-07 11:34 ` Tamar Christina
2023-11-07 14:23 ` Richard Biener
2023-12-19 10:11 ` Tamar Christina
2023-12-19 14:05 ` Richard Biener
2023-12-20 10:51 ` Tamar Christina
2023-12-20 12:24 ` Richard Biener
2023-11-06 7:38 ` [PATCH 4/21]middle-end: update loop peeling code to maintain LCSSA form " Tamar Christina
2023-11-15 0:00 ` Tamar Christina
2023-11-15 12:40 ` Richard Biener
2023-11-20 21:51 ` Tamar Christina
2023-11-24 10:16 ` Tamar Christina
2023-11-24 12:38 ` Richard Biener
2023-11-06 7:38 ` [PATCH 5/21]middle-end: update vectorizer's control update to support picking an exit other than loop latch Tamar Christina
2023-11-07 15:04 ` Richard Biener
2023-11-07 23:10 ` Tamar Christina
2023-11-13 20:11 ` Tamar Christina
2023-11-14 7:56 ` Richard Biener
2023-11-14 8:07 ` Tamar Christina
2023-11-14 23:59 ` Tamar Christina
2023-11-15 12:14 ` Richard Biener
2023-11-06 7:38 ` [PATCH 6/21]middle-end: support multiple exits in loop versioning Tamar Christina
2023-11-07 14:54 ` Richard Biener
2023-11-06 7:39 ` [PATCH 7/21]middle-end: update IV update code to support early breaks and arbitrary exits Tamar Christina
2023-11-15 0:03 ` Tamar Christina
2023-11-15 13:01 ` Richard Biener
2023-11-15 13:09 ` Tamar Christina
2023-11-15 13:22 ` Richard Biener
2023-11-15 14:14 ` Tamar Christina
2023-11-16 10:40 ` Richard Biener
2023-11-16 11:08 ` Tamar Christina
2023-11-16 11:27 ` Richard Biener
2023-11-16 12:01 ` Tamar Christina
2023-11-16 12:30 ` Richard Biener
2023-11-16 13:22 ` Tamar Christina
2023-11-16 13:35 ` Richard Biener
2023-11-16 14:14 ` Tamar Christina
2023-11-16 14:17 ` Richard Biener
2023-11-16 15:19 ` Tamar Christina
2023-11-16 18:41 ` Tamar Christina
2023-11-17 10:40 ` Tamar Christina
2023-11-17 12:13 ` Richard Biener
2023-11-20 21:54 ` Tamar Christina
2023-11-24 10:18 ` Tamar Christina
2023-11-24 12:41 ` Richard Biener
2023-11-06 7:39 ` [PATCH 8/21]middle-end: update vectorizable_live_reduction with support for multiple exits and different exits Tamar Christina
2023-11-15 0:05 ` Tamar Christina
2023-11-15 13:41 ` Richard Biener
2023-11-15 14:26 ` Tamar Christina
2023-11-16 11:16 ` Richard Biener
2023-11-20 21:57 ` Tamar Christina
2023-11-24 10:20 ` Tamar Christina
2023-11-24 13:23 ` Richard Biener
2023-11-27 22:47 ` Tamar Christina
2023-11-29 13:28 ` Richard Biener
2023-11-29 21:22 ` Tamar Christina
2023-11-30 13:23 ` Richard Biener
2023-12-06 4:21 ` Tamar Christina
2023-12-06 9:33 ` Richard Biener
2023-11-06 7:39 ` [PATCH 9/21]middle-end: implement vectorizable_early_exit for codegen of exit code Tamar Christina
2023-11-27 22:49 ` Tamar Christina
2023-11-29 13:50 ` Richard Biener
2023-12-06 4:37 ` Tamar Christina
2023-12-06 9:37 ` Richard Biener
2023-12-08 8:58 ` Tamar Christina
2023-12-08 10:28 ` Richard Biener
2023-12-08 13:45 ` Tamar Christina
2023-12-08 13:59 ` Richard Biener
2023-12-08 15:01 ` Tamar Christina
2023-12-11 7:09 ` Tamar Christina
2023-12-11 9:36 ` Richard Biener
2023-12-11 23:12 ` Tamar Christina
2023-12-12 10:10 ` Richard Biener
2023-12-12 10:27 ` Tamar Christina
2023-12-12 10:59 ` Richard Sandiford
2023-12-12 11:30 ` Richard Biener
2023-12-13 14:13 ` Tamar Christina [this message]
2023-12-14 13:12 ` Richard Biener
2023-12-14 18:44 ` Tamar Christina
2023-11-06 7:39 ` [PATCH 10/21]middle-end: implement relevancy analysis support for control flow Tamar Christina
2023-11-27 22:49 ` Tamar Christina
2023-11-29 14:47 ` Richard Biener
2023-12-06 4:10 ` Tamar Christina
2023-12-06 9:44 ` Richard Biener
2023-11-06 7:40 ` [PATCH 11/21]middle-end: wire through peeling changes and dominator updates after guard edge split Tamar Christina
2023-11-06 7:40 ` [PATCH 12/21]middle-end: Add remaining changes to peeling and vectorizer to support early breaks Tamar Christina
2023-11-27 22:48 ` Tamar Christina
2023-12-06 8:31 ` Richard Biener
2023-12-06 9:10 ` Tamar Christina
2023-12-06 9:27 ` Richard Biener
2023-11-06 7:40 ` [PATCH 13/21]middle-end: Update loop form analysis to support early break Tamar Christina
2023-11-27 22:48 ` Tamar Christina
2023-12-06 4:00 ` Tamar Christina
2023-12-06 8:18 ` Richard Biener
2023-12-06 8:52 ` Tamar Christina
2023-12-06 9:15 ` Richard Biener
2023-12-06 9:29 ` Tamar Christina
2023-11-06 7:41 ` [PATCH 14/21]middle-end: Change loop analysis from looking at at number of BB to actual cfg Tamar Christina
2023-11-06 14:44 ` Richard Biener
2023-11-06 7:41 ` [PATCH 15/21]middle-end: [RFC] conditionally support forcing final edge for debugging Tamar Christina
2023-12-09 10:38 ` Richard Sandiford
2023-12-11 7:38 ` Richard Biener
2023-12-11 8:49 ` Tamar Christina
2023-12-11 9:00 ` Richard Biener
2023-11-06 7:41 ` [PATCH 16/21]middle-end testsuite: un-xfail TSVC loops that check for exit control flow vectorization Tamar Christina
2023-11-06 7:41 ` [PATCH 17/21]AArch64: Add implementation for vector cbranch for Advanced SIMD Tamar Christina
2023-11-28 16:37 ` Richard Sandiford
2023-11-28 17:55 ` Richard Sandiford
2023-12-06 16:25 ` Tamar Christina
2023-12-07 0:56 ` Richard Sandiford
2023-12-14 18:40 ` Tamar Christina
2023-12-14 19:34 ` Richard Sandiford
2023-11-06 7:42 ` [PATCH 18/21]AArch64: Add optimization for vector != cbranch fed into compare with 0 " Tamar Christina
2023-11-06 7:42 ` [PATCH 19/21]AArch64: Add optimization for vector cbranch combining SVE and " Tamar Christina
2023-11-06 7:42 ` [PATCH 20/21]Arm: Add Advanced SIMD cbranch implementation Tamar Christina
2023-11-27 12:48 ` Kyrylo Tkachov
2023-11-06 7:43 ` [PATCH 21/21]Arm: Add MVE " Tamar Christina
2023-11-27 12:47 ` Kyrylo Tkachov
2023-11-06 14:25 ` [PATCH v6 0/21]middle-end: Support early break/return auto-vectorization Richard Biener
2023-11-06 15:17 ` Tamar Christina
2023-11-07 9:42 ` Richard Biener
2023-11-07 10:47 ` Tamar Christina
2023-11-07 13:58 ` Richard Biener
2023-11-27 18:30 ` Richard Sandiford
2023-11-28 8:11 ` Richard Biener
2023-11-30 3:47 [PATCH 9/21]middle-end: implement vectorizable_early_exit for codegen of exit code juzhe.zhong
2023-11-30 10:39 ` Tamar Christina
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=VI1PR08MB5325688E37E6A409E577E00EFF8DA@VI1PR08MB5325.eurprd08.prod.outlook.com \
--to=tamar.christina@arm.com \
--cc=Richard.Sandiford@arm.com \
--cc=gcc-patches@gcc.gnu.org \
--cc=jlaw@ventanamicro.com \
--cc=nd@arm.com \
--cc=rguenther@suse.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).